Model Extraction

An API that returns logits returns more than it means to. What is extractable ranges from a distilled approximation to, for the final layer, exact parameters up to symmetry.

[attack][stealing]

Theory

Functional extraction is distillation with the victim as teacher: query, collect outputs, fit a student. Query efficiency is the whole game, and active selection near the decision boundary beats uniform sampling by orders of magnitude.

Exact extraction is stronger and narrower. Because the logit layer is a linear map from a $d$ -dimensional hidden space to $|V|$ outputs, its image is a $d$ -dimensional subspace of $\R^{|V|}$ . Collecting more than $d$ linearly independent logit vectors and taking an SVD recovers that subspace — and hence $d$ , the hidden width — from the singular value spectrum alone.

\mathrm{rank}\bigl( [\, \ell(x_1) \; \cdots \; \ell(x_n) \,] \bigr) = d_{\text{model}}, \qquad n > d_{\text{model}}

The defences follow directly from the mechanism: return top- $k$ rather than full logits, add calibrated noise, or bind an unrecoverable per-account transform to the head. Rate limiting alone does not help against an attacker whose query budget is measured in dollars.

membership-inference prompt-injection-taxonomy

References

[1]Tramèr et al. — Stealing Machine Learning Models via Prediction APIs (2016)arXiv:1609.02943

[2]Carlini et al. — Stealing Part of a Production Language Model (2024)arXiv:2403.06634