Machine unlearning

Deleting a training record does not necessarily remove what a model learned from it. Consider an invented learner that stores the average of 2, 4 and 9: its prediction is 5. If the record 9 is removed, training afresh on 2 and 4 gives 3. Deleting the file while leaving the prediction at 5 has not made this learner equivalent to retraining. Machine unlearning asks how to remove the influence of specified training data efficiently, while keeping useful performance on the remaining task and supplying evidence that matches the intended removal criterion.[1]

How can an efficient update remove specified training influence from a model while preserving useful unrelated performance? A removal criterion must say what would count as success. For the deterministic toy learner that predicts the sample mean, the exact retraining target is easy to inspect:$$\frac{2+4+9}{3}=5,\qquad\frac{2+4}{2}=3.$$Updating the stored mean to 3 matches retraining for this learner; it establishes no general neural-network unlearning algorithm.

With randomized training, the comparison can concern distributions of trained models rather than identical parameter files, and approximate criteria must specify their tolerated discrepancy.

For image classifiers, the 2026 study separately tests final predictions and internal representations: learned numerical descriptions of an image used by the final decision layer. A linear probe trains a new classifier using weighted sums of these features, keeping the features fixed and using labeled data that includes the classes requested for removal. Recovering class predictions this way can reveal surviving discriminative features even when the model’s original output classifier stopped predicting those classes. It does not by itself recover individual training records or prove every method fails.[1] [2]

Hiding an output can leave information recoverable inside the model.[1]

The cited 2026 study recovers representations from evaluated methods. Its experiments and restricted theory do not show that every unlearning method fails.[1] [2]

Training data may later be found erroneous, harmful or unsuitable to retain. Correcting the dataset alone leaves the already trained model unchanged. An effective update would let that correction reach the learned model without always paying for training from the beginning, while checking that useful abilities based on other data survive.[1] [2]

Effective unlearning would let a correction to selected training data change the already trained model without rebuilding every useful ability from the remaining data. The mean example makes that consequence visible: removing 9 should change the learned prediction from 5 to 3. In a classifier, useful distinctions among retained classes should survive the update. Internal features matter because later training can use them again; suppressing a current output need not remove that reusable information. Retraining equivalence supplies a reference for the requested correction, while approximate methods must explain what they preserve and how closely they meet it.[1] [2]