Testing the SOTA (And Why I'm Skeptical)

After the naive methods failed to protect model utility, I turned to the cutting edge. I implemented Negative Gradient Plus (NG+), Masked Small Gradients (MSG), and Convolution Transpose (CT) from recent Deep Unlearn literature.

These are sophisticated algorithms. MSG, for instance, calculates a saliency mask to localize the “unlearning” damage only to the specific parameters that care about the target identity, freezing the rest.

And they work… sometimes.

What I discovered is a frustrating reality in the machine unlearning literature: most methods are secretly fragile. They live or die by their hyperparameters. At its default configuration, MSG’s retained accuracy collapsed to 0.010, essentially a dead model. Tuned properly, the exact same method recovered to 0.997. Same algorithm, same dataset, a hundred-fold difference in usable accuracy depending on one hyperparameter. Past benchmarks often overstate method performance because they present highly-tuned best-case scenarios as the default behavior.

To solve this for my own sanity, I engineered a custom metric called the UF-Score (Utility-Forgetting composite score).

The UF-Score automatically ranks my grid search trials based on how well the model retains its utility (accuracy on the retain set) versus how thoroughly it forgets the target (measured by Membership Inference Attack AUC). No more manual guessing. The math decides the optimal configuration.

But even with perfect tuning, I am starting to suspect these methods are hiding a deeper flaw. What happens when we delete more than one user?