OptimizationYou regularize an Adam-trained model by adding to the loss. Under SGD that equals weight decay; under Adam, weights with large past gradients are regularized less.
The best also moves whenever you change the learning rate.
OptimizationOptimization