Training a Machine Learning or Deep Learning model is as important as designing a good model and evaluating it.
Recently, Yann LeCun who is one of the giant in ML & DL released his lecture at NYU where some training tips in practice are discussed in the lecture. Those tips for training ML/DL in practice from Deep Learning lecture from Yann LeCun are:
- Use ReLu for non-linearities (tanh and logistic are falling out of favor)
- Use cross-entropy loss for classification
- Use SGD on minibatches
- Shuffle the training samples
- Normalize input (0 mean, 1 variance)
- Schedule to decrease the learning rate
- Use a bit of L1 or L2 regularization on the weights (or a combination)
- But it's best to turn it on after a couple of epochs
- Use "dropout" for regularization
- Lots more in "Efficient Backprop" (LeCun et al. 1998)
- Lots more in "Neural Networks, Tricks of the Trade" (2012 edition)
- Good initilization of weights is super important....
The paper "On Loss Functions for Deep Neural Networks in Classification" from Janocha et al. further sheds some lights on different loss functions for classification problems.