How to make your neural network work well

Ranging from things like hyperparameter tuning ,to how to set up your data, to how to make sure your optimization algorithm runs quickly

chapter one

Training, Validation & Test dataset

the work flow is:

• keep on training algorithms on your training sets
• use hold-out cross validation set to see which of many different models performs best on your validation set
• evaluate final model on test set

First trend: the way to split your dataset

1. previous era of mechine learning

• train:test = 7:3
• train:val:test = 6:2:2

2. big data era

• don't need a whole 20% of your data for val
• the val just needs to be big enough for you to evaluate, say, two different algorithm choices which one is doing better
• so, probably, if total = 1milion, 1,000,000
  • train = 980,000
  • val = 10,000
  • tets = 10,000
    So, if you have a relatively small dataset, these traditional ratios might be okay. But if you have a much larger data set, it's also fine to set your val and test sets to be much smaller than your 20%

second trend: train on mismatched train and test distributions

suggestion: make sure val and test come from same distribution

Learning video address：https://www.bilibili.com/video/BV12E411a7Xn?p=47&vd_source=cce8c915a5f402d01897a495ccc35141