Learning Curves. What to try next in ML ?

A very interesting problem in ML is : What to try next ?  Andrew Ng has some very interesting insights on this topic. (See the reference section below)

  • Nowadays most ML platforms, e.g. AzureML give the ability to do parameter sweeps.
    • Most of the time they also do cross validation when doing sweeps.
    • This simplifies model selection, the platforms will automatically select the parameters during cross validation which give the best accuracy/AUC on the cross validation dataset.
    • This is usually the 1st thing to do for pretty much all ML problems.

 

  • However, an interesting question still remains esp from a practical standpoint –
    • Should I focus more on feature engineering i.e. add more features.  OR Should I focus more on getting more data
    • For these cases I would generally use learning curves.
    • There are some nuances. So let me explain what I usually do.

 

  • Plot of Training Error v/s Cross Validation Error.
    • This usually indicates whether I am currently suffering from a high bias (underfit) problem or a high variance (overfit) problem.
    • High Bias (underfit):
      • high training error. high generalization (CV) error
    • High Variance (overfit):
      • low training error. high generalization (CV) error

learningcurve

  • High Variance (Overfitting) : Plot how the Error / Accuracy varies with increasing data.
    • A good idea here is to use log-base2 scale on the x-axis.
    • Using a log-base-2 scheme gives a good sense of how much the Error/Accuracy with decrease/increase with more data

logplot

  • Based on the intuition above the following steps can be taken. 

 

What to Try next ?

Underfit (high bias)

Overfit (high variance)

Getting More Training Examples

No

Yes

Try smaller set of features

No

Yes. But first see if you can get more training examples.

Additional features

Yes

Maybe. If we get a feature that gives a strong signal then yes add it. But also invest in more data collection in parallel.

 

Code:

 

References:

  1. https://class.coursera.org/ml-005/lecture
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s