Learning machine learning by competing

Kaggle · 2018

In 2018, machine learning was something you learned from papers, online courses and other people’s notebooks. I learned it on Kaggle, where companies publish real data and thousands of people compete to make the best predictions.

In numbers

• Time: about 8 hours per training run for the image model, overnight on a laptop.
• Money: a typical house-price prediction was off by about 12%: around $21,000 on a $180,000 house.
• Errors: an error score of 0.124 on house prices, and 79.4% of Titanic passengers predicted correctly.
• Adoption: my notebooks were public, and other competitors upvoted them 35 times.
• Scale: 1,460 houses with 79 features each, 891 passengers, 25,000 images, and a leaderboard of about 4,500 teams.

Three competitions

• Titanic: predict who survived from age, class, ticket and family. The classic first competition, and where I learned that cleaning the data matters more than choosing the model.
• House prices: predict what a house sells for from 79 features. Regression, feature engineering and a lot of plotting.
• Image processing: classify images with convolutional neural networks, on a laptop that needed a night to train each version.

From scratch first

Before using a library, I wrote linear regression myself: the formula, the gradient, the loop. The demo below is the same idea, rewritten for the browser.

Writing it by hand is slower, but you learn why a model fails: a learning rate that is too high, features on different scales, a bad split between training and test data. In 2018 each of those cost me days. Today I spot them in minutes.

Results

• House prices: top 15% of the leaderboard, with an error score of 0.124.
• Titanic: 79.4% of passengers predicted correctly.

When language models arrived four years later, I already understood the basics of how a model learns.