DATA SCIENCE, RANDOM NUMBER AND STATISTICAL PROBLEMS

These problems require you to combine your knowledge of coding with understanding of mathematics, statistics and machine learning.

Irreproducible Results

---

PROBLEM

Your colleagues are trying to use machine learning to develop a model to detect cheating in a video game. They have trained 100 different variants of a model. They perform a statistical test of the performance of each trained model against the test set and find that four of the models have statistically significantly better performance at detecting cheating than random guessing, with p-values in the range of 0.02–0.04. When they put these models into production, they are surprised to find that the models fail horribly and appear to be no better at detecting cheating than random guessing. Explain what your colleagues have done wrong.

---

You can imagine a wide variety of things that might have gone wrong here—the training and test data might not be representative...