Much of data science involves using data for some practical, business purpose. The data usually needs to be cleaned and processed and that might take a while, but it is generally close to where it needs to be. It can be incredibly exciting and engaging to work at one level back, where data is far from where it needs to be. At this level real work has to be done to transform data into a form ready to be turned into value. In this talk I will walk through this process with the example of turning raw transactional SMS text into structured personal finance features.
The work involves several steps of interest that I will present in the talk. It begins with creatively reducing and representing financial life (or any domain) with concise and clearly defined features. This is a step often overlooked in the rush to fit models and if not done well is most likely to ensure the work fails. Next is taking a difficult composite problem and breaking it into pieces. As Max Tegmark put it: “If you have a tough question that you can’t answer, first tackle a simpler question that you can’t answer.” I’ll show how machine-learning models can be combined in sequence to progressively build up structure and refine raw text into meaningful pieces of information.
While illustrating each step in the work with concrete examples (with code usually written in Python), I will focus on the generalizable process of designing and building machine learning pipelines and transforming raw data into features (using programming language/framework of choice).
Slides and more info: