Data is the new oil, and data science is the new refinery. Increasing volumes of data are being collected, by websites, retail chains, and heavy industry, and that data is available to data scientists. Their task is to gain new insights from this data while automating processes and helping people make decisions [1]. The details for how they coax real, usable knowledge from these mountains of data can vary greatly depending on the business and the nature of the information. But many of the mathematical tools they use are quite independent of the data type. This article introduces you to some of the methods data scientists use to squeeze insights from a sea of numbers.

More than Just Modeling

The term data scientist evokes associations with math nerds, but data science consists of far more than building and optimizing models. First and foremost, it involves understanding a problem and its context.

For example, imagine a bank wants to use an algorithm to predict the probability that a borrower will be able to repay a loan. A data scientist will first want to understand how lending has worked so far and what data has been collected in this field – as well as whether that data is actually available – with a view to data protection requirements. In addition, data scientists need to be able to communicate their findings. Storytelling is more useful than presenting infinite rows of numbers, because the audience is likely to be made up of non-mathematicians. The need to clearly explain the findings frequently presents a challenge for less extroverted data scientists.

Preparing the Data

What sounds simple in theory often requires time-consuming data cleaning and transformation. Data is not always available in the way you need it. For example, many algorithms require numerical data to be extracted from non-numerical data.

To separate the data, the data scientist forms categories that can be divided using either numerical distances or dummy variables, where each occurrence of a characteristic (such as male, female, and nonbinary) becomes a separate variable. As a rule, one variable can be omitted. For example, in this data set, someone can only be male if they are neither female nor nonbinary. However, erroneous user input often results in data points that could bump an algorithm off track. These data points need to be identified and cleaned up.

The data scientist also looks for variables that are genuinely relevant to the model. This is where the information gathered during the understanding phase comes into play. In an exploratory data analysis, often in a Jupyter Notebook or similar, the data scientist generates and documents the findings in order to share them with colleagues (or at least ensure that the findings are repeatable).

Choosing a Suitable Model

First and foremost, the choice of algorithm depends on the task. If data capable of training an algorithm is available, data scientists refer to this scenario as supervised learning. For instance, if you have access to historical data on loan defaults, you could use it to predict whether future borrowers will repay their loans. The variable used for training is often referred to as the target variable – in this example, this is simply whether or not a loan has been repaid. Other examples would be classifications, whether or not a birthmark is indicative of skin cancer, or whether a customer is a fraudster.