Tutorial 2: Preparing your Data

If you're familiar with Excel, you will recognise the standard structure of datasets: organised into rows and columns. While datasets vary in appearance, this basic format is consistent. 

Columns: Each column represents a variable, which can be any characteristic, number, or quantity that is measurable. Variables include attributes like an individual's age, sex, income, ethnicity, or views on a topic.

Rows: Each row corresponds to a single observation, which may include multiple fields — for instance, a participant's complete survey responses or comprehensive details of an incident. These rows are comprised of numerous variables, each represented by a column.

It is crucial to grasp this row-and-column structure, as it is a constant across various software like Excel, Jamovi, SPSS, R, and Python. Usually, your dataset will come as a .csv file. However, there are other data structures, such as .json files, but these need to be transformed into the row-and-column format before analysing the data.

Data preparation is a critical step in the data analysis process, involving cleaning and transforming raw data into a format suitable for analysis. Part of this process includes removing unnecessary columns and rows, as well as renaming variables. Here's an overview of these tasks:

For Continuous Variables:

Data Entry Errors: Check for errors in our data. Look out for impossible values. Some survey programs for example automatically record missing values as 99 or -99 or you might have a surevey for participants 18+ but have values below that. Such values need to be removed and/or corrected. 

Consistency Checks: Make sure that the data across the related variables makes sense (e.g., age, income, etc) and is conistent. 

Standardise Formats:  Make sure your continuous data follows a standard format (e.g., number of decimal places, etc).

For Ordinal Variables:

Make sure that the values are accurate. Convert the levels of ordinal variables into numerical codes, as this is necessary for certain statistical tests that require numerical input. Some programs (such as Jamovi or R) allow you to attach labels to your numbers. For example: Strongly agree = 0 

• Somewhat agree = 1

• Neither agree nor disagree = 2

• Somewhat disagree = 3

• Strongly disagree = 4

If you do not encode your levels, some tests may not function correctly. Even when tests run without errors, you must still inform the statistical program about the order of the levels.

Recoding involves converting variables from one format or structure into another. It is typically used with categorical data. We recode variables as some statistical tests and programs can only handle numerical data, so categorical data need to be recoded into a numerical format. There are several methods. 

• Label Encoding: Each unique category level is assigned an integer value. This is more compact but can introduce a new problem: the model might interpret the numerical values as having an ordinal relationship.

When analysing your data, you MUST instruct the statistical programme to recognise missing data. Otherwise, you risk obtaining wrong results. Qualtrics, for instance, designates missing data with a '-99' code. If you neglect to inform your statistical programme that '-99' signifies missing data, the computer will erroneously incorporate these values into its calculations. Other programmes such as Jamovi or Python represent missing data with NaN.

Ensure you adopt a method appropriate and consistent with the statistical programme you're utilising. For example, Jamovi identifies blank cells, 'na', or 'NaN' as missing data, requiring no additional action on your part. However, if you're dataset uses '-99', you will need to manually tell Jamovi to recognise this as the missing value code.

There's no need to eliminate all missing data, as most statistical programmes will exclude it from analysis (this will affect your sample size). Excessive missing data may trigger an error, indicating that analysis is unfeasible. Adding in missing data will increase the available data for your analysis. 

Working with Missing Data:

It's inevitable that you will encounter missing data, even after removing incomplete observations. There are two primary approaches to handle missing data:

1. Discard Observations: You may choose to discard the entire column or row containing missing data. This approach is generally not advisable if there are only a few missing values. However, if substantial data is missing from a row or column, it might be sensible to discard it. The decision is yours to make – ensure you document your reasoning.

2. Impute Missing Data: There are numerous strategies for calculating and replacing NaN values. You should carefully consider which method to use. For an in-depth discussion, consider consulting relevant literature. Essentially, you have the following options:

• • Use the Median, Mean, or Mode for continuous data. 

  • Use the Mode for ordinal and nominal data. 

  • Employ advanced techniques such as regression and machine learning to estimate the missing value more accurately.

After choosing your method, decide which data (rows and columns) to discard, then proceed to impute the missing values using your chosen method.

For a more indepth dicussion on dealing with missing values visit this website.

Transforming variables is a common operation in data analysis. There are a few common types of transformations:

• Normalization: Scaling data to a small, specified range, like 0 to 1.

• Standardization: Transforming data so that it has a mean of 0 and a standard deviation of 1 (more on this later)

• Log Transformation: Applying the natural logarithm or log10 to the data, often used to handle skewed data.

• Binning or Bucketing: Transforming continuous data into discrete bins or categories.

• Recoding: Converting categorical data into numerical format, such as one-hot encoding or label encoding for machine learning models (covered above).

• Feature Engineering: Creating new variables from existing ones, which could involve mathematical operations, aggregation, or other techniques to derive more informative or useful features.

The specific method to use depends on the nature of your data and the goals of your analysis. We will keep it simple for now and only cover recoding and binning. 

Recoding is covered above. Below we also looked at One-Hot Encoding and Bucketing/Binning in more details.

One-Hot Encoding: 

One-Hot Encoding converts each category level into a new binary variable (0 or 1). Each level is represented by a dummy variable.

How does One-Hot-Encoding work: 

One-hot encoding is a process used to convert categorical data into a numerical format by creating a binary column for each category of the variable. In one-hot encoding, each category becomes a new column, and a binary value of 1 or 0 is assigned to those columns in each row of the dataset.

Below is a step-by-step summary of how one-hot encoding works:

• Identify Unique Categories: identify all the unique categories within the categorical variable you want to encode.

• Create New Columns: For each unique category, create a new column in the dataset.

• Assign Binary Values: For each row in the dataset, fill the new columns with binary values:

  • Place a 1 in the column corresponding to the category that the row belongs to.

  • Place a 0 in all other newly created category columns.

The result is a "one-hot" because only one category column is hot (1) for each row, while all others are cold (0). This technique is widely used to prepare categorical data for many types of machine learning algorithms.

Bucketing/Binning: 

Imagine you have a laundry basket full of socks of different colours. Now, you want to organize these socks so that it's easier for you to find a pair when you need it. So you sort the socks into different bins or buckets based on their colour. All the red socks go into one bin, blue into another, and so on. This is what binning or bucketing is like in data analysis.

Now, let's relate this to data:

Bucketing or binning is like sorting socks. But instead of socks, you have a lot of numbers (data points). These could be anything: ages of people, prices of houses, temperatures, etc. When there are too many different numbers, it's hard to make sense of them. So, you put these numbers into groups (or bins) to organize them better.

Why Use Bucketing/Binning?

Simplicity: It makes your data simpler. Instead of looking at every single age in a survey, you look at age groups like 0-18, 19-35, etc.

Manageability: It makes large data sets more manageable. Imagine trying to understand the individual ages of thousands of people at once!

Analysis: It helps in analysis. It's easier to see patterns and trends when you have groups to compare, like seeing which age group buys more online.

How Does it Work

• Choose a Variable: Pick the data you want to organize, like the ages of people in a survey.

• Decide on Bin Size or Number: Choose how big each group should be or how many groups you want. For example, you could have 5-year age groups or maybe 10 groups in total.

• Create Bins: Group each data point into the right bin. For ages, anyone between 0-18 years goes into the first bin, 19-35 in the second, and so on.

• Label the Bins: Give each bin a label that makes sense, like "Teenagers" for the 0-18 group. 

• Use Your Bins: Now you can easily compare these bins to find patterns, like which age group votes for whom.

That's bucketing/binning in a nutshell – organizing your data into groups to make it easier to understand and work with.

Simple transformations can easily be done. However, often you may want to start combining variables to reduce them into one variable.

Of course, this is something that can be done. However, don't just go and add variables together. There are statistical tests, such as a Reliability Analysis, a Principle Component Analysis, or an Exploratory Factor Analysis, covered later that should be used to combine variables. These tests will show you whether the variables you want to combine are statistically correlated or not. This topic will be covered in more depth later.