Finding and handling outliers

Why this matters

An outlier is a value far from the rest of the data, and it forces a decision: is it an error to fix, or the most interesting thing in the data set? Removing a genuine outlier throws away information, such as the fraud case or the largest customer. Keeping an erroneous one can drag an average far from the truth. And the statistical rules that find outliers answer a different question from the one the analyst usually needs answered.

The exam asks you to identify outliers, choose a detection method, and justify how they are handled. This lesson uses the course's order data, which contains both kinds on purpose, and shows why detection rules and judgement both matter.

The lesson

What an outlier is, and the difference from an error

An outlier is an observation that lies an abnormal distance from the other values. That is a statement about position, not about correctness.

An outlier can be:

• a data error: a typing mistake, a unit mix-up such as grams recorded as kilograms, a sentinel value such as 9999, or a value that is simply impossible, such as a negative quantity sold;
• a genuine extreme: a real event that happens rarely, such as a bulk order from a business customer, a record sales day, or a fraudulent transaction;
• a sign of a different population: a handful of wholesale customers mixed into retail data, who should perhaps be analysed separately.

The practice order data contains both kinds. Most orders are for between one and five units. Six are genuine bulk orders of 120 to 200 units, placed by business customers. Three have negative quantities, which cannot be real sales.

Detection rules find all of these as unusual. Only context says which are wrong.

The interquartile range rule and the z-score

Two rules are standard for flagging outliers in a single column.

The interquartile range (IQR) rule uses quartiles. The first quartile (Q1) is the value a quarter of the way through the sorted data, and the third quartile (Q3) three quarters of the way; the IQR is Q3 minus Q1. Values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR are flagged. These limits are called fences, and they are the whiskers on a box plot. Because quartiles are not affected by extreme values, the rule works well on skewed data.

For the practice order quantities, Q1 is 1, Q3 is 4 and the IQR is 3, so the fences are −3.5 and 8.5. The rule flags the six bulk orders above 8.5, and two of the three negative quantities, which are −5. The third impossible value, −3, sits inside the lower fence and is not flagged.

The z-score measures how many standard deviations a value is from the mean: (value − mean) ÷ standard deviation. Values with a z-score beyond ±3 are commonly flagged. The z-score assumes roughly normal data, and it has a weakness: the outliers themselves inflate the mean and standard deviation used to judge them. In the order data, the bulk orders push the standard deviation above 9, so the z-score rule flags the six bulk orders and none of the negative quantities.

The lesson in both results is the same. A statistical rule finds values that are unusual. An impossible value that happens not to be unusual, such as −3 among quantities of 1 to 5, passes straight through. Errors need validation rules too: a quantity must be greater than zero, a date must not be in the future, an age must be below some plausible maximum.

Outliers in time series and in groups

A value can be ordinary overall and an outlier in its context.

In a time series, an outlier is a value far from what was expected at that moment. A temperature of 35 °C is ordinary in summer and extraordinary in winter. The course's sensor data contains four spikes, each far above the readings either side of it. They are extreme enough that any rule finds them; the harder case is a reading that is normal for the season and abnormal for its hour. Detection compares each value with a rolling baseline, such as the median of nearby readings, rather than with the overall average.

In groups, a value can be normal for the whole data set and abnormal for its group. A £500 order is unremarkable across all customers and extreme for a customer whose orders have never exceeded £20. Applying the detection rule within each group finds these.

A scatter plot reveals a third kind: a point whose values are each ordinary on their own but whose combination is unusual, such as a tall person with a very low weight.

Keep, cap, remove or investigate: choosing and recording why

Once outliers are found, there are four main responses.

• Investigate first, always. Look at the records: who, when, from which source. The bulk orders turn out to belong to business customers, which explains them.
• Correct or remove errors. A value shown to be impossible or mistaken is fixed if the right value can be found, and otherwise removed or set to missing. The negative quantities belong here, unless they turn out to be returns recorded as negative sales, in which case they are a different kind of record that should be separated, not deleted.
• Keep genuine extremes, and consider analysing them separately, or reporting results with and without them.
• Cap (or winsorise) extreme values at a chosen limit, such as the 99th percentile, when their exact size would dominate an analysis but the records themselves are real. Capping changes data, so it must be stated.

Whatever is chosen must be recorded: which values, how many, what was done, and why. Silently removing outliers is one of the easiest ways to make a result look cleaner and less true.

How outliers move a mean and leave a median alone

The reason outliers matter so much is that some statistics are sensitive to them and others are not.

The mean uses every value, so extreme values pull it towards them. The practice order quantities have a mean of 2.85 units. Without the six bulk orders and three negative quantities, the mean is 2.35. Nine rows out of two thousand moved it by a fifth.

The median, the middle value, depends only on order, not size. It is 2 units with or without those nine rows.

Statistics that resist outliers in this way are called robust. The median and the IQR are robust; the mean, the standard deviation and the range are not. When a data set has outliers you cannot remove, reporting the median alongside the mean, and saying why, gives a more honest picture of what is typical.