There is a statistic that circulates in every analytics team: data scientists spend about 80 percent of their time cleaning and preparing data, and only 20 percent analysing it. Clients hear that number and assume it is padding. It is not. If anything, on a first engagement with a company that has never had an analyst, it is optimistic.
Here is what that time actually goes into, and what you can do to shrink it.
What data cleaning really means
It is not one task. It is five or six distinct problems that happen to arrive together.
Definitions that disagree
The single most expensive problem, and the least technical. Sales counts a deal when the contract is signed. Finance counts it when the invoice is paid. Marketing counts it when the opportunity is created. None of them are wrong. But until someone decides which definition the company uses, no dashboard can be trusted, and every meeting relitigates the numbers instead of discussing them.
Duplicates that are not exact duplicates
Removing identical rows is trivial. The real work is deciding that Acme Ltd, ACME Limited and Acme Ltd. are one customer, and that two records with the same email but different phone numbers are one person who changed phones. Every one of those judgements needs a rule, and the rules need a human who knows the business.
Missing values that mean different things
A blank in a discount column might mean no discount was given, or that nobody recorded it, or that the field did not exist before 2022. Those three cases require three different treatments. Filling them all with zero is fast and quietly corrupts your averages.
Formats that drifted over time
Dates stored as text in three formats because the export tool changed. Currency mixed between euros and dollars in one column after an acquisition. Country codes that switched standards. These are individually small and collectively enormous.
Outliers that might be real
A single order 400 times larger than average could be a data entry error, or it could be the enterprise deal that made the quarter. Deleting it silently is as dangerous as leaving it in. Someone has to check.
Why it costs so much
None of the above can be fully automated on the first pass, because each decision needs business context that lives in people rather than systems. The analyst writes a script, finds an anomaly, asks someone, gets an answer that invalidates an earlier assumption, and rewrites. That loop is the 80 percent.
The compounding cost is worse. Skipped cleaning does not stay hidden. It surfaces three months later when a report contradicts a director’s intuition, confidence in the whole project collapses, and the cleaning happens anyway, under pressure, with an audience.
Five ways to cut the 80 percent down
- Write the definitions down before the analysis starts. One page. What counts as a customer, a sale, an active user, a churn. Get the department heads to agree in writing. This single document routinely saves weeks.
- Validate at entry, not at analysis. Required fields, dropdowns instead of free text, format constraints in the CRM. Preventing one bad record costs seconds. Finding and fixing it later costs minutes, multiplied by thousands.
- Make cleaning a pipeline, not a one-off. If the cleaning lives in a script that runs on every refresh, you pay the cost once. If it lives in someone’s spreadsheet, you pay it every month and it breaks when they leave.
- Keep the raw data untouched. Always transform into a new layer. When a cleaning rule turns out to be wrong, and one will, you need the original to go back to.
- Log what you changed and why. A short record of every rule applied. It turns an unauditable black box into something a sceptical stakeholder can inspect, which is often what it takes to get a report believed.
The reframe worth making
Clients sometimes ask whether the cleaning phase can be shortened to get to the interesting part faster. My answer is that cleaning is a large part of the value. A company that finishes this work has a reliable, shared, documented view of its own operations. Many organisations have never had that, and it changes how meetings run long before any model is built.
The analysis is what you buy. The clean foundation is what you keep.