The Fastest-Growing Job in Tech Is Mostly Cleaning Data

Data science is described as modelling and is mostly the unglamorous work that has to happen before any model means anything.

The number, and what the job is

The growth figure is real and worth stating precisely. The US Bureau of Labor Statistics projects employment of data scientists to increase 33.5 percent between 2024 and 2034, an increase of 82,500 jobs. That is the highest projected growth rate among the technology occupations in its analysis, against 3.1 percent growth across all occupations.

Now the shape of the work, which the job title does not convey.

A large share of the time goes to locating data across systems that were built for operational purposes rather than analysis, understanding how each field was actually populated and by whom, handling missing values in ways that do not distort the result, reconciling the same concept recorded differently in different systems, and determining whether the data can support the question being asked.

This is not a complaint about the field. It is the field. And it explains why data scientists with domain knowledge are so much more effective than those without: almost every judgement in that list requires knowing how the business actually works.

What the direction covers

The scope: statistics, databases, Python and R, visualisation and forecasting.

Four areas.

Data handling. Querying, joining, cleaning and reshaping, which is the daily work.

Statistics. Inference, uncertainty, experimental design and the reasoning that separates a finding from a coincidence.

Modelling and forecasting. Predictive methods, evaluation and knowing when a simpler approach is better.

Communication. Visualisation and the ability to make a technical result change a decision.

Nobody agrees what a customer is

The most common and most expensive discovery in corporate analytics, and it is a definitional problem rather than a technical one.

Ask several departments how many customers the organisation has and the answers differ, often substantially. Sales counts accounts. Finance counts billing entities. Support counts contacts. Marketing counts email addresses. A parent company with six subsidiaries is one customer or six depending on who is asked. Someone who cancelled last month is still in one system and gone from another.

The same applies to almost every important measure. Active user, order, churn, revenue recognised, employee headcount. Each is recorded by systems built at different times for different purposes by people who were solving a different problem.

Three consequences.

Reports that disagree destroy trust faster than they inform. Once two dashboards show different numbers for the same thing, meetings become arguments about the data rather than decisions about the business.

The fix is governance, not technology. Agreeing definitions, recording them, assigning ownership and applying them consistently. No tool does this, and buying one instead is the most common wasted spend in this area.

It is the work that makes everything else possible. Machine learning on inconsistently defined data produces confidently wrong outputs, and the inconsistency is invisible in the output.

The statistics people skip, and pay for

Modern tools make it easy to produce a result without understanding what the result means. Four concepts prevent most of the resulting damage.

Correlation and causation. Stated constantly and violated constantly. The practical form is that a variable associated with an outcome may be a cause, may be an effect, may share a common cause, or may be coincidence. Acting on the association assumes the first, and only an experiment or careful causal reasoning justifies that.

Selection effects. How the data came to exist shapes what it can tell you. Analysing customers who stayed tells you about people who did not leave. Surveying respondents tells you about people willing to respond. Studying successful projects without unsuccessful ones produces advice that does not work.

Uncertainty. Every estimate from a sample has a range around it. Reporting a point estimate without that range invites decisions that the data does not support, particularly when segments get small.

Multiple comparisons. Test enough hypotheses and some will appear significant by chance. Dashboards that slice by dozens of dimensions generate apparent findings continuously, and someone acts on them.

These are not advanced topics. They are the difference between analysis that improves decisions and analysis that provides authoritative-sounding justification for whatever someone already wanted.

Where this sits in the domain

Data science and analytics is the fourth of eight directions in Astra Trainer's AI, data and computing domain, and it is the prerequisite for the machine learning direction rather than a lesser version of it. Models are only as good as the definitions and data quality underneath them.

It connects to software engineering for the pipeline and production layer, and to cloud computing and DevOps for the platforms this work runs on. It also connects outward to almost every other domain, since emissions accounting, clinical data, manufacturing quality and grid forecasting are all data problems with domain requirements. You can see the eight directions here.

Why so many models never get used

A pattern familiar to anyone who has worked in this area: a model is built, performs well in evaluation, is presented, and nothing changes.

The reasons are rarely technical.

It answered a question nobody was asking. The analysis was driven by available data rather than by a decision someone had to make.

It arrived after the decision. An analysis that takes three months for a decision made in three weeks is an academic exercise.

Nobody could act on the output. A prediction is useless without a corresponding action, an owner and the authority to take it.

It was not trusted. Because the definitions disagreed with the ones the audience used, or because the method was opaque, or because a previous analysis had been wrong.

There was no path to production. The model lived in a notebook on one person's machine, and the engineering to run it reliably was never scoped.

The implication for training is that the technical skills are necessary and insufficient. The capabilities that determine whether data work changes anything are problem framing, stakeholder understanding, communication and knowing what production requires, and those are teachable to people who already have the technical half.

The roles, named

Data analysts. The largest group and the most common entry point.

Data scientists. Statistical modelling and experimentation.

Data engineers. Pipelines and infrastructure, and consistently in shorter supply than data scientists.

Analytics engineers, a role that emerged specifically to handle the definition and transformation layer.

Business intelligence developers.

Data governance and quality specialists, doing the definitional work described above.

Experimentation and causal inference specialists.

Forecasting analysts, in demand, supply chain, energy and finance.

Machine learning engineers, where data science meets production software.

Who can be trained into it

Domain experts across the business. The strongest conversion in this direction, and the most underused. Someone who has worked in operations, finance, clinical practice or manufacturing for a decade knows which questions matter and what the data is likely to be hiding. Teaching them analysis is faster and produces better work than teaching an analyst the domain.

Finance and accounting staff. Already comfortable with data, reconciliation and definitional precision, which is most of the difficulty.

Operations analysts and planners. Into forecasting, holding the process knowledge that makes a forecast credible.

Researchers and scientists. Statistical training already present, needing the commercial context and engineering practice.

Software engineers. Into data engineering, which is the scarcer role and closer to their existing skills than data science is.

Database administrators. Into data engineering and governance.

Anyone who has built a substantial spreadsheet model. Frequently holding real analytical ability and no formal route to recognising it.

What to take from this

Data science is projected to grow 33.5 percent to 2034, the fastest of the technology occupations BLS tracks, and the job is mostly data quality and definition rather than modelling.

Most organisations cannot agree what a customer is, and that is a governance problem that no tool solves.

Four statistical ideas, causation, selection, uncertainty and multiple comparisons, prevent most of the damage confident analysis causes.

Models fail to get used for organisational reasons far more often than technical ones.

And the best candidates are usually domain experts who already know which question matters, because that is the half that cannot be taught quickly.