·

by

Why you should get into Big Data in 2023

Data work has quietly stopped being a specialism and started being infrastructure. The interesting shift is not that there is more data — there has been more data every year for two decades — but that the tooling has matured to the point where a small team can do what used to need a department, and the bottleneck has moved from processing to judgement.

The hard part is no longer the pipeline

A decade ago, moving and storing large volumes of data was genuinely difficult, and much of the field’s prestige attached to people who could do it. Managed warehouses and off-the-shelf orchestration have taken most of that difficulty away. What remains is harder to automate: deciding what to measure, noticing when a number is lying to you, and explaining a result to somebody who will make a decision with it.

Where the demand actually is

Job titles in this space are notoriously unreliable, but the underlying work clusters into a few recognisable shapes:

  • Making data trustworthy. Modelling, testing and documenting the tables everyone else builds on.
  • Making data usable. Turning a warehouse into metrics people can find without asking an analyst.
  • Making data operational. Getting predictions and segments back into the tools where work happens.
  • Making data answer questions. The oldest job on the list and still the one that changes the most decisions.

What to learn first

SQL, still and always. It is the one skill that transfers across every warehouse, every tool and every job title in the field, and it is the skill that most reliably separates people who can answer a question from people who have to escalate it. After that, enough Python to move data around and enough statistics to know when a difference is a difference.

Notably absent from that list: any particular vendor’s platform. Tools rotate every few years. The habits of mind do not.

Nobody is ever asked for more data. They are asked whether the number is right, and what they should do about it.

A realistic first project

Take a dataset you actually care about — your own spending, a hobby, a public dataset about somewhere you live — and answer one specific question with it end to end. Load it, clean it, chart it, write a paragraph about what you found and what you are unsure of. That last paragraph is the part employers are looking for, and it is the part almost every portfolio project leaves out.