The Great Fragmentation — How One Job Became Thirty Titles
In my last post, I described a time when five people could build and maintain an entire data warehouse. That era ended not because the…
In my last post, I described a time when five people could build and maintain an entire data warehouse. That era ended not because the technology changed, but because someone decided it would be cheaper to divide the work.
The reasoning was simple, and at first glance, logical. If one person can write SQL, another can talk to the business, and a third can manage the project, why pay for one expensive person who does all three? Hire three cheaper ones instead. Give each a narrow role. Define the handovers. Write a process document. Problem solved.
This was the beginning of the end.
Want more practical data engineering analysis like this?
Join DWHPro Letters and get field-tested notes on Teradata, Snowflake, AI, migrations, performance, and enterprise data work. DWHPro Letters is free. Subscribe to get new issues by email.
The Invention of the Data Warehouse Assembly Line
What happened over the following decade was the industrialization of data warehouse work. Inspired, whether consciously or not, by the principles of Taylorism — the extreme division of labor that manufacturing industries had already identified as problematic decades earlier — enterprise IT departments began slicing the data warehouse lifecycle into ever-thinner specializations.
The person who once did everything was replaced by a chain: a Business Analyst who gathered requirements but did not understand the data model. A Technical Analyst who translated requirements into specifications but did not write SQL. A Developer who wrote SQL but had never spoken to a business user. A Tester who verified the output but did not understand the business logic. A Project Manager who coordinated all of them but understood none of what they were doing.
Each link in the chain was optimized in isolation. Each person was hired for a narrow skill set, trained on a narrow task, and evaluated on narrow deliverables. The system looked efficient on paper. On paper, it always does.
The Cost of Handovers
In my experience, the single most destructive consequence of this fragmentation was not the loss of broad skills. It was the introduction of handovers.
Every time information passes from one person to another, something is lost. This is not a theory. It is an observable fact in every data warehouse project I have worked on in the past twenty years. The Business Analyst writes a requirements document. The Technical Analyst reads it and produces a design. The Developer reads the design and writes the code. At each transition, context is lost, assumptions are made, and misunderstandings are introduced.
In the old model, these transitions did not exist. The person who spoke to the business was the same person who wrote the code. There was nothing to hand over. There was nothing to misunderstand.
In the new model, an entire class of problems was created that had never existed before. Requirements that were technically correct but functionally wrong. Designs that met the specification but missed the point. Code that passed all tests but did not deliver what the business actually needed. And by the time anyone discovered this — typically during User Acceptance Testing — it was too late for corrective action without high cost.
In my experience, the excuse, “but the business wanted it like this,” is simply unacceptable. It is the analyst's responsibility to determine the actual requirements, given the environment and constraints. But when the analyst has never seen an execution plan, never loaded a table, and never sat with a business user long enough to understand the difference between what they ask for and what they actually need, this responsibility becomes impossible to fulfill.
Nobody’s Problem
There is a human dimension to this fragmentation that is rarely discussed in project retrospectives, yet, in my experience, it is the most corrosive consequence of all: the complete diffusion of responsibility.
When a few people build and maintain a data warehouse, there is nowhere to hide. If the batch fails, it is your problem. If the report is wrong, it is your problem. If the business user is unhappy, it is your problem. Ownership exists because you built the thing with your own hands.
The assembly line destroyed this. When the work is sliced into narrow segments, every person in the chain develops the same reflex: point to the person before you or behind you. The Developer says: “I implemented what the specification told me to.” The Technical Analyst says, “I designed what the requirements document described.” The Business Analyst says, “I documented what the business user told me.” The Tester says: “I tested against the acceptance criteria I was given.” The Project Manager says, “All deliverables were completed on time.”
Get the next issue by email.
Everyone did their job. Nobody delivered a result.
I have witnessed this pattern in nearly every large enterprise project over the past two decades, and it follows the same script every time. Something goes wrong in production. A meeting is called. Every participant arrives with documentation proving that their part of the chain was executed correctly. The meeting concludes that the process must be improved, meaning more documentation, more handovers, and more sign-offs. The solution to a problem created by fragmentation is always more fragmentation.
The psychological effect on the individuals involved is rarely acknowledged. When a person is responsible only for a small, isolated piece of a larger system, they stop caring about the outcome. This is not laziness. It is rational behaviour. Why would a Developer invest extra effort to understand the business context when the organizational structure explicitly tells them this is not their concern? Why would a Business Analyst learn SQL when the job description makes it clear that technical knowledge is not expected? The system trains people to be indifferent to the result. And then the same system expresses surprise when the result is poor.
In the old model, pride and accountability were inseparable. You took pride in what you built because you were accountable for it. You were accountable because you understood it end-to-end. Remove the understanding, and the accountability disappears. Remove the accountability, and the pride follows. What remains is a group of people executing process steps, none of whom feel personally responsible for the system they are collectively supposed to deliver.
Thirty Titles for the Same Problem
The fragmentation did not stop at the obvious roles. As organizations grew, so did the taxonomy of job titles. What was once a “Data Warehouse Developer” split into an ETL Developer, a SQL Developer, a Data Modeler, a Data Architect, a Solutions Architect, an Enterprise Architect, a Data Steward, a Data Quality Analyst, a Data Governance Lead, a Business Intelligence Developer, a Report Developer, and a Dashboard Specialist.
Each of these roles came with a job description, a career path, a certification, and a salary band. An industry was built around maintaining this complexity. Consulting firms staffed projects by matching role descriptions to CVs, not by asking whether the person understood how a data warehouse actually worked. Training providers sold certifications for tools and methodologies that were themselves a consequence of the fragmentation. The complexity became self-sustaining.
Meanwhile, the fundamental question — “does this person understand the full picture?” — stopped being asked. It was no longer considered relevant. The full picture was nobody’s responsibility. Or, more precisely, it was everybody’s responsibility, which in practice means it was nobody’s.
The Communication Firewall
Perhaps the most damaging pattern I have observed is the deliberate isolation of developers from business users. In many organizations, a communication firewall was erected between the two. A Business Analyst or a Project Manager would serve as the sole intermediary, translating business needs into technical requirements and technical constraints into business language.
The stated reason was efficiency: business users should not be bothered with technical details, and developers should not be confused by business complexity. The actual result was the introduction of an additional translation layer, operated by someone who was frequently an expert in neither domain.
In my experience, mediation and facilitation by intermediaries is more than welcome. But firewalling — preventing developers from ever hearing the business context directly — is a recipe for misunderstanding. The developer who hears “we need the report by 8 in the morning every working day” understands the constraint differently from the developer who is told, “make it fast.” The first is a functional requirement that can be engineered. The second is vague noise that leads to arguments during testing.
The Illusion of Cost Savings
The entire fragmentation was justified by cost. Narrow specialists were cheaper per hour than broad experts. Junior developers in lower-cost locations were cheaper than senior engineers in Vienna, London, or Zurich. On a spreadsheet, the savings were real.
But spreadsheets do not capture the cost of a requirements misunderstanding that is discovered in testing. They do not capture the cost of a production incident that takes three teams two weeks to resolve because no single person understands the full system. They do not capture the cost of rebuilding a data model that was designed by someone who had never tuned a query. They do not capture the cost of engaging experienced professionals at premium rates to rescue a project that was understaffed from the start.
In my experience, overspecialization never pays off in the end. Instead, it shifts the associated costs towards the end of the project, where time constraints cause additional stress and pressure. The costs are then shifted upward, as experienced data warehouse professionals must be engaged to bring the project back on track. I have seen this pattern repeated in every major enterprise I have worked with over the past two decades. Not once have I seen it end any other way.
What Was Actually Lost
The fragmentation did not just change how the work was organized. It changed who was attracted to the work. In the generalist era, data warehousing attracted people who were curious about the entire system — people who wanted to understand the business, the data, the architecture, and the performance characteristics. These were people who could sit in a meeting with a risk manager in the morning and debug an execution plan in the afternoon.
The specialist era attracted people who were comfortable doing one thing repeatedly. This is not a criticism of the individuals. It is a criticism of the system that told them this was sufficient. When each team member contributes only a very narrow range of skills, lacks understanding of the overall operation, and is not expected to grow, the project is bound to fail.
The craft was replaced by a process. The process was managed by people who did not practice the craft. And the result was exactly what you would expect: projects that followed the process perfectly and still failed to deliver.
Roland Wenzlofsky is the founder of DWHPro, a network of senior data warehouse consultants who work in small teams, with broad skills, and with end-to-end responsibility. From architecture through development to production support, across Teradata, Snowflake, Databricks, and many others. No handovers. No fragmentation. He is the author of “Teradata Query Performance Tuning” and writes regularly about what works in data warehousing and what does not.
Trying to understand what AI means for data engineering work?
I write about the parts of IT work that are actually changing — and the parts companies still misunderstand.
Subscribe for free and keep launch access.
Written by Roland Wenzlofsky, founder of DWHPro and author of Teradata Query Performance Tuning. DWHPro has helped data warehouse practitioners for 15+ years.