The Wiring Is Free. The Judgment Is Not.
Connecting two systems was once the work. It is now the part a machine can do. What remains is the part that was always the point.
Some time ago, I argued that the market does not pay for tool lists. The argument has aged in an unexpected direction. It is no longer only the list that has lost its value. It is the act of using the tools.
There was a period when moving data from one system to another was difficult work. Drivers were unreliable, protocols were obscure, and parsing a file correctly was an achievement. Connecting a source to a destination required someone who had done it before. That period is over. The connectors are mature, the formats are standard, and a language model will now produce the boilerplate on request, correctly enough, in less time than it takes to describe the requirement. Wiring one system to another is no longer a skill the market rewards, because it is no longer scarce.
Want more practical data engineering analysis like this?
Join DWHPro Letters and get field-tested notes on Teradata, Snowflake, AI, migrations, performance, and enterprise data work. DWHPro Letters is free. Subscribe to get new issues by email.
And yet the pipelines still fail. They do not fail in the wiring. They fail in everything that the wiring does not show you.
Consider the most ordinary task in modern data work: read a stream of records, shape them, and load them into a warehouse. The demonstration takes an afternoon. The connector reads, the transformation runs, the rows arrive. Anyone can build it. A machine can build it. Then it goes to production, and the questions begin.
One record in the stream is malformed. Does the pipeline stop, or does the bad record step aside while the rest proceeds? Where does it go, and who decides what happens to it later? This is not a technical afterthought. It is a decision about whether a single bad row is permitted to halt a business process, and in a regulated environment, it is an audit question before it is an engineering one.
If you work with enterprise data platforms, migrations, performance tuning, or AI-driven delivery teams, DWHPro Letters is written for you. Get the next issue by email.
The load fails. Is the failure transient, in which case it should be retried, or is it permanent, in which case retrying it produces an infinite loop that quietly takes the whole pipeline down? The distinction is not visible in the wiring. It is a judgment about the nature of the error.
The system that delivers the data promises to deliver it at least once. At least once means more than once. More than once means duplicates, and duplicates mean the load must be designed so that loading the same record twice changes nothing. Nobody who only wired the connection would know this was required. The pipeline that ignores it does not announce the problem. It simply reports the wrong numbers.
A node restarts in the middle of a load and re-reads what it had already read. The warehouse now holds some rows twice. Whether this matters depends entirely on a decision made long before the restart.
The batch is sized for throughput and is therefore too large for the promised latency. Or it is sized for latency and is therefore too small to load efficiently. The correct size is not a setting. It is a position on a curve that has to be found, against a service level that someone must first define.
None of this is wiring. All of it is judgment.
This is the part that does not generate. A model will write the path where everything works, because that path is the common case, and the common case is what it has seen. It will not write the path where the data arrives intact and is still wrong, because that path is specific to your data, your obligations, and your tolerance for being incorrect. The happy path was always the inexpensive part. Automating it does not close the distance between a demonstration and a system. It makes the distance easier to see.
What the market pays for, then, was never the connection. It is knowing what happens after the connection succeeds. It is knowing which failures matter and which can be ignored. It is knowing what not to build, which is the most undervalued skill in the field and the one least likely to appear on any tool. Judgment of this kind is the residue of having been wrong before, in production, with consequences. It is not on a certification, and it is not in the model.
The engineer who is anxious about the tools being commoditized is anxious about the wrong asset. The tools were never the asset. The decision about when, whether, and how to use them was, and that decision still requires someone who has carried the cost of getting it wrong.
The wiring is free. It has been getting cheaper for years, and now it is free. The judgment is not, and it is not going to be.
Want more field-tested data engineering notes?
DWHPro Letters covers enterprise data platforms, migrations, AI, performance, project economics, and the career side of technical work.
Written by Roland Wenzlofsky, founder of DWHPro and author of Teradata Query Performance Tuning. DWHPro has helped data warehouse practitioners for 15+ years.