The Cost of Not Knowing What You Want
To understand what's happening in software development right now, it helps to know where the discipline came from. Not everyone does, and that's part of the problem.
Before software had its own methodology, it borrowed from engineering. This was not an unreasonable thing to do. Civil and mechanical engineering had spent a century developing rigorous processes for building complex things, and those processes worked. You did not start pouring the foundations of a bridge before the structural drawings were signed off. You did not begin manufacturing a component before the tolerances had been specified, reviewed, and approved. You worked through a defined sequence of phases: feasibility, design, specification, build, test, commission. Each phase produced documented outputs that became the inputs to the next. The logic was that complexity, managed systematically, became tractable. Skip a phase or shortcut the documentation and you were setting up an expensive problem for whoever came next.
Software inherited this model more or less wholesale. The Systems Development Life Cycle, in its classic form, followed the same sequence: requirements gathering, system design, implementation, testing, deployment, maintenance. Winston Royce wrote what is often cited as the foundational paper on this approach in 1970, describing a sequential, phase-gated model for managing large software projects. It is one of history's minor ironies that Royce, in the same paper, noted the risks of the model and argued for iteration. That part got largely ignored for the next two decades. The phases stuck. The caveat didn't.
What made the engineering analogy appealing was also what made it imperfect. Physical engineering works from relatively stable requirements. A bridge needs to carry a defined load across a defined span. The physics does not change its mind because the client attended a conference. A manufactured component needs to meet specified tolerances that are determined by how it fits with every other component in the system. The requirements are demanding and complex, but once established, they hold. The main variable is execution quality, not requirement stability.
Software requirements turned out to behave differently. The act of building a system frequently revealed what the people commissioning it actually wanted, which was not always what they had specified. Users who had signed off on a requirements document would encounter the first working prototype and recognise, often with genuine surprise, that they had described something subtly different from what they needed. Business contexts shifted during long build phases. The person who had defined the requirements was sometimes no longer in the role by the time the system shipped. The specification was thorough and the specification was, by delivery, partly wrong.
Barry Boehm gave this problem its most useful formal expression in 1981. His cost-of-change curve, published in Software Engineering Economics, showed what practitioners already suspected: that the cost of fixing a defect rises dramatically the later in the development process it is found. A requirements error caught before design begins costs relatively little to fix. The same error found during development costs significantly more. Found during testing, more again. Found in production, you are potentially looking at costs an order of magnitude greater than if you had caught it at the start. The curve is exponential, not linear. Late discovery is not just inconvenient. It is expensive in ways that compound.
Waterfall methodology was, in part, a rational response to this curve. If late discovery is expensive, make discovery happen early. Invest heavily in upfront requirements gathering. Specify completely. Get sign-off at every phase gate before proceeding. The goal was to front-load the thinking so that the building phase was, as far as possible, the execution of something already understood.
It is worth pausing here for anyone who has absorbed the prevailing tech-industry view that waterfall was simply what people did before they knew better. That view is both widespread and reductive. Waterfall was a considered response to a real and measurable problem, developed by people who understood that problem in technical and economic detail. Its limitations were not obvious at the time. They became obvious through practice, as the specific characteristics of software, the mutability of requirements, the difficulty of fully anticipating a complex system before you've built any of it, the gap between what users can articulate and what they actually need, became better understood. Writing off the methodology without understanding what it was attempting means missing something important about why what came next was necessary, and what its own limitations turn out to be.
In February 2001, seventeen practitioners gathered at a ski resort in Utah and produced a document that reshaped how the software industry thought about building things. The Agile Manifesto ran to 68 words of values and 180 words of principles. It was a direct response to what waterfall had produced in practice: projects that delivered late, over budget, and not quite what anyone had needed by the time they arrived.
The manifesto's core insight was about feedback loops. If late discovery is expensive, the answer is not to try to eliminate it through exhaustive upfront specification, complex systems resist that kind of foreknowledge, but to compress the cycle so that discovery happens earlier and more often. Instead of an eighteen-month build phase ending in the revelation that you've solved the wrong problem, you work in short iterations, deliver working software frequently, and let reality correct your assumptions before they compound. The ceremonies that came with agile adoption, the sprint cycles, the daily standups, the reviews and retrospectives, were infrastructure for exactly that: mechanisms for maintaining shared understanding continuously rather than trying to capture it once in a document and then build from it for a year.
The manifesto was also more nuanced than the way it got applied. Its four value statements each named something worth preserving on the right, processes, documentation, contracts, plans, while arguing for prioritising something else on the left. The line most people omit when they summarise it was: "while there is value in the items on the right, we value the items on the left more." A statement about priority in moments of tension, not a licence to stop thinking carefully.
That distinction got lost fairly quickly in practice, and the way it got lost is instructive. The manifesto was written by practitioners reacting against bloated, process-heavy development culture. Within a decade, Agile had become an industry. Consultancies sold agile transformations. Certification programmes issued Scrum Master credentials that you could, and many people did, add to a LinkedIn profile between their job title and their inspirational quote. SAFe, the Scaled Agile Framework, arrived to help large enterprises adopt agile principles and produced, in many implementations, something that looked remarkably like the hierarchical, documentation-heavy processes the manifesto had been written against, only with stickier notes. The philosophy became a product. Agile the noun, the thing you bought and implemented and got a badge for, quietly displaced agile the adjective, the quality of mind you were trying to cultivate.
What survived in good agile practice, even as the philosophy got industrialised around it, was the underlying requirement for rigorous thinking. The format changed. The thinking didn't, if the people involved understood what they were doing and why.
I came up through formal business analysis training before agile became the default, which meant I learned the full toolkit before I learned which parts of it I'd regularly use. The entity relationship diagram, the data flow diagram, the use case, the functional decomposition, these were not, in any good practitioner's hands, ends in themselves. They were tools for producing a particular quality of understanding. You could not draw a correct ERD without having answered fundamental questions about what existed in the system, what properties each thing carried, how they related, and what rules governed those relationships. The diagram was evidence that you'd done that thinking. The thinking was the point.
In an agile context, I was still doing exactly that thinking. Just differently. In a sprint planning session, asking the question nobody had thought to ask. In a backlog refinement, noticing that two user stories made contradictory assumptions about how something worked. In a design conversation, observing that the proposed solution didn't account for the user with no internet connection, or the name with an apostrophe, or the workflow someone had abandoned halfway through. The ceremony changed. The attention it required didn't. I never felt the BA skillset had become redundant. I felt it had become invisible, which is a different thing and in some ways more useful.
The people who had grown up entirely within agile culture and learned that the conversation was the spec sometimes had the conversation without the substance. They ran the ceremonies correctly, updated the board, wrote user stories in the right format. They didn't always know what a user story was supposed to be a placeholder for: a rigorous shared understanding of a problem, examined from enough angles that building a solution to it was unlikely to produce expensive surprises. The map was present. The territory had not been surveyed.
This matters because of what came next.
Working with AI coding tools, Codex, Claude Code, Cursor and their variants, does something genuinely new to the cost curve. The cost of generating code drops toward zero. A working implementation of a reasonably well-specified feature can be produced in minutes rather than days. What agile compressed through short iteration cycles, AI compresses further still. The build phase is no longer where most of the cost lives.
The requirement to think carefully before building never went away.
The cost of unclear thinking doesn't move at all.
AI coding tools don't push back or interrogate, they complete. And without a rigorous specification to check against, a requirement can simply disappear. Not dramatically, no error, no flag, no moment where the tool announces it has skipped something. The UI renders. The happy path works. The edge case you mentioned three prompts ago, the one that matters when the user has no network connection or returns to an abandoned workflow or enters data that doesn't fit the expected format, is gone. At best you have a smoke and mirrors interface that does nothing behind the buttons. At worst you have something that appears to function correctly until it doesn't, in production, with real consequences.
I'm anthropomorphising here, and I want to be precise about that. The model isn't pretending, in any meaningful sense. It has no intent to deceive. What it has is a completion objective and no mechanism for the kind of honest self-assessment that would cause a human developer to say "I haven't actually handled that case yet." The effect, from the receiving end, is indistinguishable from being told something is done when it isn't. Which means the rigour has to come from somewhere else. It has to come from you.
Which brings us to who, exactly, is well positioned to provide it.
The profile that thrives in AI-assisted development is not the fastest prompt writer or the most prolific generator of code. It is the person who can think rigorously about a problem before they start, specify precisely enough that the tool has something real to work from, and then inspect what comes back against what they actually needed rather than what it looks like at first glance. That is a particular combination of skills, and it does not map neatly onto any single traditional role.
The product-minded developer gets there from one direction. Someone who understands the technical constraints well enough to specify at the right level of abstraction, and who has developed enough product instinct to know what the system is actually supposed to do for the person using it. Not just what the function returns, but whether the function is solving the right problem. The technically-minded product manager gets there from the other direction. Someone with enough understanding of how systems are built to know what a meaningful specification looks like, and enough product rigour to have thought through the edge cases and the failure modes before handing anything over.
What both profiles share is the thinking that formal BA training used to be one of the few reliable ways to develop. Not the artefacts. The thinking.
Fast code generation does not remove the cost of vague intent.
The good news, if you can call it that, is that the specification AI needs is not the 80-page functional requirements document. Nobody is going back to that, and nobody should. What waterfall got wrong was not the impulse to think carefully before building, it was the assumption that you could think completely before building, that a sufficiently thorough document produced at the start could anticipate everything that would matter by the end. That assumption failed because human understanding of complex systems is partial and because the world moves. Neither of those things has changed.
What AI needs is something different in character from both the waterfall document and the agile conversation. It needs precision without the pretence of completeness. It needs enough rigour to give the tool something real to work from, combined with an iterative approach that treats each generation as a test of the specification rather than an execution of it. You are not handing over a finished brief and waiting for a finished product. You are running a rapid sequence of specify, generate, inspect, specify again, with the emphasis firmly on the inspection step, which is where your understanding of what you actually needed gets tested against what arrived.
This changes what the specification looks like in practice. A user story in the agile sense, "as a user I want to reset my password so that I can regain access to my account," is not enough. It describes the happy path and leaves everything else open. What happens if the email address is not in the system? What happens if the reset link has expired? What happens if the user has no access to the email account they registered with? What happens if they try to reset three times in a row? These are not edge cases in the dismissive sense of the word. They are the cases that determine whether the feature actually works, and an AI coding tool will not ask about them. It will implement the happy path cleanly and either ignore the rest or handle it in whatever way seemed most plausible given the patterns in its training data, which may or may not be what you needed.
The specification that works looks more like a decision table than a narrative. It states not just what the system should do but what it should do in each meaningful variation of the situation. It names the actors and their states. It defines what exists in the system and what properties those things carry. It identifies the boundaries, what is in scope and, just as importantly, what is explicitly out of scope, because an AI tool with no guidance on scope will make its own decisions about what to include, and those decisions will be optimistic. It specifies what failure looks like and what the system should do when it encounters it, because a system that fails silently or fails in a way that corrupts data is worse than a system that fails loudly and stops.
None of this requires a formal methodology. It requires the habit of asking the questions that the formal methodology was designed to force you to ask. What exists? What are its properties? What can happen to it? What should happen in each case? What should not be possible? What does correct look like, specifically enough that you would recognise incorrect?
The other thing it requires is the ability to inspect what comes back with genuine critical attention rather than relief that something arrived. The output of an AI coding tool is fluent and confident and has the aesthetic qualities of correct code. It compiles. It runs. The variable names are sensible. The structure looks reasonable. None of that tells you whether it does what you needed. That requires you to hold the output against the specification with the specific intention of finding where they diverge, which is a different cognitive mode from reading code to understand what it does. You are not trying to understand it. You are trying to falsify it.
There is something ironic about where this lands. The agile movement arose because a methodology had become so focused on the document that it lost sight of what the document was for. The shared understanding was the point. The document was evidence it existed. Twenty years on, the tools have changed beyond what anyone anticipated, and the shared understanding is the point again, except the conversation in which you build it now happens alone, before you start, written down with enough precision that something with no context and no judgment can work from it usefully.
The form keeps changing. The underlying requirement hasn't.
What has changed is the cost of getting it wrong, and it is worth being honest about what that cost looks like in practice. At small scale, with low stakes and cheap iteration, you can skip the upfront thinking, generate something, inspect it, and course-correct fast enough that the overhead is manageable. But the cost doesn't disappear, it shifts. You pay it in inspection time, in iteration cycles, in the technical debt that accumulates from code that handles the cases you tested and quietly mishandles the ones you didn't think to. At larger scale or higher stakes, that debt compounds in ways that are expensive and often invisible until they aren't.
You also pay it in frustration. In the gap between what the demos promised and what you're actually dealing with. In the feature that worked in the announcement video and doesn't quite work in your codebase. In the hours spent trying to prompt your way out of a problem that is upstream of the prompting entirely. A significant amount of the disillusionment currently circulating about AI development tools is not evidence that the tools are oversold, though some of them are. It is evidence that the thinking that makes them work has not kept pace with the enthusiasm for using them.
The tools are genuinely capable. They are also genuinely indifferent to whether you are building the right thing for the right person for the right reason. That is entirely your problem to solve before you start. The people doing well with AI-assisted development are not necessarily the most technically proficient or the fastest prompt writers. They are the people who can answer, with real specificity, who they are building for, why that person needs it, and what it would have to do to actually serve them. Everything else, the how, the what, the implementation, flows from that or it doesn't hold.
That set of questions predates every methodology in this piece. It will outlast whatever comes next.