Here is a random photo of a moongoose, as you can’t get much more agile than those furry fellows.

Impact-driven development part 4: Layered requirements in agile contexts, requirements maintenance and the conclusion

In the series so far, we’ve completed the main part of our journey to a deeper understanding of the requirements and their crucial and even foundational role in any project where more than one person is involved. However, we did leave a few loose ends and a couple of odd, unaddressed bits and pieces here and there. This final article in the series is where we take care of all of those, once and for all.

Also, this is where we finally apply what we’ve learned about requirements so far to the Agile context, and we also mention agentic engineering and post-delivery maintenance, so you might want to stick out for this one.

Layered requirements in lean and agile contexts

So, you might be saying to yourself, “This all looks brilliant so far” (aww, thanks, you’re too kind…), “but it sure as heck sounds mighty waterfall-y to me.”

Well, it does, but that is more due to how we are accustomed to looking at the requirements and not because that is actually the case. We’re all conditioned to look at any standalone requirements document as a waterfall artifact, but that's nothing more than a misconception that we all picked up along the way—the difference between waterfall and agile requirements is not really about whether the requirements documentation exists or not or about the particular format it comes in, but rather about the level of detail.

Agile methodologies adopt a progressive, incremental, level-of-detail buildup for requirements, rather than “everything will be defined at the beginning,” inherent in waterfall practices. However, you still DO need to start with a complete end-to-end picture in an agile context as well, even if it's only a fuzzy or grainy one.

And when I say “end-to-end”, that is really ALL that I imply by the term, as those “ends“ with which you start in your first iteration do not have to be the same ends (and usually, if you do things right, they won’t be) that you’ll finish with, but they have to be functionally meaningful ends nevertheless.

The big difference between agile and waterfall is in WHEN, and on WHAT SCOPE detailed discovery/design happens.

When ALL (or a significant majority of) design and discovery happen at the beginning of the project, and the requirements document contains all results of that process, we call that “waterfall”.

However, when design and discovery happen iteratively, just-in-time, and the resulting requirements are amended with new details or those details are documented in iterations themselves throughout the course of the project— that is what we call agile (or better yet, we should call that iterative because that is exactly what it is and Agility is “just” the ability to change direction between iterations, and is therefore an effect allowed for by iterativness of design and discovery in the first place).

All that famed agile collaboration, transparency, etc., you’ve heard so much over the last decade or so, happens as a necessary consequence/element of this iterative just-in-time discovery&design. Those happen because they are an integral part of the discovery process, regardless of the underlying project methodology, and not because they are somehow magical inventions of the agile manifesto. Agile manifesto in effect merely describes what naturally happens when you design, develop, and deliver iteratively rather than in one “big bang” fashion, provided you follow good, common, widely adopted, and long-known engineering principles and practices.

If we want to be extremely pedantic (and a bit harsh and oversimplifying), we could say that the rest is just window dressing and some ceremonies, and that the only true and foundational difference lies in the distinction outlined above.

But even if we refrain from such opinionated statements, if the argumentation presented above stands, then the role of the requirements, and consequently, the layered approach to engineering them, is, and should be, the same in both cases.

Now that we've got that out of the way, you might be wondering how to apply all that we’ve covered so far to your agile project.

The good news is that it applies seamlessly without requiring changes to how you approach sprints or implement your Scrum methodology (or any other agile framework).

The reason is that layered requirements engineering is a framework, which does not mandate or restrict project methodologies or contexts it can be applied to — if a methodology needs requirements (and if there is more than one person involved — they all do), layered requirements engineering will work out-of-the-box.

Why should you even care?

In my experience, most real-world approaches to creating agile requirements artifacts, such as product/project manifests and backlogs, are based on improvisation or misunderstanding of the ‘communication over documentation’ principle. The result is often incomplete and inconsistent, and as a result, estimation, planning, and ultimately, reliable value delivery are usually impeded. As an illustration, try to honestly assess when was the last time you were involved in a project where you gave the project manifest more than just a token attention and care, and where you didn’t (futilely) rely on scrum ceremonies as a solution to all your in-project woes?

Layered requirements engineering provides you with a consistent and structured way to create those artifacts, artifacts that you must create anyway — product manifest and backlog. The good part about it is that it does so without requiring any changes to your existing Scrum implementation.

Let’s illustrate all that with an example.

Building a project manifest

Every project needs a central document describing the high-level project context — in Agile, this is usually called “product/project manifest” (although its content, structure, and mere existence can and do vary wildly between projects).

The way to get a project manifest from your layered requirements is very simple — bordering on being trivial. You simply take the model layer, remove all goal statements except the top-level ones (also often called “epics”), and what you end up with is the manifest, complete with actors, high-level goals, and the project context. No changes or special approaches are required there.

Building a backlog

Now, let’s examine each actor-goal (story) and its individual sets of acceptance criteria, containing both “standalone” ones and those linking to particular formally described details.

When we do that, what we can observe is that what we have on our hands is a complete, well-formed PBI.

It has a goal that needs to be achieved, it has a list of important criteria that need to be satisfied, and every relevant detail is linked to via references in acceptance criteria — all standard elements prescribed by frameworks like Scrum.

In other words, you have a traceable, self-contained, fully implementable unit of work — exactly what the PBI should be. No changes or special approaches are required here either.

Now let’s try to meld all this together into a typical Scrum scenario.

An illustrative Scrum scenario

(Note that we’re focusing on requirements artifacts, in particular on manifest and product backlog, and we’re assuming all other ceremonies/events/techniques such as refinements. planning, story point estimates, etc., are implemented in a standard way):

*sprint 0: Creating the manifest and initial backlog ()**

You build and document the model layer as described in this article, keeping actor goals on the “epic” level without “drilling down”. You now have your manifest (model layer) and your initial product backlog (epics).

() Sprint 0 can be a contentious point as there are some in the scrum community that fully embrace the principle and some who vehemently oppose it. The author of these lines personally doesn’t care about it either way, and so sprint 0 here is used to describe “work that happens before our first 'proper' sprint with the implementation team” — nothing more than that.*

sprint 1+: Continuous discovery and delivery

You select one or more epics, decompose them into stories (actor-goals), and proceed as described in this article to discover and define acceptance criteria and incrementally build the connected detail layer.

Now you have implementable PBIs in your backlog. Select a subset relevant to the iteration’s goal, and then implement and deliver it.

And there you have it, Scrum — done in the most standard way possible — but with meaningful, consistent, and well-formed requirement artifacts.

And you didn’t have to change a thing.

The important difference between the product and implementation backlogs

There is one additional point or a distinction I’d like to make, as I've found that being oblivious to it seriously impedes the product owner’s ability to effectively steer and manage the iterative delivery process through its backlog.

There are two distinct backlog types, both of which originate from the project model, predominantly from its functional part.

The first is the product backlog — it is essentially the same as the functional model, as it is composed of PBIs, each of which describes one piece of the functional model (a.k.a. scope). This backlog does not capture broader project context, organizational constraints, team maturity, and skill level, etc. What it DOES capture is functional scope, and it does that extremely well.

However, you cannot really iteratively design and build a system just using this kind of backlog, unless you are in a very lucky position to have an extremely mature organization, team, and stable project context. And if you are that lucky, you don’t really need to worry, as things will work out in the end anyway, but for the rest of us, there is another backlog that is built just for that purpose, the implementation backlog.

Implementation backlog extends and augments product backlog in the sense that it contains all of it, with enough detail to reliably implement it, but it also contains additional, specialized PBIs that are there to specifically address and mitigate risks originating from organizational, team, and project environment limitations and particularities.

The typical (but not the only) example of such PBIs is SPIKE tickets used for research (tech. risk or team immaturity mitigation, etc.) or synchronization PBIs used for managing a disjointed project environment, etc.

Those eagle-eyed of you will notice that what I call implementation backlog looks suspiciously similar to the sprint backlog, one that results from the sprint refinement process, and that is true; they represent the same thing.

BUT there is an important difference there — as when thought only in individual sprint terms, the high-level picture often gets lost, and any spikes, research, and synchronization PBIs suddenly become “stranded on a sprint island”. What that means is that since they were discovered just before the sprint, and within a time-boxed ceremony, you have unnecessary limits on their proper design and elaboration simply due to time constraints that exist at that point in time.

You might say, “Yes, but we always have our backlog refined for at least 2 sprints in advance, so our sprints do not start right after the refinement session that addresses their scope”. Well, that is great (and now you finally know WHY that approach works), but also not very common in practice, especially for spikes and experiments, and you still have time constraints of refinement sessions in which you’d address two very conceptually different things — functional delivery and risk mitigation.

There are techniques to deal with that as well; “three amigos” sessions being just one of the many, however, you can just have implementation backlog proper right away and not awkwardly tiptoe to it with multiple good practices that are aimed at solving the effects of not having one in the first place.

The concept of implementation backlog, therefore, aims to decouple risk mitigation from individual sprint planning and free it, so to speak, to be considered independently throughout the project lifetime and not only during sprint refinement ceremonies. It also allows you to implement incremental, iterative delivery outside the formal Scrum framework in an educated way, rather than just cutting out bits you don’t find working in your organization without truly understanding why.

Additionally, this distinction clearly exposes (and forces you to deal with) two major Scrum “silent issues”. I’m talking about the magical, mythical being called a product owner that is supposed to somehow (now you know why that can never work) manage delivery just by prioritizing functional scope, and just-as-mythical self-organizing team that can somehow manage and address all those project environment issues as they appear.

Neither of these two exists “in the wild” in the vast majority of cases, so you need to address those issues and supplement needed expertise if you want to have any real measure of control over the delivery, and I find this concept of implementation backlog extremely illuminating and helpful for understanding this fact.

For example, for the product backlog, the project model is enough; however, for a good implementation backlog, you’ll need a product owner who deeply understands:

1. engineering process itself
2. strengths and limitations of the implementation team and the organization itself
3. project context (that part is covered by the remainder of the project model — the context part)

You can get someone who can do all that (business decisions, requirements engineering, and those items listed above), or you can split that role into multiple people — now that you know exactly which competences you need, instead of just one big and misleading “product owner owns backlog” statement.

In my experience, understanding this distinction and (more importantly) addressing it significantly improves the delivery management on any backlog-driven project, be it agile or simply iterative.

Role of requirements in the era of agentic engineering

If you wondered whether requirements still have their place in the era of agentic engineering, where it seems that all you need is an idea, a prompt, and the rest just comes out like magic, let me assure you that they do, and probably even more than you think.

First of all, if you truly think that all you need is “a good prompt,” you’re in for a rough awakening once you try to engineer anything non-trivial.

And second of all…

The primary role of requirements does not change with the introduction of agentic engineering practices and will not change until there are at least two humans involved in the process of software design, implementation, and delivery.

The reason is simple — the primary role of requirements is to create a mental model that can easily be shared between stakeholders, and that role is central and will remain to be central as long as humans are involved.

For implementation, though, well-defined, definition-of-ready-conforming PBIs are a prerequisite for reliable implementation and delivery (also see internal standards chapter a little later in this text), and that prerequisite does not change regardless of whether the implementation agent is a remote human or an “AI”.

I’ll illustrate this with an example — would you consider redesigning your whole requirements writing “game” if you were to add a remote developer (or three) to your implementation team?

You wouldn’t, right? (If you think that you would or should, you might have a nasty problem with your engineering pipeline, and you really should go and see someone about that, but I digress...)

My claim here is that adding coding AI agents to your implementation team — provided you are doing it correctly (see this article to get a better idea of what I mean by the term “correctly”) —should in no way shape or form have more pronounced effect on your overal engineering approach than adding a human remote dev to your implementation team, and your requirements engineering processes should really never notice the difference or need any serious adjustments.

And if you follow layered requirements principles, they won’t, and will be “right-sized” as a bonus.

Sure the level of details might vary (decrease), the iterations might be much quicker and just-in-time discovery might happen over mere minutes or hours rather than days and weeks, but all the principles will remain just the same and since the key role of requirements in any software project is to be the source for a shared mental model — as long as there are entities that needs to share the model, requirements are not going anywheree.

Requirements maintenance post-development

It’s time to address this quite significant elephant in the room; all this we’ve discussed so far (hopefully) makes sense for discovery, design, and implementation phases of the project, but the million-dollar question is “what happens to all those requirements after we deliver?”

If approached naively, the requirements quickly drift from the actual system implementation as many small changes, bug fixes, etc., are added to the system post-delivery, and implementation details change over time. The only way to prevent this (in this naive approach) is to more or less manually maintain BOTH the code and the requirements separately, and that, unfortunately, although it sounds feasible in theory, in practice, it just is not.

That said, there is one part of the requirements that does NOT change often or quickly — and is by lucky coincidence also the most compact one. I’m talking about a functional model. Think about it — the system was built to serve a functional purpose — that functional purpose does not change quickly over time. What does change over time is implementation details, an acceptance criteria is added here and there, UI gets changed a bit, or UX does — but the parts of the functional model they refer to remain largely the same.

What that allows us to do — and that is the approach the layered requirements are extremely suitable for — is to basically

“torch” all the implementation requirements on project delivery.

And when I say “torch,” I DO mean to completely get rid of:

1. scope lists
2. assumptions list
3. Implementation layer
4. Acceptance criteria

Basically, you just leave the functional model, NFRs, actors with vignettes, and the project summary. That’s it. That part remains the same, everything else will change quickly over time (details layer and acceptance criteria) or was relevant only during the implementation phase (scope, assumptions, etc)

What is left over is fairly small in size and low in detail (and therefore easily maintainable) — while still modelling the entire system and enabling any stakeholder to quickly build a mental model of the system at any point in the future.

With that said, I want to point out that any stakeholder can retain any part of the requirements they feel they need — this is fine, it is just outside this maintenance framework and therefore strictly speaking is not 100% necessary — but it’s not prohibited either.

However, there is one huge prerequisite before you can do any of that — and become free from needless requirements maintenance while still retaining the easily accessible functional model as a single source of truth when it comes to the system’s purpose and functional scope.

That prerequisite is good, comprehensive, and AUTOMATED functional test suite and a corresponding CI pipeline.

On top of that, your tests cannot be a tangled mess that only focuses on coverage, but rather they must be written and structured so that they are readable to non-coders (but also not completely a-technical) stakeholders.

That means writing proper functional “describe” and “it” statements in your tests — if you use the Jest testing framework, for example, or using Gherkin statements with ATDD tools, etc.

This is critical, don’t gloss over this — without that, the discoverability of implementation details for particular parts of the functional model becomes almost impossible or at least seriously impeded.

The logic behind ditching the written detail layer and replacing it with a functional test suite is that the implemented system’s details are fully and authoritatively defined by the implementation itself, and automated tests define/describe that particular implementation and its details. On top of it all, automated tests are (provided you have CI) “auto-verifying” in the sense that:

1. If the underlying behaviour changes and is no longer in tune with the behaviours the automated tests “fix”, your CI pipeline will fail — and will keep failing until you rectify this discrepancy
2. If your system grows without tests following that growth, your coverage will go down, which is admittedly a lot fuzzier signal than failing CI, but still a very obvious and actionable one nonetheless

None of this is a new concept; it’s called living documentation or executable specification, and we have known about it for decades now (also, we have known about the crucial importance of automated testing for roughly the same amount of time).

What usually prevented us from practising these obviously sensible and logical practices was the cost of writing comprehensive, usable test suites, which, for most teams, was prohibitively high (see the fallacy of expecting a mature self-organizing team in the previous chapter).

The good news is that that is no longer the case due to the advent of agentic and AI-assisted engineering (provided it is done as expert-driven engineering and NOT as vibe coding). Therefore, this practice is finally available to anyone who wishes to use it, and it gels perfectly with the layered requirements approach and its concept of functional model. Or at least that is true in my personal experience.

So to sum it up, the functional model + NFRs (you can have automated tests for those as well, btw. but that is besides the point) represent a compact, easy to consume “mental model builder” that changes little over time and can be consumed by any stakeholder in the future when needed. A built mental model provides grounding and the context for all the implementation details that are defined within automated tests, and that is crucial if you need to use or evaluate those details later in a proper context and not as isolated islands.

Why does having internal standards help?

Having internal standards is the most surefire way to prevent “but no one told me that I needed to write a test for it” discussions mid-development.

Regardless, in a lot of organizations, standards are often considered as something “extra”, something to have as a sign of maturity, but at the same time, often not really understanding how to properly use them, except through a “decree” and education.

And that is the shame, because in their core, internal standards and their usage are really simple and down-to-earth concepts rather than something complex that only “big boys/girls” need or know how to use.

When you think about them…

internal standards are nothing more than a pre-defined set of NFRs that apply for particular domain

For example, you might have code quality standards, SEO standards, test standards, usability standards, etc., but all of them are just requirements that apply to a project/deliverable as a whole.

Put another way,

they are just acceptance criteria that need to be satisfied for some artifact to be considered “compliant”. Nothing more, nothing less.

And that is (or at least) should be good news as acceptance criteria, their writing and their usage should be something well within our comfort zone by now, so standards really do not introduce any new complexity or special “usage instructions.”

So my advice to you is to figure out what do you want to be true for every artifact that gets delivered by implementation team, write in a form of list of acceptance criteria, make those explicitly linked to every PBI (or just create a NFR that project fully conforms to standard X, Y or Z) and you are golden (and if you use requirements forge tool, all that linking and inclusion in PBIs will be done for you by the tool :) ) and there you are — you have internal standards and what’s more imporant, you know what to do with them so they aren’t just dead letters on paper/screen.

Applicable tools

As mentioned above, layered requirements engineering is a non-opinionated framework when it comes to anything outside the internal structure of the requirements. That means that the choice of tools you’ll use to produce your requirements documentation is left completely to you.

For example, everything presented so far works perfectly fine with Jira, Azure DevOps, Trello, or just plain old Google Docs or MS Word.

The only feature of those tools that is really mandatory is good review and collaboration features. Those should include threaded commenting and the ability to assign threads to particular individuals and to explicitly resolve each opened thread with a proper audit trail. That is crucial for the review process, which is the topic that is both incredibly important and that also falls outside the scope of this particular article series.

So, although any tool conforming to the requirement above will work, here is an example of tools I typically use:

• When I write SRS documents or manifests, I use Google Docs and apply the continuous collaborative review approach whenever I can
• When I create backlogs, I use whatever tool my clients use for that purpose — usually it is Jira, Trello, or some other tool that does essentially the same thing.

For diagrams, especially those that will evolve during the project development, I use the diagram-as-code tool MermaidJS and integrate the resulting diagrams into the relevant requirements level as part of the detail layer.

If your team uses “physical” tools such as whiteboards, index cards, story maps, etc., you can continue using them without any modification. This framework is about producing the right content, not the tools you will use to document that content.

Some tools are better or more comfortable to use than others, but I’ve yet to encounter one that is so limited that it prevents applying this framework. As long as the tool has decent collaboration features, it will work well enough for our purposes.

Requirements forge

Although the framework itself is agnostic when it comes to tools, there were two main features that I missed from the tools I’ve just mentioned, and although those features were not by any means deal-breakers, and I’ve written many a good layered requirements document with Google Docs only, I still felt the itch that no tool I could find was able to scratch.

So I built one myself (or better said, I AM building one right now — it’s fully operational and I use it personally, but it is in public MVP phase, so if you’d like to try it out and give me feedback that will help me to make it better, you can check it out here)

The missing features I was referring to, and which this new tool, called (Layered) ”Requirements forge”, implements are:

1. fully hierarchical backlog management
2. auto-maintenance of referential integrity between every requirement element

In its essence, this is a tool for writing requirements following the layered requirements framework and managing those requirements over the project’s lifecycle — as a functionally complete, semantically consistent structure — rather than just a flat backlog of freeform PBIs or a large freeform textual document.

I won’t go into further details, as that will be a subject for another article, but I’d like to point out one important benefit that I think structured requirements model allows for — and which is relevant in the age of agentic or AI-assited engineering we’ve touched upon in previous chapters, and that is the ability to transform this core semantic structure into pretty much any format you need — be it just a large freeform docx for organizations that don’t tolerate anything except word documents, or (much more relevant) an complete input for implementation AI agent — with definition of done, clear testing criteria, testable functional requirements and all relevant NFRs etc.

So that’s it about this tool, once again, if you want to check it out, I’d appreciate any feedback you might give me

Conclusion

There you have it. We’ve come full circle. We started with a disaster scenario, which is uncomfortably familiar to all of us who have ever participated in non-trivial software projects, and throughout this series, we addressed every major failure point of that scenario — even if not explicitly.

The inability to MVP and lacklustre demos were addressed by impact-driven progressive delivery in part 1 of the series. We also revisited it in this one when we discussed how end-to-end does not really imply that ends remain fixed throughout the project. That is a powerful concept when you think of it — the only important part is that there is someone who can achieve something that they couldn’t before. Everything else is strictly speaking an icing on that functional “cake”.

The continuous mid-project “surprises” were addressed by explaining the role of the requirements in the software projects and how to structure them in parts 2 and 3. In particular, the shared mental model — the most powerful remedy for “but I thought that…” situations and also the most ignored aspect of requirements usage IMO.

Finally, the ability to properly “spike” and prototype during the project was addressed in this article and in part 1, when we discussed progressive delivery and clarity of insights into what is tricky on the technical level it can allow for. The thing is that prototypes, without a clear connection to the functional model and therefore without a functional purpose, ie, what the given experiments/prototype will demonstrate and which risk it will mitigate, will not be “comprehensible” to non-tech stakeholders and, to their ears, end up sounding like an abstract developer’s playtime rather than a crucial risk mitigation tool.

However, framing it in the context of relevant functional impact fixes that misunderstanding beautifully. The communication and framing are key here, and functional model and progressive end-to-end delivery are what make the framing and communication obvious and understandable to the people that “hold the wallet”.

Also, we addressed a few points that we left out of our disaster scenario (for reasons of brevity) — such as that dreaded post-delivery documentation maintenance that seems to always be in the “we’ll cross that bridge when we get there” category, and showed that the bridge is actually very obvious and the stream very shallow and was never really intimidating at all. Also, we’ve hopefully shattered that aura of “that’s not for us, we’re not Google” about having organizational internal standards. It is simple, you need them, they are not complicated, and so they ARE for you, regardless of how small you think you are. They help a lot, don’t resist them just because the word “standard” makes you think about ISO and all the (non-existent) fun that moniker brings with it.

To sum it up. Software projects do not have to be unpleasant experiences.

Sure, there will always be elements out of our control that can mess things up, but on a run-of-the-mill software project, those are surprisingly rare and often quite foreseeable and therefore preventable.

Now that you have tools and know-how, the level of comfort that you will experience on your next project is really up to you — now that you know how, it’s time to just put it all in practice and enjoy the ride.