“Either it is a well-arranged universe or a chaos huddled together, but still a universe.” – Meditations, Marcus Aurelius
The Legacy We Build Upon
The topic of lean metrics requires an understanding of the influence of Kanban and the Toyota Production System. Scrum, as an agile process framework, was built as a lightweight version of the TPS Kanban practices that anyone could use, while Atlassian developed JIRA to enable visualization of any process, regardless of its complexity.
The “priority” field originated as part of Jira’s oldest legacy as an issue-tracking system. Greenhopper was a plug-in that enabled the issue-tracking database and web services to be used for Lean-Agile teams. Greenhopper created a new front end for the data in JIRA – namely, the work-in-progress board and the product backlog – so that a Scrum team or Kanban team could use JIRA for agile. Ultimately, this became synonymous with JIRA and the today it ships with the agile front end OOTB.
In Scrum, the Product Backlog is used for prioritization, so the “priority” field has little meaning. In Kanban, the priority field is used to create swim-lanes based on Classes of Service.
Expedite Anything That Blocks Value Creation or Capture
“Blocker” would indicate that WIP limits can be violated and workers should interrupt their current work in favor of Expediting the blocker through the system so it doesn’t not cause long-running damage to continuous flow. The equivalent in Scrum is building out a process for stories, tasks, or bugs that are allowed to violate the Sprint commitment. In either case, the goal is to recognize that this is costly and unhealthy, an exception to normal rules. Unless it is a production defect, some element of the expedite cost should levied against the person demanding special treatment.
Flush the System of Any Critical Items that Disturb Continuous Flow
“Critical” would indicate that a card should “jump over” all other items at each step in the process, but it should not interrupt work or violate WIP limits. These items get to cut to the front of the line, but are not allowed to interrupt completion of existing efforts. The equivalent in Scrum is building out a process for what items, if any, are immediately prioritized to the top of the Product Backlog. It is typical for this to be a “Fixed Date” class of service – because fixed dates are the most consistent destroyer of sustainable flow, we want to get those things out of the way quickly so that the system can return to normal.
Everything Else Follows the “Normal” Process
“Major” and “Minor” and “Trivial” are typically part of the same Standard (aka normal) Class of Service. If used in Scrum, it is primarily for the benefit of the Product Owner to visualize previous conclusions about prioritization. In Kanban, these are meant to respect all WIP limits and follow a First-In-First-Out (FIFO) method at each step in the process.
A system – as we will define it – consumes resources and energy to produce something that is more than the sum of its parts. Not only does is produce value it does so in a way that sustains its own existence. If we consider Henry Ford’s early Model-T production system that assembled automobiles, the raw materials – rubber, coal, plastic, steel – were meaningless as an unformed heap. Along the way, the “intrinsic” economic value of the raw materials were destroyed and could no longer be sold for their original price as raw materials. At the time, there would have been no resale value for many of the assembly pieces, because Ford created an entirely new value network and disruptive business model to create a market that could properly assess the value of the non-luxury automobile. Yet, once assembled, the assembly line put these pieces together to create value greater than the sum of its parts.
An example of a relatively simple organic system is a single-celled organism like some species of Plankton our oceans. A plankton lacks sophisticated embryogenesis, there is no differentiation of multiple tissue types, no embedded systems, and no coordination mechanism across cells. Nevertheless, the simple biochemical processes and the internal workings that complete these processes have continued for billions of years by not only producing its own self-maintenance, but also by managing to reproduce. There is a surprising large amount of DNA for such a simple, small, organism – but why did this legacy of code begin amassing in the first place? Whether we venture to call it “divine” or not, there was certainly a spark of some kind that began an explosion that has yet to collapse back into chaos and the dark.
Even with these simple systems, where we can trace each exchange in the value-transformation process, including materials, structures, energy, and ecological context, the sum total of the Model T and the factory that produced it is more than its parts heaped separately in a pile. Our difficulty in understanding such systems is a problem of multi-fractal scaling. For now, let it suffice to say that making a variable in a system better may not result in a linear change in outcome.
A Complex System
We have major issues understanding how (or worse yet, why) a system consumes resources and energy to produce value in excess to the sum total of the elements and energy amassed in the absence of the system that produced it. This problem is only compounded when we begin embedding specialized sub-systems within an organism. In the example of an automobile factory, we could say that every cell of every person is a system, that each person is a system, and that each distinct functional area, separated by distance, is a system. The accounting and finance “system” and the inventory and assembly “system” must interplay as part of Ford Motors, a system in its own right.
So we can define a complex system as having embedded sub-systems, causing the observer to not only see that the whole is greater than the sum of its parts, but the observer may also slip into a “confusion of levels” if they attempt to manipulate a part of a system to shift the outcome of the whole. Worse yet, confusion of levels can have disastrous, non-linear results that are the opposite of the intended change due to confusion of cause and effect. When sub-systems are embedded within each other, their interrelationships may act on differing scales, either in time or place. So we must careful when attempting to improve a complex system. We must use empirical process control to chart the change in systems outcomes rather than simply optimizing subsystems in isolation.
A fractal is a pattern that repeats self-similarly as it scales. One of the most common fractal scaling patterns in nature is branching. From the trunk of a tree, to major its major limbs, to twigs, and finally leaf structures, this fractal scaling pattern enables a lifetime of growth cycles. Leaves can bud purely based on opportunism, in a relatively disposable manner. This is because the tree, as a seed, has all the legacy of generations of trees locked inside it. The tree does not aspire to be “the perfect tree” or assume that it will grow in perfect sunlight, humidity, soil pH, and water availability. The tree does not get angry when a major branch is broken off in a storm or struck by lightning. Instead, its fractal scaling pattern is prepared for intense competition for sunlight in the sky and resources from the ground. The tree’s scaling pattern has risk mitigation “built in” because it grows the same in the middle of a field with frequent rain as it does in a dense forest.
We see this branching strategy throughout nature, from ferns to human blood vessels. However, an even more effective approach to self-similarity comes from multi-fractal scaling. The ability to adaptively select between more than one repeating pattern or differentiated patterns based on scale requires a different kind of fractal: time-cycle. It is not just the branches of a tree that result in an environment-agnostic strategy for growth, it is the adaptation to cyclical daily growth, scaled to cyclical annual growth, than scaled to multiple generations of trees that grow. This final step is an important one. Multi-fractal scaling is not only the source of novelty and adaptiveness “built in” for a single tree, it repeats at an even larger scale as a species competes for dominance of a forest. Multi-fractal scaling encourages “just enough” opportunism to enable small-scale experiments that can be forgotten without loss at a greater scale, or thrive when conditions change.
Adaptive Multi-Fractal Scaling
The strength of multi-fractal scaling, from branch to tree to forest, is its total reliance on empirical process control. The legacy code is a confusing jumble of competing messages that a human mind, attempt to “engineer a perfect tree” would attempt to simplify and beautify. That legacy code, however, wasn’t written with any intention of crafting a perfect tree. That code was written to create a minimally viable reproductive system. It is built for one thing: continuous experimentation.
Continuous experimentation happens at each level of multi-fractal scaling, risking economics appropriate to its scale to find asymmetric payoffs. An Oak tree risks very little per leaf that grows over the entire course of its life. In a dense forest, however, that continuous experimentation of growing leaves higher and more broadly opportunistically based on local returns on investment can suddenly break through the forest canopy or unexpected fill the hole left by another tree’s broken limb. An Oak tree does not require centralized control of where leaves will grow or which limbs to invest in. Instead, the legacy of continuous experimentation enables multi-fractal scaling that competes locally and opportunistically.
Again, we do not need to understand what spark set this fire ablaze, we only need to see that it is still spreading and we are a part of it. Over-simplification of superficial outcomes will lead to poor decisions about inputs. Organic leadership relies on context, structure, and enablement of continuous experimentation. Organic leadership is a “pull” system that relies on scaling patterns for decentralized empirical process control. Artificial “push” systems force requirements and attempt to bandage the inevitable inefficiencies of a non-adaptive system.
A Complex Adaptive System
A complex adaptive system does not merely take in resources and energy to produce itself and reproduce itself as a unified “whole” that is greater than the sum of its parts. It does not merely embed subsystems with multi-fractal scaling and decentralized control. A complex adaptive system also operates with a continuous experimentation system built in to its normal framework of activities. When we make the leap from an Oak tree to the human body (or any other mammal on Earth), we can truly appreciate just how complicated it is to improve the health of an individual, or an entire population, when we observe the interrelationships of various physiological and socioeconomic systems and sub-systems. Creating lasting change is not only complicated in terms of finding the correct level and understanding the full ramifications across the entire system, each complex adaptive system is also continuously experimenting and will adjust against such changes based on short-run, local, decentralized opportunism.
To care for a complex adaptive system requires not only an understanding of inputs, processes, and outputs, but also the multi-fractal scaling of continuous experimentation that maintains long-run viability. When short-run economics are working against long-run viability, it is not sufficient to reward “correct” behavior to counteract short-run opportunism. Instead, we must shift the context of local decisions so that short-run opportunism serves long-run viability.
Accidents Will Happen
Accidents may seem to the observer to be unintentional, but continuous experimentation is built to test the boundaries of success, to ensure that precise empirical process data is also accurate for the needs of viability. In other words, if you’ve ever accidentally tripped and fallen, or accidentally loosened your grip on an egg and dropped it on the kitchen floor, this was a natural element of complex adaptive systems quietly running experiments.
Embedded in our own human code, our sub-systems are all built for continuous experimentation as a method of calibrating precision to accuracy, using multi-fractal scaling on short, long-short, long, and distributed cycles. A short cycle is an immediate reference point for an event, using data held in working memory, and is reactive to immediate changes. A long-short cycle compares current data to immediately recognizable patterns of events, more embedded memory or conditioned responses that have proven useful over time even if we assume the event is an occasional outlier. More significant, painful events can skew our “normal” for decades and even become passed to the next generation as part of our genetic code. A long cycle has been stored to our genetic hard drive for future generations. A distributed cycle is a socioeconomic artifact that requires a medium of exchange and may last for centuries.
As humans, our multi-fractal scaling of continuous experimentation results in the creation of complex adaptive socioeconomic systems. Our legacy code drives us toward exchange, tooling, building, and reproduction because the experiments that are in motion are far from complete.
Like our occasional fumbles and falls, our social systems produce results that appear to be accidents with no guilty party, pure coincidences of circumstance, which occur due to failed experiments. Organic leadership harnesses this natural propensity for decentralized opportunistic experimentation by encouraging it but setting boundaries for it, feeding it but ensuring checks-and-balances from opposing interpretations, and guiding it by changing context and opportunity rather than directly managing outcomes.
DevOps. SAFe. Scrum. Kanban. Did you shiver? The experts have you scared you’re doing it wrong (you know, IT – all of IT). The certificate-teachers need you to need them so they can get paid. The coaches and consultants are taxing your #FOMO (Fear Of Missing Out) on ALL THE AGILE.
It’s an impossible dream: The perfectly lean and agile software process would be just one person – a masterful engineer with great people skills and business acumen who can hang around a group of people to see what troubles them, create an intuitive software tool that relieves the pain, and collect cash from them while obtaining ultra-constructive feedback. In other words “dev” and “ops” as one person.
Ever seen that happen?
That said, every additional person involved in the flow of DevOps information is either a value-addition or a source of systemic waste. More often, everyone involved is a bit of both. As more bodies are thrown at a problem, it is likely any one of them can only spend 10-20% of their time actually adding value. The other 80% – and all overtime – is typically waste due to waiting on information from others, or over-documenting our value-add so others can consume it.
So it isn’t that IT won’t tell you the secret, if they know at all. It’s easy to crack jokes about how little other people accomplish despite working long hours, but less likely that anyone raises their hand and volunteers the truth about how much time is wasted.
– Waiting for clarification
– Waiting for updates to install
– Waiting for SVN code check-in
– Waiting for someone else to check their code, setup a VM, or answer what their code means
– Waiting for the next meeting
– Waiting for the inspiration to work on something meaningless to you
The great big DevOps secret is that no methodology will fix all that waiting. Over-specialization creates a world in which teams throw big piles of shit over the fence, each group speaking it’s own language that no other group understands. New philosophies won’t fix bad relationships, a toxic culture, and shit tools. It takes time, work, and fearless leadership.
Otherwise, Agile, Scrum, Kanban, and DevOps – or any other recipe for “when to meet” and “what to document” – are like 30-day fad diets; it’s a false hope and a gimmick and it doesn’t change the sedentary fast-food lifestyle that keeps you fat in the first place.