How to Fail at Agile, Ruin Your Culture, and Lose the Innovation Game

Cover photo via Andrew E Weber

There is one major issue that – if not properly addressed – is going to kill your company.  In fact, many companies FAIL MISERABLY when trying to break down silos, increase collaboration, and begin competing as a culture of innovators.  Every company must innovate or die.  Every product will face competition from a connected variant supported by a mobile app.  Mature enterprises who have never competed on innovation before are now suddenly competing with innovative tech companies!

Tearing down functional silos so that solution-oriented, self-contained, fully-functional and ultra-innovative collaborative teams is now THE biggest overarching challenge that will make or break enterprises that are struggling to maintain their decades-old success and strategic position.  Moving to agile in maturing industries, that are now being utterly disrupted, has become unavoidable for most companies.

The most common label for repositioning to compete on innovation is “agile” and breaking down silos is labeled “agile transformation”. 

There is a major failure that is gradually breaking many companies as their agile and high-tech endeavors fail miserably.  This is issue is caused by executive leadership’s denial and insecurity to face the terrifying reality that to truly “break down silos” is NOT (exclusively) a process transformation or change management issue.

Ready?

Here it is:  Breaking silos REQUIRES organization restructuring.

This major issue is two-fold and occurs simultaneously:

1. Failing to have “tech” report to “business”

2. Failing to have “business” people report to “tech”

There is an unwillingness to break apart and completely restructure the separation of “tech” disciplines and “business” disciplines, despite the growing number of roles that are impossible to clearly classify as one or the other.  In the sad but traditional approach massive enterprises take in the face of a major strategic decision, many companies are attempting a straddle-the-fence strategic position by attempting to leave their existing antiquated structure intact while building entirely new divisions modeled on the structure of successful tech companies.  This is likewise problematic – to succeed at this, revenue growth must accelerate enough to both maintain the old organization while building the new division(s) and the corporate efficiency gains of sharing administrative and brand resources means the new groups will adopt many of the dysfunctions of the old organization.  To the extent this is a Zero Sum Game played out against other competitors and multiple competitors are attempting this simultaneously, no one is likely to “win” the game.

What is the problem?

Let’s look at the first hard lesson facing companies attempting to compete against innovation-base startups and high tech for the first time.  They bring in consultants to help them break down silos, communicate cross-functionally for the first time, and establish a shared vision.  They bring together teams across various business functions, bring in agile coaches, and try to get everyone to collaborate.  They bring in vendors who seem to be better than them and try to “copy their homework” so to speak.  With enough cash, they may even implement widespread training and certification and bring in new employees who are from flat, agile, startups.

The problem that companies introduce when they do this, at any given level of the organization:  without restructuring, there is a “business” and “tech” person set together in equal standing.  If they get it down to two, that is an accidentally triumph.  It is more likely a three to four person group representing a) marketing/sales b) technical leadership c) requirements and project management d) creative design.  There is no clear empowerment for any of them, because mentoring, performance review, and promotion or disciplinary power are all linked to other pieces of the organization.

This diffusion of responsibility was the failure of waterfall in the first place!

Attempting to preserve a forced distinction between “business” and “tech” as well as the hierarchical position of leaders, then forcing “equals” into a nightmare arranged marriage on a per-project basis is a recipe for a slow but unmitigated disaster.

Taking a step back to “Core” Scrum

In software development today, virtually every high tech, innovation-based company is delivering products with some mix of agile values, Scrum ceremonies, Lean improvement efforts, and XP best practices.  While companies may call their process Agile, Scrum, Scrumban, Kanban, etc – let’s talk generically about “core” Scrum and why it succeeded where “waterfall” failed.

Our imaginary “Core” Scrum team is a small startup with a visionary leader who is a fantastic mix of technical proficiency, business acumen, and management skills.  We’ll call this person the Product Owner but in remember, this is the CEO, product manager, and sales-plus-marketing all at once.  For simplicity, everyone else in this company of about 12 is part of the software delivery team (all other roles like accounting are outsourced).  They are The Team, and one of them is the Scrum Master who keeps the process of delivery rolling smoothly, constantly challenging everyone to collaborate more.

That’s “Core” Scrum in a nutshell:  The Product Owner is roughly 50/50 split-focus between the Team and stakeholders, the Team only needs to converse with one voice of prioritization, and the Scrum Master keeps this one small group on-time, ever-learning, and un-impeded.  The CEO owns the company, its only product, and all stakeholder relationships (investors, customers, end users).  This is “pure” Product Ownership – the entrepreneur “lives and dies” by the sword of her decisions because the power, responsibility, and accountability for every promise and decision is firmly on her shoulders.

The purity achieved here is simple to understand:

the company role and its associated place in the superorganism’s Prestige Economy is perfectly aligned with the project role.

The Scrum Master manages no employees and builds culture, collaboration, and keeps the Product Owner’s relationship with The Team healthy and strong.  If this were a highly technically complex product, the ScrumMaster could be a very experienced architect who is can assist in code quality decisions, pair-programming to overcome tough coding challenges, and resolving any risks to the long-term sustainability of the code.  In an extremely technically complex engineering environment, it would make sense that this (dare I say) Architect is a thought-leader but not a people leader.  This Architect knows all of the code and is on speed dial for production issues.  The strength and trust it takes to successfully push the CEO to prioritize refactoring over new features, hardware investments versus new hires means the Scrum Master must be a right-hand-man to the CEO but have no agency dilemma or conflict of interest in making certain the CEO (Product Owner), the Team, and derivatively (via total process quality) the Product are at any given time healthy.

Naturally, we can assume a different but very valuable breed of Scrum Master is also possible.  If we imagine a software product where technical uncertainty is easily handled by an experienced team of engineers but the business rules are extremely complex, a business analyst may be the perfect Scrum Master insofar as Backlog Decomposition and the precise details of User Stories will require a business person with the Product Owner on speed dial who can resolve every business rule uncertainty.

Just like our anecdotal Product Owner, we see in each anecdotal Scrum Master, due to the nature of the product, the company role and the project role are perfectly aligned.

Scaling vs Scaled

This simple paradigm is compelling due to the clear accountability and laser-sharp focus  that makes it successful.  With the single, properly-sized team described above, it is easy to understand why it works and how emulate it.  Confusion tends to set in as the number of contributors and teams grow.

There is an important distinction to make between scaling this pattern and implementing the pattern at scale. 

Continuing our narrative about this successful “core” Scrum team, we can imagine the company grows organically as the product becomes increasingly successful.  Revenue is great, margins are fantastic, and more engineers join.  It becomes clear to the CEO (and Product Owner) that second product would be a perfect strategic fit.  The product, focused on different niche in a similar industry will leverage the skill set of the engineers, design patterns, and the libraries and APIs that have been built over time.  Stakeholders are excited, ready sign on the dotted line.  The engineers are thrilled.  As a group, they decide to split into two teams, each focused on their product.

This Scrum implementation has begun scaling.

The shared code and strong culture combined with the mature agility and engineering best practices allow the ScrumMaster to happily work with two teams.  Unfortunately, the CEO is now  splitting time roughly 20% stakeholders and 10% team on product #1 while 40% stakeholders and 30% team on product #2.  Days get longer and attention to the backlog is suffering.

The CEO is no longer effectively owning either product.

Her trusted colleague, the ScrumMaster talks through the risk of “Hero Scrum” and the need to delegate, and the risk of poor feedback or lack of actionable backlog when more engineers and another designer will be joining in the summer.  There is demand to have the APIs and an SDK consumed B2B.  The decision becomes clear – two experienced Product Owners are hired, one to manage the consumer apps Scrum Team and one to manage the B2B cloud platform. 

The CEO sees that leading the company’s strategy, defining new products, and the need to build a brand, while mentoring the Product Owners on great management of their categories and teams is her new “job role” and “project role” – it evolved organically, and the two still align perfectly. 

This is a happy company.  More products follow, and while not all of them are an instant success, the values, vision, and identity stay strong.  Camaraderie, disruptiveness, and culture of innovation keeps this company profitable at-scale, through a constant assurance that the job role and project role, whether a Jr. Developer or a Marketing Specialist, are always properly aligned.  People are asked to own “one big thing”, empowered to succeed, and held accountable.

Paternalism and Guild-based Hierarchies

If you work at a large company and this happy anecdote sounds ludicrously fictional, it is because implementing agile in a scaled environment – whether due to the success of an isolated team or a management decision to copy Silicon Valley – rarely looks like this.  Large enterprises in mature industries were born out of organization and management practices dating back to the British Empire and the industrial revolution, and function-based guild preservation that began in the Dark Ages. 

Blacksmith best practices evolved, certainly, but not so fast an entire practice community couldn’t keep up.  The capitalists in the industrial revolution could invest in a few big innovations, paternalistically demand new behavior from a new “guild”, and reap benefits for decades. 

Marketing, even as the internet bubbled, consumer product research and development despite the worldwide adoption of desktop computers, and a whole guild built around Information Technologies did not disrupt this proud history of dividing the troops into functional units held together by rules, history, and strong paternalistic leaders.

Transforming Project Roles While Preserving Job Roles

The United States Military is realizing, unit-based combat in an urban setting full of emerging demands and uncertain risk.  Cross-functional teams are being built tactically insert their blend of skills into situations that require empowerment and responsiveness.  The days of marching pikemen and flanking with cavalry are gone.

Enterprises are realizing the same is true of innovation-based product markets.  A level of guided group-think is needed to courageously build an innovative product for a still-emerging market segment.  The capacity to “lock everyone in a room” that you need, and risk market and technical uncertainty requires executive mandate and tenacious teams.  Limiting investment to two weeks of team effort at a time, fierce concentration, no politics, and freedom to experiment are essential.  This is a creative process that requires adventurousness and tenacity.  Let adults succeed and celebrate or fail and learn.  Otherwise management is simply holding back the potential of their most brilliant engineers.

To do this, one thing must happen above all else.  The job role must perfectly align with the project role.  If job descriptions align to guilds, as a talent “pool” of human “resources” to get allocated like widgets in a managerial accountant’s spreadsheet, the project – which is intrinsically finite – will always suffer.  The job is linked to performance and salary. 

Predictably, people behave like a faceless number on a spreadsheet when you manage them (and pay them) that way.

The move to “agile” is in large organizations is nothing more than an attempt to move to a company that has established a culture of innovation as its sustainable competitive advantage.  The four cultural choices are simple to understand, but they inevitably fail without a strategic mandate from executive leadership, and the “blessing” to fail when overcompensation in the effort to change occurs.  As a leader, tell your people these four virtues are the key to success and back it up in your daily actions:

Collaboration Focus –  Put all the right people in seats next to each other, tell them they have one goal: unlock your full potential by learning from and investing in your team as you solve tough problems.

Product Focus – A project is finite, a product is a living platform for an ongoing conversation.  Build a dialogue, tell a story, and leave a legacy.  Do you really want to control your budget and schedule, or do you want to change the world?

User Focus – The ideal information symmetry cycle:  The user Tweets a feature they want, the team codes it, knowing exactly what the Story means to the User, and the user has it in-hand within a month.  The further you alienate the producers from the consumers, the more wasteful and less innovative your organization will be.

Innovation Focus – Taking risks and learning from mistakes is mandatory.  Encourage it and live it. There is infinite content available and your well-formed team’s natural curiosity are their default state.  If they aren’t innovating, you’ve burned them and overburdened them.  Give them some slack and encourage their interests – you’ll be amazed what the unbridled human accomplishes.

Issues At-Scale:  Agency Dilemma

At scale, reorganization – not just redecorating – is going to be critical.  There will be politics to deal with.  It will take immense courage and backbone as a leader.  Middle managers will be your biggest problem until you change the seating chart and pick the right people for those seats.  Agency dilemma from the guild-based job hierarchy will ALWAYS impede the success of cross-functional teams.  Functional managers encourage non-collaborative activity.  Over-documentation.  CYA.  A lack of focus on the team and product. 

Why is this so prevalent? 

It is not simply the change management of restructuring, changing number of direct reports, flattening the organization – that is all incredibly well-known turf for large, established enterprises in mature industries.  The new political problem is two-fold and pragmatically simple:

1. Failing to have “tech” report to “business”

2. Failing to have “business” people report to “tech”

Unfortunately, the politics of putting previously separated guilds in charge one another is  not the root cause here.  You could put an abstract painter in a direct management position over a cross-functional software team so long as they had the skills to lead them to success and prestige.  Instead, it is a symptom of leadership not understanding the team-product-user relationship.  In “core” scrum described above, this was simple.  The start-up was built around it.

How you should restructure, once you “get” it, would be fairly simple: If the user has needs that are solved by a product that will need a team that must overcome technical uncertainty first and foremost – a more technical manager for that product team makes sense.  If the user has needs that are solved by a product that will need a team that must overcome market uncertainty first and foremost – a more business-savvy manager for that product team makes sense. 

In other words, when an old business problem can be solved by new technical innovation, let an engineer lead the cross-functional team.  When an old (enough) problem can be solved by technology by changing the complex business problem (think finance) – put a subject matter expert as the team manager. 

If the multiple products fit into a “category” where the users are likely to consume multiple products, scale to a director who can set strategy, build a brand, and ensure cohesion.  This person will likely be a mix of skills so significant that designating them “business” or “tech” simply makes no sense. 

Diffusion of Responsibility

This brings us full-circle back to the reason why organization restructuring needs to occur at all – multi-bossed employees are easily distracted.  Too many leaders goes hand-in-hand with sharing resources.  No one knows what any given employee is doing while no employee is terribly confident which leader to follow.  In a guild-based system, they fall back on their functional allegiances and their job description.

When you make an engineer (with experience taking new products to market) the direct manager of a cross-functional, self-contained product team that can collaboratively handle business+tech roles needed for a highly technical product – planning, design, engineering, quality, and marketing – that manager has a pulse on what everyone is doing and everyone knows who to go to when their workload gets too light.  The same is true if this manager is from a business background.  The end user is well-known, the backlog is simple to build, and the product is the professional focus of everyone in the room. 

This independence and focus is the building block of a culture of innovation.

Shared resources and functional guilds perpetuate agency dilemma, information asymmetry, and diffusion of responsibility.  Kill it by restructuring before it kills you.

Scaling Issues: Copying Enterprise Dysfunctions

To be thorough, this is not exclusively an issue faced by enterprises shifting to innovation and high-tech at-scale.  Growing from a startup to a mid-size company and beyond means personnel growth and collective skill-set via true talent acquisition.  However successful a small company was at innovation through collaborative, cross-functional problem-solving, pulling in new people is incredibly dangerous.  The temptation is fill in the top and middle with highly experienced managers and individual contributors.  Enough new hires with more than a few years experience inevitably bring with them the “baggage” of their guild-based past and the political self-protection of the toxic culture they left behind. 

If you are currently scaling, learn from the issues facing scaled transformation and protect your culture, mission, and values from the threat of new and toxic influences.  Grow teams into categories, categories into industries.  Agency dilemma, information asymmetry, and diffusion of responsibility are the inevitable dysfunctions of a company that loses its sense of identity by growing too fast through acquisitions. 

First of all, grow from your bench.  Being a small company is no excuse for constantly showing employees they will never progress in their career by hiring knowledge and leadership from the outside.  Second, when a subject matter expert can only be acquired, it makes no sense to put them in a leadership position as well – they need to spend their time educating the organization. 

To succeed in scaling a small culture of innovation, keeping every person at every level fully focused on their end user, their team, and their product is the only path.