Last year, Stormpath made the big shift from Scrum to Kanban. While we love Agile principles, the Scrum process wasn’t working for us. Kanban made our team more efficient, happier, and increased our focus on quality software. More importantly, it has become a core part of our company culture, and is now used by non-technical teams like Marketing and HR.
Kanban software development focuses on continuous delivery and drives high efficiency by limiting how much work can be done at once. Invented by Toyota and modified by David J. Anderson for software development, Kanban can have a huge impact on modern teams delivering cloud software in continuous environments.
Scrum, It’s Not You, It’s Me
Agile principles have always been important at Stormpath, and we use them on every team from Engineering to Marketing. Standups give our team transparency and eliminate blockers. Breaking work into minimum viable features gives our engineers a sense of progressing successfully. These small-step iterations – driven by user stories from our customers – help us improve the Stormpath API as we learn more about how people use it.
The sprint-based Scrum process, however, wasn’t working for us. It was too rigid, created too much overhead, and started to cause burnout on our team.
Overhead
We’re a start-up moving extremely fast with a very limited set of resources. Scrum had a lot of overhead: sprint planning was a half day event for our entire engineering team, plus preparation for the meeting for our product team. It made everyone feel involved and helped with coordination but that’s not a pain point for us. We’re already very collaborative. Sprint planning forced engineers focused on one component to sit through a discussion of unrelated components — for hours. Additionally, we would get into long (and often fruitless debates) about issue priority and scope that would drag in the whole team.
The Myth of Estimation
At its core, Scrum tries to drive productivity by imposing a time limit on work. A sprint is a bucket of available work hours that gets filled with issues based on time you think it will take to complete them. Like most teams, we have some over-estimators and many under-estimators. I can tell you with absolute certainty that 100% of us are bad estimators. Time spent trying to divine and debate a feature’s size felt like wasted time because we were probably wrong anyway. Moreover, what did it matter? If it’s a priority for our customers, it needs to get done regardless of size and, as a rule, we were always breaking it down to its smallest viable scope.
Scrum assumes you’re good at estimation
Bad estimation – rampant for many Agile engineering teams – caused a lot of problems. Issues turn out to be bigger than expected, now the sprint is overloaded. We kept finding ourselves jerry-rigging scope – even more planning meetings – to match the artificial construct of a sprint, gaming the process to very little business benefit.
Change Management Soup
Need to move fast and operate in an environment of uncertainty? Scrum complicates that. Once sprint planning is done, making changes to the sprint triggers more planning meetings. Sure, you want changes to work-in-progress to be painful as a deterrent to a meddling business team, but what about changes to work committed to but not yet in progress? What if a big bug is discovered? What if a customer call makes us rethink some part of a feature committed to in the sprint. These common situations created more meetings, frustration, and complexity in our process.
Morale and Burnout
Flawed estimation constantly led to unrealistic expectations and the team would miss their sprint commitments. The retrospectives would then turn into critiques of what we did wrong which often came back to bad estimation, priority changes, and/or feature specs. Business felt engineering missed their targets, Engineering felt business kept moving the ball. Everyone’s morale suffered.
The two week iterations were great — we felt velocity because we were always deploying something. But our productivity quickly became spiky. Crazy hours at the end of the sprint in order to get it all in a release because “we committed to it” were followed by a productivity crash the next week so people could recover. Shortening the sprint cycles might smooth out productivity, but the planning overhead limited what could meaningfully get done in a shorter sprint. Lengthening sprints reduced overall pressure, but also exposed us to greater change management risk.
We started looking for alternatives.
Enter Kanban
While in business school at Stanford, I read a lot about Toyota’s “Just-in-Time” or Kanban manufacturing process. It really is a marvel of simplicity and more importantly, it supports our velocity goals. It’s designed to be flexible, so priorities can change pretty quickly. With one quick google search I found LOTS of articles, books, and videos on Kanban for software development. Our Atlasssian Suite even supported it.
What is Kanban?
Kanban is a continuous flow process: issues enter the queue and then get “pulled” through a series of steps in the development process. Kanban is often visualized on a Kanban board and each step is represented by a column. Issues can further be organized in rows, called “swim lanes”. Here at Stormpath, we chose to use swim lanes to represent issue priority – the further down in rows you go, the lower the issue priority. Core to Kanban are Work-in-Progress (WIP) limits that cap the number of items that can be simultaneously worked on within a given stage of development. Workers pull from left to right and can only move work onto the next stage if that column has an open WIP slot.
While Scrum drives productivity by limiting the work time available in a sprint, Kanban drives productivity and velocity by limiting the number of active, concurrent issues. Time estimation is no longer part of the process.
Velocity is still a major focus but now we optimize at an issue level, now called Cycle Time. Items to the right of the board, those furthest along, get higher priority. This creates an important core value on the team. Finish what we’re already working on, before you take on more work. Stop multi-tasking. Stop context shifting.
And all the churn on priorities and requirements from the business? Gone. With Kanban, the business owns the “To Do” column, but Engineering owns all the rest. Business can go hog wild with changes while things are in To Do, but they can’t change an issue without a lot of effort once it’s moved out into another column. As a result, the engineers now feel insulated from any changes from Business prior to work being picked up.
Happy Team
Kanban made our team very happy. Whereas Scrum felt like a prescriptive, inflexible formula imposed on the engineers by management, Kanban is flexible, process-driven and owned by engineering. Our team implemented and developed how we use Kanban at Stormpath, which gave them ownership over the process and its success. The transition was very smooth: about 2 days worth of discussion and planning.
What’s more, a continuous flow model means there’s no “deadline” but instead an ever-present pressure for “velocity.” This significantly reduced the burnout we were seeing. Even better, overall productivity went up. We weren’t spending half a day every other week with the entire team planning sprints; we weren’t gaming a feature’s scope to fit into a sprint; Kanban reduced context switching, which improved both productivity and code quality. Furthermore, the Kanban board itself reduced project management overhead because everyone could see it and know what was being worked on, by who, where it was, what needed to be worked on next, and where the bottlenecks were.
Structural Focus On Quality
A major, unplanned benefit of Kanban, was a focus on software quality. Under Scrum, there was always a temptation to cut corners in order to get things into the Sprint and then shove the missing work into the Tech Debt backlog, like skipping a code review, approving 70% code coverage when our standard is 95%, etc. Tech Debt kept getting bigger, and we missed some elements in designing new features. At Stormpath, we take code quality extremely seriously, so the constant risk made everyone uncomfortable.
Under Kanban, steps that ensure code quality are baked into the process. Since we’re rarely working against a deadline, there’s rarely pressure to skip these steps. In fact, because engineering owns the process, there’s a strong pressure to stick to the process. Engineers will fight to make sure we have 100% code review and 95%+ code coverage now.
Near Zero Overhead
Sprint planning? Gone. Project Management? Minimal. Estimation poker? None. Scrum Master? Nope. We still have stand-ups, but those don’t take up a lot of time. Engineering spends most of their time building product. Product owners decide what goes in “To Do” next, and engineering takes it from there.
Kaizen vs Retrospectives
In Kanban, Scrum retrospectives have been replaced with Kaizen meetings (kaizen in Japanese means “continuous improvement”). The difference is subtle but powerful. Here at Stormpath, we have Kaizen meetings every two weeks; engineers and product owners discuss how the process should be modified to improve velocity and quality, while reducing overhead. Instead of looking back and critiquing what we did, all the energy is forward-looking and focused on process improvement. Defensiveness down, collaboration up! Everything in the process is fair game and any new idea is treated as an experiment to be tested. Most ideas are given a try, and very few are shot down. As a result, kaizen meetings have an upbeat, creative feel and the process runs very smoothly.
Kanban isn’t perfect
We’ve successfully moved to Kanban and are really happy with it, but it’s by no means a perfect solution.
The Software Tools For Kanban Are Nascent
We use and love Atlassian Jira and Greenhopper. They have some nice Kanban features, but their Kanban board is relatively new and lacks the flexibility we expected (for example, having WIP limits in swimlanes). As a result, we’ve had to conform parts of our process to what was doable in Jira and Greenhopper. This is unfortunate, because to get the most benefits from Kanban, you should be able to model your board to reflect your team’s particular needs.
Loss Of A Systematic Focus On Urgency
With Scrum, that sprint clock ticks loudly. Everyone feels the pressure to get their work done before the sprint ends. Urgency is baked in. In Kanban, urgency is more abstract and doesn’t have quite the same emotion pressure. We’ve had to impose a sense of urgency in our team through culture and management—harder but hopefully healthier.
Keep Calm and Kanban
Kanban’s power comes from its focus on getting fewer items out faster through work in progress limits. Like anything, it’s not perfect and it may not be right for everyone in every situation but ultimately, we ended up with a happy team, higher productivity, less tension between the business and engineering, higher quality software, and A LOT less overhead. Win!
Interested in Kanban? Here are a few helpful links:
- Agile Chalk Talk: Kanban and Scrum https://www.youtube.com/watch?v=Jx6_E5XxqEo
- David J. Anderson’s Kanban Book: http://www.amazon.com/Kanban-Successful-Evolutionary-Technology-Business/dp/0984521402
- What is Kanban? Kanban + Scrum. http://www.ontimenow.com/training/what-is-kanban
- Do Agile Right (Kanban): https://www.atlassian.com/agile/kanban
- What is Kanban? http://www.kanbanblog.com/explained/