Thursday, August 21, 2008

Agile 2008: Secret Sauce for Distributed Scrum

Guido Schoonheim, CTO of Xebia, presented a case study with me at Agile 2008 on how to get maximum productivity and quality out of a distributed Scrum. The solution shows how to create distributed velocity = local velocity, distributed quality = local quality, and scaling is linear. Add resources to a project and productivity per developer stays the same, something seen only in two previous case studies I co-authored. This is counter-intuitive and requires secret sauce to achieve.

Taks a look at Guido's blog
where you can find the following and more:

Agile 2008

At Agile2008 in Toronto Jeff Sutherland and myself presented our article outlining how to achieve hyperproductivity in distributed Scrum when working in an offshore situation. InfoQ recorded our presentation and will publish it online in November as the end of a series of Agile2008 talks.

Download article Download presentation

Also see this InfoQ article

Agile and Offshoring, oil and water?

If you are reading the Xebia blog chances are that you are already familiar with the benefits of Agile development. Practicing Agile (in our case Scrum combined with XP) delivers hyperproductivity combined with very high quality. The promise of offshoring in the modern IT industry is also clear: more available talent, scaling up and down without local layoffs or knowledge drain, and of course cost reduction. Together they make a killer combo!

However, Agile and offshoring seem like oil and water, they don't seem to mix. How to get a focus on individuals and interactions when your people are distributed across the globe? What is the secret sauce to use to get it running smoothly?

Sunday, August 10, 2008

The First Scrum: Was it Scrum or Lean?



Today I got a question in email about Scrum and Lean, fallout from the recent workshop I did with Mary Poppendieck at MIT. Is Scrum related to Lean? Are we finding out new things about Scrum or Lean? Are customers asking for Lean?

After thinking about the early days of Scrum I think the root of both Scrum and Lean is complex adaptive systems theory. When we created Scrum we did not talk about Lean, we talked about complex adaptive systems. I think Scrum and Lean are complementary implementations of ways to deal with physical reality where things are often not linear, not simple, and not predictable.

The first Scrum implemented all the practices we know as Scrum today plus all of what later became known as XP practices plus some secret sauce that creates hyperproductive teams.

Pair programming was used regularly with Jeff McKenna, the senior consultant, who spent half a day with each developer when he was at Easel one week a month. After one of these sessions developers were heard complaining that 75% of their code base was thrown out in a refactoring frenzy. The 25% remaining did everthing more cleanly and elegantly.

Continuous integration was the norm. It still astounds me that developers think they can program production systems without this. (They can but they suck!)

The concept of an "Ensemble" was the key to component based design. We had sophisticated component based testing strategies based on silicon chip design.

We had experienced engineers on the QA team automating tests at the component level. This was feature level testing. It was a forerunner to the smoke test that runs on every build in my later companies.

We deployed builds for use at the end of each monthly Sprint. Each build had sufficient functionality for Easel consultants to use it on real projects.

But most important, we captured the "punctuated equilibrium" effect, something I have yet to see another Scrum team do. It is one of the secrets of hyperproductive teams. Before a developer chooses a task in the Scrum meeting, the entire team discusses with the developer whether that task will be the fastest path to an observable end-use feature. This was a sophisticated implementation of "self-organization" and "test first" development.

After the first Scrum started at Easel in 1993, I began talking about the process in the internet newsgroups in 1994, particularly comp.object, and by 1995 we had a well documented thread of my discussions on Scrum in comp.object, interlaced with comments from Bob Martin, Kent Beck, and others.

So the origins of Scrum are well documented and can be researched in Google groups by searching in the archives. My original Scrum web page started in 1994 and was widely viewed on the web throughout 1995 and updated regularly until the Agile Manifesto meeting in 2001.

During 1995, I invited Ken Schwaber to take a look at the first Scrum team which led to collaborating on the first formal paper on Scrum presented in the fall of that year at the OOPSLA. We did not discuss Lean at all but did discuss complex adaptive systems.

Mike Beedle was influenced by the online description of Scrum, implemented the process in his own company, and led the effort to drive Scrum through the Pattern Languages of Programming Design conferences. This made Scrum the first (and only) formal organizational pattern that describes a complete Agile process. Recent work by Jim Coplien shows that Scrum is deceptively simply while compressing a complex array of organizational patterns in his book, "Organizational Patterns in Agile Software Development."

Looking at "scrum.txt" you will see that Scrum derives directly from best practices in new product development documented by Takeuchi and Nonaka in their 1986 HBR paper. The leading example in the paper was Honda (Toyota was not so well know at the time). Takeuchi does not specifically address Lean as we know it today.

In 1995, I commented in the comp.object newsgroup:

Usability studies ... have convinced us that only one paper in the literature describes the rapid application development environment achieved with Object Studio (the combination of Enfin Smalltalk and Synchronicity Business Object Modeling and Persistent Object Mapping).

Takeuchi, Hirotaka and Nonaka, Ikujiro. The New Product Development Game. Harvard Business Review, Jan/Feb 1986, pp. 137-146.

Takeuchi and Nonaka describe the Scrum methodology (a Rugby term) which we use internally. It is significantly different than anything you have seen because there is no project management schedule, only a committed delivery time for a release. The Japanese auto and electronic companies are using it to clean our clocks in the global economy. Did you notice that U.S. auto companies had a banner year and still lost market share in 1994?

Small teams are given objectives to meet in a fixed time frame. They take the existing software (you never build anything from scratch) and small projects (2 days to 2 weeks per person) are assigned as SynchSteps. SynchSteps are dynamic pulses generated against the existing code structure that causes mutations (as in a biologic organism).

A project team is typically 3 developers, 3 QA people, 3 doc people, and one or two users. They meet daily and all agree on steps completed and next steps. For large projects, small teams of this size build components and a SCRUM of SCRUMs meets less frequently to work out interfaces between components. Developers must be outnumbered on the team by QA and documentation people or they generate too much code too fast (malignant functionality).

The code evolves like a biological system via punctuated equilibrium. You can read about this process as modeled by Denny Hillis on a Connection Machine in the book "Artificial Life." When enough mutations occur in multiple parts of the organism, the system shifts to a higher plateau of functionality. We say at this time that a "package" is emitted. This will be a new piece of the software system we are building. For example, we are adding Use Cases to the Synchronicity product at this time and when the package appears, an objective for the release cycle is met.

A release cycle, typically six months, is built of packages which are loosely defined as objectives at the start of the release cycle.

Management must release control of the team and let it function as a self-organizing entity that grows a system like a plant. We have found that we get dramatically more functionality in less time and a much cooler product with this approach. This approach rationalizes the comments that Fred Brooks made in 1987 in his paper "There is No Silver Bullet". There, he stated that the only way to speed up development was to grow a prototype like a plant.

We track the process by the velocity of SynchStep (Sprint Backlog item) assignments versus the velocity of SynchStep completion. This gives estimate of package time completion, kind of like launching a rocket, watching its trajectory, and predicting impact point. We can adjust the impact point by lowing the arc (less functionality) or adding rocket fuel (more resources) to hit the time frame committed to management.

We monitor progress regularly via demonstrations. Nothing counts with management except delivered code that works. Packages are regularly shipped out as alpha components that people can drop into the current release of Synchronicity. Synchronicity dynamically changes, menus and all, so that people can try out the new functionality and see if they like it.

With SCRUM schedules are obsolete (developers love it) and delivery dates never slip (management loves it). If this sounds too good to be true, I suppose it is, but management will agree there is more functionality and less slippage with this process than anything they have ever seen. Our users are asking us to lengthen the product delivery schedule because we are delivering too many new releases with major upgrades
too fast. (This still doesn't stop them from complaining about not getting everything on their wish list.)

We find that the paridigm shift required for optimum utilization of the SCRUM process is significant. The object-oriented paridigm requires a shift in thinking, but Scrum requires a shift in the organization and the way people work and relate to one another.
------------------------------------------

Scrum was strongly influenced by complex adaptive systems theory - the work of Peter Senge at MIT, Christopher Langdon at the Sante Fe Institute, and many others. Also by Goldratt's constraint theory and by Rodney Brooks subsumption architecture, the basis of the company iRobot which I helped to get started shortly before Scrum began. Starting at the bottom of my original Scrum web page I commented:

"SCRUM is based on complexity theory and artificial life experiments at Thinking Machines using highly parallel simulation of systems evolution. It induces the phenomenon known as "punctuated equilibrium" seen in the evolution of biological species."

The reason the first Scrum team called Sprint Backlog items SyncSteps was that developers executed the Sprint Backlog in a carefully chosen order - that order which produced the fastest path to appearance of the next feature from the users point of view. Just as proper ordering of the Product Backlog optimizes revenue, proper ordering of the Sprint Backlog optimizes production of value. It accelerates software evolution and can produce effects seen in biological systems.

The "punctuated equilibrium" effect is achieved at Toyota using set-based concurrent engineering. As an example, Toyota does not build one radiator for a new car. They build six, and wait to the last possible moment to choose the best radiator to deploy in production. This is similar to competition between biological species in an ecosystem and the species that adapts best to the environment wins. The Scrum community has yet to implement set based concurrent engineering strategies that were used by the first Scrum team.

At Google I was asked why no Scrum teams implement this set-based conconcurrent engineering strategy. This forced me to thing hard about the problem. The answer appears to be that Scrum teams (1) have no clear and consistent component architecture, (2) the team members do not have an overall view of the architecture and state of each component, and (3) the teams have no formal method for automated testing at the component level.

However, test first development drives the software creation process in the right direction as early testing shows immediately what does not work and allows rapid deployment of alternative strategies. The key is evolution of software like a biological system based on competing implementation constructs and survival of the fittest.

Test first development is described in my paper on Scrum in a CMMI Level 5 company where accepted tests for features are described first, the feature is implemented in priority order and tested immediately. Systematic Software engineering showed this doubled velocity and cut defects in half. Test Driven Development is writing test code before writing functionality and is useful in the right hands. This tends to focus at the unit test level and helps with refactoring. As Ron Jeffries pointed out in our joint MIT workshop, he then let's the code do the talking and the architecture emerges. However, many developers cannot hear the code do the talking and can generate a mess where there is more test code than production code and more bugs in the test code than in the application. I can see the design form in Ron's mind as he works on the code and it is clear he is not the average programmer. He sees the design space alternatives unfolding as he codes and has the good judgement to take the right path into a flexible and maintainable architecture. It reminds me why Jim Copien asserts that "Architect Codes" is a highly important organizational pattern that is best practice in leading companies. (Jim also asserts TDD is broken for reasons alluded to here.)

A thoughtful analysis of the relationship between complex adaptive systems and Lean will show that both Scrum and Lean are instantiations of complex adaptive systems theory. A subset of complex adaptive systems theory is artificial life, a term invented by Christopher Langdon at the Sante Fe Institute in the 1980's. His seminal paper showing that organisms similated on a computer evolve faster as they approach chaotic regions strongly influence Scrum. In Scrum, we introduce chaos into the development process and then use an empirical control harness to inspect and adapt a rapidly involving system.

In trying to describe Scrum at JAOO in 2005, I used Lean to show why Scrum works. Scrum is a way to implement lean in building software. In fact, it has the advantage that if you follow it closely and implement well, you will be doing Lean as articulated by Mary and Tom Poppendieck sithout even understanding Lean. It is hard for people to understand complex systems theory so it is difficult to use that as a motivator. It is hard for people to understand Lean as well, but Toyota has become such a dominant force in industry that fear, uncertainty, and doubt wakes people up from their sleep.

In 1995, I worked with Hubert Smits and Jean Tabaka at Rally to develop a course for people who want to take their Scrum implementation to the next level all based on Lean as the easiest way to articulate what you need to focus on to optimize Scrum. Ken Schwaber argues that by simply focusing on the Scrum framework and the impediment list you will achieve this. This is absolutely correct. Lean is just an extra teaching tool.

There is also the paper on Scrum and CMMI Level 5 on the implementation of Scrum by Systematic Software Engineering, an implementation entirely driven by Lean. In Denmark, it is national policy to introduce lean practices in all industries. As a result, it is easy to sell Scrum as a way to implement Lean in software.

Scrum is an inspect and adapt framework which is extremely simple by design to allow the average developer at Ford Motor company to get started in a couple of days. However, it is hard to implement. Less than 10% of the Scrum teams worldwide can pass the Nokia test, primarily because they cannot deliver potentially shippable (fully tested) software at the end of a Sprint. Talking to people about complex adaptive systems doesn't seem to help them as well as talking to them about Toyota to show them how their implementation of Scrum is broken, how it is riddled with disruptions in flow (muri), stressed everywhere (mura), and waste (mudah). Using Toyota as an example of a successful company who systematically elimiates muri, mura, and mudah shows them how they might best go about fixing their process.

So Lean is helpful as a teaching tool because it derives from the basic laws of the universe which are articulated at a theoretical level by complex adaptive systems research. Scrum derives from the same fundamental principles. Lean has practical ways to deal with process mapping, the cost of delays, queuing problems, reset times, and so forth. These are all useful in dealing with constraints, global system throughput, and other factors which were a primary concern for Scrum in dealing with the complex adaptive system called software development.

Lean derives from Edward Deming and the work of others teaching the Japanese that local optimization caused by traditional management by objectives was self-destructive. Deming, after consulting with the leadership in many American corporations, concluded that management is totally broken in the U.S. causing incalculable and inconceivable damage (such as the destruction of the auto industry). The global optimization promoted by Deming and his statistical quality control practices are rooted in complex adaptive systems theory, constraint theory being a specialized area that focuses on global optimization. Professor Senge's work on systems theory at MIT is the most readable resource for these early influences Scrum.

So the bottom line is that:

1. Scrum is not a direct descendant of Lean. It derives from complex adaptive systems with an indirect linkage to Lean through Takeuchi and Nonaka.

2. Management is interested in Lean, partcularly in Europe. Building on that interest and any knowledge they have about Toyota makes it easier to describe why Scrum works.

3. Lean is a good teaching tool to show Scrum teams why their implementations are broken. For example, if you do not have fully tested code and cannot create potentially shippable code in a Sprint, you have 100% work in progress going into the next Sprint.

Excess work in progress is viewed as a horrible and intolerable blunder in a Lean operation, yet software development teams and management seem to have difficulty understanding why this work in progress will double your defects and cut your velocity in half. And even when they understand it they often lack the political will to fix it. Teaching Kaizen mind and stopping the line is useful medicine for this disfunction.

Saturday, August 09, 2008

Scrum Overview: Sometimes your waterfall manager needs one ...

A good Scrum overview is hard to find even though there are many of them out there. Here is one that might be useful next time your waterfall manager wants to know, "What is Scrum?"

Scrum
Agile Software Engineering Management
Dan Greening, Citrix Online

Scrum is a software management technique with high transparency, adaptive control, reasonably accurate release forecasting, and high productivity. However, because Scrum exposes marketers and developers to greater visibility and accountability than traditional waterfall approaches, and because it requires different management structures, some organizations encounter resistance implementing it. Scrum focuses on management, so it can be combined with developer-focused agile techniques, such as Extreme Programming. More ...

Saturday, August 02, 2008

Deep Lean and Scrum: Stockholm 25-26 Sep 2008


Crisp Academy presents
Deep Lean
Learn from the experts!
2 days with Mary, Tom, Jeff, and Henrik
Sep 25-26, 2008 Lean Scrum XP
Click here for registration!

Deep Lean is an in-depth seminar targeted to people familiar with Lean, Agile, Scrum and XP software development. This will be a unique opportunity to be in a relatively intimate conversation with Lean/Agile/Scrum/XP thought leaders for two days. We will have in-depth discussion of software development challenges, principles and practices.
Date: Sep 25 - 26, 2008
09:00 - 17:00 (breakfast from 08:30)
Agenda: Here is the detailed agenda
Location: Central Stockholm
Price: SEK 7500 + vat
Teachers: Mary & Tom Poppendieck, Jeff Sutherland, Henrik Kniberg
Contact: academy@crisp.se
+46 8 556 950 15

Examples of topics that will be covered:

* Why software projects fail
* Lean principles and Scrum
* Using Value Stream Mapping to uncover and eliminate waste
* Agile transitioning - how to get started
* Real world experience creating lean and agile companies

The price includes:

* 2-day registration covering all presentations and workshops
* All slides and course material
* Breakfast, lunch and snacks
* You are invited to dine and socialize with the speakers and volunteers at a local restaurant on friday night (all participants are responsible for their own expense).

For those needing more background, we recommend reading Mary & Tom Poppendieck’s first book: Lean Software Development - An Agile Toolkit in advance of the seminar.