Sunday, August 08, 2010

Nokia Test: Latest Version



The "Nokia Test" for Scrum teams was developed orginally by Bas Vodde at Nokia Siemens Networks in Finland. It has been updated several times and appears in it latest incarnation in Jeff Sutherland's Scrum Certification classes where he demonstrates that attending the class yields an average return on investment of 1033% for participants.

Links to all previous versions of the Nokia Test, the current scoring of the ScrumButt test, and the 1033% ROI calculations can be found in the attached presentation, "The ScrumButt Test: aka The Nokia Test."

17 comments:

tjementum said...

Hi Jeff

This is nonsense.

I've been at your course (at JAOO, Aarhus), and while the SCRUM message is brilliant, this 1033% ROI is simply a big laugh.

All these numbers are taken form thin air. At the course I attended, you was the one how made up these numbers... not the students.

First of all you say a average team cost 100K a month. Is that a 7 +/- 2 person team?

Second... you assume that only one team member has to take the course!

Third... you assume that they will be 20-50% more productive!

You know as well as I that these numbers are made up, and was now way an average... at best it was one students optimistic thought.

My problem is that, when someone how does not use SCRUM read this they are rightly skeptical, and it makes SCRUM not trustworthy.

If this was a one-time coincident, I would not comment. But it seams to me that every time you talk about the value of SCRUM, you tend to make a very naive assumptions about the reality.

Please consider being more realistic and honest.

You know... being Agile is about building trust. And this is the exact upper sit.

: Thomas

Jeff Sutherland said...

Thomas, I appreciate your conservative nature. However, I am consistently doing this test in classes all over the world and participants (many who have been doing Scrum for some time) consistently say they can improve velocity 50% by increasing their ScrumButt test score from 4 to a 6. Many of them think they can do better than that.

The average U.S. team of 7 costs about $100,000 a month. In actual fact, a 50% increase in velocity will generate a lot more than $20000 more software revenue per month.

For example, if you read Benefield's paper on Yahoo, you will see that the financial office at Yahoo determined that a Scrum coach at Yahoo generated $1,000,000 in additional revenue per year by raising 10 teams production by 35%. This is about 1000% return on investment.

This was very conservative as she points out the teams that were well coached at Yahoo got 300-400% improvement. In our venture portfolio group we have companies that have increased their velocity 300% in three two week sprints immediately after attending my class.

Now, if the attendees at the class go back and do nothing they will get zero results, so maybe that is what you are seeing. People who go to the class and then don't change anything will produce ScrumButt implementations that suck. That has nothing to do with Scrum.

Giora Morein said...

Jeff,

One of the things I have been meaning to ask you is about the "increased their velocity 300% in three two week sprints." I have seen this type of boost, but not simply from a ScrumMaster attending a 2-day class. With an integrated coach - yes.

A few people I have talked to who have attended your class walked away understanding they should resize stories at the end of the sprint they are completed in. Furthermore, they re-evaluate story sizes directly before the sprint they are planned for. We all know that stories always seem bigger later in the project.

My question is: "these teams with the 300% in three two week sprints immediately after attending my class" - did the story sizes remain constant and anchored prior to the first sprint? In not, and the story sizes are allowed to be fluid (aka, perpetually resized) then the result is story point inflation. It is a common phenomenon on projects to have the average story size increase the later in the project a story is sized. If story sizes are being inflated, the team's productivity is not increasing though Velocity appear to increase.

Giora Morein

Jeff Sutherland said...

Increase in velocity needs to be measured against stable reference stories. As noted in the Shock Therapy paper, when specific constraints on the Scrum team are enforced, a new team will consistently get a 200-300% velocity increase in three one week sprints. In London last week a new ScrumMaster achieved a 400% increase within a month of taking my class.

For most teams, an experienced Scrum coach is needed to get this effect because the new ScrumMaster cannot or will not enforce the necessary constraints.

Scrum is a simple framework with a set of constraints which if enforced with always result in a hyperproductive team. The fact that 90% of teams cannnot achieve this shows that most companies are so dysfunctional that they make it extremely difficult, so a powerful coach is needed as at MySpace.

It is easy to go hyperproductive. It is simple. And it is fast. The first Scrum did it in three sprints and anyone else can do the same if they simply execute the Scrum framework well.

Scrum is "hard" because we live in a land of mediocrity where dysfunction is OK. People have taken the blue pill and they are asleep. They will be put out of their misery whenever a Toyota shows up to take away their business.

Peter Alfvin said...

Thanks, Jeff. See my questions at http://palfvin.blogspot.com/2009/10/nokia-test-questions.html

Craig Brown said...

We made some substantial but unmeasured improvements in the first 3 months of adopting scrum. 300% is not unbelievable, but I couldn’t substantiate it as we didn’t measure what went before well.

However we did measure progress over time and by the end of 18 months we had a team that could easily achieve great outcomes. They were committed and self managed.

The team's one great flaw was always signing on for too much at sprint plan, and we kind of accepted that as annoying but okay.

The team had a velocity that was out of control and this failure to properly commit was probably the root of this problem.

I’d say that at the end of 18 months together we elevated our productivity by around 400% on top of the initial 3 month improvement.

Jeff Sutherland said...

OpenView Venture Partners discovered that taking to much into a sprint reduces acceleration. Counterintuitively, taking less initially allows you to accelerate faster and achieve higher velocity sooner. So overcommitting probably delayed your 400% improvement.

Congratualtions, you have one of the few hyperproductive teams in the world. Most companies will not remove their impediments to achieve this.

Ryuzee said...
This comment has been removed by a blog administrator.
Jeff Sutherland said...
This comment has been removed by the author.
Spike said...

The Nokia test examines RoI from the company's cost-benefit perspective, assuming the company pays and looking at the company benefits. That's entirely reasonable.

There is however another perspective, which is when the individual pays. When the individual pays, the benefit is not measured in increased deliverables vs. a fixed labour cost. It's a little harder to measure. Increased earning power would be the most quantitative measure. Increased perceived delivery value to employers is another - but presumably reduces to sustained earning power over time.

As a self employed person I also have to factor the opportunity cost of taking the days off to attend the course. But there's still not much doubt here though. All the certification has to do is keep me in work for about an extra 2 weeks or so during my lifetime, and it's more than paid for itself. :)

Vijayanand Nagaraj said...

The link to Nokia test doesn't work still. The link throws up 404 Error, Page cannot be displayed.

Jeff Sutherland said...

The link has been updated and works now.

Dale Ellis said...

Hi Jeff,

I have a couple of questions about scoring in the test:

1. Testing: It seems like Unit Testing receives fewer points than feature testing, but what if QA personnel check the functionality of features without performing Unit Tests?

2. Sprint Burndown Charts: It seems like Sprint Burndown Charts that only burn down when tasks or stories are completed receive a higher rating than the traditional method of teams estimating hours remaining for tasks - making it similar to a Release Burndown. Are there additional instructions and/or examples of how to do this kind of Sprint Burndown somewhere (the "Track Done" slide wasn't entirely clear.)

Thanks!

Dale Ellis said...

Hi Jeff,

I have a couple of questions about scoring in the test:

1. Testing: It seems like Unit Testing receives fewer points than feature testing, but what if QA personnel check the functionality of features without performing Unit Tests?

2. Sprint Burndown Charts: It seems like Sprint Burndown Charts that only burn down when tasks or stories are completed receive a higher rating than the traditional method of teams estimating hours remaining for tasks - making it similar to a Release Burndown. Are there additional instructions and/or examples of how to do this kind of Sprint Burndown somewhere (the "Track Done" slide wasn't entirely clear.)

Thanks!

Dale Ellis said...

Hi Jeff,

I have a couple of questions about content in the slides:

1. How do you score QA if feature testing is performed without unit testing?
2. How does the new Sprint Burndown chart system recommended work – the one that shows completed work as opposed to remaining estimated hours? Is that supposed to be done as a substitute for an hour burndown or as an addition? It seems like it would lack the granualarity necessary to anticipate additional required hours.

Thanks!

lIlya Pavlichenko said...

Hello, Jeff
I would like to ask the same question:

How do you score QA if feature testing is performed without unit testing?

So generally - can we score higher than the "bad thing" in a list or not?

Jeff Sutherland said...

Since 2008, the leading trainers in the world at Scrum Inc. and the Scrum Foundation have recommended avoiding hours as hours are error prone, they generate bad release plans, they take a lot of time to estimate, and they must be reestimated for every team, every person, and every time they are looked at. They do not help a team perform and they slow a team down.

My focus on testing is acceptance test driven development. Acceptance tests need to be good enough that they will surface any unit test failures whether or not unit testing has been performed.