Friday, August 03, 2012

Time Tracking is Anti-Scrum: What do you do when you need it for billing?



In general, tracking hours slows teams down and confuses the Product Owner about release dates. Over 50 years of research shows that hours have high error rates and variability and relative point estimation is better. Even worse, some of our consulting partners bill in hours when they are hyperproductive and running at five times the production rate of waterfall teams on the same project. Yet they only get paid for the hours they work! They leave 80% of the money they should be getting on the table. This is very bad business management to negotiate contracts like this.


However, sometimes consultancies and even product companies find they have to bill in hours or collect hours due to unenlightened management. In this case, Scrum data can be collected in a way to give more accurate hourly estimation and completion than traditional planning. In the early years at PatientKeeper, we were experimenting with hyperproductive Scrum and wanted detailed hourly estimation. For several years, we gathered tens of thousands of data points until it taught us we could go faster with point estimation alone (at that time we abandoned any hourly tracking).


When we implemented an automated Type C Scrum back in 2002 at PatientKeeper, we did a user analysis of what to ask developers to minimize their administrative time (waste) and lean out the organization. I'm not aware of any other user studies of this type on eliminating administrative waste for developers.

We came up with a counterintuitive answer. While all we wanted to know was how to update the Burndown Chart, our developers told us not to ask about time remaining on a task for the following reasons:

1. It took more than 60 seconds to answer the question for five tasks and 60 seconds was the design criteria for the project reporting system.
2. It made them think too hard and they did not want to be bothered when they thought the resulting estimate was not necessarily accurate.
3. They felt that the estimate of time remaining had high variability and therefore was high risk. If the current estimate was exploding they knew what time they had invested and percent complete. Anything else was pure guesswork.

These were high powered professional engineers. We had several Ph.D.s on the team from MIT and elsewhere. The founders of the company were all MIT graduates. As a result, my view was that they gave the "right" answers to the questions based on best practice and not an answer you would get from inexperienced, perhaps ill-motivated engineers, who knew little about Scrum.

By taking the time invested and percent complete on each task touched each day, the system autocalculates time remaining to update the Scrum Burndown Chart. If a task is 50% done and one day is invested, the time remaining is assumed to be one day. The new total estimate for the task is two days. The original estimate for the task is archived in the system for historical analysis. The Burndown Chart automatically recalculates every estimate for every item touched during a day.

Time remaining to Sprint completion is all that is reported publicly and the team remains focused on what it takes to get "Done" at the end of the Sprint and does not even feel the questions asked relate to time reporting in any way. It is simply the "leanest" way to get the data needed to update the Burndown Chart.

A side effect of this approach is that consulting companies that bill customers on an hourly basis can generate billing from our automated system without filling out any time sheets.

It was generally agreed in a Scrum training class in Copenhagen yesterday with experienced project leaders operating at CMMI Level 5 at IBM and elsewhere that asking developers to fill out time sheets reduces team productivity by at least 10%. They hate to do it, it is duplicate work, it is obviously "waste", and is only done by companies that do not have a clue about lean product development. I have consulted at some companies where time reporting is so complex, developers make up time sheets, the company lies to customers as a result, and senior management make bad decisions on erroneous data, compromising the company's success in the marketplace. My final report to senior management was that 50% of their entire development productivity was lost because of the time sheet implementation.

Any project leader that has any sense will not "waste" 10% or more of development team productivity. They will move heaven and earth to avoid it and get more features done sooner with higher quality.

The recommendation to software development organizations that have to bill hours is:

1. Abolish all time reporting.
2. Institute the part of Type C Scrum reporting described above. You do not have to implement a Type C Scrum to do this. How to do it is detailed in the current draft of the book, "The Scrum Papers." For those willing to give editorial feedback, send me email and a copy of the latest draft will be made available.
3. Autogenerate time reporting to the customer from the Type C Scrum approach. Never use it to report development progress.
4. This approach gives more accurate information than any time reporting system ever created:
- Developers never "make up" the data on the time sheets.
- Exact time is available for every task.
- It provides a "micro-costed" analysis of work in progress.
- You never lie to customers. They are billed only on real work.

For those organizations not billing in hours, we recommend abandoning time tracking completely and go to only relative point estimate. 

Teams that do this have more accurate estimates and higher velocity. See why story points are better than hours.

27 comments:

Michael said...

Jeff,

Great blog! Learned a lot (as usual). When I saw your "evolution" comic, it reminded me of the one I published just last week at www.implementingscrum.com -- while it talks about the evolution of a team member on a scrum team, I thought your readers may enjoy it.

The link is:

http://www.implementingscrum.com/cartoons/cartoons_files/implementingscrum-20070226.html

Thank you.

- mike vizdos
www.michaelvizdos.com
www.implementingscrum.com

Indu - kvpsc@yahoo.com said...

Really practical, great read.

Indu - kvpsc@yahoo.com said...

This a great read, very practical .

Guillermo Schwarz said...

Sorry, my English is not that good, what is "a nieve preference"?

Pavel said...

Interesting.
While obviously some companies tend to implement overcomplicated time reporting, you still need "something".

Something to measure iteration progress.

Something to measure team performance.

Something to measure personal performance (you will probably want to reward top performers differently than people on the other side of the list).

Something that makes a predictable and controllable process, even at the cost of certain productivity loss. This is a trade-off, not a waste of course. Just balance it out.

In the end of the day, contractual terms are very much explicit, and you can hardly speak in terms of "more", "sooner" and "higher". That's too vague.

Jeff Sutherland said...

Iteration progress in Scrum is measured by Story Points, a measure of features done. Hours mean nothing except the customer has to pay more for stuff they cannot confirm is done.

In the best companies in the world, progress is measured in cost per story point. This is also the measure of team performance. The customer acceptance tests stories at the end of every Sprint and only pays for stories that are done and ready to deploy. They could care less how many hours the team had to spend. That is the teams problem. The consulting vendor assumes the risk for poor performance.

This is called walking the talk or putting your Agile mouth where the money is.

Individual bonuses are counterproductive on Agile teams. They should be team bonuses. Individuals should be compensated at market rate for their skill levels.

For contractual terms that support this strategy, see my presentation at Agile 2008 at:
http://submissions.agile2008.org/node/282

A complete paper will be uploaded for the conference by 15 June.

Bill said...

Jeff,

Is there somewhere I can download your paper on Agile contracts?

Thanks,

Bill House

Jeff Sutherland said...

The Agile contract work is going on at:
http://jeffsutherland.com/scrum/2008/11/agile-contracts-working-group.html

My Agile 2998 presentation on this can be found at:
http://jeffsutherland.com/scrum/2008/08/agile-2008-money-for-nothing.html

K Jackson said...

Great blog! It really challenged some of my thinking.

I just finished a fixed priced contract where we tracked time at the level of themes rather than individual stories or tasks. This worked pretty well as there wasn't much overhead in using the timesheets for billing and at the end of the project we could compare our initial estimates to the actual hours using the timesheets.

I've often found it better to ask "What's left on this story/task?" rather than "What percentage complete is it?" for tracking real progress in a Burndown chart. "What's left" helps to illicit additional tasks and it focuses on the right question - Can we still get this story done by end of the sprint?

Sharmila said...

yes I would like to review "the Scrum paper" however I have some questions:
1. If time tracking is anti-scrum, what is it that proves that we eliminated "waste"
2. How do you measure rework

I agree that some % of data is error prone but this data is just to facilitate the decision and not to rely on the data for decisions. Practically "no time booking" works in a perfect team of mature people however this is not always the case. Secondly I remember a say "if u can not measure something you can not improve it". So time tracking is just a measure and I think it is worth the time. If time tracking is aborted, it does not really guarantee you that that 10% adds to productivity.

I somehow contradict the thought of time tracking is anti-scrum, if everyone owes everything, no one has anything.

Eaglesnest said...

I tend to agree with Sharmila, and would like to see "the Scrum paper" too. Also my question is - how do you determine "time invested" if it is not time spent? Do you mean just automatically base it on time elapsed for that task since the sprint started? How can that be accurate? In addition, you needed "remaining %", you would still need to ask them to log that, right? If so, they are still required to think, how do one determine that is not going to be 10% inefficiency as well?

Jeff Sutherland said...

Please read and understand the article on how PatientKeeper used Scrum to track time. This is part of the Scrum Papers which you can download at http://jeffsutherland.com/scrumpapers.pdf - when we are all on the same page we can have a good discussion about this.

scrumcrazy said...

I found this quote helpful in the Scrum papers link that Jeff provided:

"If the initial estimate is 2 days, for example, and no work has been accomplished, the days
remaining are 2 days. If a developer has invested 1 day and states that it is 25% complete,
GNATS calculated the days remaining as 3 days. Initial estimates are automatically expanded
based on real time data."

I'm hopeful that this quote will help others as well.

scrumcrazy said...

I found this quote helpful in the Scrum papers link that Jeff provided:

"If the initial estimate is 2 days, for example, and no work has been accomplished, the days
remaining are 2 days. If a developer has invested 1 day and states that it is 25% complete,
GNATS calculated the days remaining as 3 days. Initial estimates are automatically expanded
based on real time data."

I'm hopeful that this quote will help others as well.

scrumcrazy said...

Jeff,

Thanks so much for writing, updating, and publishing the Scrum Papers link here.

I've only skimmed it so far, but I find it highly useful already.

I do find it odd that "Scrum Type A" is waterfall. Is this the same thing as saying "Scrum Type A" is very similar to what some people call "Scrummerfall" ?

Eaglesnest said...

Thank you for posting the paper. I understand the point about automatically calculating remaining time. My question is about the fact that you still require time spent before the remaining time can be calculated.

On page 171 of the paper, it states below:
"It was determined that only three questions would be asked as developers could answer them without thinking, they could give a gut level response:
• What is the initial estimate for this task if it is a new task?
• At this moment, how much time have you spent on this task?
• At this moment, what percent complete is this task?".

So they are still required to think about how much time they've spent. Correct? What am I missing? Thanks.

Jeff Sutherland said...

In the early years at PatientKeeper I had them track time spent on a task only. After several years and about 10000 data points we found that on the average tasks were completed at 85% of the estimate, i.e. tasks were completely earlier on average. When the management looked at this data, they asked why were projects late if tasks were early. They agreed that the only way projects were late was that the wrong tasks were completed, i.e. developers were working on the wrong thing. Since we were very disciplined on working only on the backlog in prioritiy order, they concluded that the business was giving them conflicting priorities causing projects to be late. From that time forward they realized time was worthless as a predictor of project success in an environment with multiple competing projects. PatientKeeper graduated to story points and this helped the teams hit live dates for hospitals at the end of every sprint. This is why we no longer recommend hours for anything.

Eaglesnest said...

Would this mean you no longer recommend hour burndowns and budget burnups? Other than story point burndown and burnup. What other metrics do you recommend for project performance monitoring?

Jeff Sutherland said...

Our team at ScrumInc is a good example. We track only points and velocity. From that we know how many points the team can produce per sprint. We know the cost of a sprint. We also look at revenue per point which determines revenue generation per sprint. This is a hyperproductive team that can do almost any project in 50 points or less. All of this comes up in a metrics dashboard for running an agile company. We will be posting information on this at scrumlab.scruminc.org in good time. If people post questions on scrumlab it will acclerate our efforts.

Eaglesnest said...

Thanks. What about for new teams or teams that are not so hyperproductive and are still trying to get their sizing right from one effort to the next? Any suggestions on how to best determine and estimate what they can more accurately fit into a sprint in the beginning of projects where they may not know about velocity yet and no hour estimates are used. How do you also in that case best estimate release timeline at release planning time?

Jeff Sutherland said...

As we emphasize consistently in our Scrum training, in the past we taught people how to estimate how much to take into their first sprint and found that they always got it wrong. A team does not know their velocity until they complete at least two and preferably three sprints and know how many story points they have actually completed. For what to do next, see http://scrumlab.scruminc.com/index.php?/topic/25-how-much-should-new-teams-take-into-a-sprint/

Joe Little said...

Hi Jeff,

Great post. Very much along the lines of what you have been saying for years.

To clarify for others and to be sure myself (that I do not mis-remember now)...
is your current advice now, for BEGINNING Teams, that they use hours to estimate the tasks in the Sprint? (And SP for stories.)

(I think I could infer 'yes' from what you said previously.)

Clearly, your advice for experienced teams is different. If I state it as this, am I correct?? (a or b)
a. Use fractional SP for tasks. And only burndown a task when the task is complete.
b. (For somewhat more advanced teams): Have yet smaller stories and have tasks (task cards maybe), but no 'tracking of the tasks' -- just burndown the SPs when the STORY is complete (for Sprint Burndown purposes)?

Thanks, Joe

Jeff Sutherland said...

Joe, I'm now recommending to new teams that they use only points. Since their stories tend to be larger they need to do tasking. They should split point among the tasks. Getting them off hours from day one helps them be more successful.

Unknown said...

Hi Jeff,

One major issue we are encountering when using SCRUM is how to evaluate the developers' performance... For example on yearly basis, how the evaluation is produced to reflect the peformance of a specific developer working on 1 or more projects and is member of SCRUM Team....

Jeff Sutherland said...

It is a really bad idea to try to evaluate developers on time or points. If you try to double the points of each developer on the team you could take forever. If you try to double the points of a team you can do it in two sprints.

Unknown said...

Well... From the management perspective, we have to respect their point of view where they might need to evaluate individual developers. i know the team is acting as 1 unit which is great. how do you think can the management evaluate their employees over time....?

Jeff Sutherland said...

As CEO I have banned performance appraisals as the research shows they clearly demotivate employees. Direct and immediate feedback works. Our employees say they get a perfomance appraisal in every retrospective every week. It is up to the team to deal with non-performing team members and is a breakdown of the Scrum process when management has to step in.