As an agile shop, we are constantly asked what metrics we track. My answer is that we don’t. What?
We do have health indicators that we keep an eye on. I see indicators and metrics as different things. Metric is a comparison against some number and based on that comparison it is good or bad. Indicators shows a specific state. The readings from indicators are just readings for that specific time.
Similar to the gauges in a plane, our indicators provide information that we use to make corrections and improvements. For example, the altimeter tells the pilot the altitude of the plane. If the altitude is showing a very low number, it could be good or bad depending on what the plane is doing. It could be parked or it could be about to land, so we expect the altitude to drop. And that is okay. The pilot uses a lot of gauges and instruments similar to the altimeter to help fly the plane.
Similar to the pilot, we use these health indicators to enable the team to achieve high performance. Depending on the team, the indicators have different meaning and they are not meant to be used to compare one team against another. They are also not meant to measure individual performance. The information gathered is meant as the starting point for further conversations that leads to discoveries, understandings and potential changes.
Below are the indicators that we use:-
Once a quarter, we will look at the last 8 burndown charts. We look for common trends among the 8 charts.
- Are there a lot of cliffs? i.e. Are we closing stories only at the end of the sprint?
- Are there a lot of work pull in at the end of the sprint? i.e. Are we pulling in the correct amount of work?
- Are there new work introduce into the sprint? Constant interruption?
- Are we swarming enough?
- Are we silo in how we work?
Committed Points vs Completed Points
We collect the number of points committed at the beginning of the sprint and the number of points that was completed at the end of the sprint. We see if there are trends to how many times a squad under commit or over commit. Indicators or discussion points from these data, are:
- Does the team have a fear of failure?
- Are we under committing? Or over committing?
- Are team stretching themselves so that they can be better? What is the percentage of times did the team completed everything that they commited to?
Team Velocity Trends
We look at the velocity trends to see if the squads are moving in the right direction. Overall, we expect to see squad increasing their velocity over time.
- Is there a drop in velocity? Why?
- Is there a gain in velocity? Why?
- Are we inflating the story points?
- Does the curve makes sense?
360 Individual Feedback
At a quarterly interval we send out 360 feedback to everyone in the tribe. Individuals are asked to provide feedback on squad members and also those that work closely with them. We ask the individuals to rate how the person they work measures up against our values. The collective data allows us to see if the squad is aligning with our values. This data also allows us to reflect on where the squad has been and areas where there may be potential improvement.
Another survey that we send out is about the team health. In this survey, we ask individuals to rate how the Scrum ceremonies are running and how each of the supporting roles (managers, product owners, scrummasters, etc.) and channels (chapters) are working for the team. The data allows us to reflect on where we can improve and help with improvement for supporting roles and channels.
Points Per Person Per Sprint
The most controversial data that we collect, is the number of points per person per sprint. These data points helps us reflect on the trends similar to velocity.
- What can we do to improve?
- Is there an upward trend or downward trend?
- What causes these trending?
What do we do with these indicators? We share the data with each individual team and facilitate conversations and discussions on what does each one of these indicator could potentially mean.
The basis for all of these indicators is that each tells a story. These gives us some insight into what happen in the past. We look at them and as a team figure out what we can tweak or change so that we can get better.