A treatise on Measuring Agile Performance

Introduction

      Measuring Agile sounds like an oxymoron, Isn't it? Why would you like to collect metrics on projects that follow the 'Agile' methodology? Do you really need to measure some aspects of your project? You have a product owner, the scrum master and the end users who would ultimately define the quality of your software. They are responsible folks and they would track the work. No metrics needed. Also, would this not take away the flexibility that 'Agile' proclaims? Well, yes and no! I'd rather take the middle path. 
      Yes - because many organizations require that the team members 'generate' metrics that the management can ingest and track the performance. The organizations overburden the teams to track the metrics that is not relevant for the project. In doing so the agile stops being agile and starts resembling the heavily metric based development methodology like the CMMi. Now, I am not proclaiming that CMMi or other similar methods are flawed and agile is superior. You have to take a step back and first decide which development methodology suits your project. Once you have made a choice to follow the agile practices then you should track your metrics that are most useful to your project. Yes if you are obsessed with collecting metrics for its own means then it will tax your team. Excessive love of metrics will do more harm than good. So collect only what is needed to manage the project. Discard everything else.
      No, measuring will not take away the flexibility of Agile development if you make the process of metrics collection relevant, simple and automatic. After all you still need to know how your team is performing and if it is meeting the customer's needs. Keep few metrics that you feel is needed and collect the data on it. Any such collection exercise has to justify why it is being done. All the collected metrics should eventually be used as a feedback loop to plan the future better or to avoid certain mistakes. Any metrics that does not immediately benefit the project can be ignored. 
     This post is intended to define and suggest the metrics that you should be using to measure the Agile projects. Though you can use the exact same metrics that I propose, you are encouraged to come up with your own set. This post in essence not only tells gives you a 'fish' but also teaches you how you should go about 'fishing'. It not only intends to give you a fixed set of ideas to work with but also encourages you to you use your own imagination and come up with your own set of rules. 

Manifesto for Agile Software Development

Before we get carried away and start declaring all sorts of metrics and data, it is worthwhile to have a look at the 'Agile Manifesto'. The manifesto by itself doesn't suggest that we must have a metric for measurement, but it doesn't discount it either. That doesn't mean that you go about collecting every datapoint that you find in your project. It has to be as less as possible and you have to be as targeted as you can. Here is the manifesto:

We are uncovering better ways of developing

software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on

the right, we value the items on the left more. 

The Agile Principles

Before we proceed further, it helps to glance at the "Agile Principles" once again. Unfortunately in our quest to measure every single aspect of software development we have all been guilty of going against these principles. Note that some words are marked as bold. This is to emphasize those words. The reason we are looking at the principles is to make sure all our metrics align to them. These principles will be used to make sure our metrics align with them. The 12 "Agile Principles" are:

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.	7. Working software is the primary measure of progress.
2. Welcome changing requirements, even late in  development. Agile processes harness change for the customer's competitive advantage.	8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
3. Deliver working software frequently, from a  couple of weeks to a couple of months, with a preference to the shorter timescale.	9. Continuous attention to technical excellence  and good design enhances agility.
4. Business people and developers must work  together daily throughout the project.	10. Simplicity--the art of maximizing the amount  of work not done--is essential.
5. Build projects around motivated individuals.  Give them the environment and support they need, and trust them to get the job done.	11. The best architectures, requirements, and designs emerge from self-organizing teams.
6. The most efficient and effective method of  conveying information to and within a development team is face-to-face conversation.	12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Tenets for deriving the ‘Agile’ metrics

Once you have decided to put some metrics in place it is time now to decide how to choose or define them. Agile itself does not tell you which metrics you should use. So you are free to do it on your own. It helps however to put in some checklist and rules that your metrics should confirm to when deciding on the metrics. 

1. The metrics should be in-line with the 'Agile Principles'. This is obvious. If it not inline then we are not following the Agile process. Metrics that hamper frequent delivery or does not welcome changes, like creating change requests and then measuring those, should never be considered.  
2. Any metric that contradicts or is in conflict with the 'Agile Principles' or the 'Agile Manifesto' should be abandoned as it violates the basic tenet of the Agile Development. 
3. The metrics should drive cooperation. Any metric that threatens team fabric should be modified or discarded altogether. Some agile teams work with a seperate set of folks based upon the function e.g. - development and test. Ideally the same user should do both, but in case your team has different folks, do not put a metric that has conflicting interests for individual team members. 
4. KISS (Keep it Simple, Stupid)
5. The metrics should be reevaluated after the release is complete. Anything within the release amounts to monitoring and for that you need tools such as burn-down, Kanban, etc. which is the scrum master's job. Thus anything that is used to track progress or work should not be a part of the metrics. That comes under project management.
6. Metrics should come out automatically without spending efforts to get them. The team should not work separately to generate them - like entering individual timesheets or measuring manually the success rate of the test cases, or documenting the review defects, etc. Such metrics leave a lot of scope to manually 'adjust' the data to one's liking. Besides a lot of effort goes in making up such data. These do not help in measuring the agile project.

What not to Measure

 Sometimes we knowingly or otherwise measure things that is either not needed or can cause conflicts within the team. We have already discussed earlier that such metrics should be avoided altogether. Here are a few concrete examples of what you should not measure as a part of the agile metrics. 

1. Conflicting Metrics: Any metric that creates conflicts within the team and does not provide a common goal should not be measured. Some organizations create sub-teams within the agile team such as development and test. This by itself is fine, as the individuals are specialists in some field or the other. But many organization also put in conflicting metrics such as for developers the defects (unit or integration) should be very low, whereas the testers are incentivised and encouraged to open as many defects as possible. This usually creates conflict between the team. This might be a good idea in processes where failure is not acceptable. On the other hand Agile promotes fail-fast approach. If your project is not mission critical, such metrics will add a layer of incompetence where testers might create defects to increase their metrics. This would take away a lot of time in arguments and unnecessary bureaucracy. I would certainly advice against it, because in short delivery cycles, you would not like to lower the productivity of your team. 
2. Reviews: You can choose to review the code but code review defect logging is not recommended. You can choose to log the system test defects but this is not a part of the metrics as they contradict the fail fast philosophy. 

Introducing - “The GREEKS”

      We will be measuring the agile principles - 1,2,3,7,8,9 & 12. All others fall under the scrum master's role and we needn't measure them. Measuring these metrics would being your project inline with the 'Agile Principles' too. You just need to make sure that you do not go overboard in your passion to pursue the metrics. Have a look at the principles that we will measure once again. 

    Let me propose a set of metrics called 'The Greeks' (a name borrowed from Options trading), that will generate metrics in line with the above principles. Let us meet our family-of-four. If you feel these are not adequate  for your project go ahead and create a few of your own. Be cautious however that too many metrics will cause confusion, and you should be ready to discard some of the 'Greeks' mentioned here. 4 metrics are ideal. More than that becomes a crowd. 

The Greeks
Release Efficiency	Rho (ρ)
Turnover Efficiency	Eta (η)
Software Efficacy	Delta (δ)
Team Efficiency	Alpha (α)

Meet Rho (ρ) – The Release Efficiency measure

Agile Principles inline with Rho (ρ)

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

3. Deliver working software frequently, from a  couple of weeks to a couple of months, with a  preference to the shorter timescale.

     

      ρ measures how fast you are churning out the desired requirements. There could be multiple factors that could contribute to faster delivery. Automation is one of the core activity to achieve agility. For effective continuous delivery model you must attain the highest possible automation that is achievable in your project. You must also try to attain shortest possible delivery cycle churning out the software as fast as possible.

Rho measures the ratio of the percentage of automation coverage from release to release. If the percentage of automation test case, a.k.a coverage, in the latest release was a and for the previous release was a', then,

ρ= (a-a′)/a = 1-a′/a

a can be calculated as mean ratio of all automations. e.g. If you have x% of JUnits & y% of Selenium then a = (x + y)/2. Or maybe you would like to choose only one. Use whatever works for your project.

Let us go about deriving ρ mathematically. You will see that a is not the only thing that matters. There are other factors too, just in case you'd like to measure them. Let us represent the efficacy of your release by e. It will be proportional to the automations you have done for your project. You can measure it by the percentage coverage of automation. Let us call it a. Then,
e ∝ a

If r is the no. of releases/deployments you do in a set time (say a month, or 3 months) then the efficacy would be proportional to the number of releases you give.
e ∝ r
If you think that the low production downtime is important for your release you can add it here. Any other factor that can influence the efficacy can be added here and then resolved. You may argue that some of the variables would not be 'proportional' to e, but would follow other mathematical function. If that is the case, then you will have more complex mathematical problem to solve. Remember KISS. We are  not trying to formulate a law of physics here. So if it is more or less proportional, we are taking the above rule for the sake of simplicity.

Combining the constructs, we get

e ∝ ar

e = Kar     (K is the proportionality constant)

For the previous release, we can calculate the efficacy as

e' = Ka'r'  (K will be constant here too)
Thus, Rho (ρ) can be calculated as
ρ = (e - e')/e = K(ar - a'r')/Kar = (ar - a'r')/ar

For most of our release processes the number of releases that we give in a fixed time-box would be constant, so r = r',
Thus, our formula reduces to the one above.
ρ = (a - a')/a

Points to Remember

• Measures release efficiency how fast can a release be delivered
• Its value is "usually" between 0 and 1, theoretically it can be negative too
• High ρ implies that you can either deliver faster by decreasing the release cycle or take up more work keeping the release cycle constant. 
• Remember ρ is not only for test automation but can be a measure of automations, overall. 
• ρ is the ratio of percentage of automation achieved with respect to the previous release. 
• Negative ρ implies slowing down

Some reasons for low Rho(ρ)

• Not writing test cases for features
• Lack of automation in the system

Meet Eta (η) – The Turnover Efficiency measure

Agile Principles inline with Eta (η)

2. Welcome changing requirements, even late in  development. Agile processes harness change for  the customer's competitive advantage.

η measures how flexible you are in your delivery. How much change can your scrum team accommodate? It is just the ratio of the planned stories vis-a-vis what you have delivered. The count of stories might not be a valid measure as not all stories are made equal, rather, you would like to add up the total story points that you planned and what you finally delivered. Note like all other metrics this is also a release metric and should not be to measured after every sprint, but after every release.  

This is how Eta is calculated

η= Sd/Sp

Sd  No. of Story Points(SP) delivered
Sp  No. of Story Points(SP) planned

When stories are removed from the planning find the total removed story points lets call it epsilon ε, then

η= Sd/(Sp-ε)

ε is not the SPs that are moved to next release

Points to Remember

• Measures the number of SP planned versus delivered.
• Its value is between 0 and 1
• High Eta is good for your project, low ones BAD.
• You must strive to maintain Eta as close to 1 as possible.
• If the Eta is decreasing over a period of time brainstorm the reason behind it.

Some reasons for low Eta

• Underestimation
• Dependencies
• Competency
• Architecture

Meet Delta (δ) – The Software Efficacy measure 

Agile Principles inline with Delta(δ)

7. Working software is the primary measure of progress.

Delta (δ) measures the quality of your product. Does your product work? Ultimately the user is responsible for defining the quality of your software. Remember this is not a unit or system test defect. Delta is calculated as,

δ=  Σd/Sd

Σd Total (or sum of all) ‘unique’ production defects

Sd  No. of SP delivered

** You can add the UAT defects to Σd if the UAT is performed by the actual user of the software.

Points to Remember

• δ measures the quality of the software produced
• The end users are the people who tell if the software works.
• Find the number of ‘unique’ production defects or tickets that can be pinned to the release. No need to weight them. KISS. 
• A large delta is bad for your software.
• Delta is not a testing bug. 
• Delta could be a L&P (Load and Performance) defect seen in production but not in test scenario .
• Ideally Delta should be zero. But …Alas !! 
• Delta should be brought in subsequent release as a User Story. Anything that is not worth fixing later doesn’t count for Delta. The UAT δ can be fixed in the same release if there is adequate time, or can be moved to next release as a User Story. 
• If you plot Release as x-axis and Delta as y-axis, then make sure it is a decreasing function

Some reasons for high Delta

• Code complexity 
• Team Competency (lack of)
• Underestimating and then scrambling to deliver without adequate testing

Meet Alpha (α) – the Team Efficiency measure

Agile Principles inline with Alpha(α)

8. Agile processes promote sustainable development.  The sponsors, developers, and users should be able  to maintain a constant pace indefinitely.

Alpha (α) measures the efficiency of your team. If measures if your output is increasing or decreasing with time. To measure my value you need to get the per-capita story point output of the team. The per-capita story point is needed to level the playing field if your team size changes regularly. If the team is constant release after release then you can simply take the total story point as a measure. In the generic example of team sizes changing on a release basis the per-capita story-point output is given by omega (ω) as:

ω= Sd/N ,where

Sd is number of story points delivered,

N is the team size – can include scrum master or not, always use the same measure of inclusion or exclusion

If current release’s output is ω and previous one is ω', then α is calculated as:

α = ω – ω’  = ( Sd/ N  -  S′d/N′ )

The value of α could be positive or negative. Positive is a positive sign for your project. Negative value tells that your efficiency has gone down compared to previous release. 

Points to Remember

• Measures the efficiency of the team in delivering the software. You can also call me acceleration.
• It can also be negative
• To measure its value you need to get the per-capita story point output of the team.
• Negative implies that your output has lowered as compared to the previous release 
• If you plot alpha α (y-axis) against ‘release’ over time, then a horizontal line is better than an increasing curve. Why ?? Answer below. 
• If the alpha is high, then you would take more work in the next releases anyway, and the curve would flatten automatically. 
• The optimal case is that α gives a horizontal line output. That tells you have utilized surplus efficiency of your team.

Some reasons for low Alpha 

• Team Competency (lack of)
• Team, not working as a team. 
• Keep a tab on the Scrum Master

Now that you have the metrics, what are you going to do about it?

9. Continuous attention to technical excellence  and good design enhances agility.

12. At regular intervals, the team reflects on how  to become more effective, then tunes and adjusts  its behavior accordingly.

The metrics collected should be utilized and put back as a feedback loop to make the system more efficient. If you could effectively do that then your metric has solved its purpose.  

Once you have the metric, here are what you should be doing:

• Formulate a Hypothesis
• Adjust the numbers into the next release
• Avoid past mistakes
• Keep a tab on past performance

Adjusting future releases  

I will demonstrate how that can be done by giving an example below. Your hypothesis might not be the same as mine. So use this as an illustration and create your own. Looking at the figure below, each release throws out a set of rho, eta, delta and alpha. Use them to adjust some of your planning. In this case we will adjust the intake measured in story points. 

In case of δ - it should be fed into the backlog and will appear as a story anyway. This is the feedback mechanism for δ. 

For η (eta) should be maintained at values close to 1. When η goes down check your estimation methods, team velocity and maybe plan trainings, try replacing few members or maybe increase the velocity by adding more people.

Let us now use the other Greeks to adjust the intake. If N is the total no. of team members, V is the average per day velocity and D is the total number of working days in a release, then the total story points that you can plan to take in a release is:

S = N * V * D

The this has to be adjusted by the α – that is you can now take more work (or less is α is negative). Adjusting,

S = N * (V * D + α)

The value of α will not increase on release basis as you would offset it by taking more work.

Rho (ρ) can be used to decrease the release cycle. If you wish to maintain the cycle, then use the surplus efficiency to take more user stories. Assuming ‘t’ is the percentage of time spent in testing and deployment (some use t = 20%, some 25%, and some 30%). The net percentage time you save is (t * ρ). You can adjust this to take in more User Stories. Increase the velocity.

S = N * [{ V + (1+t*ρ)}* D + α]

You can come up with other formulae. The idea is to formulate a feedback loop and adjust the intake. 

Conclusion

This is a simplistic proposal on what metrics we would need. Of course you can come up with many more, in fact hundreds of them if you try. But too many metrics will overwhelm the team and will encourage them to fudge the data. Remember the aim of agile is to generate a working software and not collecting metrics. 

Should you come up with your own ideas, i'd certainly like to hear from you. Please shoot out a mail to me or write in the comments below. 

Author : Gaurav Vishal