A True Metrics: Running Tested Features (RTF)

Ron Jeffries, one of the founding fathers of Agile wrote an excellent summary of what he considers a truly valuable metric, RTF or Running Tested Features. You can view the article here.

From the article:

What is the Point of the Project?

I'm just guessing, but I think the point of most software development projects is software that works, and that has the most features possible per dollar of investment. I call that notion Running Tested [Features], and in fact it can be measured, to a degree.

Imagine the following definition of RTF:

1. The desired software is broken down into named features (requirements, stories) which are part of what it means to deliver the desired system.
2. For each named feature, there are one or more automated acceptance tests which, when they work, will show that the feature in question is implemented.
3. The RTF metric shows, at every moment in the project, how many features are passing all their acceptance tests.

How many customer-defined features are known, through independently-defined testing, to be working? Now there's a metric I could live with.

5 Essentials for Software Development Metrics

I ran across a whitepaper today by 6th Sense Analytics that summarizes the essentials of a software development metrics program. The paper doesn't specify the metrics to collect, just the properties they should all have. An overview is provided below. You can find the full whitepaper here.

From my perspective, for any metric to be useful, it needs to help the Project Manager make decisions. As this whitepaper states, all metrics should be actionable. If it's not actionable, then it's not useful.

Introduction
Today's approach to metrics calls for automation over process and in-process adjustments over post-mortem analysis. Information should be made available to all stakeholders throughout the lifecycle.

To be effective, metrics must be properly planned, managed and acted upon. What is measured, how it’s collected and how it’s interpreted are the difference between brilliant insights and blind alleys on the path to metrics-driven software development.

The key is to ensure metrics are meaningful, up-to-date, unobtrusive, empirical and actionable.

#1 - MEANINGFUL
Emerging activity-based software development metrics solutions focus on a simple and fundamental unit of measure: Active Time. Understanding the actual time being applied to these activities across a portfolio of systems and projects can provide an important level of insight that enables organizations to understand project velocity, relative investment, sequencing of activities, and opportunities and risks. It also provides a uniform basis for comparison across projects and teams.

Select metrics that will enable you to steer your projects in a meaningful way.

#2 - UP-TO-DATE
It is important to look for metrics that can be captured automatically during the execution of the process itself. Activity-based metrics solutions are able to automatically capture metrics during the conduct of the software development process, ensuring that the metric is consistently based on up-to-date data.

#3 - UNOBTRUSIVE
The process of collecting data for your metrics program should be seamless and unobtrusive, not imposing new processes or asking developers to spend time collecting or reporting on data.


#4 - EMPIRICAL
Activity-based metrics solutions capture data as software development processes are executed, eliminating all of the issues that compromise the integrity and accuracy of data. Additionally, the use of the Active Time metric ensures data consistency; an Active Hour is the same in Boston, Bangalore, Mumbai and Beijing.

#5 - ACTIONABLE
It is critical that the metrics you gather inform specific decisions during the course of your software development projects. Avoid information that is nice to know, but doesn’t help you make decisions or solve problems.

The litmus test for any metric is asking the question, “What decision or decisions does this metric inform?” Be sure you select your metrics based on a clear understanding of how actionable they are and be sure they are tied to a question you feel strongly you need to answer to effect the outcome of your software development projects.

It is also critically important to ensure that you are able to act on and react to metrics during the course of a project.

Finally, be sure that metrics programs are inclusive and that data is available to all stakeholders within the software development lifecycle. Data that is widely available is empowering.

Standard Progress Reporting & Tracking

A simply way to improve visibility for customers into projects is through reporting and metrics.

Timeline

The first thing that all stakeholders want to know is how does the timeline look. Are we on track? Are there any delays? What is our next milestone? I create a simple timeline in Excel (you can use Visio if you prefer) and color code it to indicate the health of each iteration.

Parking Lot

Provide visibility into the progress of features of the system. Use the same color coding system as the timeline. Each feature has a number of UCs associated with it and you can show the percentage of those UCs that have been delivered to the client.

Productivity
Report the team's productivity after each iteration. Also show the target for the team. I report function points per hour because I want to see the productivity going up.

Resource Burn Rate

Our customers need to know how many resources we have any our burn rate. They have budgets they need to manage too. In this example, I am only reporting past periods, but it could easily be extended to any future timeframe. I build this from the Ramp Plan. I also report the resource count (onshore and offshore) by role.

Burndown & Burnup

Use a burndown chart to track the remaining effort against plan. Use a burnup to track the % complete against plan. I color code the bars to indicate where we went off track. I also add call outs to explain large changes. I report the data in a small table (e.g., planned effort, actual effort remaining, difference).


Testing

There are several reports that you can create to provide visibility into testing. This example represents our defect trend by severity over the course of an iteration.

Attrition Rate
Most of our clients are concerned about turnover and losing key resources. We have introduced the following graphs to track our attrition rate per quarter. We also list the resources that left and the reasons for leaving.

Other Metrics Under Consideration

• Planned vs. actual features delivered by iteration
• Velocity (# of FPs delivered by iteration)
• Defect discovery rate
• Defect density (defects per FP)

Scrum vs. RUP - The winner is...

...the Team!!!!

Back in August, I talked about how agile was working with my offshore team. Over the past few months we've had our ups and downs, but the good thing is that we've learned from our mistakes and continued to modify the process to fit our project's needs. All those iteration assessments (or lessons learned or retrospectives) that we conduct really pay off if you take the time to adjust based on what you learn.

Overall, we've ended up modifying our strictly RUP process to accomodate for some of the Scrum methodologies that we are comfortable with as an organization. Here is an update:

Potentially Releasable Software

• The team was having difficulty with this concept
• The team was killing themselves trying to get the defect counts within the acceptance criteria
• We came up with a compromise that is working well.
  (1) Iterations are now about 3 months long versus 5 weeks
  (2) We deliver an "in progress" build to the customer at the end of each month to get early feedback; this build always includes UCs that have been completed during that month and other in progress UCs.
  (3) We commit to meeting the quality criteria by the end of the iteration

Product Backlog

• We've altered the Product Backlog to be a combined Release Roadmap and Release Planning document
  
• It serves the same purpose, however it has made planning easier since we have added more columns to track builds, effort per build, contingencies, rework, etc.

Planning

• This was the biggest problem area and the area where we've seen the most improvement. It's amazing how good planning improves morale!
  
• We now have one master Iteration Backlog for the team as a whole.
  
• Each of the teams manages their own Project Plan and these roll up into a Master Project Plan.
  
• From the Master Project Plan I can get the progress of the build and the progress of the iteration.
• We are using both burndown and burnup charts. This makes is very easy to see the progress and gives the customer visibility into the project.
• On the burndown charts, I track the remaining effort against the target.
• On the burnup charts, I track the percent complete against the target.
• The actual data that generates these charts changes to red if something is behind.
  

Sample Iteration and Build Burndowns

Sample Iteration and Build Burnups

5 Reasons for Project Management Failure

Adapted from PM Network, October 2006

1. Organizations lack the resources, know-how and commitment to manage culture change.
2. Executives says they want good project management methologies, but their actions don't back up their words.
3. The methodology isn't scaled appropriately to the size of the project -- making project management seem cumbersome and unfocused.
4. Organizations fail to plan up front.
5. Senior managers think project management is a software tool.
   

It's take courage to initiate change. Managers need to be willing to take risks if they want projects to run better. SCRUM provides a framework for addressing these failures.

Changing the Mindset of an Organization

I have been working hard to convince people (i.e., management) within my organization that the benefits and values that Agile brings to organizations and projects far outweigh any possible side effects (I guess some warped people could see hyperproductivity and making everything visible as negative side effects).

We have a global agile process group and everyone on the team is in complete agreement that our current RUP process would benefit by adapting some Agile/Scrum techniques. Here's our current game plan:

1. How do we convince the organization of the benefits? How do we convince them that change is good?
   
2. Who do we get to sponsor the initiative at the executive level? We need someone with the courage to act differently, to find out if the environment will support us.
3. How do we incorporate Agile & Scrum best practices into our corporate culture and our projects sooner rather than later?

Any suggestions or experience with these questions would be greatly appreciated (both good and and comments welcome).

Scrum Core Values

I believe these should be posted somewhere for everyone on the team to see. It's very easy for them to lose sight of them otherwise.

Commitment

• Be willing to commit to a goal
• Support & encourage commitment
• The team has the authority to decide how to do the work it has selected

Focus

• Do your job
• Focus all of your efforts and skills on doing the work that you’ve committed to doing
• Don't worry about anything else
• Once you're focused, all of your time is spent looking for and trying solutions to bring order to the problems

Openness

• Keep everything about the project visible to everyone
• Scrum removes the ability to dissemble
• Responsibilities are clear, authority is allocated, and everything is visible
• Scrum counters interference; No one is allowed to add work to a Sprint once it is underway
• It’s better to produce something than it is to pursue many alternatives, please everyone, and produce nothing

Respect

• Individuals are shaped by their background and their experiences
• Respect the different people who comprise a team
• The team adjusts and adapts to meet its commitments for a Sprint:
  (1) Who does what is up to the team
  (2) The team commits as a whole and sinks or swims together
• Do your best, remember everyone else is doing his/her best, and help your teammates

Courage

• Have the courage to commit, to act, to be open, and to expect respect
  
• It requires courage to act differently; Courage to see find out that the environment will support these values; Courage to be willing to find out that relying on one’s own judgment is acceptable – even admirable
  
• Courage is having the guts, the determination, to do the best you can
  
• Courage is the stubbornness not to give up, but to figure out how to meet commitment

Eureka: The Daily Scrum

We have a team at the client site that is holding daily scrum meetings. I am onsite this week and was listening to the conversation (as a chicken from the other room). What I heard caught me off guard; it was a status meeting where they would actually go through and update each task in the Sprint Backlog.

I took the ScrumMaster to lunch and discussed with him the purpose and benefits of the 3 questions:

1. What did you do yesterday?
2. What will you do today?
3. What is preventing you from accomplishing your goal?

He was very receptive and acknowledged that he needed to make a change. We held a daily scrum this morning and went through the 3 questions with each team member.

The difference was remarkable. The meeting turned into a conversation rather than a status meeting. Team members were offering help to each other. They finally get it that they are a team and that they are measured as a team.

If anyone is running a project without the daily scrum, they are really missing out on significant opportunities.

Agility & Discipline Made Easy

I was turned on to the book "Agility and Discipline Made Easy" by a colleague of mine. This book is a good follow up to the book "The Rational Unified Process Made Easy." The primary focus of the book is to provide you with best practices for making projects more agile (with focus on the Unified Process).

The book outlines six fundamental principles (best practices) that characterize the most successful projects in the software development industry. Many of these principles are found to some degree in most iterative and agile processes:

1. Adapt the process
2. Balance stakeholder priorities
3. Collaborate across teams
4. Demonstrate value iteratively
5. Elevate the level of abstraction
6. Focus continuously on quality

Each principle is then broken down into sets of practices for:

• Supporting Iterative Development
• Improving Your Ability to Manage Requirements
• Improving Your Ability to Do Architecture, Design, and Implementation
• Improving Your Ability to Do Testing

I highly recommend this book for those of you that are experienced with RUP and want to make your processes more agile.

Set the Foundation in the Inception Phase

Jon Kern has written an excellent overview of what you need to accomplish in the Inception phase of a project in order to set the proper foundation for all future work.

Here is a high-level summary of his article.

To set the proper foundation, you need to have the following defined:

• Prioritized list of features for Release 1 (at a minimum)
• Domain model
• Technical architecture
• Automated build process
• Coding guidelines
• Version control system
• Issue tracking system
• Requirements management system
• Collaborative workspace (e.g., blogs, wiki)

Jon properly notes that there are two critical pieces to the Inception process: the domain workshop and the technical workshop.


The requirements-gathering process takes place in the Domain Workshop. The Technical Workshop is used to ensure the architecture and development techniques are defined. At the end of this process, we will have the necessary ingredients to form a Release Plan and can start the iterative development in earnest.

In the Domain Workshop, the artifacts we should typically end up with include:

• Business definition
  - Problem statement
  - Vision document
  - Simple cost/benefit check
  
• Feature set
  - Basic description
  - Roughly prioritized
  
• Domain model (initial cut)
  
• "Non-functional" requirements

In the Technical Workshop, the artifacts we should typically end up with include:

• Architecture vision
  
• Things we chose not to do (and why)
  
• Object models
  
• Sequence diagrams
  
• Deployment diagram
  

If all of these goals are accomplished, the team will be able to move forward quickly with a list of features in hand, the technical architecture determined, a set of coding guidelines, a set of automated build scripts and a domain model from which to start coding.

This is certainly a recipe for success!!!

Agile & Blended Shore Projects

My whitepaper is finally published. It's titled "Adapting the Agile Unified Process to Blended Shore Projects." You can read it here.

I have put nearly all of my recommendations into practice and they are working. Our next step is to collect more metrics to help us plan the project.

New Metrics

• Use Function Point Analysis to create estimates by feature
• Review & update FPA estimates after each iteration
• Record the number of function points per feature in the Product Backlog
• Record the actual effort for each feature after it is implemented
• Team Productivity = Total # of function points implemented in an iteration
• Track actual effort per function point
• Track team productivity over time; the trend should be that the team's productivity increases as they become more familiar with the domain and technology

What's Working

• Product Backlog as the master feature set
• Iteration (Sprint) Backlog for each team
• Daily Scrum
• Iteration Assessments (Sprint Review Meetings)
• Iteration Planning Sessions (Sprint Planning Sessions)
• Early and frequent delivery to QC within an iteration
• Storyboards
• Modeling
  

Challenges

• No automated test tools are available for Avalon/WinFX
• Technical reviews of requirements are not conducted soon enough; we plan to bring more technical resources onshore to close the communication gap
• Tracking rework and changes; we have revised our change control process so we have a record of the changes requested and the effort to implement each

This is fun and it's working well!!!

My next assignment

Since I am the only certified ScrumMaster in our company, I have been tasked with creating a "plan" for how I would run an agile/scrum project with our blended-shore model. Things I need to consider:

1. How will this affect/be different than our existing RUP process?
2. Would applying this "plan" to a project jeapordize our CCMi Level 5 certification, if at all?
3. What training would people on the project need?
4. What communication mechanisms should the team use?
5. Do I need to have the "best" people place on this project? What skills sets will I need?
   
6. Team size
7. Offshore team location (India, Vietnam, or both)

Fortunately for me, there has been much research done already in this area and I have plenty of materials to help me get started.

Maybe I'll post my plan here along the way to get some feedback (assuming anyone actually reads this blog).

Welcome to The SCRUM Blog!

I am a recently Certified ScrumMaster (CSM) and I have been implementing SCRUM (and Agile) practices and techniques on my projects for the past year. I created this blog in the hopes that it will serve as a discussion area for those of us wanting to implement Agile and SCRUM. I welcome your comments.