Don't just test, change how you code

When deciding on your testing plans and your process for handling bugs, keep in mind this question:

Which practices will go beyond reporting a snapshot of current quality?  Which practices will drive positive changes in how we develop going forward?

One unfortunate pattern you see in many software teams is:

1. Have a set of programmers who are responsible for making new functionality appear as quickly as possible.
2. Have a set of testers who are responsible for quality.
3. Have the programmers "hand off" new features to testers as soon as possible, so they can move on to the next new feature they need to implement.
4. Don't have testers interact with programmers (review design, do preliminary testing, create/review test plans) until the feature is 'done', and even then have minimal interaction.
5. Have the primary communication between testers and programmers be the bug database.  The only real 'output' of the testers is to create bugs in the bug database.
6. A manager will prioritize the entries in the bug database and an engineer will be assigned to resolve each high priority bug.
7. Metrics are kept for inflow, outflow, total bug counts for each priority level.  But other than that no analysis is done to look for patterns in the bugs.  Does a bug represent one of a common type? Are parts of the codebase much more prone to having bugs reported against them?  Typically those questions are not answered.
8. If there is a concern that the quality is not high enough, the reaction is to increase the number of testers and the volume of testing.

This approach is far from optimal, of the various issues here are three that stand out:

• 'responsibility' for quality should be on everyone not just on the testers who measure it.
     
• this process reinforces a "reactive" approach where the team is driven by what is "urgent" (handing off the current feature, responding to the bug report) rather than what is "important" (improving your skills and practices to increase the baseline productivity and quality of the team going forward).
     
• this process has no feedback loop that drives the team to improved results

I do believe in having dedicated testers and I do believe in the importance of a bug database and I do believe in tracking bug counts.  However, those things by themselves aren't enough.  You need to go beyond these things to implement QA practices that drive continuous improvement in how you develop, and not just improvement in how you test.

What specific practices make sense depend on your circumstances: size of team, maturity of developers, nature of software being developed, etc.  Here are some possibilities to consider:

• Require automated unit test to be delivered by the developer before new code is considered 'done' and ready for integration. This does not just result in a useful test, it makes developers think harder about their implementation decision and about how their code may be used.
   
  
• To identify potential side-effects require a critical sub-set of all automated tests (not just the ones for the new functionality) to be run as part of the check-in process.  Require that check-ins be held until these tests pass.  This does not just result in fewer broken builds, it encourages developers to think about cross module dependencies.
   
• Require that the entire universe of automated tests be run as part of daily (or more frequent) build.  If any test fails have the engineers who checked-in during that window stop their work and fix the build before continuing.  This practice reinforces the message that it is more productive to fix bugs in existing code first, before adding new code.
   
• Measure code coverage generated by automated tests on a daily basis.  Have specific numerical goals for target code coverage.  Have developers (not testers) be the ones responsible for achieving those goals.  This does not just ensure that you have a robust test suite, it also pushes developers to go back and remove redundant code and to simplify the code that they keep.  Both those steps in turn make the code easier to understand and to maintain.
   
• For bugs found after the developer believes the code is done, in the bug database track the nature and resolution of those bugs.  What part of the codebase was it in?  Was it a logic error, a syntax error, a misunderstanding of the language runtime behavior, a misunderstanding of some public interface, a misunderstanding of requirements, a performance deoptimization, etc?  Look for patterns in this info, share it with the team as a whole and see what conclusions can be drawn.
   
• For each bug in the bug database that was found through testing, ask the developer who resolves it "is it likely that there are similar undiscovered bugs in the codebase that could be found via inspection/search/analysis?"  If yes, make a specific trackable work item to do that inspection/search/analysis.
   
• For a sub-set of the bugs in the bug database have the relevant engineer go in front of the team and present the "lesson learned" from that bug.  Such presentations may be short (5-10 minutes, 1-3 slides) and tagged on to some other meeting that is occurring anyway. A steady stream of such presentations (3-6 a week) is best.
   
• For each non-trivial check-in have the developer write 1-5 bullets that answer "what is the most likely problem to be triggered by this change?" and 1-5 bullets on "suggested system testing to exercise this code."   These bullets can be very terse, it is ok if someone who wants to take action based on one of these bullets has to go back and ask the developer some questions.  However, even the tersest comments, made at the moment that that code is freshly in mind, can be incredibly helpful.
     
• Have developers take part in system testing.  It would be a mistake to have every developer take part in every system test.  However, it is also a mistake for developers to never help exercise the system as a whole.  They need to understand in a tangible way the system as a whole and not just their piece.  Also having a sense of the challenges facing the testers will drive them to pay more attention to the quality of their code and to do a better job of communicating with the testers.

There are some common threads in the above practices.  One thread is to get developers to foresee where bugs are going to be and resolve them before that bug is found by testing.  Another thread is to move away from "whip it out and hand it off" approach and have practices that encourage developers to review their code multiple times from multiples perspectives.

                                                                                 copyright 2007 Kerry Champion