A Simple Matter of Software

Need for nimbleness

The interesting thing about the Microsoft-Yahoo merger to me is not that two companies that lost out to Google now are combining, but rather why they lost to Google in the first place.

 

Here is one common thread in the commentary:

"Yahoo management has not been nimble, Microsoft has not been nimble, Google has a management team that is nimble and dynamic."  - SF Chronicle

Over the past few years, "Yahoo became less nimble ... " -  Diab

"I'd add that since Google is a faster growing, more nimble company than Microsoft or Yahoo ... " - Techsmart

"Word has it that Yahoo's once nimble and entrepreneurial culture has turned sluggish and bureaucratic. The company's organizational structure and compensation system appears to be rewarding silo behavior that inhibits change."  - Steve Tobak

"... what some see as an inability to respond to more nimble (though considerably larger) Google" - CNET

Falling back into a "comfort zone" and building mechanisms (bureaucratic or otherwise) that "ensure stability", is a typical human response to success.  In its time Yahoo had plenty of success and slowed their rate of change as a result. 

For a Internet service business where the "network effect" applies and there are "very low switching costs" for the end user.  It is critical that you respond to success by staying nimble and accelerating your rate of progress.  Otherwise someone else will see what you have achieved so far and take the initiative to eat your lunch.

 

Simplify to find the essence

"Our lives are frittered away by detail; simplify, simplify."
                                                              — Henry David Thoreau

Michael Watkins on Urgent vs. Important

Michael Watkins has a new post here with specific tactics to use when you find that the "urgent crowds out the important".

There are some good suggestions here.  However, I believe Seth Godin's article is the more visceral description of this syndrome.

Leverage

To quickly generate dramatic growth in usage ventures need to use "leverage" to their advantage.

Marc Andreessen provided a simple definition of leverage, while discussing which businesses fit the "venture-backed startup model":

Startups that have a credible potential to be sold or go public for a 10x gain on invested capital within 4 to 6 years of the date of funding should consider raising venture capital.

Most other startups should not raise venture capital. This includes: ... startups where there's no inherent leverage in the business model that could result in a 10x gain in 4 to 6 years ...

Notably, there are many fine businesses in the world -- many of them highly profitable, and very satisfying to run -- that do not have leverage in their model that makes them suitable for venture capital investment.

By leverage in this context, I mean: the ability to make something once (a piece of software, a chip design, a web site) and sell it (directly or indirectly) to a lot of people (1,000 business customers or 10 million consumers) -- which leads to the classic "hockey stick" revenue projection.

While the above is acknowledged as true in the abstract, in practice actual startups often lose sight of this point and create patterns where there is not enough leverage.

Here is one typical software venture scenario:

• Start-up with no revenue and no market presence feels intense pressure to land some key large customers to establish their credibility and prove the value of their business proposition.
     
• Those initial large customers are hesitant to work with a small unproven company.  The start-up responds by saying "we may be small, but that is an advantage as we can be nimble and quickly deliver to you a product/service that exactly matches your specific individual requirements"
     
• This pitch succeeds and lands some important initial customers.  Because they see something working (after a long period of struggling) the sales team repeats the same pitch to other potential customers.
     
• Under pressure to move quickly the software team does not build a configurable platform but rather does customer-specific source modifications to the core product to deliver on what each customer expects.  Which leads to multiple source branches, high maintenance cost.  low ability to deploy new features across all customers, high incremental costs per each new customer, etc.

The above scenario is common and can lead directly to falling into a low leverage trap.

There are absolutely times when doing a low leverage special feature for a specific launch customer is the right thing to do.  The question is, do you want this to be a rare exception or is it a pattern you want to replicate?

Here are two models:

1. .“On-demand individualized customer service" model focused on directly  responding to unique needs of large customers, through a negotiated delivery plan.  “Professional services” companies such as IBM Global Services and Infosys fit this model.  As does a large part of the Defense industry: companies such as SAIC.
      
   
   B.
2. "Market service" model focused on executing roadmap that delivers high leverage: creating a single unified service/product that meets the needs of the broad market. Examples here include: eBay, Amazon, Google, Adobe, and Microsoft.

Successful businesses can be built using either model.  However, the "market service" model has higher leverage, higher valuations, and greater probability of leading to fundamental changes to how we work and live. 

If you have the opportunity to pursue this "market service" model you should clearly do so, even if it means extra effort and extra risk.

If your business does not lend itself to pursuing the "market service" model that is ok too, you can build a good business creating custom solutions for large customers.  However, that approach will mean slower growth than a successfull "market service" model and you need to make sure your team and investors understand that.

The one thing that absolutely does not work is to be mis-aligned:  you claim you are aimed at one of the models and some of your activities reflect that, but a large part of your actual behavior fits the other model the one you are not targeting.  This is a recipe for painful failure.

Committing to building a high leverage model takes real discipline: the engineering team has to avoid taking tempting (but non-scalable) shortcuts, the sales team will sometimes need to pass on a customer which goes against all instincts.  However, if you can recognize the difference between urgent and important, and instill that self-discipline you will be well rewarded.

                                                                               copyright 2007 Kerry Champion

What does a teaching hospital have to teach a software team?

Stanford Hospital is one of the better known institutions in Silicon Valley.  It is a  teaching hospital where some of the best learn their profession.  Every day the doctors and staff take the time to share what they know and improve their skills.  They do this through a variety of mechanisms: doing rounds together, meeting to review patient outcomes, reviewing charts and asking questions, writing papers, presenting at conferences, etc.  They do all this while literally dealing with life and death issues, and while working inhumanly long hours.

Often in software teams you will hear comments such as these:

• I don't have time to train the new guy, just have him read the code.
• It is too much hassle to include a comment explaining my checkin.
• There is not enough time to talk the whole team through the architecture of the new module and explain why we made the decisions we made.
• We just can't seem to fit in the code reviews.
• Oh I never bother filling in the "root cause" field when closing a bug in the bug base, no one is interested.
• What is the point of doing a debriefing now that the release is over, no one will pay attention to the results.
• etc. etc.

Sometimes my response is:  "Staff at teaching hospitals work long hours under intense stress.  Just down the road at Stanford they still find time to fill in charts, go on rounds, go to postmortems, and all the rest.  If they can both 'do and teach' why can't we?"

One of the things that makes teaching hospitals so effective is they don't just teach from books (medical school takes care of that) they teach through considered examination of real life situations.  Software teams have the opportunity to do the same thing.

Teaching a rapidly evolving body of knowledge, requires observation, reflection and articulation.  Software teams are often very bad at all three, rushing from the latest new feature to fixing the customer critical bug, without taking the time to gather observations, reflect on that data drawing useful conclusions, and then capture those insights into a form that can be transmitted again and again.

To make your team the equivalent of a teaching hospital, to make it a "teaching team", requires several elements:

• Management must buy in to the idea that investing in these practices now will pay off with greater total throughput over time. Then team leaders must go on to implement the specific practices: code reviews, bug analysis, project debriefings, etc.
   
• Team members must organize their own work and adapt their attitudes to avoid creating a culture where the "urgent" gets all attention and the "important" gets ignored.
   
• Everyone must constantly ask themselves "ok we have solved this immediate problem, what lesson should we learn from it and how should we spread that lesson around?"

Failure to make a "teaching team" is one of the key leading indicators that you are bound to repeat the same mistakes over and over.

                                                                               copyright 2007 Kerry Champion

 

Don't just test, change how you code

When deciding on your testing plans and your process for handling bugs, keep in mind this question:

Which practices will go beyond reporting a snapshot of current quality?  Which practices will drive positive changes in how we develop going forward?

One unfortunate pattern you see in many software teams is:

1. Have a set of programmers who are responsible for making new functionality appear as quickly as possible.
2. Have a set of testers who are responsible for quality.
3. Have the programmers "hand off" new features to testers as soon as possible, so they can move on to the next new feature they need to implement.
4. Don't have testers interact with programmers (review design, do preliminary testing, create/review test plans) until the feature is 'done', and even then have minimal interaction.
5. Have the primary communication between testers and programmers be the bug database.  The only real 'output' of the testers is to create bugs in the bug database.
6. A manager will prioritize the entries in the bug database and an engineer will be assigned to resolve each high priority bug.
7. Metrics are kept for inflow, outflow, total bug counts for each priority level.  But other than that no analysis is done to look for patterns in the bugs.  Does a bug represent one of a common type? Are parts of the codebase much more prone to having bugs reported against them?  Typically those questions are not answered.
8. If there is a concern that the quality is not high enough, the reaction is to increase the number of testers and the volume of testing.

This approach is far from optimal, of the various issues here are three that stand out:

• 'responsibility' for quality should be on everyone not just on the testers who measure it.
     
• this process reinforces a "reactive" approach where the team is driven by what is "urgent" (handing off the current feature, responding to the bug report) rather than what is "important" (improving your skills and practices to increase the baseline productivity and quality of the team going forward).
     
• this process has no feedback loop that drives the team to improved results

I do believe in having dedicated testers and I do believe in the importance of a bug database and I do believe in tracking bug counts.  However, those things by themselves aren't enough.  You need to go beyond these things to implement QA practices that drive continuous improvement in how you develop, and not just improvement in how you test.

What specific practices make sense depend on your circumstances: size of team, maturity of developers, nature of software being developed, etc.  Here are some possibilities to consider:

• Require automated unit test to be delivered by the developer before new code is considered 'done' and ready for integration. This does not just result in a useful test, it makes developers think harder about their implementation decision and about how their code may be used.
   
  
• To identify potential side-effects require a critical sub-set of all automated tests (not just the ones for the new functionality) to be run as part of the check-in process.  Require that check-ins be held until these tests pass.  This does not just result in fewer broken builds, it encourages developers to think about cross module dependencies.
   
• Require that the entire universe of automated tests be run as part of daily (or more frequent) build.  If any test fails have the engineers who checked-in during that window stop their work and fix the build before continuing.  This practice reinforces the message that it is more productive to fix bugs in existing code first, before adding new code.
   
• Measure code coverage generated by automated tests on a daily basis.  Have specific numerical goals for target code coverage.  Have developers (not testers) be the ones responsible for achieving those goals.  This does not just ensure that you have a robust test suite, it also pushes developers to go back and remove redundant code and to simplify the code that they keep.  Both those steps in turn make the code easier to understand and to maintain.
   
• For bugs found after the developer believes the code is done, in the bug database track the nature and resolution of those bugs.  What part of the codebase was it in?  Was it a logic error, a syntax error, a misunderstanding of the language runtime behavior, a misunderstanding of some public interface, a misunderstanding of requirements, a performance deoptimization, etc?  Look for patterns in this info, share it with the team as a whole and see what conclusions can be drawn.
   
• For each bug in the bug database that was found through testing, ask the developer who resolves it "is it likely that there are similar undiscovered bugs in the codebase that could be found via inspection/search/analysis?"  If yes, make a specific trackable work item to do that inspection/search/analysis.
   
• For a sub-set of the bugs in the bug database have the relevant engineer go in front of the team and present the "lesson learned" from that bug.  Such presentations may be short (5-10 minutes, 1-3 slides) and tagged on to some other meeting that is occurring anyway. A steady stream of such presentations (3-6 a week) is best.
   
• For each non-trivial check-in have the developer write 1-5 bullets that answer "what is the most likely problem to be triggered by this change?" and 1-5 bullets on "suggested system testing to exercise this code."   These bullets can be very terse, it is ok if someone who wants to take action based on one of these bullets has to go back and ask the developer some questions.  However, even the tersest comments, made at the moment that that code is freshly in mind, can be incredibly helpful.
     
• Have developers take part in system testing.  It would be a mistake to have every developer take part in every system test.  However, it is also a mistake for developers to never help exercise the system as a whole.  They need to understand in a tangible way the system as a whole and not just their piece.  Also having a sense of the challenges facing the testers will drive them to pay more attention to the quality of their code and to do a better job of communicating with the testers.

There are some common threads in the above practices.  One thread is to get developers to foresee where bugs are going to be and resolve them before that bug is found by testing.  Another thread is to move away from "whip it out and hand it off" approach and have practices that encourage developers to review their code multiple times from multiples perspectives.

                                                                                 copyright 2007 Kerry Champion

Development via trial and error

My daughter has a t-shirt that says on the front:

There are 10 kinds of people in the world.

On the back it says:

Those that understand binary and those that don't.

She is in Junior High and thinks this is hilarious.  It has been a long time since I was in Junior High but I also think it is hilarious.  I love it when you can make good use of base-2 nomenclature. 

In any case, on to the real topic. 

There are 10 kinds of programmers in the world: those that code via trial and error; and those that don't.

When I first started programming as a teenager I was a trial and error coder.   Luckily I later worked with some consummate professionals who showed me a better way.

Unfortunately lots of folks who are nominally professional software engineers still use the trial and error approach.  The following is a typical scenario:

1. Grab some existing piece of code that has some overlap with what you are trying to accomplish
2. Modify that code so that it is at least somewhat in line with your current goal
3. Try to compile it, not really expecting it to work
4. Sure enough it has syntax errors.  Don't bother to re-read the all the code, just go in and fix the individual errors as reported by the compiler
5. Assume the compiler/linker errors/warnings are very accurate.  Only as a last resort go and pull the language or linker book off the shelf to read anything about required syntax and recommended usage.
6. Repeat steps 3-5 until you have a clean build
7. Now run your code, not really expecting it to work
8. Have some basic test that illustrates the most common use case for what you are trying to implement.
9. Manually execute that test.
10. If that test fails (which it almost certainly will the first several times), don't bother to go back and re-read all the code with this failure in mind.  Instead single-step through the debugger to see where things go off the tracks. 
11. As you step through using the debugger you will eventually realize that something is wrong.  Whatever code you are looking at when you make this realization, assume that is the code that needs to change.  Go in and modify that code to handle the current failure case. 
12. Don't consider the possibility of refactoring to solve the problem.
13. Always assume you must add more code to fix the problem, don't consider the option of simplifying the logic to address the issue.
14. Don't spend much time thinking about the side effects of your change, assume it is easier to catch those via testing than by thinking hard about logic and data structures.
15. Don't bother analyzing the error to see if it is just one example of a more widespread issue.  Assume any analogous errors that exist elsewhere in the code will be found via testing, don't bother trying to find them via inspection.
16. Now repeat steps 3-11 until your test case passes.
17. As soon as your test case passes, declare victory and stop work on this code.  Tell QA to start testing the functionality.
18. Don't bother to write automated unit tests or to do code coverage analysis
19. There is an urgent need for you to go back and debug problems in your previous code (that you wrote using this same approach).  Assume this need to fix what supposedly was working fully justifies your decision to not spend time on: reviewing your new code with others, refactoring based on new insights, executing multiple test scenarios, developing automated tests, searching for analogous bugs via inspection, etc. etc.

The problems with the above approach should be obvious.  However, it is shockingly frequent that this is exactly how commercial programming gets done.

Sometimes this is because the engineer is simply incompetent, however much more often it is other factors that drive this behavior.  For example:

• Management consistently sending the message "what really matters is getting new functionality 'working' as soon as possible" and never sending the message "sound programming practices will pay off in the long run and deserve our attention and time"
     
• The normal human tendency to prefer being driven by what is urgent, rather than basing our efforts on proactively defining what is important.  To break out of this habit requires conscious support from the leaders within the team.

I think it is critical to explicitly talk to the team about why "trial and error" programming is a bad idea, to set up practices that encourage a more mature approach, and to make it part of your recruiting process to screen out this type of programmer.  I will talk more about these last two points in subsequent posts.

                                                                                 copyright 2007 Kerry Champion

Make New Mistakes

There are some bits of wisdom that are so often repeated that we become desensitized to the need to take action based on that truth.

For example, we have all heard and agreed with some version of:

"Insanity is doing the same thing, over and over again, but expecting different results." - Rita Mae Brown

“Progress ... depends on retentiveness. Those who cannot remember the past are condemned to repeat it.” - George Santayana

"Always make new mistakes" - Esther Dyson

Yet it is common for software ventures to repeat the same mistakes made in the past.  Why is that?

I don't claim to know the full answer, but here are some contributing factors:

1) Reactive vs. Proactive mind-set

If you manage by waiting for bad things to happen and then reacting to them you are more likely to fall in the same bad patterns again and again.

If you manage by proactively defining and driving an agenda for change you are more likely to avoid old mistakes and discover new ones.

2) Deferred Gratification and Impulse Control

Organizations with poor impulse control tend to repeat the same mistakes again and again.  Software ventures that master the art of deferred gratification will have better long term results.

3) Wish-based Planning

You often see old mistakes repeated anew, when organizations make plans based on how they "wish" things were rather than based on a more realistic assessment.

Lots of examples of this sort of behavior in:

• assessing feedback from customers
• dealing with unpleasant truths
• assessing how feature commitments will impact your ability to innovate
• defining your market
• deciding how to throttle your spending plan
• etc. etc.
  

4) Urgent vs. Important

Organizations that only react to the urgent and never focus on the important, tend let avoidable mistakes occur and than run a fire-drill to respond to them. They then repeat the process again and again.

It is critical that you know how to distinguish urgent from important and why that matters.

5) Low Investment in Lessons Learned Exercises

One important activity that is often pushed aside in favor of more urgent ones, is the "lesson learned exercise."  This goes by different names in different organizations:  "after action report", "debriefing", and "postmortem" (my least favorite term).  In all cases it involves a conscious systematic process to look at what as been going well, what has been going badly, and how the team can learn and improve.

Investing in a "lessons learned" process will decrease the chance of repeating old mistakes.

6) Split Incentives

If different parts of the organization have different incentives and priorities you can easily create situations where specific individuals are motivated to lead the team into making the same mistake again and again.

If you build an engineering team that is motivated by a desire to "build frameworks" and you don't clearly communicate the organizational imperatives that might motivate them to "buy the framework and build a solution on top" than you will repeatedly err on the side of investing too much in building frameworks and not enough in building solutions.

If you have a sales team where there cash compensation is based 100% on "signing new contracts" and 0% on "keeping customers happy through deployment and use", than you will consistently make mistakes around over-committing and under-executing.

In both the above examples the motivations of parts of the team are out of alignment with the organization as a whole.

   

                                                                                 copyright 2007 Kerry Champion

Urgent vs. Important

Most things which are urgent are not important, and most things which are important are not urgent. - Dwight D Eisenhower

Understanding the difference between urgent and important is critical to making a software venture work.

Seth Godin is more articulate in describing this dynamic than I could be:

The easiest way to deal with change and with all the anxieties that go with it is not to deal with it at all. The easiest thing to do is to allow the urgency of the situation to force us to make the decisions (or take the actions) that we'd rather not take. Why? Because then we don't have to take responsibility for what happens. The situation is at fault, not us. ...

Urgent issues are easy to address. They are the ones that get everyone in the room for the final go-ahead. They are the ones we need to decide on right now, before it's too late.

How can you tell if you're too obsessed with urgent?

Do senior people at your company refuse to involve themselves in decisions until the last minute?

Do meetings regularly get canceled because something else came up?

Is waiting until the last minute the easiest way to get a final decision from your peers?

Smart organizations ignore the urgent. Smart organizations understand that important issues are the ones to deal with. If you focus on the important stuff, the urgent will take care of itself. ... You will succeed in the face of change when you make the difficult decisions first.

Organizations manage to justify draconian measures--laying people off, declaring bankruptcy, stiffing their suppliers, and closing stores--by pointing out the urgency of the situation. They refuse to make the difficult decisions when the difficult decisions are cheap. They don't want to expend the effort to respond to their competition or fire the intransigent VP of development. Instead, they focus on the events that are urgent at that moment and let the important stuff slide.

A quick look at the gradually failing airlines, retailers, and restaurant chains we all know about confirms this analysis. They're all content to worry about today's emergency, setting the stage for tomorrow's disaster.

Modern Managers have the concept laid out as an annotated matrix, what has come to be known as the Eisenhower Matrix.  They also have a definition of terms that gets right to the point:

Important activities have an outcome that leads to the achievement of your goals.

Urgent activities demand immediate attention, and are usually associated with the achievement of someone else’s goals, or with an uncomfortable problem or situation that needs to be resolved.

   

                                                                                 copyright 2007 Kerry Champion

Does your team pass the marshmallow test?

Predicting the adult success of a 4 year old is a tricky business.  Understanding how income differentials arise in primitive economies is a tricky business.  Guiding a software venture to long term success is also a tricky business.  I believe there is at least one common lesson that applies to all three.

It is very hard to look at a 4 year old child and predict how successful they will be as an adult.  At that age all the normal aptitude and intelligence tests don't seem to be good predictors of ultimate life outcomes.  However an exception are tests of deferred gratification, such as the famous marshmallow experiment.  Wikipedia does a good job of summarizing:

The marshmallow experiment is a famous test of this concept conducted by Walter Mischel at Stanford University and discussed by Daniel Goleman in his popular work. In the 1960s a group of four-year olds were tested by being given a marshmallow and promised another, only if they could wait 20 minutes before eating the first one. Some children could wait and others could not. The researchers then followed the progress of each child into adolescence, and demonstrated that those with the ability to wait were better adjusted and more dependable ...

... Shoda, Y., Mischel, W., Peake, P. K. (1990). Predicting adolescent cognitive and self-regulatory competencies from preschool delay of gratification: Identifying diagnostic conditions. Developmental Psychology, 26(6), 978–986.

An analogous study was done to try and explain the rise of income differential.   An experiment was conducted among the Tsimane', a group of Amerindians who live in the Amazon.  The Economist summarized it well in Economics and Anthropology - Patience is a virtue, Feb 8th 2007:

... Exactly why some people were able to accumulate more than others has been something of an anthropological mystery ... the main hypotheses have been luck, intelligence and aggression. These all, of course, play their part, but now a fourth phenomenon has been added to the list: patience

... One phenomenon that is almost unique to humans is deferred gratification ... the researchers offered all 151 adults in two Tsimane' villages a choice between receiving a small amount of money or food immediately, getting a larger amount if they were willing to wait a week, and getting a larger amount still in exchange for several months' wait for payment. ... Five years later, Dr Reyes-Garcia and her colleagues came back again ... and found that those who had shown most patience in the original experiment had also seen their incomes increase more than those of their less patient counterparts ...

The same concept applies to software ventures. Organizations that master the art of deferred gratification will have greater success over time.

There are plenty of times when you will be tempted to do something that feels good at the moment, but won't be the optimal step for the long term.  Previous posts talked about situations where it makes sense to pass on a customer or hold off committing to additional features, even though there is a lot of immediate pressure to take those steps.  Sometimes you just have to say no to the marshmallow right in front of you, to get more marshmallows later on.

However, like any good idea it is possible to misapply it.  It is critical to remember that software enabled ventures usually have limited windows of opportunity, so that being too late often means failure.  If you don't quickly package up a useful set of features, if you don't close a critical mass of reference customers early, you may miss your window altogether and never have a chance to collect any marshmallows.  Implicit in the concept of "deferred gratification" is the idea that you are selectively choosing to defer some benefit only in cases where that deferral increases the total benefit.  Buying into the concept of deferred gratification is not a license to lose your sense of urgency.

Using deferred gratification to your advantage is just one part of a broader proactive leadership style.  The most success goes to those who don't just react to opportunities and problems that pop-up in front of them, but rather take charge of the agenda and proactively manage the team with a clear vision of where they are going.

                                                                                 copyright 2007 Kerry Champion