A Simple Matter of Software

Becoming A Craftsman

Interesting new book on craftsmanship. 

All craftsmanship [is] skill developed to a high degree.  By one commonly used measure, about ten thousand hours of experience is required to produce a [craftsman] ... as skill progresses, it becomes more problem-attuned ... whereas people with primitive levels of skill struggle more exclusively on getting things to work  ... At its higher reaches, technique is no longer a mechanical activity; people can fell fully and think deeply what they are doing once they do it well.

Producing a chair that holds a persons weight does not make you a master carpenter, producing a software program that doesn't crash does not make you an architect. 

Many programmers simply progress through more sophisticated forms of trial and error without ever really mastering their craft.  That is a shame.

Changing Behavior - Focus on what you want not what you don't want

Slate has a good article on changing people's behavior.  This article is about children's behavior but most of the ideas apply just as well to a 40 year old as a 4 year old.

You begin by deciding what you want the [person] to do, the positive opposite of whatever behavior you want to stop. The best way to get rid of unwanted behavior is to train a desirable one to replace it. ...

Then you tell the [person] exactly what you would like him to do. ...

Whenever you see ... what you would like, or even something that's a step in the right direction, you not only pay attention to that behavior, but you praise it in specific terms ...

If you don't see enough of the desirable behavior, then you can work on it using simulation ... Your objective is to arrange for as much reinforced practice as possible, ...  A brief but intensive period featuring lots of reinforced practice, often somewhere between a couple of weeks and a month, can make long-lasting or even permanent changes in ... behavior.

Going ballistic never helps, but explanation aimed at improving ... understanding can actually play a useful part in this approach. When combined with reinforced practice, explanation has been proven to speed up the acquisition of behavior.

Your mission should you choose to accept it

How a great communicator defines a big audacious goal:

We will "put a man on the moon and return him safely by the end of the decade."
                                                                    - John F. Kennedy

How the typical business executive would define the same goal:

"Our mission is to become the international leader in the space industry through maximum team-centered innovation and strategically targeted aerospace initiatives."

Thanks to Made to Stick for creating this example.

Too many software ventures have mission statements that sound like the latter.  Don't try to define your mission so it covers all the bases and supports every contingency.  Instead focus on defining your mission in terms that are simple, concrete, and emotionally compelling.

How to retain the people you want to retain

Marc Andreessen has a good post on how to retain the people you want to retain:

It's about winning.

Let me explain:

Companies that are winning -- even really big, old ones -- never have a retention problem.  Everyone wants to stay, and when someone does leave, it's really easy to get someone great to take her place.

Companies that have a retention problem usually have a winning problem . . .

. . . Great people want to work at a winner.

All the raises, perks, and HR-sponsored "company values" drafting sessions in the world won't help you retain great people if you're not winning -- not even the $6,000 heated Japanese toilets in all the restrooms, the $30,000 Olympic lap pool out back, and the free $4 bottles of organic orange juice in all the snack rooms.

I would just add two comments.

1) “Winning” is not enough, the people inside need to viscerally feel that they are winning

"Winning" but having people not "feel" that they are winning does happen more often than you would suppose.  People get caught up in all the daily challenges and issues and lose the big picture.  I have been at start-ups that are hitting it out of the park (tens of thousands of users, tens of millions in revenue, a good team of people who work together well, coming up on IPO, a solution that is certain to be in growing use for the next decade) and yet there is a percent of the population that have closed their eyes and can't see the value of what they are creating.   

In these cases, where some of the people don't understand/appreciate what they have got, I have seen good people leave to do something "new".  Only to see the hard work on their new project amount to nothing (most new software ventures fail and result in a bunch of lovingly crafted software that does not get used long term) while their old company builds success upon success.  This can be the cause of substantial regret.

2) People who don’t care about working at a winner are people who you will eventually be better off without.

Though be careful to distinguish between people who will never care about the team's success and people who are justifiably frustrated about some important aspect of their environment.   People who will never care probably shouldn't be long term members of the team.  On the other hand, for the people who could care if only some fundamental dysfunction is addressed, you should keep them around and fix the dysfunction.

                                                                               copyright 2007 Kerry Champion

Surveillance and diligence

As I mentioned in a previous post I recently heard Atul Gawande talk about his book "BETTER: A Surgeon’s Notes on Performance".  He made observations about why some teams have much better results (in terms of patient medical outcomes) than others.

He highlighted two traits that seemed to be central to that success: "surveillance" and "diligence".  These traits are incredibly important in the software world as well, for much the same reasons as touched on by Atul.

I am not sure that "surveillance" is the best possible phrase to use, I prefer to think in terms of "active observation."  Whatever the label the concept is critical.  Here are some points to keep in mind when practicing active observation:

• Don't depend solely on your own observations, enlist the broader team to take part in the process and listen carefully when they share what they are seeing.
   
• Don't accept everything you hear at face value, question and probe, search for validating or contradictory data points.
   
• Make sure you have a good consistent data set that lets you make comparisons over time and comparisons between different people and different projects.  When there is variation in that data set probe into it, even if the variation seems small.
   
• Take advantage of your insights into well known patterns of misleading reports.  Some folks are modest and pessimistic and consistently make things sound worse than they are.  Others may tend towards self-justification.  In any case you need to look for these patterns and take them into account as you get new reports from the same source.
   
• As you form conclusions based on these observations, test that conclusion, make predictions based on that conclusion and go look at the relevant data.  If your new data points don't support your prediction, than your conclusions are probably wrong.

Atul makes some key points regarding "diligence":

• Diligence can be defined as "the constant and earnest effort to accomplish what is undertaken."
   
• "People underestimate the importance of diligence as a virtue" perhaps because on the surface it sounds mundane and simplistic.
     
• However, diligence is a "prerequisite of great accomplishment" and "one of the most difficult challenges facing any group of people who take on tasks of risk and consequence"

I agree wholeheartedly with these points and would add just a couple more regarding the practice of diligence. 

• It is unreasonable to expect diligence from someone who does not understand why their tasks matter in the bigger scheme of things.
     
• Having relevant and easy to understand metrics that are shared with the entire team, encourages diligence and continuous improvement.
     
• Developing team practices that enable folks to get into a good rhythm and spend most of their time "in the zone" is critical to encouraging diligence.  There are lots of things (too many meetings, unproductive meetings, inconsistent goals/feedback, constantly switching assignments, etc.) that will pull folks out of the productive zone and discourage them from trying.
   
• An out-of-context or irrelevant response to anxiety or social tension is called a displacement behavior.   I believe this term was originally from animal behavior studies, but it certainly applies to humans as well.  For example you may feel anxious about a deadline, doing the relevant work just reminds you of how hard it will be to make the deadline, which makes you even more anxious, so you find yourself reading Slashdot or reorganizing the source tree instead of actually working on the relevant implementation.  Team leaders need to give folks an open door and when they express anxiety or point out tension, leaders need to help them work-around those issues so they can pursue the tasks that matter instead of getting sucked into an eddy.   

Here is Atul's description of a Doctor who has mastered the arts of surveillance and diligence.  This describes an examination of a Cystic Fibrosis patient (Janelle) by her Doctor (Warwick).  Even though the specifics of this situation are far from the specifics of a software project, the mindset and basic skills shown here are extremely relevant.

Warwick pulled out her latest lung-function measurements. There’d been a slight dip, ... Three months earlier, Janelle had been at a hundred and nine per cent (she was actually doing better than normal); now she was at around ninety per cent. Ninety per cent was still pretty good, and some ups and downs in the numbers are to be expected. But this was not the way Warwick saw the results.

He knitted his eyebrows. “Why did they go down?” he asked.

Janelle shrugged.

Any cough lately? No. Colds? No. Fevers? No. Was she sure she’d been taking her treatments regularly? Yes, of course. Every day? Yes. Did she ever miss treatments? Sure. Everyone does once in a while. How often is once in a while?

Then, slowly, Warwick got a different story out of her: in the past few months, it turned out, she’d barely been taking her treatments at all.

He pressed on. “Why aren’t you taking your treatments?” He appeared neither surprised nor angry. He seemed genuinely curious, as if he’d never run across this interesting situation before.

“I don’t know.”

He kept pushing. “What keeps you from doing your treatments?”

“I don’t know.”

“Up here”—he pointed at his own head—“what’s going on?”

“I don’t know,” she said.

He paused for a moment. And then he began speaking to me, taking a new tack. “The thing about patients with CF is that they’re good scientists,” he said. “They always experiment. We have to help them interpret what they experience as they experiment. So they stop doing their treatments. And what happens? They don’t get sick. Therefore, they conclude, Dr. Warwick is nuts.”

“Let’s look at the numbers,” he said to me, ignoring Janelle. He went to a little blackboard he had on the wall. It appeared to be well used. “A person’s daily risk of getting a bad lung illness with CF is 0.5 per cent.” He wrote the number down. Janelle rolled her eyes. She began tapping her foot. “The daily risk of getting a bad lung illness with CF plus treatment is 0.05 per cent,” he went on, and he wrote that number down. “So when you experiment you’re looking at the difference between a 99.95-per-cent chance of staying well and a 99.5-per-cent chance of staying well. Seems hardly any difference, right? On any given day, you have basically a one-hundred-per-cent chance of being well. But”—he paused and took a step toward me—“it is a big difference.” He chalked out the calculations. “Sum it up over a year, and it is the difference between an eighty-three-per-cent chance of making it through 2004 without getting sick and only a sixteen-per-cent chance.”

He turned to Janelle. “How do you stay well all your life? How do you become a geriatric patient?” he asked her. Her foot finally stopped tapping. “I can’t promise you anything. I can only tell you the odds.”

In this short speech was the core of Warwick’s world view. He believed that excellence came from seeing, on a daily basis, the difference between being 99.5-per-cent successful and being 99.95-per-cent successful. Many activities are like that, of course: catching fly balls, manufacturing microchips, delivering overnight packages. Medicine’s only distinction is that lives are lost in those slim margins.

And so he went to work on finding that margin for Janelle. Eventually, he figured out that she had a new boyfriend. She had a new job, too, and was working nights. The boyfriend had his own apartment, and she was either there or at a friend’s house most of the time, so she rarely made it home to take her treatments. At school, new rules required her to go to the school nurse for each dose of medicine during the day. So she skipped going. “It’s such a pain,” she said. He learned that there were some medicines she took and some she didn’t. One she took because it was the only thing that she felt actually made a difference. She took her vitamins, too. (“Why your vitamins?” “Because they’re cool.”) The rest she ignored.

Warwick proposed a deal. Janelle would go home for a breathing treatment every day after school, and get her best friend to hold her to it. She’d also keep key medications in her bag or her pocket at school and take them on her own. (“The nurse won’t let me.” “Don’t tell her,” he said, and deftly turned taking care of herself into an act of rebellion.) So far, Janelle was O.K. with this. But there was one other thing, he said: she’d have to come to the hospital for a few days of therapy to recover the lost ground. She stared at him.

“Today?”

“Yes, today.”

“How about tomorrow?”

“We’ve failed, Janelle,” he said. “It’s important to acknowledge when we’ve failed.”

With that, she began to cry.

Warwick’s combination of focus, aggressiveness, and inventiveness is what makes him extraordinary. He thinks hard about his patients, he pushes them, and he does not hesitate to improvise.

I particularly appreciate the comment "... combination of focus, aggressiveness, and inventiveness is what makes him extraordinary"  That certainly applies to the software world as well.

                                                                               copyright 2007 Kerry Champion

Ineptitude or ignorance

in·ept   \i-ˈnept\
1 : lacking in fitness or aptitude : unfit  <inept at sports>    
2 : lacking sense or reason : foolish        
3 : not suitable to the time, place, or occasion : inappropriate often to an absurd degree <an inept metaphor>        
4 : generally incompetent

ig·no·rant   \ˈig-n(ə-)rənt\
1 : destitute of knowledge or education <an ignorant society>
2 : lacking knowledge or comprehension of the thing specified <parents ignorant of modern mathematics>  
3 : generally unaware,   uninformed

I recently heard Atul Gawande talk at Kepler's.  He has written "BETTER: A Surgeon’s Notes on Performance".  The issues of how to achieve peak performance, how to achieve optimal outcomes, was central to his talk.  Some of the lessons he discussed seem very applicable to  the software world.

For example one phrase that caught my attention was the question of "ignorance vs. ineptitude".

He makes the case that the fundamental barrier to optimal outcomes changed during the last century.  In the 19th century the real issue was ignorance.  Understanding the germ theory, understanding the mechanisms of metabolism, understanding genetics, etc. etc. were the most important challenges to improving real world medical care.  In other words the issue was ignorance.

In the 21st century the situation is fundamentally different.  There is still plenty to learn, it is not the case that ignorance is banished.  However, now the biggest near term opportunity for improved performance is addressing issues of ineptitude, rather than ignorance.  The most immediate "bang for the buck" comes from addressing issues where the knowledge required is present but the problem is effectively applying that knowledge.  The barriers in these cases are diverse: organizational, social, economic, and emotional.  Issues that fall into this category include:

• Creating drug resistant microbes through misuse of antibiotics.  (We will see more of these types of scares as a result.)
• Failure to maximize longevity of cystic fibrosis and diabetes patients through failures in "disease management" practices. 
• In-hospital medical mistakes, such as giving out the wrong drug dosage (one study suggests more die of medical mistakes in hospitals than from car accidents).
• Ineffective practices to encourage consistent pill taking regimes.  (I know this sounds trivial but it is a significant source of bad outcomes.)
• etc. etc.

All of these issues can be addressed today without any medical breakthrough required.  What is required is:

• Understanding of human nature and what it is in human nature that gets in the way of applying what we know.
• Having the will, focus, and discipline required to act and to not stop till you have achieved improved results.

I believe similar points can be made about software development.  Way too many software teams consistently operate far below peak performance, way too many software projects have far from optimal outcomes, and the biggest barrier is ineptitude, not ignorance.  Too often there are perverse incentives, or emotional reactions or consistent behavioral issues that stop us applying lessons we have already learned.  Which in turn leads to us make the same mistakes over and over again.

                                                                               copyright 2007 Kerry Champion

Unexamined life is not worth living

"An unexamined life is not worth living."
                  --  Socrates

This is obviously not literally so, yet there is some fundamental truth here.

There is no question that every human activity benefits from thoughtful observation and reflection.  More specifically, for activities where we are consciously trying to achieve peak performance and continuous improvement, such observation and reflection is an absolute requirement. 

Software ventures clearly fall in this latter category.  A mature and evolved organization will proactively work against an agenda that is driven by thoughtful self-observation.  If you find that is not the case, if you find that you are constantly focused on reacting and firefighting, if you are always responding to the urgent and never working on the important, than I recommend you  remember old Socrates: step back and examine your work life.

                                                                               copyright 2007 Kerry Champion

What does a teaching hospital have to teach a software team?

Stanford Hospital is one of the better known institutions in Silicon Valley.  It is a  teaching hospital where some of the best learn their profession.  Every day the doctors and staff take the time to share what they know and improve their skills.  They do this through a variety of mechanisms: doing rounds together, meeting to review patient outcomes, reviewing charts and asking questions, writing papers, presenting at conferences, etc.  They do all this while literally dealing with life and death issues, and while working inhumanly long hours.

Often in software teams you will hear comments such as these:

• I don't have time to train the new guy, just have him read the code.
• It is too much hassle to include a comment explaining my checkin.
• There is not enough time to talk the whole team through the architecture of the new module and explain why we made the decisions we made.
• We just can't seem to fit in the code reviews.
• Oh I never bother filling in the "root cause" field when closing a bug in the bug base, no one is interested.
• What is the point of doing a debriefing now that the release is over, no one will pay attention to the results.
• etc. etc.

Sometimes my response is:  "Staff at teaching hospitals work long hours under intense stress.  Just down the road at Stanford they still find time to fill in charts, go on rounds, go to postmortems, and all the rest.  If they can both 'do and teach' why can't we?"

One of the things that makes teaching hospitals so effective is they don't just teach from books (medical school takes care of that) they teach through considered examination of real life situations.  Software teams have the opportunity to do the same thing.

Teaching a rapidly evolving body of knowledge, requires observation, reflection and articulation.  Software teams are often very bad at all three, rushing from the latest new feature to fixing the customer critical bug, without taking the time to gather observations, reflect on that data drawing useful conclusions, and then capture those insights into a form that can be transmitted again and again.

To make your team the equivalent of a teaching hospital, to make it a "teaching team", requires several elements:

• Management must buy in to the idea that investing in these practices now will pay off with greater total throughput over time. Then team leaders must go on to implement the specific practices: code reviews, bug analysis, project debriefings, etc.
   
• Team members must organize their own work and adapt their attitudes to avoid creating a culture where the "urgent" gets all attention and the "important" gets ignored.
   
• Everyone must constantly ask themselves "ok we have solved this immediate problem, what lesson should we learn from it and how should we spread that lesson around?"

Failure to make a "teaching team" is one of the key leading indicators that you are bound to repeat the same mistakes over and over.

                                                                               copyright 2007 Kerry Champion

 

Out of the comfort zone

Wikipedia defines "comfort zone" as a set of "artificially created mental boundaries, within which an individual derives a sense of security."

It is natural for humans to build a comfort zone for themselves and in many situations this is an excellent life strategy.  When our Paleolithic ancestors were wrestling hyenas for the right to scavenge the best bits off the gazelle carcase, finding a secure niche and staying in it made a lot of sense.  So long as you are getting your basic needs met in the valley you are in, why move to the next valley over?

Unfortunately finding a comfort zone that meets our basic needs and staying there is not the best strategy for guiding a software team to deliver world-class performance.

Our paleolithic ancestors defined their comfort zones in terms of what geographic area they would search for food; and what animals they would fight vs. which they would fly from.

Software teams tend to define their comfort zone in terms of how they respond to change. 

• Will they limit themselves to incremental changes to existing product or will they create category defining new products? 
• Will they obsessively cling to keeping the current team as is, or will they reorganize the team and add new members as needed to address new challenges? 
• Will they be primarily motivated by risk-aversion or will they be primarily motivated by reward-seeking? 
• Will they train others and share responsibility for their code, or will they have a very proprietary attitude towards the code they wrote?
• etc. etc.

Often  these limitations are not driven by any rational analysis of risks and rewards, but rather is driven by specific previous bad experiences, a desire for control, and a general free-floating anxiety.

Given the natural human tendency to define a fairly constricted comfort zone, it is part of a leader's responsibility to know when and how to break down those boundaries.

Of course creating change for changes' sake or pushing people just to see them squirm is not productive.  Being a "loose canon" who has a new "priority of the week" is not how to guide a team to real success.

Instead your should carefully and consciously call your shots.  You should push people out of their comfort zone when doing so is directly tied to addressing organizational imperatives that are critical to your overall success.  And you should take the time to explain why there is a direct connection between these changes and those imperatives.

Even as you change what needs to be changed, you need to highlight to the team what is remaining stable.  Often people will fixate on the changes and risks, and lose sight of elements which are remaining stable and resources which will provide extra resiliency if things don't go exactly as planned.  Take the time to remind folks of these stable elements and resilient resources.

Let's talk through an example of when taking people out of their comfort zone was the right thing to do.  I was SVP of Development at a software company that had recently acquired another firm of about the same size.  We were just a few miles down the road from each other so we could combine offices.  Each of the predecessor companies had a stable software engineering team run by a VP of Engineering.  In both cases the teams liked and trusted their VPs and the VPs had intimate knowledge of the teams.

After the merger I had several goals including:

1. Maintain rapid progress in the development of both the original and the acquired product lines
2. Find opportunities to integrate the product lines
3. Form an objective assessment of the relative strengths of the two VPs, of the two teams, and of specific individuals on each team

Because of the first goal I did not want to just immediately tear apart both teams and form a new fully integrated team.  Also, until I had fully succeeded at the third goal I as not in a good position to ensure that such a drastic reorganization would be successful.

Therefore, what I did in the short-term was to maintain the teams as-is but to ask the two VPs to switch jobs.  This certainly pushed people out of their comfort zones, everyone involved was hesitant to see this change made and everyone needed to be talked into it.  Each team would have a brand-new top boss and each VP would have a wholly new team.

While people were uncomfortable with this change they accepted it because we could show how it made sense given the goals:

1. By just changing the two VPs, but not reorganizing the teams overall, we could maintain enough continuity to allow rapid progress on both the original and the acquired product lines
2. The switch of the two VPs forced them to rapidly learn the "new" team's product line and technologies which put us in a good position to assess what integration points made sense.
   
3. Also the switch of the two VPs put me in a much better position to get objective assessments of each sub-team and individual; as we now had two senior folks (the VPs) who could provide relevant analysis about everyone.  It also gave me a chance to assess the VPs themselves as I could see if switching them had much impact on the relative productivity of their teams.

The above is an example of a change that was highly useful to make and was directly driven by our imperative goals.  However, it was also a change that was non-obvious and was perceived as a "surprise".  This change was outside those "artificially created mental boundaries" that defined our current comfort zone, that is why it was a "surprise" and why we almost did not implement it.

                                                                                 copyright 2007 Kerry Champion

Development via trial and error

My daughter has a t-shirt that says on the front:

There are 10 kinds of people in the world.

On the back it says:

Those that understand binary and those that don't.

She is in Junior High and thinks this is hilarious.  It has been a long time since I was in Junior High but I also think it is hilarious.  I love it when you can make good use of base-2 nomenclature. 

In any case, on to the real topic. 

There are 10 kinds of programmers in the world: those that code via trial and error; and those that don't.

When I first started programming as a teenager I was a trial and error coder.   Luckily I later worked with some consummate professionals who showed me a better way.

Unfortunately lots of folks who are nominally professional software engineers still use the trial and error approach.  The following is a typical scenario:

1. Grab some existing piece of code that has some overlap with what you are trying to accomplish
2. Modify that code so that it is at least somewhat in line with your current goal
3. Try to compile it, not really expecting it to work
4. Sure enough it has syntax errors.  Don't bother to re-read the all the code, just go in and fix the individual errors as reported by the compiler
5. Assume the compiler/linker errors/warnings are very accurate.  Only as a last resort go and pull the language or linker book off the shelf to read anything about required syntax and recommended usage.
6. Repeat steps 3-5 until you have a clean build
7. Now run your code, not really expecting it to work
8. Have some basic test that illustrates the most common use case for what you are trying to implement.
9. Manually execute that test.
10. If that test fails (which it almost certainly will the first several times), don't bother to go back and re-read all the code with this failure in mind.  Instead single-step through the debugger to see where things go off the tracks. 
11. As you step through using the debugger you will eventually realize that something is wrong.  Whatever code you are looking at when you make this realization, assume that is the code that needs to change.  Go in and modify that code to handle the current failure case. 
12. Don't consider the possibility of refactoring to solve the problem.
13. Always assume you must add more code to fix the problem, don't consider the option of simplifying the logic to address the issue.
14. Don't spend much time thinking about the side effects of your change, assume it is easier to catch those via testing than by thinking hard about logic and data structures.
15. Don't bother analyzing the error to see if it is just one example of a more widespread issue.  Assume any analogous errors that exist elsewhere in the code will be found via testing, don't bother trying to find them via inspection.
16. Now repeat steps 3-11 until your test case passes.
17. As soon as your test case passes, declare victory and stop work on this code.  Tell QA to start testing the functionality.
18. Don't bother to write automated unit tests or to do code coverage analysis
19. There is an urgent need for you to go back and debug problems in your previous code (that you wrote using this same approach).  Assume this need to fix what supposedly was working fully justifies your decision to not spend time on: reviewing your new code with others, refactoring based on new insights, executing multiple test scenarios, developing automated tests, searching for analogous bugs via inspection, etc. etc.

The problems with the above approach should be obvious.  However, it is shockingly frequent that this is exactly how commercial programming gets done.

Sometimes this is because the engineer is simply incompetent, however much more often it is other factors that drive this behavior.  For example:

• Management consistently sending the message "what really matters is getting new functionality 'working' as soon as possible" and never sending the message "sound programming practices will pay off in the long run and deserve our attention and time"
     
• The normal human tendency to prefer being driven by what is urgent, rather than basing our efforts on proactively defining what is important.  To break out of this habit requires conscious support from the leaders within the team.

I think it is critical to explicitly talk to the team about why "trial and error" programming is a bad idea, to set up practices that encourage a more mature approach, and to make it part of your recruiting process to screen out this type of programmer.  I will talk more about these last two points in subsequent posts.

                                                                                 copyright 2007 Kerry Champion