A Simple Matter of Software

Agile Manifesto - Annotated

I have been thinking recently about the Agile Manifesto. 

Here it is with some annotations. 

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Right on!  This gets to the heart of the "adaptive" vs. "predictive" debate.  The post here makes a similar point phrased in terms of "evolutionists" vs. "creationists" using analogies from Richard Dawkins.

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Changing requirements is a reality and you need to embrace that and learn to succeed within that reality. 

However, I have two concerns with the way this particular item is presented. 

First "welcoming change late in an iteration" can directly conflict with your ability to achieve "early and continuous delivery of valuable software".  The manifesto leaves it as an exercise for the reader to figure out how to balance those conflicting desires.  It can be done, but there is no guidance here how to do it.

Second recognizing the reality of changing requirements doesn't give you a free pass to put off detailed requirements definition.  You still want to push hard to define the requirements as early as possible and then only change them based on "new information". Thrashing a decision because of internal indecisiveness is a bad idea, adapting as you get new insights into what works best for the user is a good idea.

The good thing about short iterations is that you can "freeze" and "reopen" requirements quickly (with each iteration).  Still at some point you have to at least temporarily freeze the requirements to get the iteration out the door.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Right on! This is a direct corollary of point 1 above.  Of course some initiatives will take longer than a "couple of months", but that does not mean they can't deliver working software along the way.

Business people and developers must work together daily throughout the project.

Right on! 

Engineers working in a vacuum will produce some beautiful code, which may be irrelevant to the problem that needs to be solved.  In the case where the engineers themselves fit the target audience  (programming tools, search engines) then engineers can produce great things without the business folks.  But those cases are the exception not the rule.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

100% agree.

However, when you have multiple motivated individuals they often don't agree on how to proceed forward.  The manifesto leaves as an exercise to the reader: how do you get alignment and get past such disagreements.   

The most efficient and effective method of  conveying information to and within a development team is face-to-face conversation.

It is true that face-to-face is the most powerful way to communicate an idea and it is true that face-to-face is often underused.

However, there are plenty of times when face-to-face is not the best option for "conveying information" especially information that needs to survive for a period of time. 

Tribal knowledge is a powerful thing, but if all information only exists as tribal knowledge in peoples' heads than you will have some real issues in achieving "sustainable development" (see two items below).

Working software is the primary measure of progress.

Right on!  Such a simple but powerful concept.

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

100% agree.

Continuous attention to technical excellence and good design enhances agility.

100% agree.  An interesting question here is how you go beyond "attention" to "measurement".  It is much easier to keep your attention focused when you have some measure of how well you are doing.

Simplicity--the art of maximizing the amount of work not done--is essential.

Right on! 

It is easy to state the value of simplicity, it is harder to apply it in practice.  Though Dilbert had a good example of simplicity in action that has some lessons for the software world.

The best architectures, requirements, and designs emerge from self-organizing teams.

This is true. 

However, it begs the question: what do you do when the team fails to "self-organize"?  Give up and go home?  Hopefully not.  How to catalyze "self-organization" is left as an exercise for the reader.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

100% agree.  Any complex task that is worth doing demands a certain amount of self-reflection.

By the way, one implication of this last point is that you should make your behavior suit your particular needs and experiences even if that means not exactly following agile development method as described in the various books, training classes, etc.

                                                                               copyright 2007 Kerry Champion

Are you agile?

I ran across an amusing checklist to measure "Agility" in the agile programming sense.

Don't agree with every single point implied by this list, though most rang true.  Here are some favorites.

======================================================================

You are not yet Agile if:

“Test-driven” still refers to your car.

Developers only develop, testers only test, and managers just manage.

Simplicity is presumed to be simple.

======================================================================

A more serious checklist to measure agility is available from the same source here.

Surveillance and diligence

As I mentioned in a previous post I recently heard Atul Gawande talk about his book "BETTER: A Surgeon’s Notes on Performance".  He made observations about why some teams have much better results (in terms of patient medical outcomes) than others.

He highlighted two traits that seemed to be central to that success: "surveillance" and "diligence".  These traits are incredibly important in the software world as well, for much the same reasons as touched on by Atul.

I am not sure that "surveillance" is the best possible phrase to use, I prefer to think in terms of "active observation."  Whatever the label the concept is critical.  Here are some points to keep in mind when practicing active observation:

• Don't depend solely on your own observations, enlist the broader team to take part in the process and listen carefully when they share what they are seeing.
   
• Don't accept everything you hear at face value, question and probe, search for validating or contradictory data points.
   
• Make sure you have a good consistent data set that lets you make comparisons over time and comparisons between different people and different projects.  When there is variation in that data set probe into it, even if the variation seems small.
   
• Take advantage of your insights into well known patterns of misleading reports.  Some folks are modest and pessimistic and consistently make things sound worse than they are.  Others may tend towards self-justification.  In any case you need to look for these patterns and take them into account as you get new reports from the same source.
   
• As you form conclusions based on these observations, test that conclusion, make predictions based on that conclusion and go look at the relevant data.  If your new data points don't support your prediction, than your conclusions are probably wrong.

Atul makes some key points regarding "diligence":

• Diligence can be defined as "the constant and earnest effort to accomplish what is undertaken."
   
• "People underestimate the importance of diligence as a virtue" perhaps because on the surface it sounds mundane and simplistic.
     
• However, diligence is a "prerequisite of great accomplishment" and "one of the most difficult challenges facing any group of people who take on tasks of risk and consequence"

I agree wholeheartedly with these points and would add just a couple more regarding the practice of diligence. 

• It is unreasonable to expect diligence from someone who does not understand why their tasks matter in the bigger scheme of things.
     
• Having relevant and easy to understand metrics that are shared with the entire team, encourages diligence and continuous improvement.
     
• Developing team practices that enable folks to get into a good rhythm and spend most of their time "in the zone" is critical to encouraging diligence.  There are lots of things (too many meetings, unproductive meetings, inconsistent goals/feedback, constantly switching assignments, etc.) that will pull folks out of the productive zone and discourage them from trying.
   
• An out-of-context or irrelevant response to anxiety or social tension is called a displacement behavior.   I believe this term was originally from animal behavior studies, but it certainly applies to humans as well.  For example you may feel anxious about a deadline, doing the relevant work just reminds you of how hard it will be to make the deadline, which makes you even more anxious, so you find yourself reading Slashdot or reorganizing the source tree instead of actually working on the relevant implementation.  Team leaders need to give folks an open door and when they express anxiety or point out tension, leaders need to help them work-around those issues so they can pursue the tasks that matter instead of getting sucked into an eddy.   

Here is Atul's description of a Doctor who has mastered the arts of surveillance and diligence.  This describes an examination of a Cystic Fibrosis patient (Janelle) by her Doctor (Warwick).  Even though the specifics of this situation are far from the specifics of a software project, the mindset and basic skills shown here are extremely relevant.

Warwick pulled out her latest lung-function measurements. There’d been a slight dip, ... Three months earlier, Janelle had been at a hundred and nine per cent (she was actually doing better than normal); now she was at around ninety per cent. Ninety per cent was still pretty good, and some ups and downs in the numbers are to be expected. But this was not the way Warwick saw the results.

He knitted his eyebrows. “Why did they go down?” he asked.

Janelle shrugged.

Any cough lately? No. Colds? No. Fevers? No. Was she sure she’d been taking her treatments regularly? Yes, of course. Every day? Yes. Did she ever miss treatments? Sure. Everyone does once in a while. How often is once in a while?

Then, slowly, Warwick got a different story out of her: in the past few months, it turned out, she’d barely been taking her treatments at all.

He pressed on. “Why aren’t you taking your treatments?” He appeared neither surprised nor angry. He seemed genuinely curious, as if he’d never run across this interesting situation before.

“I don’t know.”

He kept pushing. “What keeps you from doing your treatments?”

“I don’t know.”

“Up here”—he pointed at his own head—“what’s going on?”

“I don’t know,” she said.

He paused for a moment. And then he began speaking to me, taking a new tack. “The thing about patients with CF is that they’re good scientists,” he said. “They always experiment. We have to help them interpret what they experience as they experiment. So they stop doing their treatments. And what happens? They don’t get sick. Therefore, they conclude, Dr. Warwick is nuts.”

“Let’s look at the numbers,” he said to me, ignoring Janelle. He went to a little blackboard he had on the wall. It appeared to be well used. “A person’s daily risk of getting a bad lung illness with CF is 0.5 per cent.” He wrote the number down. Janelle rolled her eyes. She began tapping her foot. “The daily risk of getting a bad lung illness with CF plus treatment is 0.05 per cent,” he went on, and he wrote that number down. “So when you experiment you’re looking at the difference between a 99.95-per-cent chance of staying well and a 99.5-per-cent chance of staying well. Seems hardly any difference, right? On any given day, you have basically a one-hundred-per-cent chance of being well. But”—he paused and took a step toward me—“it is a big difference.” He chalked out the calculations. “Sum it up over a year, and it is the difference between an eighty-three-per-cent chance of making it through 2004 without getting sick and only a sixteen-per-cent chance.”

He turned to Janelle. “How do you stay well all your life? How do you become a geriatric patient?” he asked her. Her foot finally stopped tapping. “I can’t promise you anything. I can only tell you the odds.”

In this short speech was the core of Warwick’s world view. He believed that excellence came from seeing, on a daily basis, the difference between being 99.5-per-cent successful and being 99.95-per-cent successful. Many activities are like that, of course: catching fly balls, manufacturing microchips, delivering overnight packages. Medicine’s only distinction is that lives are lost in those slim margins.

And so he went to work on finding that margin for Janelle. Eventually, he figured out that she had a new boyfriend. She had a new job, too, and was working nights. The boyfriend had his own apartment, and she was either there or at a friend’s house most of the time, so she rarely made it home to take her treatments. At school, new rules required her to go to the school nurse for each dose of medicine during the day. So she skipped going. “It’s such a pain,” she said. He learned that there were some medicines she took and some she didn’t. One she took because it was the only thing that she felt actually made a difference. She took her vitamins, too. (“Why your vitamins?” “Because they’re cool.”) The rest she ignored.

Warwick proposed a deal. Janelle would go home for a breathing treatment every day after school, and get her best friend to hold her to it. She’d also keep key medications in her bag or her pocket at school and take them on her own. (“The nurse won’t let me.” “Don’t tell her,” he said, and deftly turned taking care of herself into an act of rebellion.) So far, Janelle was O.K. with this. But there was one other thing, he said: she’d have to come to the hospital for a few days of therapy to recover the lost ground. She stared at him.

“Today?”

“Yes, today.”

“How about tomorrow?”

“We’ve failed, Janelle,” he said. “It’s important to acknowledge when we’ve failed.”

With that, she began to cry.

Warwick’s combination of focus, aggressiveness, and inventiveness is what makes him extraordinary. He thinks hard about his patients, he pushes them, and he does not hesitate to improvise.

I particularly appreciate the comment "... combination of focus, aggressiveness, and inventiveness is what makes him extraordinary"  That certainly applies to the software world as well.

                                                                               copyright 2007 Kerry Champion

Unexamined life is not worth living

"An unexamined life is not worth living."
                  --  Socrates

This is obviously not literally so, yet there is some fundamental truth here.

There is no question that every human activity benefits from thoughtful observation and reflection.  More specifically, for activities where we are consciously trying to achieve peak performance and continuous improvement, such observation and reflection is an absolute requirement. 

Software ventures clearly fall in this latter category.  A mature and evolved organization will proactively work against an agenda that is driven by thoughtful self-observation.  If you find that is not the case, if you find that you are constantly focused on reacting and firefighting, if you are always responding to the urgent and never working on the important, than I recommend you  remember old Socrates: step back and examine your work life.

                                                                               copyright 2007 Kerry Champion

What does a teaching hospital have to teach a software team?

Stanford Hospital is one of the better known institutions in Silicon Valley.  It is a  teaching hospital where some of the best learn their profession.  Every day the doctors and staff take the time to share what they know and improve their skills.  They do this through a variety of mechanisms: doing rounds together, meeting to review patient outcomes, reviewing charts and asking questions, writing papers, presenting at conferences, etc.  They do all this while literally dealing with life and death issues, and while working inhumanly long hours.

Often in software teams you will hear comments such as these:

• I don't have time to train the new guy, just have him read the code.
• It is too much hassle to include a comment explaining my checkin.
• There is not enough time to talk the whole team through the architecture of the new module and explain why we made the decisions we made.
• We just can't seem to fit in the code reviews.
• Oh I never bother filling in the "root cause" field when closing a bug in the bug base, no one is interested.
• What is the point of doing a debriefing now that the release is over, no one will pay attention to the results.
• etc. etc.

Sometimes my response is:  "Staff at teaching hospitals work long hours under intense stress.  Just down the road at Stanford they still find time to fill in charts, go on rounds, go to postmortems, and all the rest.  If they can both 'do and teach' why can't we?"

One of the things that makes teaching hospitals so effective is they don't just teach from books (medical school takes care of that) they teach through considered examination of real life situations.  Software teams have the opportunity to do the same thing.

Teaching a rapidly evolving body of knowledge, requires observation, reflection and articulation.  Software teams are often very bad at all three, rushing from the latest new feature to fixing the customer critical bug, without taking the time to gather observations, reflect on that data drawing useful conclusions, and then capture those insights into a form that can be transmitted again and again.

To make your team the equivalent of a teaching hospital, to make it a "teaching team", requires several elements:

• Management must buy in to the idea that investing in these practices now will pay off with greater total throughput over time. Then team leaders must go on to implement the specific practices: code reviews, bug analysis, project debriefings, etc.
   
• Team members must organize their own work and adapt their attitudes to avoid creating a culture where the "urgent" gets all attention and the "important" gets ignored.
   
• Everyone must constantly ask themselves "ok we have solved this immediate problem, what lesson should we learn from it and how should we spread that lesson around?"

Failure to make a "teaching team" is one of the key leading indicators that you are bound to repeat the same mistakes over and over.

                                                                               copyright 2007 Kerry Champion

 

What you want to see and what you need to see.

Scott Maxwell does an excellent job explaining the 10 best ways to lie with metrics.

Seth Godin recently posted on the disconnect between common metrics and what really matters.

Our previous post on how market definitions can be used to reaffirm existing biases is getting to a similar point.

Behind these cautionary tales the real issue is selectively using data to reinforce what you want to see.  Instead of using data to discover what you need to see.

Sometimes what you want to see is a specific conclusion, such as "yes it makes sense to inject another $10m into our current product focus, no changes needed there".

Sometimes what you want to see is a high degree of action and precision regardless of the conclusion:  "Look at all the metrics, look at the precise numbers, this is one organization that is really on top of things ... well no, we are not sure yet what action plan makes sense given these numbers ...".

In both cases you are missing the real goal: having a open minded data-driven process that sometimes leads you to surprising conclusions, but always leads you to decisive action.

                                                                                 copyright 2007 Kerry Champion

A software venture with no servers?

Is it possible to build a software venture with no servers to be managed? And more importantly no server software to be managed.

For awhile now I have had a personal goal to prove that the answer is yes.  Wouldn't it be great to have every resource that you need, run somewhere else across the net by experts who do nothing else.  With you spending zero cycles on: how to install it, how to upgrade it, how to back it up, etc. etc.

The benefits of this for a stand-alone startup are obvious.  But I think this approach can be valuable even within a big company, if you are trying to create a fast-moving independent project within that bigger organization.

I recently ran across a couple presentations that highlight some relevant resources.  They left out important resources (such as QuickBooks online) but are still useful references.

Andy Forbes has a good list of such resources. 

 

Also Greg Papadopoulos was a quick list buried in one of his presentations (it is about slide 12)

Greg makes a strong case that these sorts of online services will be more and more important in the near future.

One advantage of this approach is that it wipes a whole class of "urgent" issues off your plate.  Mostly IT and administrative issues.  A fundamental struggle in software ventures is the struggle of "urgent vs. important".  Anything that helps you focus more on the truly important issues is a good thing.

In a previous post we argued that it is important to be selective about what risks you take, and match the risks to the goals.  Push-hard and take risks where the pay-off from that risk directly addresses your audacious goals and defining questions.  However in all other aspects of what you are doing, avoid risks and do what is proven to work.  That logic applies here.  Use those online services that are proven and stable and reduce risk and distraction.  Don't dive in to use some shaky beta on-line service, if that service is just part of your machinery and is not changing what you can offer to your customers.

                                                                                 copyright 2007 Kerry Champion

Never Your Code Alone

Giles Bowkett added a fresh perspective on the Software as Legos Meme yesterday.  One of his points is that you should not assume too quickly that you have to create a new building block from scratch, but you should first look harder at what you already have.

This thinking about building blocks reminds me of an important observation that is often overlooked.

When studying how software engineers actually spend their time it is a truism that they spend only a  minority of their time designing specific algorithms, defining specific data structures and writing fresh code. 

So where does the rest of the time go?  Well there are several ways to slice it. 

It is often pointed out that more time is spent on maintenance and evolution than on greenfield creation.  That is certainly true.

It is also pointed out that more design time is spent on refining requirements than on actual internals design.  That is also true.

However, in this posting I am interested in another point.  Less often noted is the fact that they typical developer needs to spend more time dealing with the interfaces he interacts with than he does dealing with the core code he authors.  Recognizing that this is true, encourages developers to make targeted investments up front that will save them time and improve quality in the end.

By "interfaces" I mean all interfaces: system interfaces (OS functionality, UI toolkits), major application components (databases, user directories, security libraries), shared data structures, and the internal interfaces to other components under development.  Most engineers tend to underweight the importance of having a truly complete understanding of the interfaces they are using, blithely assuming the actual behavior will be obvious from the declared interface and they will just figure it out as they go along.  This leads to: expensive to fix bugs discovered late in the process, a tendency to rebuild functionality that already exists in another module, and a tendency to constrain user functionality because you are not getting full leverage from the interfaces that are available.

When assigned a new module a responsible developer will go through and read all the code in that module and make sure he understands the code he will be modifying.  He will then go on and write (or extend) some unit tests that demonstrate he really understands that code.  However, he typically does not invest comparable effort to understand the interfaces his code is using.  It is worthwhile to take some steps up front: carefully reading the interfaces, understanding the implementation behind those interfaces, and where needed writing test code that shows the actual behavior of ambiguous interfaces. 

Investing  the effort to understand not just your code but the interfaces you have available will save time and effort in the long run.  Often this kind of investment is pushed aside by more "urgent" tasks, but this is one of those times you must find a way to get "important" tasks done, even if they don't feel "urgent" at the moment.

                                                                                 copyright 2007 Kerry Champion