An Assembly of Pieces

James Anderson Merritt's piecemeal thoughts and observations, and the occasional attempt to put some of the pieces together.
Write to James!

All material except cited quotations Copyright (C) 2004-2008 by James Anderson Merritt. All rights reserved.
Wednesday, March 17, 2004

There in the left-hand bar, you'll notice that my name is an active link, which takes you to my very modest personal home-page. At the bottom of said home page is a hit-counter, which never exceeded the low 100s over the course of the two or three years it has been on the home page ... until recently.

A few weeks ago, I noticed that my home page "hits" were suddenly numbering in the low thousands. The surge in hits came -- coincidentally, I now believe -- after a couple of fellow bloggers told me that they had linked to this blog because they liked some of the commentary here. At first, I thought that those other blogs were simply encouraging large crowds of people to visit mine, and that some of those people were electing to follow the link to my home page. As the days went on, however, I saw that the counter was continuing to rack up thousands of hits, every day. As I write this, in fact, there have been over 40,000 hits since I first noticed the uptick in the counter. It strained credulity to think that the other blogs were driving that much traffic to mine, and that all of the visitors were clicking through to my home page. Yet, I had no better theory to explain the increase in hits.

At some point -- around hit 15,000 or so, as I recall -- I got the bright idea of testing the "click-through" theory by putting a "tip jar" on my home page. It was fairly straightforward, to add the necessary Paypal button code to my home page script. I figured that, if people were getting to my home page by clicking through from my blog, some of them might value my opinions enough to encourage me to continue writing, by leaving a dollar or so. Maybe one or two might leave a dollar just for the hell of it, to reward my chutzpah, or even simply to try out the mechanism. Yet, many days and over 25,000 hits later, I have not received a single penny via the tip jar.

The empty tip jar proves little, of course. I have no way of knowing whether people are actually clicking through from the blog; or, if they are, whether they like these columns; or if they do, whether they value them enough to leave a tip. Perhaps the hit counter mechanism is simply broken, spinning freely, and my home page is getting the same handful of visitors per day that it always got before. One thing is certain. My experience thoroughly disproves the old internet canard that goes something like this: "If we got x number of hits per day and if only y% of them left a dollar/bought our product, we'd be rolling in cash in no time!" Or if it doesn't disprove it, then we should conclude that the appropriate value of y is zero, or at least an infinitesimally small number.

My favorite theory is that my webpage is being visited by web robots (search engines, FBI terrorism sniffers, SPAM trawlers, and the like). This would explain the consistently large number of daily hits (as well as the empty tip jar -- as far as I know, robots don't have credit cards). But there is still the open question of why? Why did the number of hits suddenly take off and why does it continue to increase at a sustained high rate after many weeks? If web robots are the culprits, what could they possibly be seeking at my home page, which keeps bringing them back after they failed to find it the first thousand times? Or, if they found something, why come back, over and over again, to see exactly the same thing, since I change the home-page far less frequently than I update the blog? The mysteries of the machine mind are subtle, indeed.

Sunday, March 14, 2004

I want to talk now about something a little less earth-shaking: the nature of software development and the development process. (Those with no interest in geeky things can skip this entry, but there's nothing here that is all that technical; you might even find it interesting!) Let me start my telling you a little story:

The Experience of Past Lives

Before I went into documentation, I was a developer in a corporate setting; before that, I was an independent developer.

The One Man Band

As an independent, I wore all the hats: designer, coder, tester, maintenance programmer, documentor, tech support rep, ad copywriter, salesman, marketer, shipping clerk, and bookkeeper, just to name a few. In that circumstance, the documentation for my product included the comments in my code, the notes in my engineer's journal, the user manual and release notes that I wrote, and items of correspondence between me and current or prospective customers. My own memory and understanding of the product, its purpose, and the details of its implementation, were the glue that held everything together, filling in the documentation gaps. That worked for a while, until I had to start supporting multiple versions over the long term. Then, I at least had to improve my internal documentation of the several versions I was supporting, so I could keep everything straight for my customers. My approach worked well enough that my little product gained respect and reputation, eventually inspiring job offers to work as a developer within various corporations.

With A Little Help From My Friends

Developing software as a corporate employee, I had to give up most of the hats, but in return, I was better able to see clearly the discrete stages of product development, which I had long been used to treating holistically, as a unit. I also began to appreciate how crucial internal documentation was to cooperation between contributors within a team, and handoff between teams, as a product moved from stage to stage on its way out the door. Few of the collaborators in the corporate development process could carry inside their heads the deep and broad understanding of their complex product and its evolution that I, as an independent developer, had maintained on behalf of my more modest products. With various forms of solid internal documentation, and no small amount of handholding during the handoff periods between teams, we tried to maintain that level of product knowledge as a group. Needless to say, we weren't always successful.

A Russian Doll Factory

At the time I "came of age" as as developer, fractals were all the rage in mathematics and computer science. Perhaps my pervasive exposure to a core idea of fractals -- that you could create complex patterns from smaller parts that resembled each other and the whole in fundamental ways (i.e., micro and macro were essentially the same despite apparent superficial differences) led me to observe the "fractal nature" of product development.

As I looked at successful products, I noted that each stage of development (or each subcomponent project) yielded a complete package of materials, which was a mini-product in its own right: working software; descriptions of design, usage, and construction details; an assessment of quality and a record of the tests or evidence in support of that assessment; and so forth. There was always a period of "handoff" or "rollout," too, during which the producers at one stage would educate the consumers at the next stage, enabling the latter to incorporate the product as raw material in their own subsequent products. A typical development effort might be several stages deep and tens (if not hundreds!) of subprocesses wide.

To the extent that producers at any stage provided a complete mini-product and did their handoff right, they enjoyed freedom and opportunity. They could move on to the next version of their current product, or tackle something else altogether, without worrying about getting the midnight call at home, or having to postpone or truncate a vacation, because a production line had to be stopped, or a customer had lost money, due to a software defect. To the extent that incomplete products went on to the next stage, or handoff was sloppy, the producers at the previous stage always had to be ready to stop everything, in order to help people at subsequent stages get around earlier-stage deficiencies, which tended to snowball into major problems downstream. Repairing, working around, or "documenting around" defects later in the process always entailed pain and expense. Any exceptions seemed the result of dumb luck -- most usually, the luck of people who were able to move on before the repercussions of earlier-stage failures became apparent.

Lessons Learned

I can summarize the key conclusions that I drew about software development
from those experiences, as follows:

1. In a successful development effort, every developmental stage or component development sub-project yields a mini-product, which must be complete, in and of itself, including working software; descriptions of design, usage, and construction details; an assessment of quality and a record of the tests or evidence in support of that assessment; and so forth. The design of the software and the nature and level of polish of its accompanying documentation will be appropriate for its role as raw material in the next stage of development.

2. Every product roll-out to the end-user is the culmination of several chains of smaller-scale mini-product roll-outs, each adding value and polish to the product and building momentum toward the final "birth."

3. Every component lead developer/producer is a mini-product manager, responsible for producing a complete package before rollout, educating the customer during handoff/rollout, and supporting the product after the rollout.

4. It is not true to say that a component is "done" before the complete package is delivered; incorporating such mis-information in project schedules or status reports is the source of much pain, misunderstanding, delay, and financial loss.

5. Conversely, to the extent that you do not actively apportion tasks and divide labor to permit a large job to be completed as a collection of mini products (not just separate tasks or components, but complete mini-products), you are setting yourself up for failure; the power of "fractal vision" -- the ability to recognize the mini-products in the larger one, and to provide for their separate production and completion -- is key to success.

Over the years, I have looked for noteworthy exceptions to those conclusions, and found none. Have you? Has anyone had similar experiences to mine in software development, or markedly different experiences that have given them a different view of the world? I'd like to read your take on the essential features of software development and the development process, as you understand them. I think that by sharing experiences and lessons learned, we can develop and refine our own process so as to better match reality and provide for satisfactory results.

In any case, dear visitor, thanks for reading my story; I hope to see yours someday. In any case, I appreciate your time and attention.


  This page is powered by Blogger, the easy way to update your web site.  

Home  |  Archives