Maurice’s selected thoughts.

It’s Friday and it seems like my last few posts have been quite serious (complete with references even) so today I’m just going to mention a few Web sites that are useful for doing research on doing research, and mention an interesting research workflow.

The first site I want to call attention to is MacResearch. If you are doing research and you use a Mac you need to check out this site. It includes reviews of research oriented products, as well as a forum, script repository, tutorials, and news. It was this site where I found DevonThink and DevonAgent, two very useful research tools. I also discovered BibDesk there, although I was already using JabRef so that hasn’t been as big an impact as DevonThink. There are several articles on software for electronic research notebooks that I highly recommend if you are trying to get your research organized.

The other site you should check out if you are doing any kind of online literature review is ResearchBuzz. ResearchBuzz is pretty much what it says it is “News about search engines, databases, and other information collections.” The author of the site is Tara Calishain who also authored several books on searching including Google Hacks and more recently Information Trapping: Real-Time Research on the Web, which is on my current reading list. I think this site stands fairly well on it’s own, but reading any of Tara’s books adds value.

Finally, if you start reading Information Trapping and take a look at the combination of DevonThink and DevonAgent you should get a feel for a new workflow for doing online research. This probably isn’t new too everybody, but it was new to me last week. I’m fairly certain that this combination of technique and technology will change the way I do literature reviews. Though I’ll still be doing a lot of reading at least the management of the process can be streamlined. Basically, the idea is to allow DevonAgent to search for information, and import it into DevonThink where it is quite easy to sort through it all - separating the wheat from the chaff. DevonAgent can be setup to remember what it has already seen, and run at a scheduled time. So you effectively end up with a research inbox, where all the latest search hits are stored and await processing. Oh, and since DevonThink includes a little AI it turns out that the more information you add the better the program gets at automatically classifying it. You must check it out!

Much has been written about process improvement in the software industry. The SEI has developed the CMM (now known as CMMI) for about twenty years, yet according to a survey of 2000 government and commercial organizations about 50% are still CMM Level 2 or lower [1]. This is not surprising considering estimates of project failures typically run very high (some estimate over half). The higher levels of CMM are designed to insure repeatability of successes as well as identification of and improvement on failures, so we might assume that organizations operating at a higher CMM level have fewer failures. This leads to an interesting question. Why would a company not improve their process by means of application of CMM KPA’s or other forms of process improvement?

To answer that question we need to look at the evolution of a software business. The industry is filled with stories of two guys starting out in a garage hacking together something they think is cool. The size of the product starts small, the enthusiasm of the people high, and the number of people on the development team low. This sounds like a formula for success so for the sake of argument let’s say it is, the product is a hit, and there are talks about an IPO. It isn’t hard to see that eventually the size of the product that is being developed is much larger, customers demand it because they need it to do more. So we scale up our product, our next release has roughly twice the requirements of the original. We know our team can handle it because this is the team that developed the original version from nothing. Isn’t that right?

No. The complexity of a software system is not well understood early in the development process. Even if all requirements are known (not realistic but perhaps a best case lower bound for unknown requirements) there does not currently exist any cheap and easy way to estimate the development time of the system accurately. There are well known models but they have a well known failing, they work well for systems that are similar to those that have already been developed. Not exactly a revolutionary concept, if you know how long it took to develop a similar system you can make a good estimate. This doesn’t really help our team who is working on extending a completely new product unlike anyone else’s.

Ok, so we have a successful product and team who is now working on the next version of the product that has twice as many requirements as the first. We know the product is more complex but we don’t know by how much. Now we return to our original question, why don’t we engage in a process improvement process? I believe the winning answer is human nature. Our experience includes the previous success, but we learn more from failure. Why? We are not in the habit of asking what we could have done better unless we see failure. So every software development project should include a lessons learned.

What does this have to do with process improvement? It’s simple, most of the big software improvement processes are, well they are big. There are metrics and paperwork and a lot of other artifacts generated that require many resources. This doesn’t help the small software development business who can’t justify the expense or doesn’t want to burden the talented engineering staff with what they see as a bullshit management process. So the question becomes how do you move toward a process improvement culture (one that asks what they can do better even when they succeed) with out alienating everyone on your existing successful teams.

I believe the answer is to make lessons learned along with qualitative statements about the state of the process a part of the project. All of the fancy and big process improvement models include quantifiable metrics that are used to evaluate the existing process and compare to the new process when it is implemented. This is necessary because we have to know where we were and where we are in order to measure any change between the two. However, we don’t need to necessarily make the measure quantitatively. We can exchange the precision of a metric for the efficiency of our talented engineers “gut” feeling, which may more accurately reflect reality. We lose the precision of the quantitative models but we gain reduced overhead and acceptance from our most important resource (the engineering team), while introducing the much needed process improvement concepts into the company culture. With a company culture that is primed for process improvement we will have an easier time introducing full blown process improvement when we do have the resources available to implement it.

References

[1] R. N. Charette. Why software fails. IEEE Spectrum, September 2005.