Nassim Taleb’s book The Black Swan has enormous implications for the human sciences, including the psychology of scoping and projecting software timelines. (If you haven’t read my previous post that introduces what a Black Swan is, I recommend you do so (click here) before continuing.) I want to use this post to relate a few additional lessons that this book brings to the table.

1. Watch for the nerd effect.

The nerd effect refers to the “mental elimination of off-model risks, or focusing on what you know. You view the world from within a model. Consider that most delays and cost overruns arise from unexpected elements that did not enter into the plan – that is, they lay outside the model at hand – such as strikes, electricity shortages, accidents, bad weather, or rumors of Martian invasions. These small Black Swans do not seem to be taken into account. They are too abstract – we don’t know how they look and cannot talk about them intelligently.”

2. Humans are bad at predicting timelines.

One of Taleb's examples: “Researchers have tested how students estimate the time needed to complete their projects. In one representative test, they broke a group into two varieties, optimistic and pessimistic. Optimistic students promised twenty-six days; the pessimistic ones forty-seven days. The average actual time to completion turned out to be fifty-six days.”

3. In it comes to projects, when you're off you're off.

If what Taleb says is correct, then every project manager should know about this one: “Let’s say a project is expected to terminate in 79 days…On the 79th day, if the project is not finished, it will be expected to take another 25 days to complete. But on the 90th day, if the project is still not completed, it should have about 58 days to go. On the 100th, it should have 89 days to go. On the 119th, it should have an extra 149 days. On day 600, if the project is not done, you will be expected to need an extra 1,590 days.” Basically we're looking at an exponential curve here.

And a few personal observations:

4. Because of the number of variables involved, software development is particularly susceptible to Black Swans.

I think the main implication here is to have a project plan that acknowledges the existence of Black Swans. It also means having the guts to say to the client, “I don't know when this project will finish definitively, we may have to bump things back because of all the wild, unpredictable stuff that happens in projects, such as x, y, z.” Taleb calls this turning Black Swans into Grey Swans.

5. There are no positive Black Swans in software development – only negative ones.

A positive Black Swan is akin to winning the sweepstakes, a negative Black Swan is akin to being swindled out of all your money. In the world of software projects, sizable delays can arise for no predictable reason; it is not very often that a project finishes a couple months ahead of schedule, in my experience.

6. Being unprepared for Black Swans gives new meaning to development “sprints.”

Failing to accommodate for Black Swans in project plans turns 40 hour work weeks into 80 hour work weeks, with scaled back functionality.

I kept a diary of all the Black Swan events (that is, highly irregular and - at the time - unpredictable events that make a big impact) that happened to me over the course of the last month. The results are interesting, especially when you add up the time.

  • A crisis situation arose with another client, and I was suddenly pulled away from my current project for two weeks to work on it

  • I experienced a two day delay trying to resolve an intractable symbols file error that (I think) was related to a problem with IIS

  • I needed an external hard drive to transfer a big file to my computer, but the task had to wait for a day while I tracked one down

  • There was one day where, instead of spending eight hours coding as I had anticipated, I unexpectedly spent six hours helping people around the office

All told, in the period of a month, I spent about half my time on unexpected issues. To see if this is representative of what normally happens I'm going to continue this diary for an entire year. The results should be interesting, especially if I can group Black Swan occurrences into different fractals.* As for managing the lower order fractals and "expected unexpecteds," Joel from joelonsoftware.com has a great blog post on that.

*A fractal is a pattern that, when scaled up or down, contains smaller or larger copies of itself respectively. The more scaled down a fractal, the more often it occurs. For instance, Taleb describes a rock as fractal; pebbles are scaled down rocks, mountains are scaled up rocks. There are many more pebbles than mountains. Make sense? Basically a fractal is a scalable pattern of something. When it comes to software project planning, Black Swan occurrences share the same basic pattern of creating more work and pushing back deadlines. They are fractal because they share the same basic properties at varying occurrence rates which are proportionate to how impactful they are. Looking back at my diary, the Black Swan of suddenly needing an external hard drive falls into a lesser fractal category (with a greater occurrence rate) than does the occurrence of a two week project delay (with a lesser occurrence rate). It is possible to predict rates of fractal occurrences (using what Taleb calls "exponents"), if you are comfortable with extremely wide margins for error.