What would you give for the ability to predict the future? a lot I presume. Knowing the future (and being free to act on it) can make you rich, successful, almost indestructible. What would you give to know the future with 90% probability of being right? well, if you can participate in the lottery every week 90% pretty much guarantees success, definitely valuable. What about 50%? 10%? 0.01%? In theory at least, anything that’s better than random is worth it, especially if you can try the lottery an infinite amount of times. Unfortunately, in practice there are some fundamental problems which make premonition less valuable. We cannot retry the scenario over and over again especially if there is a chance of ruin, we don’t know the accuracy of our predictions, and predictions aren’t free - in fact, they tend to be fairly expensive. If that’s not bad enough sometimes our predictions can be worse than random, and we need another meta-prediction about the accuracy of our prediction.In preparing for battle I have always found that plans are useless, but planning is indispensable.
Despite all of these problems, predicting the future remains extremely valuable. So valuable, that evolution has produced many future predicting organisms. Our brains are a sophisticated device for predicting the future, constantly evaluating information and forecasting. Our predictions are vague, probabilistic and expensive - but they do work, at least well enough to give us an evolutionary edge. They are mostly better than random, which is why we have meta-prediction abilities which try to minimize the cost of error: fear, risk aversion and hope. It isn’t perfect, but it does work. Probabilistically.
Like many natural abilities humans have, industrialization has produced a formal analog to predicting the future: planning. Actually planning is more than future prediction, it is also the evaluation of multiple possible timelines and deciding on a course of action - the plan. But planning is so intertwined with future prediction that it makes no sense to speak of one without the other. The problem with planning, like many other industrial activities, is that people often confuse the activity with the artifact; planning with the plan. Like all things contingent on accuracy, the plan gets more and more useless as prediction errors compound. The further into the future the plan is, the more useless it is. The less accurate the predictions, the faster the plan is obsolete. It is worth noting that accuracy isn’t only about the probability of error, but also the magnitude of error, which takes into account the payoff and the size of errors. But planning, and particularly continuous planning, is still useful even if the plan is not. Planning is about expanding your options, where as the plan is about fixating them. Let me elaborate with a military example, as a tribute to ol' Ike. Suppose we have two mountain passes leading to our territory which we predict the enemy will use to attack. We cannot defend both due to a shortage in troops and supplies, and 50% chance of losing the battle isn’t acceptable. What are we to do? we must expand our options. rather than deciding up front which passage to defend, we will put a small force in both passageways and keep a large, mobile force in the back. Our plan is to slow down the enemy and send reinforcements as fast as possible. The plan is shit of course, but planning has revealed many things of value. For example we now understand our success hinges on early detection and so we will invest heavily in reconnaissance. To slow down the enemy we will look for whatever means we can find, mines, mud, demolishing the road, etc. Planning has expanded our options, but the plan, once executed, will rapidly eliminate them - once reinforcements are sent to either passage, if that attack turns out to be a diversion we are screwed. Expensive gamble that one, and so we will defer the execution of the plan as much as possible, as well as include contingencies and options in the plan in case we are wrong. Nothing will work according to plan of course, because battle is highly unpredictable; but our acts of predicting and option expansions will pay off.
Do not conclude that plans are generally useless; Battle is kind of an extreme case, our lives mostly consist of rather predictable things. Industrialized activities, of necessity, are highly predictable - and it is there the plan has the most value. But the thing about plans, and planning, is that you very rarely see good ones. A good plan assumes failures, contains reserves and contingencies, red lines and points of no return. A good plan is is not linear but branches off to multiple options. But when you ask a project manager for the plan, you get this:
In 15 years in the tech industry I have rarely seen anything that remotely looks like a plan. Don’t feel too bad about this, most army plans are like Lord of the Rings - a spectacular work of fantasy fiction containing many verses in a made up obscure language. But as ol' Ike said, it isn’t the plan we’re interested in, it’s planning, right? Well, I haven’t seen a lot of good planning either. Actually I rarely saw planning of any kind. Planning is much more than marking a target and working your way backwards. Planning is about expanding your options, remember? it’s collecting data, making predictions, branching over different realizations of those predictions and speculating about resources that could help. Planning is as much about what not to do as it is about what to do. Planning is about cramming up as many possible futures into a plan which tries to narrow these futures into one future. Planning is fan-out where the plan is fan-in. A common misconception is that planning is making decisions up front and the plan is simply the list of decisions we need to apply. It is the opposite: good planning is about deferring as many decisions as possible, as far as possible into the future, balancing the costs; if you could afford trying everything at once you would be guaranteed success. But the cost of maintaining parallel timelines is high, and so we push only some decisions into the future to preserve optionality. You will always know more in the future, but options will run out faster than knowledge will accumulate - optionality is highly valuable and in limited supply. The decision which and how many options to preserve usually needs to be done in advance.
Deferring decisions into the future requires a lot of thought and information. Planning is expensive both in time and resources - often more than the execution of the plan. This in itself would be a fairly plausible explanation as to why we see so little of it, but does not explain why so many people are convinced they are planning when in fact they are doing nothing of the sort. My conjecture is that this is due to that basic confusion of the planning with the plan - it is easy to confuse a plan derived from planning and “a plan” derived as an afterthought, if all you do is give it a superficial look.
The planning paradox
Briefly stated, the planning paradox is this:
The less likely things are to unfold “according to plan”, the more valuable the planning
The reasoning behind the paradox is quite simple; If things are predictable everything is likely to go according to plan, and naturally less planning is required. If things are unpredictable, we can pretty much assume things will spiral out of control quickly making the plan useless and thus much more planning is needed. In other words, fan-out is an opposite and complementary strategy to fan-in.
The problem of course, is the meta-plan; When should we invest in planning more than in the plan? it’s mostly a question of predictability, and often there is a hard cap on the time and resources we can spend on planning and so we must resort to short lived highly generic plans (emergencies, anyone?). It’s a question of ROI that is too often ignored, most people defaulting to either no planning at all or excessively planning everything. Both should be considered wrong, as this is highly contextual. Personally I found the Cynefin framework to be helpful as the meta-plan: if things are clear, invest in a plan rather than planning. if things are complicated, bias towards planning. When things are complex, you can pretty much ignore the plan and focus on continuous planning and learning. If things are chaotic, you have no time to plan - aggressively cap planning and use general plans until you are able to transition to another domain.
The great Agile misconception
Close your eyes and imagine what planning looks like: researching data, brainstorming with people, playing with models and prototypes, getting feedback from customers and revising until it becomes clearer what to do. Close your eyes again and imagine what a plan looks like: documentations, contracts, processes and tools. Now go read the Agile manifesto, it’s all right, I’ll wait. See how it maps to planning and plan? the way I read the Agile manifesto is
We value planning more than the plan. We value continuous planning even more.
If you are working in the complicated or complex domains, as most software practitioners do most of the time, this makes total sense. Sadly people have read this as “we shouldn’t plan”, and although I haven’t asked the fabled bunch I’m pretty sure that’s not what they meant. Viewed as such, this explains why so many (project) managers find Agile difficult to stomach - they are all about executing a plan - which is important, no doubt. This is encouraged by a focus on tangible “results”, but the planning yields options - most of which will not be realized and will have no results. The execution of the plan is easy to attribute to a person, where as the road not taken not so much. Plan execution can be valuated in short term, the value of planning in long term. An organization that values planning and agility is an organization that rewards good decision making rather than good results; because rewarding good results not only assumes that results are a direct consequence of execution rather than luck (and Nassim Taleb has quite a bit to say about that) but also that execution is more important than optionality. When you hold execution of the plan above all, linear plans will fair because they are always the easiest and cheapest to execute. Pretty soon you forget that there is nothing sacred about following the plan; Moreover, deviating from the plan is the trump card of options - you can always play it but think twice before you do because you can only play it once (and a good plan will have a failure condition telling you to use that card, scrap it and move on). Linear thinking prevails and optionality wanes, and when failure does happen it becomes an emergency.
It is no wonder that organizations managed solely by results are plagued with security breaches and systems failures, because no one is rewarded for preparing for a database split brain that never happened or a hacker that never attacked. They have entire departments dedicated to fighting ever escalating fires. It is the natural consequence of valuing the plan over the planning. An old Army saying is that many medals and honors are a sure sign of a fuckup - heroism isn’t required when everything goes well. As a parting thought, I will leave you to ponder what the abundance of medals and honors (often in the form of bonuses and promotions) in our industry might signal to an Army veteran.