Project Planning in the Age of AI: What Can History Teach Us?

Working notes toward a longer discussion. This is a living document.

The Estimation Problem

Every project begins with a guess. We call it an "estimate" to give it dignity, but Daniel Kahneman would call it what it is: a prediction made under uncertainty, distorted by cognitive biases we cannot feel operating.

In Thinking, Fast and Slow, Kahneman describes the planning fallacy — our systematic tendency to underestimate the time, cost, and risk of future actions while overestimating their benefits. This isn't a character flaw. It's how human cognition works. System 1 (fast, intuitive) generates an optimistic anchor. System 2 (slow, analytical) is too lazy to adjust it far enough.

The result is predictable: projects run late. Not randomly late — systematically late, in the same direction, by roughly the same factor.

The question is: can we do better? And specifically — can AI agents, with access to structured memory and historical data, break the cycle?

What Precision Demands

Simon Winchester's The Perfectionists traces the history of precision engineering — from John Wilkinson's cannon borer to the nanometer tolerances of modern semiconductors. A recurring theme: precision improves only when you measure the gap between intention and outcome, then feed that measurement back into the process.

Wilkinson didn't bore better cylinders by trying harder. He bore better cylinders by measuring the deviation of each one and adjusting the machine. The feedback loop is the precision.

Software estimation has no such loop. We estimate, we execute, we ship (or slip), and then we estimate the next thing as if the last one never happened. The data exists — in our git logs, our issue trackers, our CI pipelines — but we never close the circle.

The KT Framework Applied to Estimation

Kepner and Tregoe gave us a framework for structured thinking: Situation Appraisal, Problem Analysis, Decision Analysis, and Potential Problem Analysis. Their insight was that most failures come not from lack of intelligence but from lack of structured process — people jump to solutions before understanding the situation.

Applied to estimation, the KT framework asks:

Situation Appraisal: What do we actually know? What data do we have from previous similar tasks? What is the scope of our uncertainty?

Problem Analysis: When our estimates were wrong, why were they wrong? Was the task more complex than anticipated? Did we underestimate integration effort? Did rework from failures consume the buffer?

Decision Analysis: Given what we know about our historical accuracy, which estimation method gives us the best prediction? Raw intuition? Three-point estimates? Reference class forecasting? Monte Carlo simulation?

Potential Problem Analysis: What could make this estimate wrong? What would we do if it is? Where are the risks concentrated?

Most teams skip straight to "how long will this take?" and answer from intuition. KT says: slow down. Examine the situation. Look at the data.

The Data We Already Have

Consider what a modern development environment produces automatically:

Source	Signal
Git commit history	When work actually started and ended
Issue tracker (open → close)	Elapsed calendar time per task
CI pipeline history	Build time, failure rate, rework cycles
DORA metrics	Deployment frequency, lead time, change failure rate, recovery time
Agent memory (Neo4j)	What was known, what was decided, what was learned

We measured our own DORA metrics today. Deployment frequency: elite. Lead time: elite. MTTR: elite. Change failure rate: 68% — low performer.

That 68% is not just a quality metric. It's an estimation input. If two-thirds of deployments fail and require a fix-forward cycle, then every task estimate needs a rework multiplier. Ignoring this is like Wilkinson ignoring that his bore drifts left — the error is systematic and correctable, if you measure it.

Reference Class Forecasting

Kahneman and Tversky proposed reference class forecasting as the antidote to the planning fallacy. Instead of asking "how long will this task take?", ask: "how long did similar tasks take?"

The method:

1. Identify a reference class of comparable past tasks
2. Obtain the statistical distribution of outcomes in that class
3. Use that distribution as the baseline prediction
4. Adjust only for specific, articulable differences

This is powerful because it bypasses the inside view (our optimistic story about this particular task) and forces us to confront the outside view (what actually happened last time, and the time before that).

For software teams, the reference class is rich:

• "Refactoring" tasks: how long did previous refactors actually take vs. estimate?
• "Bug fixes" after failed deployments: what's the average rework time?
• Tasks assigned to a specific agent: what's their personal estimation bias?

Three-Point Estimates and Monte Carlo — Necessary but Not Sufficient

Our scheduling tool, tj, already supports three-point estimates (optimistic, most likely, pessimistic) and runs Monte Carlo simulations using PERT-Beta distributions. This gives us P50, P80, and P95 confidence intervals.

But three-point estimates have a hidden assumption: that the estimator is well-calibrated. If your "pessimistic" estimate is actually your "most likely" outcome (a common pattern), then even 10,000 Monte Carlo simulations will produce confidently wrong predictions.

The simulation is only as good as the distribution it samples from.

Closing the Loop: Calibrated Estimation

What we need is a feedback mechanism with three components:

A. Retrospective Comparison

After a task completes, compare estimate to actual:

tasks:
  - id: implement-sse
    effort: { optimistic: 2d, likely: 4d, pessimistic: 8d }
    actual: 6d

From many such comparisons, compute:
- Bias factor per resource: mean(actual / likely) — are they consistently 1.3x or 0.8x?
- Spread accuracy: do 80% of actuals fall within the estimated P20–P80 range?
- Task-type patterns: refactors take 2x, bugfixes take 0.5x of estimated effort

B. Calibrated Distributions

Feed the bias factors back into future estimates. If an agent historically underestimates by 1.4x, shift their PERT distribution rightward before running Monte Carlo. The math is straightforward:

calibrated_likely = raw_likely * bias_factor
calibrated_pessimistic = raw_pessimistic * spread_factor

The result: Monte Carlo predictions that reflect demonstrated accuracy, not assumed accuracy.

C. Automatic Data Collection

The richest approach: don't ask humans (or agents) to self-report actuals. Import them from the system of record.

• Git: first commit on branch → merge to main = working time
• Gitea: issue opened → issue closed = calendar time
• Woodpecker: pipeline count per issue = rework cycles

This removes the burden of manual retrospectives and ensures the data is honest.

The Agent Memory Connection

Here is where AI agents change the game. A human developer forgets their estimation patterns. They don't track that they underestimate "refactoring" tasks by 40% and overestimate "adding tests" by 20%. The bias operates invisibly.

An AI agent with structured memory can:

1. Remember the estimate it gave for each task
2. Record the actual outcome when the task completes
3. Compute its own calibration factors — per task type, per project, per complexity level
4. Apply those factors automatically to future estimates

The agent doesn't need to be told it has a bias. It can discover the bias from data, and correct for it. This is reference class forecasting, automated.

Our shared memory graph (Neo4j) already stores project context, decisions, and agent knowledge. Extending it to store estimation-vs-actual pairs is a natural step. The calibration factors become part of the agent's evolving knowledge — improving with every completed task.

The Vision: Task, History, Knowledge

The connection we're building toward:

New Task → "Have we done this before?"
              ↓
        Reference class lookup in memory
              ↓
        Historical bias factors applied
              ↓
   Agent's calibrated three-point estimate
              ↓
        Monte Carlo simulation (tj)
              ↓
      P50 / P80 / P95 with earned confidence
              ↓
            Execute
              ↓
        Record actual outcome
              ↓
      Update memory → improve next estimate

Each task feeds forward into better predictions for the next one. The agent's knowledge compounds. The estimates get more honest — not because the agent tries harder, but because the feedback loop enforces it mechanically.

This is Winchester's lesson from precision engineering: you don't improve by wanting to be more accurate. You improve by measuring the error and correcting for it, systematically, every time.

What This Means for Project Planning

Traditional project planning tools (MS Project, Jira, etc.) treat estimates as inputs and schedules as outputs. They never question whether the inputs are any good.

A calibration-aware planning tool inverts this: past outcomes are inputs, and the schedule reflects demonstrated capability rather than hopeful prediction.

For a small team with AI agents, this means:
- The planning tool gets smarter with every sprint
- Estimates are grounded in evidence, not anchoring bias
- Confidence intervals actually mean what they claim (P80 = 80%)
- The team can trust its own predictions — and stakeholders can too

Open Questions

• How many data points are needed before calibration factors are reliable? (Likely 10-15 per reference class, per standard statistical guidance.)
• Should calibration be per-agent, per-task-type, or both? (Both — with fallback to global when data is sparse.)
• How do we handle tasks that have no historical reference class? (Fall back to uncalibrated estimates with wider uncertainty bands, and flag them explicitly.)
• Can we detect when a team's capability has changed (new tools, new architecture) and reset calibration? (Track calibration drift over time — a sudden improvement in bias factor signals a capability shift.)

References

• Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux. — Planning fallacy, reference class forecasting, System 1/System 2, anchoring bias.
• Kahneman, D. & Tversky, A. (1979). "Intuitive Prediction: Biases and Corrective Procedures." TIMS Studies in Management Science, 12. — The original reference class forecasting paper.
• Winchester, S. (2018). The Perfectionists: How Precision Engineers Created the Modern World. HarperCollins. — Feedback loops as the mechanism of precision improvement.
• Kepner, C.H. & Tregoe, B.B. (1981). The New Rational Manager. Princeton Research Press. — Structured analysis: situation appraisal, problem analysis, decision analysis, potential problem analysis.
• Forsgren, N., Humble, J. & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press. — DORA metrics, capabilities of high-performing organizations, data-driven improvement.
• McConnell, S. (2006). Software Estimation: Demystifying the Black Art. Microsoft Press. — Cone of uncertainty, estimation techniques, calibration through historical data.
• Flyvbjerg, B. (2006). "From Nobel Prize to Project Management: Getting Risks Right." Project Management Journal, 37(3). — Applying reference class forecasting to large-scale projects.
• DORA Team / Google Cloud. (2024). Accelerate State of DevOps Report 2024. — Current benchmarks for elite/high/medium/low software delivery performance.