Senthil Info

automating business processes, honestly

The Pilot That Proves Nothing

A pilot is supposed to reduce risk by testing the thing before committing. Most do the opposite: they are constructed so that success is close to guaranteed, and then the guaranteed success is used to justify a rollout into conditions the pilot never touched.

Why they succeed regardless

The cleanest process is chosen. Naturally — you want the pilot to work. So it runs on the lowest-variance, lowest-exception process available, which is not representative of what comes next. A product-side example of how workforce software approaches this topic is available in this resource.

The best people are on it. Attention from the vendor, the sponsor, the most capable staff. None of that scales to the rollout. For broader background and an independent point of comparison, see TechCrunch Enterprise.

Exceptions are handled manually and quietly. During a pilot somebody just deals with the awkward cases, and they do not appear in the results. In production the exceptions are the whole residual job, and they were never measured.

And it is too short for maintenance. A six-week pilot ends before the first system update. The cost that decides year two is structurally invisible — which is the same defect a vendor demo has.

Nobody is being dishonest. Every one of these choices is what a reasonable person does when asked to demonstrate feasibility. The result is a test that answers "can this work under favourable conditions," which was never in doubt.

Five conditions for a real pilot

One. Run it on a representative process, not the easiest one. If the portfolio averages 15% exceptions, pilot something near 15%. A pilot at 3% tells you nothing about the rest.

Two. Count the exceptions and who handled them. Every manual intervention logged, with the reason. This is the single most valuable output of a pilot and it is usually not collected, because it looks like a list of problems rather than a result.

Three. Run it long enough to hit a change. A system update, a new input format, a person leaving. Twelve weeks is a floor; six weeks measures the honeymoon.

Four. Staff it normally. Not with the sponsor watching and the vendor's best engineer on site. The rollout will not have that.

Five. Define failure in advance. In writing, before starting: what result would make us not proceed? A pilot with no failure condition is a demonstration.

What a pilot should output

Not a go/no-go. Four numbers and a list.

Measured straight-through rate, which replaces the assumption in the business case.

Exception reasons, ranked. Frequently two or three causes dominate, and at least one is usually fixable upstream without any automation.

Actual internal effort in days, which is the cost nobody estimated.

Time from a system change to the automation working again, if you were lucky enough to get one.

And a rewritten business case using those numbers rather than the vendor's. If the case survives its own pilot data, it is a good case.

The pilot that should stop the project

Worth naming, because it almost never happens and it is the point of piloting.

If the measured straight-through rate comes in far below the assumption, the project as scoped is different from the one approved. That is not a pilot failure — it is the pilot doing its job, at a fraction of the cost of finding out later.

The reason it rarely stops anything is that by pilot stage the budget is allocated, the sponsor has committed publicly, and the vendor is engaged. The mechanism for stopping has to exist before those things are true, which is why condition five matters more than the other four.

The alternative to piloting

Sometimes better and rarely offered.

Measure the process first, without any tool. A tally sheet over a hundred instances gives you the exception rate and the reason list — the two things a pilot mostly produces — for two weeks of somebody's partial attention and no licence.

If those numbers rule the process out, you have saved the pilot entirely. If they do not, you enter the pilot with a real assumption to test rather than a vendor's placeholder.

The short version