At a glance
- 1.Measure your baseline before the pilot starts: how long does one procedure take today, from first draft to approval?
- 2.Measure the whole chain in the pilot — recording, correction, review, approval — not only the time the software needs.
- 3.Write down, before the first upload, which errors would rule a tool out. That keeps the final decision from drifting toward whatever went wrong last.
- 4.Hold a joint review after the first three to five SOPs. Problems found in week one can still be fixed; problems found in the final meeting can only be judged.
- 5.A pilot that ends with a clear “no” has done its job. A pilot that ends with “we need more time” usually was not set up to produce an answer.
A pilot should end with a decision you can defend — for or against a tool. What we have learned about the weeks between kickoff and final meeting, and a scorecard you can use with any vendor, including us.
We sell SOP software, so take this with that in mind. But we have also run pilots that ended without the answer either side needed, and every time the cause was in the design of the pilot rather than in the software or the team. This article is the checklist we now use ourselves, written so that a buyer can use it with any vendor.
Why we wrote this
It is based on our own pilots, not on a market study. Where we give numbers, they are what worked for us.
A pilot can also end with a clear yes: in our 14-plant rollout, the decision followed a single pilot plant.
Before the kickoff: three numbers and one list
1. Your baseline
How long does one procedure take today? Count everything: walking the process, writing, photos, formatting, review rounds, approval. The industry-typical range for a manually written work instruction is four to eight hours, but your number is the one that matters. Measure two or three real ones.
Without a baseline, “it still took an hour” can mean a 90 % saving or none at all.
2. Your success criterion
“Does it save us time?” is not a criterion. Something like this is:
For procedures of up to ten minutes, a trained recorder gets from video to an approval-ready SOP in less than a quarter of today's time, including correction and review.
Write it down in the kickoff. Both sides can check against it mid-pilot.
3. Your volume
How many procedures need documenting in the next twelve months, and how many people will record them? A tool that saves 80 % on twenty procedures is a different decision from one that saves 50 % on four hundred.
4. Your knock-out list
Which errors would rule a tool out, regardless of time saved? For example:
• A value that was stated in the recording is missing from the draft, and the tool does not flag it.
• A safety-relevant statement is reversed.
• The tool cannot produce the second language your site requires.
• Confidential screen content cannot be masked.
Agree on this list before you see any drafts.
During the pilot: measure the whole chain
The software part — upload to draft — is usually minutes. The interesting time is around it. For every SOP in the pilot, record four numbers:
| Stage | What to measure | Typical trap |
|---|---|---|
| Recording | Minutes of filming, including retakes | Retakes not counted |
| Correction | Minutes editing the draft | Edits made outside the tool, in the exported PDF |
| Review | Minutes a second person spends checking | Reviewer re-watches the whole video |
| Approval | Days until release | Waiting time mistaken for work time |
Two patterns tell you a lot. If correction time is high and edits happen outside the tool, the team is working around it — find out why. If review means re-watching every video, reviewers do not trust the drafts — find out which error broke the trust. Both are fixable early and expensive late.
The joint review after the first three to five SOPs
This is the most useful hour of the pilot. Sit down with the people who recorded the first drafts and go through each one next to its video. You will see two kinds of problems:
• Input problems: long silent stretches, conversations on camera, the camera several meters behind the operator, a second-language narration.
• Tool problems: values dropped, statements reversed, wrong language, unreadable pictures. These belong to the vendor.
Separating the two in week one is fair to both sides. A team that only discovers in the final meeting that its recording habits cost half the detail cannot fix them anymore, and a vendor that only hears about a dropped value at the end cannot fix that either.
When the team changes
In larger organizations, the project team changes during a pilot more often than not: holidays, a new role, a restructuring. When it happens, the person who inherits the pilot did not choose the tool and was usually not in the training. They will — reasonably — judge it on what they see.
Agree in advance on what happens then: a short live training for the new owner, and a fresh look at the success criterion. A link to an online course is not a handover.
The scorecard
Fill this in at the end of the pilot. It works for any SOP or work-instruction tool.
| Question | Answer |
|---|---|
| Baseline time per procedure (today) | |
| Average total time per SOP in the pilot (record + correct + review) | |
| Saving against baseline | |
| Success criterion met? | Yes / No |
| Knock-out errors found | List |
| Share of SOPs approved without re-watching the full video | |
| Second language usable without manual rework? | Yes / No |
| Main input problems found in the joint review | List |
| Main tool problems found in the joint review | List |
| Were tool problems fixed during the pilot? | Yes / No / Not reported |
| Decision | Go / No go / Extend with a defined question |
When to decide against a tool
A good evaluation should be able to end in “no”. Reasons we would accept, as a buyer:
• A knock-out error appeared and the vendor could not fix it within the pilot.
• After input problems were addressed, the total time per SOP was still not meaningfully below your baseline.
• The recording effort the tool needs does not fit how your teams work — for example, technicians who work alone in noisy places with no way to narrate.
• Your volume is too small to justify the licence.
Training effort and ease of use are part of the product. If a tool only works for people who were in a training session two months ago, that is a fair finding too.
Pilot checklist
1. Baseline measured on two or three real procedures.
2. Success criterion written down.
3. Knock-out list agreed.
4. Owner named, plus a deputy for absences.
5. First five procedures and their recorders named.
6. Joint review scheduled for week one or two.
7. Recording spread across the pilot, not saved for the final week.
8. Plan for a team change: live training, criterion re-checked.
9. Scorecard filled in at the end.
Frequently asked questions
- How long should an SOP software pilot run?
- In our experience, three to four weeks of active use are enough if the first joint review happens early. Longer pilots rarely give better answers; uploads tend to cluster at the end.
- How many people should take part?
- Three to six people who record and review, plus one owner who can decide. More users without more structure mostly adds noise.
- What should we measure besides time?
- How many drafts could be approved without re-watching the full video. It is the best single indicator of whether reviewers trust the tool.
- Is a pilot the same as a proof of concept?
- No. A proof of concept shows that something works in principle. A pilot should show whether it works for your team, on your procedures, within your time budget.