The principle: fit the rigor to the stakes
A perfect evaluation of every skill would be too costly and too slow. Nobody would finish. So CDD asks one question about each competency before choosing how to evaluate it:
What happens if we are wrong about this person?
- If the answer is "a minor inconvenience," a lower-confidence evaluation is the right call. It is fast, cheap and good enough.
- If the answer is "money is lost, someone is hurt, or a life is at risk," we need very high confidence. Use higher-fidelity methods even though they cost more.
Spending too little on a high-stakes skill is reckless. Spending too much on a low-stakes skill is waste, and it slows people down for no benefit.
A simple way to choose
- Low stakesa mistake is easy to spot and fix
- Confidence needed
- Reasonable
- Typical methods
- Self-check against a rubric, short quiz, peer or champion sign-off, a brief guided conversation
- Cost
- Low
- Medium stakesa mistake costs time or rework
- Confidence needed
- Good
- Typical methods
- Live demonstration with a trained evaluator, work sample reviewed against a rubric, a project
- Cost
- Moderate
- High stakesfinancial, health, safety, legal, or life-threatening
- Confidence needed
- Very high
- Typical methods
- Observed performance in realistic conditions, more than one observation, more than one evaluator, trained and calibrated evaluators, re-evaluation at set intervals
- Cost
- High
Notes on the table:
- A self-check is a legitimate tool for low-stakes skills. It is not a valid tool for high-stakes ones. People often can't judge their own skill accurately. They may overrate or underrate themselves, and they may report differently when pay or promotion is involved.
- "Sat through the class" is the weakest evidence. Attendance shows exposure, not skill. It may be enough for a trivial skill. It is never enough for a critical one.
- Knowledge tests show what someone understands. Observation shows what someone can do. For skills that matter, observe.
How to raise confidence when you need to
Confidence goes up when you do more of these:
- Write a clear rubric. The more specific the guidance, the more evaluators agree with each other.
- Describe good and poor behavior. For each competency, note what passing looks like and what failing looks like.
- Use a recording sheet. Evaluators note the evidence they saw, not just a verdict.
- Train evaluators. Practice on a recorded or trial run, and compare notes.
- Observe more than once. One observation can be luck.
- Use more than one evaluator for the highest-stakes skills.
- Re-evaluate on a schedule. Skills fade. Set an expiration on the badge.
How to test whether an evaluation can be trusted
Before you rely on an evaluation for something important, check it.
- Compare known groups. Have some of your strongest performers and some of your weakest take it. The results should reflect the difference. If they don't, the evaluation isn't measuring the skill.
- Compare evaluators. Have a few evaluators assess the same performance and discuss differences until they agree. Repeat until the disagreement shrinks.
- Ask the people evaluated. A short, anonymous survey right after an evaluation will tell you whether it felt fair and accurate.
- Pilot first. Run a new evaluation with a small group at the same level as the people who will take it. Check the clarity of instructions, the time limit and the difficulty.
What to do next
- Rate each competency's stakes (low, medium, high) when you write it.
- Choose the evaluation method to match.
- Write down why you chose it, so the next person understands.
Sources
The methods in "How to test whether an evaluation can be trusted" draw on Sanghi, The Handbook of Competency Mapping, chapters 4 and 6.