How to Test Whether an Analysis Prompt Works

To test an analysis prompt, collect a small set of inputs where you already know the right answer, then run the prompt and score the output against that answer key. A prompt that reads well and scores poorly is a prompt that flatters you.

The key does not need to be perfect. It needs to be fixed in advance, so you cannot move the target after you have seen the result.

Build a small answer key

Pick ten videos you understand well. For each, write down the one finding a good analyst should reach, such as the reason retention dropped at minute four.

Keep the key short and specific. If you cannot state the expected finding in a sentence, the question may be too vague for the model to answer at all.

Score the output, not the vibe

Track the counts across prompt versions. A change that raises hits and lowers fabrications is progress, even if the prose reads a little rougher.

• Hits: findings that match the key or add a defensible one.
• Misses: findings that contradict the key or ignore the obvious cause.
• Fabrications: claims with no row or transcript behind them.
• Format breaks: replies that ignore the requested shape.

A prompt you have not tested is a guess wearing a confident face.

Iterate on one thing at a time

Change a single element per round, then rerun the same key. Adjust the role, then the examples, then the format, so you can tell which change moved the score.

When two versions tie, keep the shorter one. Less instruction leaves less room for the model to misread you.

Key takeaways

Retest after any model update. A prompt that scored well last month may behave differently on the next version.

• Fix a small answer key before you run any prompt.
• Score hits, misses, fabrications, and format breaks separately.
• Change one element per round and rerun the same test.
• When scores tie, prefer the shorter prompt.