I Audited Our EPA 608 Practice Test Bank: 59% of the Correct Answers Were Option B

Here's a test you can run on any multiple-choice question bank in about five minutes. It will probably tell you something uncomfortable.

I ran it on the question bank behind our EPA 608 practice test, the one HVAC technicians use to prepare for the federal refrigerant handling certification. The result:

  A:   31  (13.7%)
  B:  134  (59.0%)
  C:   52  (22.9%)
  D:   10  (4.4%)
Chi-square vs uniform: 155.7 (df=3, 7.81 is the p=0.05 cutoff)
Correct answer is the longest option: 67.0%
"All of the above" correct: 4/4

Fifty-nine percent of correct answers were in position B. Correct answers were in position D less than one time in twenty. And in two out of three questions, the right answer was simply the longest option.

Then I ran it on our full-length mock exam file:

  A:  100  (100.0%)
  B:    0  (0.0%)
  C:    0  (0.0%)
  D:    0  (0.0%)
Chi-square vs uniform: 300.0 (df=3, 7.81 is the p=0.05 cutoff)
Correct answer is the longest option: 79.0%

Every single correct answer was option A.

This post covers the audit script, why it happened, and the fix. If you build any kind of quiz, exam prep or e-learning product, check your bank before you read the rest.

Why this happens

Nobody sat down and decided "make the answer B." Question authors have habits:

• Type the correct answer first, then come up with distractors. If nothing reorders them, the answer is always A. That's what happened to the mock exam file.
• Avoid putting the answer first because it "feels too easy," and drift toward B or C instead. That's a well-known human tendency when people try to place answers "randomly."
• Write the correct answer carefully and the distractors quickly. Correct answers come out longer, more specific and more hedged ("90% of the refrigerant or 4 inches of vacuum"), and distractors come out short ("50%").

None of this matters if the app shuffles options when it displays them. Ours shuffled the question order but rendered the options in the order they were stored. So the authors' habits went straight to the user's screen.

Why it matters for a practice test specifically

A biased practice test doesn't just leak points. It teaches the wrong skill.

A candidate who takes enough of those practice tests will, without ever noticing, learn "when in doubt, pick B" or "pick the long one." Their practice scores go up. Then they sit the real EPA 608 exam, which is written and checked by a certifying organization, and the trick stops working. Each section is pass/fail at 18 of 25, so a couple of points of fake confidence per section is exactly the margin that decides the result.

A practice score is only useful if it predicts the real score. Bias breaks that.

The audit script

The bank is a JSON array of questions. Each question has options and an answer that matches one option's text:

{ "id": "TYPE1_004", "options": ["...", "...", "...", "..."], "answer": "90% of the refrigerant or 4 inches of vacuum" }

// audit.js: node audit.js bank.json
const fs = require('fs');

const bank = JSON.parse(fs.readFileSync(process.argv[2], 'utf8'));
const LETTERS = ['A', 'B', 'C', 'D'];

const positionCounts = [0, 0, 0, 0];
let longestIsCorrect = 0;
let aotaCorrect = 0, aotaTotal = 0;

for (const q of bank) {
  const idx = q.options.indexOf(q.answer);
  if (idx === -1) throw new Error(`${q.id}: answer not found in options`);
  positionCounts[idx]++;

  const lengths = q.options.map(o => o.length);
  if (q.answer.length === Math.max(...lengths)) longestIsCorrect++;

  if (q.options.some(o => /all of the above/i.test(o))) {
    aotaTotal++;
    if (/all of the above/i.test(q.answer)) aotaCorrect++;
  }
}

const n = bank.length;
const expected = n / 4;
const chiSquare = positionCounts.reduce((sum, obs) => sum + (obs - expected) ** 2 / expected, 0);

console.log(`Questions: ${n}`);
positionCounts.forEach((c, i) =>
  console.log(`  ${LETTERS[i]}: ${String(c).padStart(4)}  (${(100 * c / n).toFixed(1)}%)`));
console.log(`Chi-square vs uniform: ${chiSquare.toFixed(1)} (df=3, 7.81 is the p=0.05 cutoff)`);
console.log(`Correct answer is the longest option: ${(100 * longestIsCorrect / n).toFixed(1)}%`);
console.log(`"All of the above" correct: ${aotaCorrect}/${aotaTotal}`);

Three checks:

Position distribution with a chi-square test. With four options, each position should hold about 25% of correct answers. The chi-square statistic measures how far the observed counts are from that. With 3 degrees of freedom, anything above 7.81 means "this would happen by chance less than 5% of the time." We were at 155.7, which is not a borderline result.

You don't need a stats library. The statistic is one reduce, and the critical value for df = 3 is a constant you can hard-code.

Longest-option rate. If length carried no information, the correct answer would be the longest option roughly a quarter of the time, a bit more once you count ties. 67% means length is a strong clue.

"All of the above." In our bank, every question that offered "All of the above" had it as the correct answer. Small sample, but candidates notice patterns like that quickly.

The fix, part 1: a real shuffle

The obvious fix is to shuffle options at render time. The obvious implementation is wrong:

options.sort(() => Math.random() - 0.5); // don't

A comparator that returns random results breaks the assumptions of the sort algorithm, and the resulting permutations aren't uniform. How biased they are depends on the engine and its sort implementation. I measured it on Node 22 by shuffling ['A','B','C','D'] 100,000 times and recording where A ended up:

sort(() => Math.random() - 0.5): 36.2%  17.0%  15.6%  31.3%
Fisher-Yates:                     25.0%  25.1%  24.9%  25.0%

With the sort trick, the first option still lands in position A more than a third of the time. On a bank where the answer was authored first, that's leaving a lot of the bias in place.

Fisher-Yates is five lines and provably uniform:

// Fisher-Yates: every permutation equally likely
function shuffle(array, rng = Math.random) {
  const a = array.slice();
  for (let i = a.length - 1; i > 0; i--) {
    const j = Math.floor(rng() * (i + 1));
    [a[i], a[j]] = [a[j], a[i]];
  }
  return a;
}

The rng parameter lets you pass a seeded generator in tests, so a shuffled exam can be reproduced exactly.

The fix, part 2: pin the options that refer to other options

You can't shuffle "All of the above" into position A. It stops making sense. Same for "Both A and B" and "None of the above." Those get pinned to the end, and only the rest are shuffled:

const PINNED = /^(all|none|both) of the above$|^(a|b) and (b|c)$|^(a|b), (b|c),? and (c|d)$/i;

function shuffleOptions(question, rng = Math.random) {
  const pinned = question.options.filter(o => PINNED.test(o.trim()));
  const free = question.options.filter(o => !PINNED.test(o.trim()));
  return { ...question, options: [...shuffle(free, rng), ...pinned] };
}

Options like "A and B" are the worst case: after a shuffle, the letters point at different options. Pinning keeps the option in place, but the letters inside it are still wrong. The real fix is to rewrite those questions so options don't refer to each other by letter. The audit is a good moment to find them.

The fix, part 3: compare answers by text

If the app records "the user picked index 2," shuffling breaks scoring. Index 2 means something different on every attempt. Our bank already stored the answer as option text, so scoring is selected === question.answer no matter what order the options appeared in. If yours stores indices, migrate to text or stable option ids before you turn on shuffling, or historical results become meaningless.

What shuffling doesn't fix

Shuffling removes position bias completely. It does nothing for the longest-answer bias or the "All of the above" pattern. Those are in the content, and only editing fixes them:

• Rewrite distractors so they're as specific as the correct answer. "50% of the refrigerant" next to "90% of the refrigerant or 4 inches of vacuum" gives it away. "80% of the refrigerant or 4 inches of vacuum" doesn't.
• Make sure "All of the above" is sometimes wrong, or remove it.
• Re-run the audit after each batch of edits and watch the longest-option rate fall toward 25 to 30%.

I'd add the audit to CI with a threshold, for example fail if the chi-square for authored positions exceeds the cutoff on newly added questions, or if the longest-option rate for a section goes above 40%. That catches regressions when someone adds a batch of new questions next quarter.

The takeaway

Shuffle the questions and the options, with a real shuffle. Then audit what shuffling can't fix. It takes an afternoon, and it's the difference between an EPA 608 practice test that measures what someone knows and one that measures how well they've learned the author's habits.

Our app was shuffling questions the whole time and still sending a 59% B / 100% A pattern straight to users. That's the scary part about this kind of bug: nobody files a bug report about getting answers right.