Accuracy report · version 1.5 · October 9, 2026

How often Pedagrade is right, and how often it's wrong

A review you can't verify is just an opinion. So we test Pedagrade on quizzes and lessons where we already know the answer, and publish what it catches, what it misses, and how often it raises a false alarm. Weak spots included.

0good questions falsely accused of a wrong key or two right answers, in 371 question checks
94%of planted wrong answer keys caught (100% on our hand-written quizzes)
97%of questions rated at the exact Bloom's level, on our main set of 432
89/96lessons fully correct on alignment: the right problems found, none invented

How we test

Every test uses material where the right answer is known before Pedagrade sees it. We run the same service you use.

How each test runs
Test 1: a planted mistake
Plant a known mistakeA wrong key, a joke option, a giveaway, or a missing key
Run Pedagrade, the same service users getThe right answer is known before Pedagrade sees it
Caught?How often it found a problem we planted
Test 2: nothing planted
Leave questions cleanSome stay clean on purpose
Run PedagradeThe same service you use
False alarm?How often it flagged something that was fine

Quizzes: 371 question checks

326 questions from a public, MIT-licensed academic question bank, kept only where we verified the answer key, plus 15 course questions we wrote, each checked 3 times. Most quizzes get one planted mistake: a wrong key, a joke option, a giveaway, or a missing key. Some stay clean on purpose.

Bloom's levels: 432 questions

Across 24 course topics, from feedback skills and phishing to medication safety and food safety. Each question is written to demand one level. 120 use a misleading verb, like "Evaluate which of these is part of..." for a recall question.

Lesson alignment: 103 lessons

96 lessons with known objectives and one planted problem each: a quiz that tests an objective too easily, questions from an unrelated topic, both, or none. Plus 7 hand-built lessons from real courses, each checked 3 times.

Results by check

"Caught" is how often it found a problem we planted. "False alarms" is how often it flagged something that was fine.

Caught, by check

Same numbers as the table below. Where the table gives a range, the bar shows the low end.

StrongGood, noisyWeak
Wrong answer key23 planted, 371 checks
94% to 100%False alarms: 0
Missing answer key19 planted
100%False alarms: 0
Practice with no feedback3 planted
100%False alarms: 0
Answer giveaway19 planted
88% to 100%False alarms: 12 of 326 clean questions
Wrong options nobody would pick20 planted
94% to 100%False alarms: 33 of 326 clean questions
Two right answers3 checks (small test so far)
1 of 3False alarms: 0
Bloom's level432 questions
97% to 99% exact; 98% to 99% within one levelFalse alarms: n/a
Misleading verbs120 questions
100%False alarms: n/a
Lesson alignment103 lessons
89 of 96 fully right; 7 of 7 real lessons, every checkFalse alarms: rare, counted in the 7 misses
CheckTested onCaughtFalse alarms
Wrong answer keyThe marked answer isn't the right one23 planted, 371 checks94% to 100%0Strong
Missing answer keyOne question has no key when the others do19 planted100%0Strong
Practice with no feedbackLearners see right or wrong, but not why3 planted100%0Strong
Answer giveawayThe right option stands out, like being far longer19 planted88% to 100%12 of 326 clean questionsGood, some noise
Wrong options nobody would pickA joke or obviously wrong distractor20 planted94% to 100%33 of 326 clean questionsGood, noisy
Two right answersMore than one option can be defended3 checks (small test so far)1 of 30Weak, see below
Bloom's levelThe level a question actually demands432 questions97% to 99% exact
98% to 99% within one level
n/aStrong
Misleading verbsA higher-sounding verb on a simpler question120 questions100%n/aStrong
Lesson alignmentTaught, practiced, and tested at the objective's level103 lessons89 of 96 fully right
7 of 7 real lessons, every check
Rare, counted in the 7 missesStrong

Testing status by framework

Status as of October 10, 2026.

FrameworkWhat Pedagrade checksStatus
AlignmentEvery objective taught, practiced, and assessed (Backward Design and UbD)Tested89 of 96 test lessons fully right
Bloom's TaxonomyMeasurable verbs, and questions at the objective's levelTested97% to 99% exact on the main set of 432; 87% on a second, harder set worded differently
Quiz qualityAnswer keys, wrong options, giveaways, feedbackTested94% to 100% of wrong keys caught; never called a correct key wrong in 371 checks
AccessibilityContent basics: alt text, captions, color, link text, instructions that rely on shape or positionTested81% right on 94 new test courses (from 63% before the October 5 fix); no false alarms on text-only courses
Merrill and GagnéReal tasks, prior knowledge, worked examples, practice, feedback, transferTestedOn 20 realistic expert-rated courses (2 runs each): every Merrill and Gagné gap caught (8 of 8 runs), no false flags on good courses. 91% to 100% right on 168 new test courses
Mayer's principlesMultimedia and cognitive loadTestedOn 20 realistic expert-rated courses (2 runs each): Mayer gaps caught in 5 of 6 runs, no false flags on good courses. 97% right on new test courses
UsabilityInstructions, navigation, clarityBetaNot yet tested
Plain languageSentence length, passive voice, reading level, acronyms, wordy phrasesBetaMeasured in code against the Federal Plain Language Guidelines, not by the AI. Not yet tested on real courses
Universal Design for LearningMore than one way to take in content, show learning, and stay engaged (based on CAST's UDL Guidelines 3.0)BetaEach point counts only when the course text proves it with a quote. Not yet tested on real courses
Keller's ARCS modelAttention, relevance, confidence, and satisfaction: does the design motivate learnersBetaEach point counts only when the course text proves it with a quote. Not yet tested on real courses
Retrieval practice and spacingRecall practice with feedback, spread across the course, plus a cumulative reviewBetaSpacing measured from the alignment map; other points count only with a quote from the course. Not yet tested on real courses
Kirkpatrick evaluation readinessIs the course built so all four levels can be measured: reaction, learning, behavior, resultsBetaLevel 2 read from the alignment map; other levels count only with a quote from the course. Not yet tested on real courses

Where it's weaker

We'd rather you hear these from us.

1

It under-reports "two right answers."

Pedagrade is built to stay quiet when it isn't sure. That's why it had zero false accusations in this test. The cost: on our small test of questions with two defensible answers, it spoke up 1 time in 3. We chose that trade on purpose.

2

Its suggestions about wrong options can be picky.

On questions with nothing planted, it suggested a stronger wrong option about 1 time in 10. Some of those are fair. These show as suggestions, not errors, and bringing the rate down is on our list.

3

It misses some "tested too easily" cases.

7 of 96 test lessons had a problem it missed, mostly a quiz question that asks learners to recall what the objective asks them to compare or judge.

4

Courses with gaps could score too high. Fixed October 8.

On three test courses with real alignment gaps, the score used to land between 84 and 92. Since October 8, gaps the alignment map finds cap the score: for example, a course with no practice anywhere caps at 72. On our 20 realistic test courses, 16 of 20 now score in the expected range. The map can still miss a real link, so read the list of gaps, not just the number.

5

Scores move a little between runs.

Reviewing the same course twice usually lands within about 2.5 points. The findings themselves are more stable than the score.

6

It's less sure of itself on harder subjects.

We ran a second, harder set: 1,593 quiz questions from subjects like law, engineering, and economics, none of which it had seen. It questioned a correct answer key 4 times, and 2 of those questions have a defensible second reading. On 1,536 Bloom's questions worded differently from our main set, it rated the exact level 87% of the time. These are what we're working on next.

What Pedagrade can't do

Our rules

We'd rather miss a problem than accuse something that's fine.

Every finding points to the text it's about, so you can check it.

We publish these numbers after each round of testing, including the ones that get worse.

If it gets your course wrong, tell us. Real mistakes become new test cases.

Found a mistake? Email support@pedagrade.com with what it said and what was right.

Changes