Automation Assist
A desktop tool that automates the mechanical half of my own QA work, so the half needing judgment gets my full attention instead of whatever is left over.
- Tests
- 225
- Deliberate regressions
- 16 2 initially survived
Why it exists
I tracked where my time actually went, and course QA was the biggest line by a distance. Most of it was mechanical. Reading every slide looking for doubled words, terminology that had drifted between modules, menu labels that no longer matched the slide they pointed at.
None of that needs a person, and all of it fails silently if nobody does it. So the tool took the mechanical checks and I kept the judgment calls.
What is in it
| Tab | What it does |
|---|---|
| SCORM QA | Static analysis of a course package. Structure, SCORM API wiring, terminology, duplicates, spelling and spacing, questions, screen inventory, and a generated manual checklist |
| SCORM Runtime QA | Drives a packaged course in a real browser and verifies tracking. Unfinished, deliberately |
| Website | WordPress update reporting, a cache clear for a known Elementor rendering bug, and a site health smoke test |
| Regression | Wraps a Playwright suite so it runs from one button |
| Scheduler | Read-only view of what runs automatically and when |
Accessibility checks live inside SCORM QA rather than as their own feature. The courses it inspects ship a rich-media path and a screen-reader-friendly path in the same package, and the checks only apply when a course is marked dual-path. So the tool has to know what kind of course it's looking at before it can judge it.
Three decisions I would defend
No check ever reports a guaranteed pass
Every result is either a hard failure or information that narrows down human review. Nothing returns a green tick meaning "this is fine."
The SCORM API check is the clearest case. If
cmi.completion_status isn't set, that's a guaranteed failure
and nothing else proceeds until it's fixed. If it is set, I still
verify it at runtime, because a package can be configured correctly and
fail live anyway. A tool that told me it had passed would be worse than a
tool that said nothing, because I'd believe it.
The WordPress flows fail hard on purpose
Anything involving a login is built to break rather than recover. Logins periodically surface things that need a person. An email verification prompt, a policy change notice, an interstitial about upcoming changes. Swallowing those and carrying on would mean automating straight past the one thing that actually wanted attention.
A hard failure is the notification. Graceful degradation would have quietly deleted the signal I built the thing to catch.
Vocabulary is separated from logic
The static checks change constantly as I find new things worth checking. Every word list, terminology pair, and regex lives in a single data module with no logic in it, so adding a term never means reading around a check implementation to find where it goes. The part that changes weekly stays apart from the part that changes rarely.
What is deliberately not here
The Playwright regression suite that the regression tab drives is private. Those specs encode how an internal LMS behaves, down to certificate rules, scoring thresholds, and reporting filters, and that isn't mine to publish.
What's in the repository is the runner. The tab, the subprocess handling, and the folder resolution. Point it at a Playwright project and it works. Without one, that tab says the folder is missing and nothing else is affected.
How I tested it, and how I tested the tests
225 tests, and none of them need a course package or a network connection.
The checks get fed hand-built ScormData objects from a
fixtures module, so the suite runs in a clean checkout instead of
depending on content I can't publish.
Here's what they cover.
- Detection thresholds, because a check that cries wolf gets ignored, and an ignored check is worse than no check. The duplicate tests assert as much about what should not be flagged as about what should. Titles legitimately reused across modules, short UI labels, the expected twice-over of a dual-path course.
- Accessible-path detection, because every accessibility check is built on top of it, including the parallel-structure rule for courses whose accessible scenes carry no title.
- The two failure modes that have actually bitten me. A WordPress history replay rendering differently from a live run, and the Playwright project path resolving into PyInstaller's temp directory in a frozen build.
Then I checked whether the suite was worth having. I introduced sixteen
deliberate regressions one at a time and the tests had to catch each one.
Dropping a @dataclass. Downgrading the
cmi.completion_status failure to a warning. Reintroducing the
frozen-path bug. Stripping the word boundaries out of terminology
matching. A green suite proves nothing on its own. It has to be shown to
go red.
Two of the sixteen survived. Both pointed at assertions checking how many sections had run when they should have been checking which checks ran. The exercise paid for itself before it finished.
The one I keep coming back to
SCORM runtime QA is the biggest manual time sink in my job and I've prototyped a way out of it many times over. The problem ought to be tractable. A course can determine internally whether something was clicked or answered correctly, which makes it deterministic, and anything deterministic ought to be automatable. In practice SCORM is old, and authoring tools expose their items differently and inconsistently at runtime.
Every attempt has failed and every attempt has narrowed the problem, so it gets picked up again every couple of months, usually right after it's cost me another week. What keeps me on it isn't the repetition. This is a deterministic problem being checked by hand, so every manual pass is a workaround for something that shouldn't need one.
One thing about the coverage
It's deliberate rather than complete. The tests go where the risk is, meaning thresholds, accessible-path detection, and the two bugs that have actually cost me a day. Much of the interface layer is left alone. I'd rather say that than quote a percentage implying otherwise.