Project Showcase: pytest-gremlins, Fast-first mutation testing for Python
Code coverage is the metric everyone tracks and nobody questions. Hit 90% and CI goes green. Ship it.
But coverage only measures what your tests execute. It says nothing about what they verify. Your tests can run a line 1000 times and still miss a bug on that line.
Mutation testing fixes this. It injects small bugs: change >= to >, flip True to False, swap + for - and checks if your tests catch them. If they don't, you found a real gap.
The problem: existing Python tools (mutmut, cosmic-ray) take 15+ minutes on modest codebases. They rewrite files, reload modules, and run your entire test suite for every single mutation. Nobody actually uses them in practice.
I built pytest-gremlins to make mutation testing fast enough to run during development. Three key decisions:
- All mutations embedded at import time, toggled via environment variable (no file I/O)
- Only tests covering the mutated line actually run
- Results cached by content hash, unchanged code skips entirely
Benchmarked against mutmut: 13.8x faster with parallel + caching enabled.
pip install pytest-gremlins
pytest --gremlins
It'll show you exactly which lines your tests touch but don't actually verify. Open source, MIT
0 Comments