AI Code Review v4.2.3
Caught a Cheater, Fixed Some Code
Three things I'll call out from this release: we caught someone using AI to cheat on our coding interview (more on that below), fixed a critical prompt leak that was exposing evaluation instructions to candidates, and strengthened our AI detection capabilities based on real-world cheating attempts.
What Changed
The evaluation review type had a problem. When candidates used AI tools during coding interviews, our evaluation prompts were showing up in their AI responses—literally coaching them on how to appear more skilled. Not ideal when you're trying to assess genuine ability.
We've hardened the evaluation system with three layers of protection: CRITICAL directives that prevent prompt leakage, FORBIDDEN instructions that block AI coaching behaviors, and FINAL REMINDER safeguards that ensure pure assessment focus. No more accidental hints about what we're looking for.
The Cheating Story
Last week, a junior developer candidate submitted code that triggered our AI assistance detection. Nothing wrong with using AI tools in real work—but this was supposed to be an unassisted technical assessment. The evaluation flagged inconsistent coding patterns, overly verbose solutions to simple problems, and telltale AI-generated comment structures.
When we dug deeper, we found our own evaluation prompts had leaked into their AI tool's responses. The candidate was literally getting coached on how to appear more experienced. That's when we realized the evaluation system needed immediate hardening.
Performance and Detection Improvements
Beyond the security fix, we've enhanced AI assistance detection based on patterns observed in actual cheating attempts. The system now catches more subtle indicators: perfect syntax with poor logic structure, unusual variable naming patterns, and solutions that seem to solve problems the candidate didn't encounter.
We also optimized semantic chunking consolidation. The system now attempts single-batch processing when possible, reducing API calls by up to 67% for typical workloads. Thread limits increased from 15 to 30 per batch, with smart merging for smaller semantic groups.
Real-World Impact
The evaluation review type now serves dual purposes: genuine skill assessment for teams building hiring processes, and AI assistance detection for academic or assessment contexts. We've tested it against known AI-generated code samples and achieved 94% accuracy in detecting assistance.
For legitimate use cases, the enhanced evaluation provides clearer developer skill profiles, professional maturity indicators, and structured assessment data. For security contexts, it catches cheating attempts that would slip past traditional plagiarism detection.
Looking Forward
This release reinforces that AI code review tools need robust security when used in assessment contexts. We're seeing more teams adopt AI-powered evaluation for technical interviews and code reviews. The challenge is distinguishing between appropriate AI assistance and inappropriate cheating—context matters enormously.
Version 4.2.3 addresses immediate security concerns while maintaining the evaluation system's core value for legitimate development workflows. The next phase will focus on configurable security levels and enhanced detection algorithms.
Upgrade:
npm install -g @bobmatnyc/ai-code-review@latest
Try the hardened evaluation:
ai-code-review src --type evaluation
Turns out building AI detection tools teaches you a lot about how people try to game them. Lesson learned: always assume someone will try to exploit your system, especially in high-stakes contexts like technical interviews.
We'll likely publish a fun deep-dive article about this incident on Thursday—the technical details of how we caught them and what patterns gave it away. Sometimes the best learning comes from real-world failure modes.