.png)
Key insights from “AI Can Make Mistakes,” presented by Anna Chernova, Software Development Engineer in Test at Ingage Partners.
Artificial intelligence is helping software teams accelerate development, automate repetitive tasks, and improve productivity. But as AI becomes more integrated into everyday workflows, an important question remains: How do we know when we can trust its output?
At MomentumDevCon 2026, Anna Chernova, Software Development Engineer in Test (SDET) at Ingage Partners, presented AI Can Make Mistakes, exploring how seemingly accurate AI-generated results can introduce unexpected errors and what development teams can do to catch them.
Through examples from her testing experience, industry research, and AI-generated content, Anna shared practical insights into maintaining software quality as AI becomes a larger part of the development process.
When AI Gets the Details Wrong
Anna shared several examples illustrating how AI can produce convincing results while quietly introducing inaccuracies.
In one claims-processing workflow, AI was used to transfer customer information into a table. Although the process appeared successful, the system had modified unusual email addresses and phone numbers to match more familiar patterns.
In another case, AI summarized student feedback from a software course. When students provided no response, the system generated positive feedback instead.
Both examples highlight a critical concern: AI can produce polished, seemingly complete results without accurately preserving the underlying information.
These risks become more significant in software development. Anna referenced research presented at USENIX Security 2025 that examined 16 code-generating models across 576,000 code samples. Approximately 19.7% of package recommendations referenced nonexistent packages.
This behavior creates opportunities for slopsquatting, a software supply chain attack in which malicious actors register fabricated package names that AI tools might recommend to developers.
Why Traditional Testing Still Matters
AI-generated errors can be especially difficult to identify because they often appear reasonable within the surrounding context.
Anna experienced this firsthand while using AI to update automated tests. She later discovered that a critical product price field had disappeared for certain product and filter combinations. More concerning, the tests that would have detected the issue had also been removed.
Because the original task was unrelated to pricing, the changes were easy to overlook.
The example demonstrates why AI-assisted development requires careful review of changes beyond the immediate task. A model may not recognize which functionality is business-critical unless those expectations are explicitly defined.
Practical Ways to Strengthen AI Testing
Anna emphasized that established software testing practices remain essential. Teams can build on those foundations with additional methods for evaluating AI behavior.
- Maintain deterministic tests. Continue validating schemas, permissions, API contracts, data parsing, and business rules.
- Build a golden dataset. Begin with 20–30 representative scenarios, including common use cases, edge cases, and previously identified failures. Run evaluations repeatedly to assess consistency.
- Validate AI evaluators. Compare AI-generated assessments against human judgments, particularly for high-impact scenarios.
- Review test coverage. Investigate tests removed or modified by AI, and consider mutation testing to verify whether existing tests detect defects.
These practices help teams benefit from AI-assisted development while maintaining confidence in software quality.
A Final Lesson: Count the Legs
Anna concluded with an AI-generated cookie design featuring a cat. An AI evaluator approved the image, despite the cat having too many legs and an extra eye.
The humorous example captured an important challenge: An AI system can produce an incorrect result, and another AI system can mistakenly approve it.
As organizations integrate AI into development workflows, clear testing standards, meaningful human oversight, and continuous evaluation remain critical to delivering reliable software.
Anna's closing advice captured it perfectly:
Test the AI. Test the judge. Test the tests. And when everything passes, count the legs.
