Why You Cannot Trust LLM-Powered Resume Scoring

HackerRank open-sourced its candidate screening tool. Turns out, the scores are noisy, gameable, and structurally biased.

When HackerRank open-sourced its hiring-agent repository in June 2026, it was framed as a transparency move for AI-driven recruiting. The tool is a Python CLI that processes PDF resumes using Large Language Models to calculate candidate scores. Within days of release, the community audited the code and found a significant flaw: the system is non-deterministic, and the scoring logic is trivial to manipulate.

Implementation Details

The tool uses PyMuPDF to convert PDFs into Markdown, then executes six separate LLM calls to extract structured data across categories like work history, skills, and projects. It also integrates GitHub signals to evaluate code commits. You can run the collector using standard Python patterns:

git clone https://github.com/interviewstreet/hiring-agent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python score.py /path/to/resume.pdf

The Reproducibility Problem

Independent testing revealed that identical PDF files yield drastically different scores. In one audit, a single resume processed 100 times returned scores ranging from 66 to 99. If a firm sets a hiring cutoff at 85, a high-quality candidate would be auto-rejected roughly 65 percent of the time simply due to model variance. This is not a configuration error; it is a fundamental challenge when chaining LLM prompts for subjective judgment. While checklist-based categories like tech stacks remain stable, qualitative assessment of architecture or engineering judgment fluctuates with every inference pass.

Prompt Injection via Hidden Text

A more severe finding is that the pipeline is vulnerable to prompt injection via hidden PDF text. Because the parser extracts all characters without verifying visibility, a candidate can include white-text blocks containing keywords like Google or Meta internships. These fabricated credentials are treated as factual during the extraction phase, allowing an otherwise junior resume to receive an inflated score. The tool lacks any enforcement mechanism to distinguish between human-readable qualifications and hidden malicious context.

Reference

HackerRank Open-Sourced Its ATS. The Score Depends on the Roll of the Dice