AI evals and learnings from AWS automatic medical coding in Revenue Cycle Management (RCM)

LLM / NLP based Medical coding workflow

A benchmarking study conducted by the New England Journal of Medicine (NEJM), a division of the Massachusetts Medical Society, evaluated the performance of general-purpose large language models (LLMs) including GPT-3.5, GPT-4, Gemini Pro, and LLaMA-70B for medical coding tasks. The study concluded that LLMs performed poorly in this domain. However, the models assessed were not purpose-built medical LLMs; they lacked training on specialized clinical datasets. Purpose-built models such as Google Med-PaLM, AWS Comprehend Medical, Azure Healthcare NLP, ClinicalBERT, and BioGPT are specifically designed for healthcare contexts and may offer significantly better performance.

Based on direct experience implementing AWS Comprehend Medical, natural language processing (NLP) and medical LLMs can automate a substantial portion of straightforward coding scenarios. Complex cases still require human review to ensure compliance and accuracy. This section provides definitions and a recap of a prior proof-of-technology initiative that illustrates a viable path forward.

Automatic medical coding refers to the AI-driven assignment of diagnosis codes (ICD-10) and procedure codes (CPT/HCPCS) from clinical documentation. This reduces manual entry errors and accelerates claim submission. For example, in an urgent care visit, a physician’s SOAP note describing symptoms (ear pain, fever), diagnosis (otitis media), and treatment (amoxicillin) can be automatically mapped to ICD-10 and CPT codes. Accurate coding is essential for reimbursement and compliance; codes must align with documentation and adhere to billing rules to avoid denials.

A quick recap:

AWS Comprehend Medical is a managed NLP service that extracts medical information from unstructured text. It can identify entities such as conditions, medications, procedures, and anatomy, and map certain entities to standard codes. For example, the InferICD10CM API detects diagnoses in text and returns ICD-10-CM codes with confidence scores, while recognizing traits such as negation and uncertainty. It also links to ontologies such as ICD-10-CM for diagnoses and RxNorm for medications.

Integration to RCM: Comprehend Medical can be embedded in RCM workflows to assist coders or automatically code simple cases. Clinician notes can be processed in real time — upon patient checkout, the API can return suggested codes within seconds, expediting claims submission. The service is HIPAA-eligible, with encryption and audit logging for PHI compliance.

Strengths of AWS Comprehend NLP:

• Out-of-box capability: Recognizes a wide range of clinical terms without custom training.
• Consistency: API-driven and suitable for high-volume, real-time pipelines.
• Scalability: Fully managed infrastructure with automatic scaling.
• Efficiency: Reduces manual effort and improves coding turnaround time.

Limitations of AWS Comprehend:

• Limited Code Coverage: Supports ICD-10-CM and RxNorm, but not CPT/HCPCS procedure codes. Procedure coding requires supplementary models or rules.
• Lack of Customization: The model cannot be fine-tuned with proprietary datasets or adapted for specialty-specific terminology.
• Recall Gaps: May miss inferred diagnoses not explicitly stated in text, leading to false negatives and potential claim denials.
• False Positives: Without context filtering, could suggest codes for negated or uncertain conditions.
• Explainability Constraints: Provides entity spans and scores but limited rationale for code selection, complicating audit justification.

Other Medical Models:

• Google Med-PaLM / Healthcare NLP: Maps text to SNOMED CT and UMLS, with reasoning for coding consistency.
• Azure Text Analytics for Health: Extracts medical concepts and maps to coding systems; similar in scope to AWS Comprehend.
• Domain-Specific Models: Examples include ClinicalBERT, fine-tuned on clinical datasets like MIMIC-III for improved ICD code prediction.

Architecting a LLM Model driven Medical Coding System: (in AWS Cloud)

Given AWS Comprehend’s limitations, a custom LLM-based system offers adaptability, improved accuracy, and continuous learning.

Domain Tuning for Higher Accuracy: A general-purpose model (like GPT-4, etc.) can be fine-tuned on a corpus of clinical notes and their corresponding true codes (e.g., a hospital’s historical data or a public dataset like MIMIC-III with discharge summaries and ICD codes). Fine-tuning or few-shot prompting with specialized clinical data teaches the model the language and patterns of coding, greatly improving performance

Comparison with Off-the-Shelf Models: A custom-tuned LLM has the potential to outperform both out-of-box general LLMs and static services. Research indicates that when provided with tools or context, LLM performance jumps significantly. For example, one experiment used a retrieve-and-rerank approach (retrieving candidate ICD-10 codes for a given term and then using an LLM to pick the best match) and achieved near 100% accuracy on a test set, vastly outperforming a vanilla GPT-3.5 model that managed only 6% on the same task.

Continuous Learning and Adaptability: Unlike Comprehend Medical, a custom LLM system can be designed to continuously learn. New data can be annotated (with human coder oversight) and used to periodically retrain or fine-tune the model, so it stays up-to-date with evolving medical knowledge and coding standards. For instance, when ICD-10 or CPT releases new codes or guidelines each year, those can be quickly incorporated into the model’s knowledge.

Contextual Reasoning: Modern LLMs are capable of understanding narrative context and making inferences. A fine-tuned model can use the full context of a patient’s record (history, presenting symptoms, labs, etc.) to decide on codes in a way that a simple entity extractor cannot. For example, if a note describes symptoms that strongly point to a particular diagnosis not explicitly stated, a welltrained LLM might infer that diagnosis should be coded (or at least suggest it with rationale). LLMs can also produce an explanation of why a code is suggested, by highlighting the supporting evidence in the note — this can greatly aid human reviewers in validating the AI’s output.

Retrieval-Augmented Generation (RAG): We can further boost a custom model’s performance by integrating a vector database of official code descriptions, guidelines, and historical cases to provide grounded references during inference. This means, when coding a note, the system can retrieve relevant information from an external knowledge base (for example, the official ICD-10 code descriptions, coding clinic guidelines, or a database of common coding scenarios) and provide that to the LLM as additional context. In a custom solution, one could maintain a vector database of medical codes and their definitions or past similar cases. When a new note comes in, the system finds the top relevant code candidates (based on semantic similarity of the text to code descriptions or past examples) and gives those as hints to the model. This way, the model isn’t generating codes purely from its internal memory (which can lead to hallucinations or slight mismatches); instead, it grounds its output in real, valid codes and guidelines.

Improving Compliance & Reducing Denials: By rebuilding the solution with an LLM can incorporate rules and learn from outcomes, we can specifically target compliance. The model can be trained to avoid upcoding or downcoding by learning the appropriate level of coding from examples (e.g., learning the distinctions between a level-3 vs level-4 visit code based on documentation). If certain code combinations often cause denials (for example, a procedure that needs a certain diagnosis code present, or mutually exclusive codes billed together), these patterns can be included in training or a post-processing rule engine.

AWS Architecture: LLM-Powered medical coding System

Reference architecture for an AI-driven medical coding system: The pipeline is divided into, (1) Data Acquisition, (2) Model Development, and (3) Inference & Feedback. Phase 1: longitudinal patient records (structured EHR data, unstructured notes, past claims history) and referential data (coding guidelines, ontologies) are ingested into a data lake. (or Healthlake) Phase 2: Iterative model development: a foundation model is selected and fine tuned on the collected data to create a specialized “Medical Coding LLM”, optionally enhanced with a knowledge base for retrieval. The model is validated and deployed as an inference service. Phase 3: Shows the live inference endpoint integrated into the billing workflow (automated medical billing agent), and a feedback loop where human review outcomes and new data continuously feed back into model improvement.

Simplified “happy path” workflow:

1. When a patient encounter is completed and the provider signs off their note/patient chart in the EHR, the clinical note data is sent to the coding system (this can be triggered via HL7 interfaces, FHIR APIs, or a periodic batch export).
2. An orchestration service (could be a simple serverless function or workflow engine) takes the note and prepares the input for the model. Preparation may include pulling additional context (e.g. patient history or problem list if that’s relevant to coding) and performing a retrieval query on the knowledge base.
3. The note (and any retrieved context, such as “likely codes: X, Y, Z with descriptions”) is fed into the LLM model endpoint to generate a coding output. We might prompt the model to return results in a structured format (e.g. JSON with code and maybe a snippet of supporting evidence from the note for each code).
4. The raw output from the model is then processed. If the model provided justification for each code, the system can log that or even present it to a human reviewer. The system can also apply any business rules at this stage.
5. A human coder in the loop especially in early phases or for complex cases. The model can produce a suggested set of codes, and a human coder reviews and either approves or corrects them. This “human in the loop” step is crucial for both quality assurance and for gathering feedback data. When a human corrects the AI’s output, those corrections are fed back into the system (this becomes labeled data for retraining). Over time, as the model improves, the fraction of cases needing manual review can decrease, but you likely always keep at least some oversight to catch novel situations.
6. Once the codes are finalized (by AI or after human confirmation), they are used to populate the billing claim (e.g. the CMS-1500 or UB-04 form, or an electronic 837P/837I claim transaction). The coded claim can then be submitted to payers much faster than traditional processes.

Human-in-the-Loop, Active Learning, and Continuous Evaluation

By combining human oversight, active learning, and rigorous evaluation — we create a system that is resilient and improves over time. In fact, the human-in-loop and metrics are part of AI governance which is critical in healthcare: it ensures the AI does not operate unchecked and that we have traceability for decisions.

Human-in-the-Loop for Quality Control: At least initially, every AI-generated code recommendation should be reviewed by a human coder or a clinician. This manual oversight acts as a safeguard to ensure no obviously incorrect or non-compliant code goes through. The human’s role is to verify and correct the AI’s suggestions. If the AI missed a secondary diagnosis that’s documented, the human adds it. If the AI suggested a code that doesn’t actually have support in the note, the human will remove it or adjust it. This review not only protects against errors but also builds trust with the coding staff and stakeholders — they see that the AI is not a black box making unchecked decisions, but rather a smart assistant.

Active Learning Loop: The essence of active learning is that the model improves by learning from its mistakes. Every time a human corrects the AI (or a claim is denied and later corrected), those examples should be fed back as “hard examples” for training. We especially focus on cases the model was uncertain about or got wrong — those are high-value learning opportunities. The system can also intentionally route some ambiguous cases for human labeling even if the model had a guess, to ensure a diverse training set (this is sometimes called uncertainty sampling in active learning — ask humans for labels on cases the model isn’t sure about).

Continuous Evaluation Framework: Apart from training on new data, we need to continuously evaluate how the system is performing using a stable set of metrics and test cases. We establish an evaluation framework with the following elements:

Accuracy and Error Rates: We specifically watch the model’s accuracy in terms of coding correctness. If the model predicts 5 codes for a note and 4 are correct, 1 wrong, that’s 80% accuracy on that case. At scale, we might use F1-score as a combined measure of precision/recall for multilabel output. We also log the false positive rate (cases where model suggested an unsupported code) and false negative rate (cases where the model missed a code that should have been there).

Validation Dataset: A fixed set of example clinical notes with known correct codes (this can be a mix of typical cases and challenging edge cases) is kept aside as a benchmark. After each model update (or periodically, e.g. weekly), the model’s predictions on this set are computed. We track metrics like exact match rate (did the model get the entire set of codes exactly right), recall (did it get all the necessary codes?), precision (did it output any codes that were wrong?). These metrics show if improvements are being made and ensure new model versions don’t silently degrade.

Billing Compliance Checks: As part of evaluation, we include metrics for compliance. For example, we track if the model ever suggests codes that violate coding rules (we can simulate an audit by running the model’s output through an auditing software or rules engine). We measure the percentage of AI-coded cases that pass all compliance checks versus those that would be flagged. This metric is important for billing integrity.

Speed and Turnaround Time: Since one of the goals is faster claims processing, we evaluate the speed of the system. This includes the technical inference time (how many seconds to process a note through the model) and the overall workflow time (from encounter completion to claim ready). Suppose manually coding used to take 48 hours on average after a visit; with AI assistance we might get that down to a few hours or real-time. We can track the average coding turnaround time pre-AIand post-AI.

Denial Rates and Revenue Impact: A key ultimate metric is the denial rate for coding-related issues. We compare the percentage of claims denied for coding errors (or queries from payers for clarification) before vs. after the AI system. Ideally, accurate and consistent coding reduces denials. We also measure capture rate of billable items — e.g., are we now capturing more diagnoses or procedures that were previously missed (under-coded)? If the AI helps coders catch things like a billable counseling session that was documented but forgotten, that directly increases revenue.

Financial metrics such as increase in revenue capture or reduction in lost charges can be reported. Another metric measured is RVU (Relative Value Unit) utilization, since RVUs determine reimbursement, ensuring all relevant RVUs are captured (without overcoding) is crucial . The AI’s effect on total RVUs billed per encounter could be monitored, with the expectation that it should go up in a compliant manner (reflecting more complete coding).

Continuous Monitoring and Alerts: The evaluation framework would ideally include dashboards and alerts. For example, if accuracy on the validation set drops after a new update, or if the model starts suggesting a code that was rarely used before (could indicate a drift or an error), the team gets alerted. If claim denials spike or a certain payer starts rejecting something that the AI suggests, that triggers an investigation. This continuous eval ensures the system remains reliable.