Comparing AI Model Performance in Genetic Diagnosis: A Case Study

Created by Fotor AI

Introduction

Since the release of ChatGPT, the integration of Large Language Models (LLMs) into existing workflows has become a hot topic across various industries. As a bioinformatician specializing in genetic diagnosis, my goal is to explore how LLMs can be leveraged to enhance the efficiency of medical geneticists in their daily work.

Working in mainland China, I’m limited to LLMs deployed there for production purposes. However, I want to use advanced models like ChatGPT, Claude.ai, and Meta LLM as benchmarks. This comparison serves not only as a reference point but also as a way to gauge the current state of AI capabilities in specialized fields like genetics.

Before embarking on a comprehensive evaluation with a large dataset, I decided to conduct a focused comparison based on a single use case. This approach allows for a more nuanced understanding of each model’s strengths and limitations in a real-world scenario.

The Task at Hand

The inspiration for this comparison comes from a common task in genetic diagnosis reporting. Medical geneticists often need to craft a concise paragraph describing how pathogenic variants in a specific gene can lead to a disease. This description typically synthesizes information from authoritative sources like the OMIM (Online Mendelian Inheritance in Man) database and GeneReviews.

This task presents several key challenges:

1. Cross-lingual synthesis: The input data is primarily in English, drawn from expert knowledge bases, but the output needs to be in Chinese to serve local medical professionals.
2. Accurate translation of medical terminology: This is particularly challenging as there may not be sufficient bilingual medical corpora to train these models effectively.
3. Adherence to specific formatting requirements: The output must follow a predetermined structure to maintain consistency in medical reports.

Methodology

To set up this comparison, I first needed to prepare the input data. Using the OMIM API, I queried and extracted gene-phenotype mapping relationships and molecular descriptions for a specific gene (SLC2A1 in this case). The gene-phenotype relationships were formatted as a markdown table, while the molecular description was left in its raw XML format.

OMIM Phenotype-Gene Relationships

OMIM Molecular Genetics

GLUT1 Deficiency Syndrome 1 In patients with a transport defect of glucose across the blood-brain barrier, consistent with GLUT1 deficiency syndrome-1 (GLUT1DS1; {606777}), {43:Seidner et al. (1998)} identified heterozygous mutations in the SLC2A1 gene ({138140.0001}-{138140.0003}). Two of the patients had been reported by {9:De Vivo et al. (1991)}. {20:Klepper et al. (2001)} reported a father and 2 children from separate marriages who were affected by GLUT1 deficiency, and confirmed autosomal dominant transmission by identifying a heterozygous mutation in the GLUT1 gene (G91D; {138140.0006}). The father developed generalized tonic-clonic seizures and myoclonic seizures at age …

Output is truncated.

Next, I created a standardized prompt template using LangChain, a popular framework for developing applications powered by language models. The prompt, written in Chinese, provided four key instructions:

1. Describe the gene-disease relationship, emphasizing how pathogenic variants can lead to associated conditions. Include the inheritance pattern.
2. If patient information is available, summarize the potential clinical presentations.
3. Format disease names as: Chinese name (English name) [MIM: OMIM ID].
4. Output as a single paragraph, not exceeding 500 words.

from langchain.prompts import ChatPromptTemplate

template = """作为遗传学专家，你的职责是阅读基因相关的英文信息，并总结归纳成基因疾病相关性的描述。请遵循以下准则：
1. 以基因名称为主语进行描述，需求强调基因的致病变异会导致相关疾病，并说明遗传模式。
2. 如果基因信息中包含疾病的患者信息，需要简要总结疾病的患者可能存在的临床表现。
3. 疾病名称需要为中文名（英文名）[MIM:omim id]的格式。
4. 不分行，以一个段落进行输出；字数不超过500字。
<<表型关系/phenotype map>>: {phenotype}
<<基因信息/molecular genetics>>:{molecular_genetics}"""

prompt = ChatPromptTemplate.from_template(template)

Running code

I then selected a mid-tier model from each of the popular AI platforms to run this prompt:

1. Doubao-pro-32k-240615 from Volcegen (ByteDance)
2. Qwen-plus from Alibaba
3. Llama3–70b-instruct from Meta (deployed on Volcengine)
4. GPT-4o-mini from OpenAI
5. Claude-3-opus-20240229 from Anthropic

doubao-pro-32k-240615

from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
# Doubao-pro-32k-240615
llm_model = "ep-20240821102124-nl9ck"
doubao_llm = ChatOpenAI(temperature = 0.0, 
                model=llm_model,
                openai_api_key=os.environ["ARK_API_KEY"],
                openai_api_base="https://ark.cn-beijing.volces.com/api/v3")
chain = prompt | doubao_llm | StrOutputParser()
response = chain.invoke({'phenotype': phenotype_markdown_table, 'molecular_genetics': molecular_genetics})
display(Markdown(response))

qwen-plus

from langchain_community.chat_models.tongyi import ChatTongyi
from langchain_core.output_parsers import StrOutputParser
qwen_llm = ChatTongyi(model="qwen-plus", temperature = 0.0)
chain = prompt | qwen_llm | StrOutputParser()
response = chain.invoke({'phenotype': phenotype_markdown_table, 'molecular_genetics': molecular_genetics})
display(Markdown(response))

llama3–70b-instruct

from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
# Doubao llama3-70b-instruct
llm_model = "ep-20240821130256-nwhdr"
llama3_llm = ChatOpenAI(temperature = 0.0, 
                model=llm_model,
                openai_api_key=os.environ["ARK_API_KEY"],
                openai_api_base="https://ark.cn-beijing.volces.com/api/v3")
chain = prompt | llama3_llm | StrOutputParser()
response = chain.invoke({'phenotype': phenotype_markdown_table, 'molecular_genetics': molecular_genetics})
display(Markdown(response))

gpt-4o-mini

from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
llm_model = "gpt-4o-mini"
openai_llm = ChatOpenAI(temperature = 0.0, model=llm_model)
chain = prompt | openai_llm | StrOutputParser()
response = chain.invoke({'phenotype': phenotype_markdown_table, 'molecular_genetics': molecular_genetics})
display(Markdown(response))

claude-3-opus-20240229

from langchain_anthropic import ChatAnthropic
from langchain_core.output_parsers import StrOutputParser
claude_llm = ChatAnthropic(model='claude-3-opus-20240229',temperature = 0.0)
chain = prompt | claude_llm | StrOutputParser()
response = chain.invoke({'phenotype': phenotype_markdown_table, 'molecular_genetics': molecular_genetics})
display(Markdown(response))

Results

The outputs from each model varied significantly in quality and adherence to the given instructions. Here are their raw outputs.

doubao-pro-32k-240615

SLC2A1 基因的致病变异会导致多种疾病。其遗传模式包括常染色体显性遗传、常染色体隐性遗传。

特发性全身性癫痫易感性 12（Epilepsy, idiopathic generalized, susceptibility to, 12）[MIM:614847]，患者可能存在轻度至中度智力障碍、轻度共济失调、肌阵挛、运动诱发的阵发性运动障碍等临床表现。

肌张力障碍 9（Dystonia 9）[MIM:601042]，患者多在儿童期发病，表现为阵发性舞蹈手足徐动症和进行性痉挛性截瘫，多数存在一定程度的认知障碍，还可能有癫痫、偏头痛、共济失调等症状。

婴儿起病的严重 GLUT1 缺乏综合征 1（GLUT1 deficiency syndrome 1, infantile onset, severe）[MIM:606777]，患者可能有全身性强直阵挛性癫痫、肌阵挛性癫痫、轻度痉挛性双侧瘫痪、发育迟缓、中度智力障碍、小脑性共济失调、锥体束征等。

儿童起病的 GLUT1 缺乏综合征 2（GLUT1 deficiency syndrome 2, childhood onset）[MIM:612126]，患者可能有精神运动发育迟缓、中度智力障碍、水平眼球震颤、构音障碍、肢体共济失调、反射亢进、肢体肌张力障碍姿势等。

伴有神经系统缺陷的 stomatin 缺陷性冷性血细胞增多症（Stomatin-deficient cryohydrocytosis with neurologic defects）[MIM:608885]。

qwen-plus

SLC2A1基因的致病变异可导致多种神经系统疾病，包括苯丙糖转运蛋白缺乏症1型（GLUT1DS1；606777）、苯丙糖转运蛋白缺乏症2型（GLUT1DS2；612126）、与SLC2A1相关的神经缺陷性冷水溶血症（SDCHCN；608885）、特发性全身性癫痫12型易感性（EIG12；614847）及肌张力障碍9型（DYT9；601042）。这些变异主要通过常染色体显性遗传模式传递，少数情况下也可通过常染色体隐性方式遗传。苯丙糖转运蛋白缺乏症1型表现为婴儿期严重的症状，包括发育迟缓、肌阵挛发作、智力障碍、共济失调和腿部锥体束征等。该疾病还可能导致低脑脊液葡萄糖水平（hypoglycorrhachia）。苯丙糖转运蛋白缺乏症2型则通常在儿童期发病，特征为发作性运动障碍、共济失调、构音障碍等症状，且部分患者无癫痫发作。与SLC2A1相关的神经缺陷性冷水溶血症表现为新生儿期出现神经缺陷和红细胞膜阳离子泄漏，导致假性高钾血症和红细胞溶解。特发性全身性癫痫12型易感性表现为早发性失神发作，部分患者伴有轻度至中度智力障碍、轻微共济失调或运动诱导的发作性运动障碍。肌张力障碍9型则表现为儿童期发作的舞蹈手足徐动症和进展性痉挛性截瘫，伴随认知障碍、癫痫发作、偏头痛等症状。上述疾病均与SLC2A1基因突变有关，表明该基因在维持正常神经系统功能方面起关键作用。

llama3–70b-instruct

SLC2A1基因的致病变异会导致多种疾病，包括 GLUT1缺乏症候群1（GLUT1DS1，606777）[MIM:606777]、GLUT1缺乏症候群2（GLUT1DS2，612126）[MIM:612126]、 idiopathic generalized epilepsy 12（EIG12，614847）[MIM:614847]、Dystonia 9（DYT9，601042）[MIM:601042] 和 Stomatin-deficient cryohydrocytosis with neurologic defects（SDCHCN，608885）[MIM:608885]。这些疾病的遗传模式为 autosomal dominant 或 autosomal recessive。

GLUT1DS1的患者可能存在发育迟缓、智力障碍、癫痫、肌张力障碍、共济失调和视力障碍等临床表现。GLUT1DS2的患者可能存在发育迟缓、智力障碍、癫痫、运动障碍和视力障碍等临床表现。EIG12的患者可能存在癫痫、智力障碍和运动障碍等临床表现。DYT9的患者可能存在舞蹈症、痉挛性麻痹和智力障碍等临床表现。SDCHCN的患者可能存在神经系统损害、红细胞膜缺陷和低血糖等临床表现。

gpt-4o-mini

SLC2A1基因的致病变异与多种疾病相关，包括葡萄糖转运缺陷综合征1（GLUT1缺乏症1，GLUT1DS1）[MIM:606777]、葡萄糖转运缺陷综合征2（GLUT1缺乏症2，GLUT1DS2）[MIM:612126]、特发性全身性癫痫易感性12（EIG12）[MIM:614847]、以及肌张力障碍9（Dystonia 9，DYT9）[MIM:601042]。SLC2A1基因的突变通常以常染色体显性遗传模式传递。GLUT1缺乏症1的患者可能表现为全身性强直-阵挛性癫痫、发育迟缓、轻度智力障碍、运动协调障碍等症状；而GLUT1缺乏症2的患者则可能出现运动发育延迟、横向眼震、言语不清、肢体共济失调等表现。特发性全身性癫痫易感性12的患者通常在4岁之前出现缺失性癫痫，可能伴有轻度至中度智力障碍和运动失调。肌张力障碍9的患者则可能在儿童期出现阵发性舞蹈样运动和进行性痉挛性下肢麻痹，且大多数患者存在一定程度的认知障碍。SLC2A1基因的突变导致的葡萄糖转运缺陷，影响了血脑屏障的功能，进而引发神经系统的多种病理表现。

claude-3-opus-20240229

SLC2A1基因的致病变异可导致多种疾病。杂合突变可引起婴儿期发病的1型葡萄糖转运蛋白缺乏综合征（GLUT1 deficiency syndrome 1, infantile onset, severe）[MIM:606777]，常染色体显性或隐性遗传；儿童期发病的2型葡萄糖转运蛋白缺乏综合征（GLUT1 deficiency syndrome 2, childhood onset）[MIM:612126]，常染色体显性遗传；常染色体显性遗传的9型原发性扭转痉挛（Dystonia 9）[MIM:601042]；常染色体显性遗传的特发性全身性癫痫易感性12型（Epilepsy, idiopathic generalized, susceptibility to, 12）[MIM:614847]；以及常染色体显性遗传的神经系统异常的缺乏Stomatin的冷凝集素血症（Stomatin-deficient cryohydrocytosis with neurologic defects）[MIM:608885]。患者的临床表现包括癫痫发作、运动障碍如舞蹈病和肌张力障碍、发育迟缓和智力低下等。SLC2A1基因编码葡萄糖转运蛋白GLUT1，突变会影响GLUT1蛋白的细胞内定位、降低葡萄糖转运功能，从而导致神经系统能量代谢障碍和红细胞膜离子通透性异常。

In my opinion, Claude 3 Opus emerged as the clear winner, producing results that required minimal revision before use. GPT-4o mini followed closely, with the other models requiring more substantial editing to meet the task requirements. Here’s a breakdown of their performance:

1. Translation Accuracy: GPT-4o mini and Claude 3 Opus excelled in accurately translating disease names, including complex medical terminology. Doubao pro and Llama3 70b struggled with abbreviations like “GLUT1,” while Qwen plus provided incorrect Chinese translations.
2. Phenotype Description: All models successfully summarized the clinical presentations associated with the gene variants, demonstrating a good grasp of the medical information provided.
3. Formatting Adherence: Only Doubao pro and Claude 3 Opus strictly followed the specified disease name format. The other models deviated to varying degrees.
4. Output Structure: Surprisingly, both Doubao pro and Llama3 70b ignored the instruction to provide the output as a single paragraph, a relatively simple requirement that the other models managed to follow.

Conclusion

This study offers a glimpse into how different AI models handle specialized medical tasks. While the comparison has its limits — the models aren’t exactly equivalent in size or training — it clearly shows that some AI tools are much better than others at tackling complex, field-specific problems.

As AI keeps improving, it has great potential to help streamline work in specialized areas like genetic diagnosis. However, this study highlights why it’s crucial to thoroughly test and carefully choose AI tools before using them in important medical work. Not all AI models are created equal, and picking the right one can make a big difference in real-world applications.