Transformer-based LLMs can Outperform Standard ML Approaches in Single-Cell RNAseq Annotation

Abstract

With the rise of generative AI, there has been a shift in computation from deterministic to probabilistic tooling. LLMs provide the means to accelerate routine biological tasks such as scRNAseq analysis. In this study, the cell type annotation accuracy of real and simulated PBMC data was compared between non-transformer (ScType) and transformer-based tools (GPT-4o and GenePT). GPT-4o was shown to perform as well as, and sometimes better than, ScType. However, this came at the cost of large run-to-run variability, which underlines the importance of further research into the reliability of these black box models moving forward.

Introduction

Biology has been undergoing a sequencing revolution that promises to reshape our understanding of it. Since Robert Hooke discovered the cell, the basic biological unit of life, we have sought to comprehend it. Transcriptomics has emerged as a powerful approach, offering an unbiased view of how cells respond through their gene expression profiles. The extension of this to the cell level through single-cell RNA sequencing (scRNAseq) has enabled us to study gene expression at cellular resolution.

A central challenge in scRNAseq analysis is cell type annotation, which is the process of identifying and assigning cell types based on their gene expression data. This process is required for analyzing cell-type variations within cell populations which is critical to many biological contexts, including disease characterization and drug discovery efforts.

Manual annotation is expertise-dependent, and even with the proper expertise, it requires a significant amount of time and sufficient computational resources. Multiple tools have been developed to aid in this endeavor, such as ScType¹. More recently, tools leveraging transformer architectures and embeddings have been produced. This blog explores a tool from each of these categories, GPT‑4o (zero-shot LLM), GenePT (LLM embeddings), and ScType (automated standard pipeline with curated database lookup), to help researchers tackle the cell annotation task.

Using a frequently downloaded scRNAseq dataset, the author-reported cell labels were set as the ground truth. Cells were also simulated from this dataset while preserving the distribution of gene counts. The accuracy of reproducing the ground-truth labels was measured for each tool.

Data and Simulation

The data used in this study contained Peripheral Blood Mononuclear Cells (PBMCs), as described in the Seurat Guided Clustering Tutorial. This dataset, freely available from 10X Genomics, is of high quality and commonly used to acquaint oneself with scRNAseq analysis, including the cell annotation task. Using this dataset for modeling, a new scRNAseq count matrix with defined cell types was simulated with SPARSim².

The reason for simulating data was two-fold. First, simulation allowed testing the tools’ capabilities on data that is guaranteed not to have been seen during any LLM’s training. Second, it provides controlled scenarios where tools can be tested multiple times on different datasets, allowing measurement of their performance consistency. The simulated dataset contained an equal number of cells to maintain the relative abundance of each cell type, and each dataset comprised 9 unique cell types.

In one task, clusters of the original and simulated data were assigned using GenePT and ScType. In another task, cell types of the original and simulated data were assigned directly by GPT-4o from gene expression profiles, skipping clustering (Figure 1).

Figure 1: Flow of data and tools used in the study.

The results of the various tools used in the study are presented in the following sections, and the details of the steps and methods are outlined in the “Methods” section. The first tool is non-transformer-based, while the tools that follow are transformer-based.

Automated Pipeline with Curated Database Lookup: ScType

ScType¹ is a marker database and cell-type identification tool that enables accelerated cell-type annotation. The ScType annotation pipeline takes in scRNAseq count data, preprocesses the data (normalization, outlier removal), clusters the data using classical ML techniques such as shared nearest neighbor (SNN), and then assigns cell type annotation using the ScType database.

To assess accuracy, GPT-5 was used to resolve slight naming differences and cases where predicted annotations were synonyms or subtypes of the ground truth (see the “Methods of Name Resolution” section). A design choice made was to penalize annotated cells with increased specificity (e.g., counting incorrect “Naive B” cell annotations when the ground truth is “B” cell ). Here, predictions are not awarded for over-interpreting a cell type that could be another type in reality (e.g., the ground truth “B” cell could be a “Memory B” cell and not the predicted “Naive B cell”). An argument could be made that counting all subtypes of B cells as B cells is more accurate, as it reflects their hierarchical relationship, but the more conservative choice was made in this study.

ScType performed better on the original PBMC dataset compared to the simulated dataset (Table 1). These performance values served as the benchmark against which the other methods were compared.

Table 1: ScType performance on original and simulated data.

Zero-shot LLM: GPT-4o

LLMs are powerful tools that can be used to accelerate the cell annotation process by requiring less effort and biological expertise. Following the methods used by Hou & Ji³, OpenAI’s GPT‑4o was used to assign labels directly from gene expression data without clustering by providing the marker genes per cell. Here, “marker genes” are the names of the genes that are the most expressed in the cell.

We observed relatively high accuracies for GPT‑4o across original and simulated data and different numbers of gene markers. To assess the stability of GPT-4o’s performance, the same data was run a total of 4 times for each case and averaged.

This showed that although the performance of GPT-4o could be substantially higher given an appropriate selection of the marker list size, the variability between runs was significant. This is a critical difference compared with ScType, which is deterministic, showing the same result from the same input. Both the average performance and stability across GPT-4o runs were maximized for intermediate marker gene numbers (100 and 500 marker genes) (Figures 2A and 2B).

Figure 2A: GPT-4o can outperform ScType at annotating original PBMC data. Bar heights correspond to average values and error bars are standard deviations.

Figure 2B: GPT-4o generally outperforms ScType at annotating simulated PBMC data. Bar heights correspond to average values and error bars are standard deviations.

The F1 scores showed a similar trend to total accuracies (Supplementary Figure and Table 2C/2D).

Notably, there were some cases where GPT‑4o outputted “Unknown” or “General PBMC” as an annotation. However, despite this limitation and the general variability between runs, GPT-4o’s average performance being capable of exceeding that of ScType places it as a powerful zero-shot option for cell-type annotation.

LLM with Guardrails: GPT‑4o + Word Bank

Extending the approach of Hou & Ji³, GPT-4o’s performance was assessed when provided a word bank of names that the LLM must choose from. Given the challenges of cell type name resolution, it was expected that this would increase the accuracy of the tool. Although this approach prevents novel cell-type identification, it would not be unreasonable in cases where the researcher is confident in the cell types found in their sample.

Providing GPT‑4o with a list of cell types circumvented the need for name resolution. This wouldn’t have been the case if the LLM had chosen names outside of the word bank, but somewhat surprisingly it adhered to the selected terms in all trials. However, there were some cases where some of the cell types in the word bank were not assigned to any cells. The missingness of a specific cell type was uncorrelated with how much it appeared in the dataset, with each trial missing different cell types. Surprisingly, for most cases the word bank did not improve accuracies and even dropped in certain cases (Figures 3A and 3B). Note that these annotations with a word bank were only run a single time each despite likely suffering from the same variability as the annotations without a word bank. As such, this analysis only provides a suggestion as to how adding a word bank may affect performance.

Figure 3A: Providing GPT-4o with a word bank tends not to enhance the annotation accuracy of original PBMC data but can still outperform ScType. “Random” is the absolute accuracy for selecting a random cell type from the ground truth possibilities for every cell.

Figure 3B: Providing GPT-4o with a word bank tends not to enhance the annotation accuracy for simulated PBMC data but can still outperform ScType. “Random” is the absolute accuracy for selecting a random cell type from the ground truth possibilities for every cell.

In summary, zero-shot cell type annotation with OpenAI’s GPT‑4o can outperform the established method ScType, with challenges arising in reproducibility across trials.

Utilizing LLM Embeddings for Clustering: GenePT

GenePT⁴ still utilizes the transformer architecture, but uses a different transformer model to extract embeddings (text-embedding-ada-002). Leveraging ChatGPT embeddings, GenePT, as described by the authors, is a single-cell foundation model that addresses both gene-level and cell-level biology tasks.

Briefly, the simulated per-cell count data was transformed into embedding vector representations with GenePT and clustered with UMAP or PCA. The top 1000 gene markers for each cell were passed through the transformer model (Figure 4). Appropriate clustering was then evaluated by calculating Adjusted Rand Index (ARI), which compares cluster partners with the ground-truth cell types from which the simulated cell was modeled upon. The ARI of UMAP and PCA on the non-transformed, normalized count data was also calculated to provide a benchmark.

Figure 4 from Chen,Y. and Zou,J. (2024): The textual representation for each cell is passed through the embedding model to prepare it for clustering.

The ARI scores for GenePT transformed data revealed significantly lowered clustering performance (Figure 5). This was true across multiple dimensionality reduction and clustering techniques tested (Supplementary Tables 8A and 8B).

Figure 5: GenePT reduces performance relative to clustering the normalized simulated data for both dimension reduction techniques.

To determine if annotation accuracy could be improved by aggregating cell expression at the cluster level, GPT‑4o was provided the top 1000 marker genes per cluster determined by the best-performing dimensionality reduction and clustering techniques. Both accuracy and F1-score were calculated (Figure 6).

Figure 6: The top 1000 marker genes are taken for each cluster and passed through GPT-4o for cell annotation.

Comparing the per-cell GPT-4o performance, clustering the normalized simulated data before GenePT annotation reduced the accuracy by over 50% and GenePT transformation reduced performance even further (Table 2). While the performance could be suffering from a non-ideal number of gene markers per cluster, it is more likely to come from improper clustering (Figure 5) and the introduction of noise in the case of the GenePT transformation.

Table 2: Normalized simulated and GenePT accuracy.

Discussion

Table 3: GPT-4o can perform better at cell-type annotation than ScType for both original and simulated datasets. Providing GPT-4o with a word bank tends not to enhance the annotation accuracy, and the performance of GenePT + GPT-4o falls short compared to the other tools. GPT-4o, GMM UMAP + GPT-4o, and GenePT-s GMM PCA + GPT-4o are averages over multiple runs, and the best marker gene argument was taken from each tool.

LLMs have provided acceleration to biological tasks, such as scRNAseq analysis, with tools like GenePT and direct usage of GPT-4o. In this study, the capability of these tools to aid cell type clustering and annotation was compared with that of an established, non-transformer based method, ScType.

Overall, GPT-4o can perform better at cell-type annotation than ScType, as shown with both original and simulated PBMC datasets. Surprisingly, additional information provided through a word bank did not enhance performance. There was also a notable drop in performance for simulated datasets compared to the original data, which could be explained by the distortion of between-gene relationships that can take place during SPARSim simulation. Potential improvements to these tools could come from reducing between-run variability and using marker genes defined by experts or differential expression rather than just abundance.

It was observed that providing intermediate numbers of gene markers per cell reduced between-run variability. A potential explanation for this is that too few markers did not provide enough information, while too many markers introduced noise. This effect has been described before in the context of degrading LLM performance with very large input token sizes⁵. The variability of GPT-4o between consecutive runs is a considerable drawback, but could possibly be reduced by setting a parameter called “temperature”. The higher the temperature, the more evenly possible responses are weighted. Thus, higher temperatures result in more diverse outputs, whereas lower temperatures result in more deterministic outputs. This parameter is adjustable via the OpenAI API, but not in the web interface where the GPT-4o analysis was done. Performing cell type annotation on tens of thousands of gene marker data can be costly through the API, which is one of the reasons the web interface was used. Further research should assess whether setting the temperature to a low value can reduce between-run variability.

It was also observed that GPT‑4o occasionally ignored the instruction to not assign “Unknown” as a cell-type and sometimes assigned very general annotations such as “General PBMC”. LLMs are more prone to hallucinate when the data is noisy or ambiguous or when they lack a nuanced understanding of specific domains general-purpose models like GPT‑4o. For cell types distinguished from one another by marker genes that are more lowly expressed, the LLM may struggle to assign the proper annotation provided only the non-distinctive highly expressed genes. Using marker genes defined by experts or differential expression rather than just abundance may potentially reduce the hallucinations of these models.

Given the sparsity of scRNAseq data, clustering is a common approach for aggregating cell information before attempting annotation. Clustering approaches leveraging the embeddings of LLMs, like GenePT, offer the possibility of providing additional context to overcome the challenge of sparsity. However, here both clustering normalized simulated data or GenePT transformed data were found to reduce GPT-4o annotation accuracy. Imperfect clustering, as exacerbated by the GenePT transformation, would be expected to reduce aggregate cell-type clarity rather than sharpen it, and this would reduce overall annotation accuracy. The performance of the GenePT transformation could also be improved by tinkering with the number of gene markers selected per cell. It should also be noted that for extending this approach to more realistic cases where the number of unique cell types in a dataset is not known, clustering techniques such as HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) and OPTICS (Ordering Points To Identify the Clustering Structure) could be used to allow the number of clusters to arise from the data itself.

It is worth mentioning that all the results above were obtained using a zero-shot approach, meaning that specific examples to guide the model were not provided in the prompt. While the goal here was to test the out-of-the-box capabilities of GPT‑4o, it is expected that performance can be significantly improved with additional information given to the model. Potential examples include providing a list of specific gene markers related to each cell type or fine-tuning with additional datasets containing the same cell types as the dataset being analyzed. With other LLM models, such as Google Gemini and DeepSeek, newer GPT versions, and agentic systems such as Biomni⁶, it will be interesting to see if these models can improve the zero-shot annotation of scRNAseq data.

Currently we are in the LLM era, trained on more data than we have ever had before and with significant applications to a variety of fields. As such, crucial biological research tasks such as scRNAseq cell type annotation are becoming increasingly accelerated, saving precious time and resources for researchers. With the current shift in computation from deterministic to probabilistic tools (generative AI), it is important to weigh any enhancements against the variability they incur. In some cases, such as creative endeavors, this variability may be beneficial. However, science already has a reproducibility crisis, and caution should be taken wherever adding additional variability is involved. With the aim of full automation, research into the interpretability and trustworthiness of these black box models will be important moving forward. Until then, human researchers must play a crucial role in validating LLM outputs for biological tasks like cell annotation.

Other Tools (scGPT, Biomni)

There are countless tools that can be used to annotate scRNAseq data. Two other modern tools in this blog are discussed below: scGPT and Biomni.

scGPT⁷ is based on a generative pretrained transformer trained on a repository of over 33 million cells. According to the paper, “to fine-tune the pretrained scGPT for cell type annotation, a neural network classifier takes the scGPT transformer output cell embedding as input and outputs categorical predictions for cell types”. A model fine-tuned on a partition of healthy human immune cells and evaluated demonstrated strong alignment with the cell type annotations and achieved 0.85 accuracy. For this simple study, an attempt was made to extract cell annotations using scGPTHub, a tool that eliminates the need for GPU investments and provides an analysis report after submitting a dataset to the pre-trained model. For the dataset used here, multiple errors were encountered when waiting for a result, and a GitHub Issues was submitted in the scGPT repo. The Issues page does not appear very active and may take a while to be resolved.

Biomni⁶ is a “general-purpose biomedical AI agent designed to autonomously execute a wide spectrum of research tasks across diverse biomedical subfields”. According to the authors of the paper, Biomni was evaluated on “eight new biomedical benchmarks spanning genetics, genomics, microbiology, pharmacology, and clinical medicine” with scRNAseq cell annotation included in these tasks. “Across all tasks, Biomni outperformed the base LLM by an average relative performance gain of 402.3%, the coding agent by 43.0%”. This suggests that the limitations of LLM-led scRNAseq annotation may be reduced with a biomedically specialized agentic approach.

Acknowledgement

I would like to thank Ian Newman and the Su Lab at Scripps Research for their guidance and funding for this study.

Methods

Note: The main sections in this blog were completed before the release of GPT-5. The newer model was used only for cell name resolution.

SPARSim

Both original and simulated datasets had 2638 cells and 13714 genes with ground-truth cell-types covering “B”, “CD14+ Mono”, “CD8 T”, “DC”, “FCGR3A+ Mono”, “Memory CD4 T”, “NK”, “Naive CD4 T”, and “Platelet” types.

For the SPARSim code, an R package used for simulating scRNAseq count matrices:

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/simulation.R

Functions

Helper functions were used in the study for the different cell annotation tools:

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/functions.py

Methods of Name Resolution

Code used for cell name resolution methods:

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/cell_matching.py

The success of the cell annotation tools was measured by the accuracy of output annotations compared to ground truth. Initially, this comparison was evaluated as direct matches (== in python), incremented by 1 if correct, and divided by the total number of cells. An issue with this exact match approach was that some of the labels outputted were synonyms or subtypes of the ground truth (e.g., CD4+ T vs. Naive CD4 T), but were scored as incorrect due to not being an exact match. To account for these cases, 4 alternative approaches were evaluated: soft match, SapBERT, Translator, GPT‑4o resolver.

Soft match was a generous approach that simply checked if the tool’s predicted annotation was contained within the true annotation string (”in” operator in python). This approach struggled in cases where an annotated cell was contained on the character level, but was unrelated to the ground truth cell. An edge case, such as “T” being in the cell “Platelet”, was counted as correct by the soft match approach when it should have been counted as incorrectly annotated.

The SapBERT approach evaluated semantic similarity between known and predicted annotations utilizing SapBERT, a domain-specific language representation model pre-trained on large-scale biomedical data. Two vector embeddings, one for the original cell name and the other for the predicted cell name, were extracted from SapBERT. Then, cosine similarity was calculated and values above the threshold of 0.8 or 0.9 were labeled as matches.

Supplementary Figure 7: Visualization of SapBERT vectors and similarity scores.

The Name Resolver API by Translator, is a service that takes lexical strings and attempts to map them to identifiers from a vocabulary or ontology. The “GET /lookup” endpoint, with parameters “autocomplete” = “false” and “biolink_type” = “biolink:Cell” was used to indicate that the input string is a complete phrase and the matched terms should only be from the biolink:Cell ontology. The GET /lookup endpoint allows the user to extract similar cell type names for the given cell. The highest scoring resolution was used for match scoring.

The final approach involved using an LLM to resolve cell names. OpenAI’s GPT‑5 (gpt-5–2025–08–07) was asked to compare pairs of cells, original cell name and outputted cell name, and give a confidence score between 0 and 5. Any score equal to or greater than 3 was counted as correct. OpenAI’s GPT-4o (gpt‑4o-2024–08–06) was also used here to test the capability of the model, but after manual inspection of cell pairs, the model seemed to count incorrect subtypes correctly. For example, “B” was counted as a subtype of “Naive B” rather than the other way around. The following prompt was used in the OpenAI API call:

“You are a biological domain expert. Compare the two cell types: {original_cell} vs. {output_cell}. Return only an integer 0–5 using the following rules: 1. If {output_cell} is a umbrella term (e.g., PBMC, leukocytes, splenocytes), then cap the score at 2 regardless of hierarchy. 2. Otherwise: 5 if identical or synonyms; 3–4 if {original_cell} is a subtype exactly 1–2 levels below {output_cell}; 1–2 if {original_cell} is a subtype 3+ levels below {output_cell}; 0 if {output_cell} is a subtype of {original_cell} or unrelated;”

The outputs stored from the LLM outputs can be found below:

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/data/gpt_resolver_df.csv

Manual review revealed that the GPT-5 method, utilizing the OpenAI API, performed best at limiting false negatives (counting a failure when the name is actually a subtype or a synonym). For example, this method was able to resolve subtype names such as “Dendritic” to “DC” and “FCGR3A+ Mono” to “Monocyte”. “Exact” and “Soft” methods were too strict and would count subtypes as failures. Translator was not as effective as GPT-5 in minimizing false negatives. SapBert, an embedding model trained on biological data, also misses cases such as these by assigning low similarity scores. It is essential to note that, regardless of the resolution method, false negatives still occur; however, the GPT-5 method reduced them most significantly.

To provide a more thorough example, both correctly and incorrectly labelled data for the SapBERT, Name Resolution, and GPT‑5 methods are displayed below. Each key is the name of the original cell, and its value is a dictionary of incorrect cell names outputted by the tool.

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/data/gpt_resolver_example.txt

scType Method

Both original and simulated data were used to get the accuracy of the labels from this tool.

Code used to get the ScType annotations given a .csv file of gene rows and cell columns:

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/scType_annotation.R

GPT‑4o Method

To discover the number of gene markers that yields the best performance, sets of 10, 20, 50, 100, 500, or All gene markers were utilized, where “All” includes every gene expression for the cell. 1000 out of the 2638 total cells in the dataset were sampled for the “All” gene markers due to file size analysis restrictions for GPT‑4o.

For both the original and simulated data, the following prompt was used along with an uploaded .txt file where each row represented the gene markers for that cell. Note that N is the total number of cells in the dataset:

“Identify cell types of Peripheral Blood Mononuclear Cells (PBMC) using the following markers. Identify one cell type for each row (there should be N total). Only provide the cell type name and all cells must be identified and none can be labeled “Unknown”. Store them in a downloadable .txt file.”

This prompt takes inspiration from Hou & Ji’s³ paper with some key differences. When running the original prompt in the paper, around a quarter of the returned cell types were annotated as “Unknown”. By adding the guardrail of not allowing GPT‑4o to return “Unknown” labels, the number of “Unknown”s returned was significantly reduced. Additionally, the instruction to return results in .txt file format made it easier to manage them. It is interesting to note that GPT‑4o ignores the instruction of not assigning “Unknown” for certain gene marker datasets. The reason why this may occur was further discussed in the “Discussion” section.

The “write_gpt_file” function in “functions.py” was used to create the input .txt file. “num_unique_cells” was the number of unique cell types in the dataset, the assert statement was used to make sure all cell types are represented for the “All” gene maker analysis where only a sample is used, and “trunc_index” was the number of marker genes to use for each cell. The file was attached with the prompt above to get the cell annotation labels. Below is what the first 3 rows of the file looked like for the top 10 gene marker .txt file:

B2M MALAT1 TMSB4X RPS2 RPL11 RPL13A RPS12 RPL27A RPS6 RPL10 MALAT1 RPS2 RPL10 RPL13A RPS6 B2M RPS8 TMSB4X RPS12 RPS18 MALAT1 TMSB4X B2M RPL10 RPL13A RPS18 RPS27A RPL19 RPS8 JUNB

The updated prompt below was used for the word bank scenario, where N is the number of cells in the dataset:

“Identify cell types of Peripheral Blood Mononuclear Cells (PBMC) using the following markers. Identify one cell type for each row (there should be N total). You must only choose from one of the following cell types and can’t assign “Unknown”: B, CD14+ Mono, CD8 T, DC, FCGR3A+ Mono, Memory CD4 T, NK, Naive CD4 T, Platelet. Store them in a downloadable .txt file.”

GenePT Method

The process to create the GenePT-s embeddings, a specific type of GenePT embedding used, involved creating a textual representation of the cells in the dataset by taking the top N marker genes for each cell, then passing this into OpenAI’s model (text-embedding-ada-002) to get the cell embeddings. 1000 gene markers, an upper end gene marker number from the GPT‑4o results, were used to balance between choosing an N that is not overfit to a particular result, but also falls in a range of values that did work well for the per-cell GPT‑4o tool. This N value was also used in this GenePT notebook.

The “get_genept_embeddings” function, mentioned in the “Functions” section, extracted these embeddings given a dataset of cells as rows and genes as columns. To get a general understanding of the performance of the GenePT-s embeddings on cell annotation, the Adjusted Rand Index (ARI) was used on both the simulated data and GenePT-s embeddings after running the K-means and Gaussian Mixture Model (GMM) algorithms. Perfectly agreeing assignments have a score of 1, random a score of 0, and complete disagreement has a score of -1.

Both PCA and UMAP were used to reduce the high-dimensional data in order to prepare it for the K-means and GMM clustering. Here, UMAP was used on top of PCA for better UMAP performance as PCA provides a more stable and de-noised input for UMAP. The “n_components” argument for PCA was determined by plotting an elbow plot of explained variance and looping a list of “n_components” based on the values around the elbow point.

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/genePT_ARI.py

To get the ARI values for the GenePT-s embeddings, the same code was used after replacing the “file_path”.

Taking the best dimensionality reduction and clustering algorithms from for each normalized simulated and GenePT-s data (Supplementary Table 8A and 8B), the top 1000 gene markers between the cells in each cluster were chosen. This was run 3 more times for each dataset with the following prompt, where N is the number of cell types in the dataset:

“Identify cell types of Peripheral Blood Mononuclear Cells (PBMC) using the following markers. Identify one cell type for each row (there should be N total). Only provide the cell type name and all cells must be identified and none can be labeled “Unknown”. Store them in a downloadable .txt file.”

The code used (GMM + UMAP is used for the example):

https://github.com/ChanyoungPark07/Cell_Annotation/blob/main/genePT_GPT4_annotation.py

Supplementary Figures and Tables

Below is the table format for the figures used in prior sections as well as additional figures. The table number corresponds to the figure number used.

Note: The 2 tables above were the result of the per-cell GPT-4o tool mentioned in the “Zero-shot LLM: GPT-4o” section. The GPT-5 row in the tables were taken with the other 3 runs and averaged to produce Tables 2A and 2B below.

Supplementary Table 2A: GPT-4o can outperform ScType at annotating original PBMC data.

Supplementary Table 2B: GPT-4o generally outperforms ScType at annotating simulated PBMC data.

Since there are cell class imbalances within the dataset, the F1-scores were also calculated. F1-score is an important metric for imbalanced datasets as absolute accuracy does not tell the whole story of a model’s performance. For example, in the hypothetical case of there being 100 cells with 90 part of type 1 and 10 in the other, if the model predicts type 1 for all the labels even though it misses the labels from the other type, the absolute accuracy would still be a high 90%. The F1-score would be significantly lower, as the model misses the entire type. As such, it can provide a more nuanced representation of performance.

The average F1-scores showed a similar pattern to the absolute accuracies, where the intermediate marker genes performed better than the extreme ends. GPT-4o was capable of outperforming ScType for intermediate (100–500) input marker genes, while more extreme inputs did not always outperform ScType (Figures 2C and 2D).

Supplementary Figure 2C: GPT-4o can outperform ScType at annotating original PBMC data. Bar heights correspond to average values and error bars are standard deviations.

Supplementary Figure 2D: GPT-4o generally outperforms ScType at annotating simulated PBMC data. Bar heights correspond to average values and error bars are standard deviations.

Supplementary Table 2C: GPT-4o can outperform ScType at annotating original PBMC data.

Supplementary Table 2D: GPT-4o generally outperforms ScType at annotating simulated PBMC data.

Supplementary Table 3A/B: Providing GPT-4o with a word bank tends not to enhance the annotation accuracy of original and simulated PBMC data but can still outperform ScType.

Supplementary Table 8A: Normalized simulated ARI results used in the GenePT section.

Supplementary Table 8B: GenePT-s embedding ARI results used in the GenePT section.

References

1. Ianevski, A. et al. (2022) Fully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data. Nature Communications, 13.
2. Baruzzo, G. et al. (2019) Sparsim single cell: A count data simulator for scRNA-Seq Data. Bioinformatics, 36, 1468–1475.
3. Hou, W. and Ji,Z. (2024) Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis. Nature Methods, 21, 1462–1465.
4. Chen,Y. and Zou,J. (2024) Simple and effective embedding model for single-cell biology built from ChatGPT. Nature Biomedical Engineering, 9, 483–493.
5. Levy,M. et al. (2024) Same task, more tokens: The impact of input length on the reasoning performance of large language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 15339–15353.
6. Huang, K. et al. (2025) Biomni: A general-purpose biomedical AI agent.
7. Cui, H. et al. (2024) scGPT: toward building a foundation model for single-cell multi-omics using Generative AI. Nature Methods, 21, 1470–1480.