Generative AI Approach for De Novo Molecule Design: The Tech, The Frontiers and The Challenges.

Photo by julien Tromeur on Unsplash

Drug discovery has always been constrained by the fundamental challenge of an astronomically large chemical space.

With an estimated 10⁶⁰ possible drug-like molecules, traditional approaches, which is largely based on iteratively enumerating and screening existing libraries, a very expensive and rigorous process that barely even scratch the surface.

This is where generative AI is changing the game and reshaping drug research and development by drastically shortening early-stage research and development (R&D) timelines and cut costs by using machine learning (ML) and generative models to accelerate tasks, compared with traditional approaches long reliant on cumbersome trial-and-error.

What Is De Novo Molecule Design?

“De novo” literally means “from scratch”, and in the context of drug and molecule design, refers to the computational generation of entirely new chemical structures that can serve specific biological and physicochemical purposes.

Rather than asking “Which molecule fits this target?”, we ask: “What molecule should exist to solve this problem?”

Generative AI models help us answer this question by providing models such as the VAEs, GANs, diffusion models that are uniquely suited to explore this space.

Variational autoencoders (VAE) and generative adversarial networks (GAN) are AI models capable of generating new, unique molecules that looks similar to but fundamentally different from molecules provided in training data. Both networks use dual neural networks to accomplish this task with different approaches.

Let’s go through some of these Generative AI approaches and techniques.

Variational autoencoder Autoencoding is a machine learning concept where artificial neural networks are trained to reconstruct its own input, these neural networks are referred to as autoencoders.

A variational autoencoder is a type of autoencoder that uses variational inference to compress or encode data before accurately reconstructing or decoding it, retaining all of the most important features (variables) and components.

All autoencoders use a dueling neural network architecture to compress and decompress data. The key feature that makes a variational autoencoder different from other types is that it uses variational inference, a machine learning technique that uses optimization to create a complex probability field. The model uses this data to recreate a probabilistic approximation of the original molecule that retains the key variables but represents an entirely new entity.

This model, therefore, enables the smooth exploration of the vast chemical space.

How does a variational autoencoder work? A variational autoencoder contains two neural networks — an encoder and a decoder. The input goes to the encoder first, which identifies the data’s latent variables. Latent variables represent points of information that, while not directly observable, explain how the data distribution underlies its features. Next, the encoder within a VAE calculates the mean and variance of the data using a statistical distribution. This allows the AI to compress the data into a lower-dimensional space, retaining the most meaningful information and removing noise.

The compressed data then arrives at the bottleneck, which acts as the encoder’s last layer and the decoder’s first layer. The decoder uses Gaussian noise or the Gaussian distribution of the latent data, to reconstruct the data in a novel and unique way.

Generative adversarial network A generative adversarial network (GAN) is also an AI model that generates novel content from an input, but it operates differently from a VAE. Instead of encoding and decoding the input, a GAN contains dueling neural networks — a generator, and a discriminator — that work against each other to create a novel structure based on training data.

These two neural networks play different roles: the generator creates fake content, and the discriminator spots the fake content. A conditional GAN allows us to add conditions for the novel content the model produces.

How does a generative adversarial network work? After you provide a GAN model with a large amount of training data, the generator can create new content — a chemical structure in our context — that looks similar to its training data yet represents a new or unique piece of content. The discriminator will attempt to spot the generated or fake content. The generator will keep trying, learning to produce better representations. The discriminator will continue to reject the generator’s attempts, learning to become more accurate at spotting AI-generated content.

This gamified process continues back and forth until the generator can “fool” the discriminator, which is to say that the generator produces a piece of content convincing enough that the discriminator can’t distinguish the fake content from the real training content data. This “winning answer” becomes the output.

This technique enables us generate molecules with high novelty.

Diffusion Model Diffusion models follow a unique idea I find very interesting.

In simple but accurate term, a diffusion model is a probabilistic generative model that learns to create new data by reversing a gradual noise-adding process.

How does Diffusion Model work? The core idea is impressively simple; If we slowly destroy a molecule by adding noise until it becomes corrupted or random or invalid, if we can train a model to undo or reverse that destruction step by step, then we can generate new molecule from pure noise.

Now, what is this “noise”? In diffusion models, noise is random perturbations added to data in a controlled mathematical way. In other words, Noise is deliberate randomness added to destroy structure gradually, not arbitrarily.

Noise is not the same thing in every diffusion mode, if our data is for example atomic coordinates, noise would literally be small random shifts in atom positions. while in other models such as in Molecular Graph noise could mean:

Corrupting node features (atom types, charges) Corrupting edge features (bond types) Masking or perturbing adjacent atoms

Why do we add noise at all? Because generation becomes easier if we approach it as making something from a corrupted or invalid starting source and making it cleaner to get a valid molecule. This is much easier than trying to Generate a valid molecule from nothing.

Diffusion model function in two steps. First step is destroying structure slowly by adding noise and then learn how to reverse that destruction in such a way that we end up with similar but a novel molecule.

Language-based molecular design using SMILES or graphs Let’s talk about language-based molecular design, one of the most practical and currently deployed AI approaches in computational drug discovery.

What is language-based molecular design? Language-based molecular design treats molecules as sequences or structured “string” and applies Natural Language Processing (NLP) -style models, such as transformers, LSTMs, and foundation models to Learn chemical rules, generate new molecules, optimize molecular properties, and also predict activity or synthesizability.

The key idea is, if a model can learn grammar and semantics of human language, it can also learn the of chemistry.

There are several ways of representing molecular structures for Large Language Model to make sense of it, however, the two dominant molecular representation are the SMILES strings and Molecular graphs.

SMILES — Simplified Molecular Input Line Entry System an encoding technique that helps us encode a molecule as a string. Example:

Aspirin → CC(=O)OC1=CC=CC=C1C(=O)O

From a language model’s perspective, Atoms becomes tokens. Bonds, branches, rings have unique syntax and a valid molecule is essentially a grammatically correct sentence.

SMILES are powerful, but chemistry is not truly linear. That’s where Graph-Based Language Models come in. This method represents molecules such that it preserves important chemical attributes such as valency, connectivity and chemical constraint. In these models, Nodes are used to represent atoms, Edges represents bonds, Node features could include atomic number, charge, hybridization, while Edge features could represent bond order, aromaticity.

The Frontiers: From Models to Molecules to Clinic Artificial intelligence (AI) has progressed from experimental curiosity to clinical utility, with AI-designed therapeutics now in human trials across diverse therapeutic areas.

Impressive computational drug breakthroughs in pharma have come from a number of companies.

Insilico. Insilico Medicine exemplifies end-to-end AI-driven biotech, integrating AI at every stage from target discovery to clinical trial design.

The company’s approach is to significantly reduce time and cost in drug R&D by computationally exploring the biological and chemical space faster than traditional methods.

What makes this a real-life, operational use case (not just research hype) is that drug candidates discovered or designed by these AI engines have progressed through real animal studies and into human trials, a milestone only a few companies have achieved.

February 2021, Insilico Medicine announced the nomination of a preclinical candidate for a novel antifibrotic target discovered using AI.

The novel molecule designed by AI for the first time bridging biology, chemistry, and clinical trial outcome prediction with its “Pharma.AI” platform, and have reported that the first healthy volunteers have been dosed in a first-in-human (FIH) dose trial of ISM001–055 — a potentially first-in-class small molecule inhibitor of a novel biological target developed by Insilico Medicine for the treatment of idiopathic pulmonary fibrosis (IPF), an irreversible progressive orphan disease that affects an increasing number of people globally.

With no cure currently available This small molecule showed promising efficacy for IPF and a good safety profile that led to its nomination as a preclinical drug candidate with drug progressed from target discovery to phase I in 18 months.. The molecule as of January 2026, has now completed the Phase-IIa Clinical trial in China, and is still undergoing same trial in the USA.

In addition to ISM001–055, Insilico’s other clinical programs with molecules generated from Artificial Intelligence includes:

SM3312: immunomodulator for COVID-19/viral infections; have had phase I completed. ISM3091: USP1 inhibitor for cancer; got IND approval in 2023, and now entering phase I.

Exscientia. Exscientia is one of the leading technology companies turning generative AI and machine learning into practical drug discovery results by tightly integrating human medicinal chemistry expertise with advanced computational models.

It has pioneered what’s often described as the “Centaur Chemist” paradigm — pairing AI algorithmic design with human domain expertise to design better molecules faster.

This collaborative framework accelerates the discovery–make–test–learn loop that drives real-world drug discovery.

The platform uses deep learning models trained on vast chemical libraries and experimental data to propose new molecular structures that satisfy precise target profiles, including potency, selectivity, and absorption, distribution, metabolism, and excretion properties.

Exscientia’s first AI-designed molecule was created for Sumitomo Pharma to treat obsessive-compulsive disorder. It took just 12 months to discover and represents its first ever AI-designed drug candidate to enter clinical trials, and now has several molecules in trial phase including a next-generation mucosa-associated lymphoid tissue lymphoma translocation protein 1 (MALT1) inhibitor (EXS-73565) is also progressing through IND-enabling studies with encouraging preclinical data presented at the European Society for Molecular Oncology Congress in 2023.

Recursion. Recursion Pharmaceuticals has taken a distinct AI approach centered on phenomics, the large-scale mapping of cellular phenotypes (visual readouts of cell biology) to discover drug opportunities in an unbiased manner.

Instead of starting with a specific protein target, Recursion’s platform probes disease biology by systematically perturbing cell models and analyzing the results with computer vision and ML.

Recursion’s phenotype-first strategy has enabled both drug repurposing and novel drug discovery. A notable example is REC-994, a previously “shelved” compound identified by Recursion’s AI platform that reversed cellular defects in cerebral cavernous malformation, a rare neurovascular disease, and advanced to a phase II trial.

Notably, in November 2024, Recursion acquired Exscientia, bringing in Exscientia’s pipeline of AI-designed oncology compounds. By mid-2025, the combined company identity narrowed to 6 active development programs, 4 in oncology and 2 in rare diseases.

BenevolentAI. BenevolentAI has been a pioneer in applying AI to target identification and drug repurposing.

Its platform centers on a vast knowledge graph that integrates scientific literature, biomedical databases, omics data, and clinical information. By applying natural language processing and graph ML,

BenevolentAI’s system can propose novel links between genes, diseases, and compounds that might not be obvious to human researchers. The goal is to generate hypotheses for either new targets in disease or new uses for existing drugs.

A flagship success of this approach was the 2020 identification of baricitinib, which is an already-approved Janus Kinase inhibitor for rheumatoid arthritis, as a potential treatment for COVID-19. In this case, Benevolent’s algorithms flagged the potential of baricitinib from its knowledge graph in a matter of days. Baricitinib went on to show a significant mortality benefit in patients with COVID and received emergency use authorization, becoming one of the first AI-predicted therapies to reach patients.

This demonstrated the power of AI–driven repurposing, especially in an urgent scenario.

By 2022, it had advanced at least 2 novel small molecules into the clinic: BEN-8744, an oral PDE10 inhibitor for ulcerative colitis and BEN-2293, a topical pan-Trk inhibitor for atopic dermatitis (eczema), which unfortunately failed clinical trials and becoming the first high-profile failure of an AI-derived drug in clinical trials. After failing to show significant efficacy in during clinical trials.

The Reality Check: Challenges and Limitations in AI-Driven Drug Design Despite its transformative potential, the full integration of AI into practical drug discovery workflows is faced by several critical challenges. A major obstacle is the limited interpretability of many high-performing models, particularly the deep neural networks used for prediction or molecular generation.

Compounding the issue are persistent data quality and scarcity concerns, as public and proprietary datasets frequently contain noisy or sparse bioactivity labels, inconsistent assay protocols, and target annotation, all of which compromise model robustness and impair generalizability in multi-target settings.

Most public datasets (e.g., ChEMBL, BindingDB) are heavily biased toward well-studied targets, limiting the data available for novel or multi-target efforts.

Another practical concern is the synthetic feasibility of generated compounds. Many AI-generated molecules, while chemically valid, are synthetically infeasible due to unstable intermediates, overly complex substructures, or low-yield routes. This disconnect arises because most generative models prioritize idealized property optimization over real-world tractability.

To address this, retrosynthesis tools such as ASKCOS and AiZynthFinder, and reaction-based generation strategies, are being integrated into modern pipelines to enforce synthesis-aware design.

Final Thought. Generative AI is not a shortcut or guarantee to approved drugs, however, it is a force multiplier for scientific creativity.

The companies winning today are not asking, “Can AI design a drug?” but “How do we design better drugs with AI?”.