Handwritten Text Synthesis and Recognition

Can an AI learn to write before it reads?

Escola Politécnica de Pernambuco - Universidade de Pernambuco - Recife, Brasil.

After several years and gray hairs, a lot has happened. I completed my master's degree, my doctorate, got married, moved houses, and relocated to a new country. Today, I'm happy to share that I have finally published my final PhD contribution.

Over the years, I often imagined when and how this post would be written. I questioned if the creative blocks and the lines of code would ever be made public. In this text, I will share the journey of this doctoral project, detailing its stages, core ideas, and reflections. While looking back at the major milestones, I will also dive into the technical aspects.

So, here it is. The paper titled “HTSR-Pollen: Handwritten Text Synthesis and Recognition System to Overcome Data Scarcity” presents an overall view of this project. For those interested, here are the links to the paper and to the repository, which is the same repository from my first post:

• Paper: *HTSR-Pollen: Handwritten Text Synthesis and Recognition System to Overcome Data Scarcity*
• Code: *https://github.com/arthurflor23/handwritten-text-recognition*

The project aimed to develop a method for generating synthetic handwriting images to use during the training of an offline Handwritten Text Recognition (HTR) system, in order to improve its recognition accuracy. I can say that the final result went well beyond that initial scope. So, grab a coffee and enjoy the read.

One of the first stages of the doctoral program was to conduct and publish a systematic literature review in my research area. It was the first time I had worked with a well-defined search protocol, and it was a lot of work.

• Paper: *Data Augmentation for Offline Handwritten Text Recognition: A Systematic Literature Review*

The review aimed to identify related works focused on data augmentation methods applied to handwritten text recognition in data-scarce scenarios. It is worth mentioning that the scope was intentionally strict, since we focused only on data augmentation applied to text recognition within a low-volume data context. Everything outside that scope was excluded.

In this context, we used the following scientific databases: ACM Digital Library, IEEE Digital Library, Science Direct, Scopus, and Springer. The timeframe covered from January 2012 to December 2024. Initially, the window was set for 10 years; however, as the project progressed, we had to update it several times, extending the range to include 2023 and 2024. I believe it ended up being much more work than I expected.

Following the defined evaluation criteria and a peer-review process, we selected 43 works from thousands of papers. From this, a few key perceptions stood out. We observed a clear publication trend related to the theme, starting in 2016 and continuing through 2024, the volume grew from 1 to 10 publications per year. I believe this number is even higher today.

Distribution of the Selected Papers by Publication Year

We also noticed that most papers focused more on word images than on other recognition levels, such as characters, lines, or paragraphs. In general, the works tend to use word-level images to generate new samples, which can later be expanded to lines and then paragraphs.

Proportion of Recognition Levels Used by the Papers

Among the datasets used, researchers prioritize IAM and RIMES, as they are the most established and serve as the primary benchmarks for text recognition. Another frequently used base is the CVL-Database, which is common in studies focused on writer identification. Other datasets appear less often, usually in research involving specific languages.

Proportion of Datasets Used by the Papers

Regarding approaches, many of these works use Digital Image Processing (DIP) alongside the optical model, as it is a flexible and low-cost technique, though limited by text structure. Transfer Learning (TL) is also explored, but on a smaller scale, facing challenges when adapting between datasets with very different patterns. Finally, the use of deep learning for Data Synthesis (DS) has seen growing application in recent years.

Proportion of Approaches Used by the Papers

Within the context of deep learning for Data Synthesis, the main approaches included Transformers, Generative Adversarial Networks (GANs), and Diffusion models. It was interesting to see the early papers applying Transformers, then gradually moving toward GANs, and finally toward Diffusion. At that time, only a few works used Diffusion, but I believe its use has increased in recent years.

By the end of the systematic review, I had a clearer picture of how the literature on handwriting synthesis was evolving. The field started with word generation and then moved toward line-level generation, varying textual content and applying methods to interpolate writing style, which I found very interesting.

Advances in Handwritten Text Synthesis

At that point, you might think “That's it, the path and solution are clear. Integrate a synthesis model with a recognition model and everything will work”. To be honest, I thought the same.

One of the main goals behind the proposed integration was for the synthesis model to generate images that were already ready for the recognition stage, eliminating the need for post-processing or intermediate save-and-load cycles. However, a deeper look into the available papers and code revealed a mismatch with those requirements.

The synthesis models analyzed were built to the word level, which required post-processing to compose lines and paragraphs. The images also went through a preprocessing step for width normalization, since those models were trained to generate images of a fixed-size per character.

Furthermore, training with line images became impractical because of the GPU memory usage. While word images of size 32 × 256 pixels were already enough to fully occupy a 24 GB GPU with a batch size of 4 samples, processing line images of size 128 × 1024 pixels with a batch size of 8 represented an even greater technical challenge for the available hardware.

It hit me like a bucket of cold water. The works that previously seemed like a solution became only a theoretical foundation. Instead of using synthesis models from the literature, I decided to develop my own. The new goal was to create a synthesis model according to my needs: a compact architecture, able to operate directly at the text-line level, without pre- or post-processing. All this in a 24 GB GPU.

BRESSAY: A Brazilian Portuguese Dataset for Offline Handwritten Text Recognition

Along the way, I had an unexpected but very rewarding spin-off. During the systematic review, we realized that there was no Brazilian Portuguese dataset among the related works. At the same time, we had a collection of 1,000 essay images stored away, but without the proper labeling for recognition. That's when the opportunity arose.

I had the pleasure of working with an incredible research group, including students from several universities in Pernambuco (Brazil), to organize and label everything from scratch. The goal was to present a dataset as a paper at the International Conference on Document Analysis and Recognition (ICDAR 2024).

I admit that joining that project at that moment was difficult and risky. I was already halfway through the PhD timeline, still without concrete results from the synthesis model, and under a lot of pressure. Fortunately, with a strong team and a generous dose of exhaustion, we made it happen.

We even created our own annotation tool, and the effort resulted in two papers: one presenting BRESSAY and another describing the recognition competition we launched at the conference. It was hard, but it was worth it.

• Paper: *BRESSAY: A Brazilian Portuguese Dataset for Offline Handwritten Text Recognition*
• Paper: *ICDAR 2024 Competition on Handwritten Text Recognition in Brazilian Essays - BRESSAY*
• Dataset: *A Brazilian Portuguese Dataset for Offline Handwritten Text Recognition (BRESSAY)*

Academic performance in Brazil, especially through essay writing, can be a turning point in a student's life. The National High School Exam (Exame Nacional do Ensino Médio - ENEM) is a good example of this, serving as the main gateway to higher education. With the advancement of HTR systems, the potential to make the evaluation of these essays faster and more efficient is enormous, but the path is full of challenges. The nuances of handwriting, such as corrections, strike-throughs, and highly varied writing styles, make automatic recognition a very complex task.

The dataset, as well as the competition, focused on recognition at three levels (line, paragraph, and page), using 1,000 essay images from different authors. Working with this data brought real-world challenges, as the images have varying sizes and resolutions depending on their source. Another point was the labeling of errors, erasures, and out-of-position text. We decided to label everything using a special tagging system.

Examples of Handwritten Pages from BRESSAY

The experience of organizing a competition at an international conference was incredible. First, because you get to meet the authors you have spent years reading and citing. Second, because of the weight of the contribution: providing a Brazilian Portuguese dataset for the research area.

After months of balancing all these demands, the BRESSAY cycle came to an end. It was time to return to the PhD project and finally finish it. However, after so much effort, finish what exactly?

HTSR-Pollen: Handwritten Text Synthesis and Recognition System to Overcome Data Scarcity

With full focus on the synthesis model and the requirements defined, the development took a few months, but the learning lasted much longer. Today, part of that study is already consolidated in a paper, with a more polished and concise approach.

• Paper: *HTSR-Pollen: Handwritten Text Synthesis and Recognition System to Overcome Data Scarcity*

Along the way, I remember feeling stuck on concepts and code many times. Every time I finally unlocked one stage, another knot would appear. It's that feeling that the more you learn, the more you realize you know nothing. Each new discovery about a topic made me realize that my previous understanding was incomplete. It was a real emotional roller coaster.

Overall, the idea was consolidated into three stages within the same workflow: (i) Generative, where the synthesis model creates new images based on input features or random noise to increase diversity; (ii) Optical, responsible for extracting visual features from the synthetic images and decoding them into text; and (iii) Language, which acts on refining the final text output of the system.

HTSR-Pollen: Handwritten Text Synthesis and Recognition System

Reaching the final version of the synthesis model was, without a doubt, the greatest challenge. After a lot of trial and error, I identified some of the main reasons why synthesis models tend to be so expensive in this context:

• Redundancy and Scale: the use of large and generic deep learning models in parallel within the same pipeline. In many cases, these models are not specialized and end up introducing redundancy.
• Excessive Depth: models with too many layers and an too many number of filters for feature extraction at each level.
• Spatial Self-Attention: applying self-attention to spatial features consumes a lot of GPU memory due to the image size.
• Diffusion Models: besides being slow to train and infer, these models process the image at its full size throughout the entire generation, pushing GPU consumption to the limit.
• Flow Conflicts (Horizontal vs. Vertical): many recognition models transpose images to extract features, while synthesis models operate horizontally. Combining both approaches in the same training ends up duplicating content in memory.
• Processing Inefficiency: the need for pre- and post-processing to adjust both input and output. Since many models work with fixed widths per character, everything has to be readjusted so that the handwriting looks natural rather than a rigid grid.

That said, it wasn't all smooth sailing, and each topic required a technical choice. To overcome these bottlenecks, my decisions were:

• Redundancy and Scale: instead of using generic models, I developed a new and more efficient optical model (FlorV2). Also, I reused its compact encoder as the base for the other auxiliary models.
• Excessive Depth: I adopted a minimum of the minimum philosophy. I started with the smallest possible number of layers and filters, planning to increase complexity only if the results were unsatisfactory.
• Spatial Self-Attention: I replaced self-attention with a residual convolutional layer with gated blocks. This allowed me to highlight spatial features without the restrictive memory cost.
• Diffusion Models: even though they are the current trend, I opted for Conditional Generative Adversarial Networks (CGANs). They are lighter, faster, and delivered the performance I needed.
• Flow Conflicts (Horizontal vs. Vertical): I standardized the models to work with horizontal images only.
• Processing Inefficiency: I removed both pre- and post-processing. To make the handwriting look natural, I created a resizing layer inside the synthesis model itself. It uses a mask to guide the content resizing, avoiding the fixed character grid in the output images.

In addition, I changed the standard line size to 64 × 1024 pixels instead of the traditional 128 × 1024. It may seem like a small detail, but together with the other changes, it was what made it possible to reach the performance I was looking for. The result was the training of full text lines with batch sizes of 8 samples on a single 24 GB GPU, using around 2× fewer trainable parameters than GAN-based synthesis models and even 10× fewer than Diffusion-based models.

Overview of the Proposed Generative Module

Following the GAN concept, I used a Discriminator model to distinguish between real and synthetic images, and a Generator based on the BigGAN approach. The idea here is to start the generation from a small patch and, at each convolutional block, gradually expand it until the target image size.

Overview of the Proposed Handwritten Text Generator and Discriminator Models

The Generative module, however, doesn't work alone; it requires a few auxiliary models during training. The Handwriting Recognition model (HTR) is responsible for transcribing the text, while the Writer Identification model (WID) uses the style extractor to capture handwriting style features. These two models are essential, because the HTR guides the synthesis toward generating readable text, while the WID guides the preservation of cursive style.

Finally, there is the Style Encoder. It acts as a bridge, taking the style features extracted from the images and projecting them into a latent space that follows a normal distribution, inspired by the concepts of Variational Autoencoders (VAEs). This is a key point, because it allows the generator to also produce cursive styles randomly, creating new handwriting patterns without depending on a reference image.

Overview of the Proposed Handwritten Text Recognition, Writer Identification, and Style Encoder Models

For the training strategy, I decided to simplify what the literature often treats in five stages into only two. Instead of training each model separately, all models are trained together from scratch. To keep this combination of influences from becoming a mess, I used a dynamic weight balancing strategy: the content and cursive-style reconstruction losses adjust according to the training gradient, while recognition and writer identification follow an uncertainty-based weighting approach. In other words, at the beginning of training, when the model is still lost, these tasks carry less weight, and as learning progresses, they begin to guide the synthesis more strongly. This fine adjustment is what helps maintain stability without requiring an eternity of processing time.

In case you haven't realized it, we are still in the synthesis module. Once trained, we can integrate the generator model only into the optical module and use real-time synthesis to train new optical models.

In the optical module, the roles are reversed. If before the recognition model was only an auxiliary component helping the synthesis process, here it becomes the protagonist. The full focus is to train the optical model from scratch to predict text as accurately as possible. The interesting part is that the synthesis generator enters as a training partner, generating diverse handwriting samples to feed the optical model. This provides a good amount of flexibility, since we can combine synthetic data with traditional data augmentation, allowing the recognition model to learn from a variety of writing styles that would be very difficult to obtain from real data alone.

Overview of the Proposed Optical Module

To complete the workflow, there is the Language module, which works as a spelling reviewer for the final text. Instead of training another model from scratch or relying on external dictionaries, I used a GPT model as a post-processing tool. The idea is simple, the recognized text is sent to the model, which receives a restricted instruction to correct only obvious mistakes.

With the workflow defined, it was time to plan the experiments, and I confess I might have gone a bit overboard here. I decided to test the proposal across different datasets, training variations, optical models, and methods for data scarcity, such as data augmentation and transfer learning. The result? More than 35,000 GPU hours and almost two years of constant concern. It was a real stress test.

Overview of the Experimental Design

Regarding the results, I first performed a grid search, a brute-force exploration, across parameters and transformation methods for the data augmentation pipeline. What surprised me the most was the impact of perspective variations. I had never explored this method deeply in previous projects, and the result was impressive. By itself, it reduced the Character Error Rate (CER) by 16.1% and the Word Error Rate (WER) by 13.2%.

Average Error Reduction by Augmentation Method (Higher Is Better)

At the end of this exploration, we reached an ideal combination of methods, which guaranteed a total reduction of 22.9% in CER and 17.7% in WER. The recipe involved elastic transformations, perspective, shearing, scaling, and rotation, each with its parameters. Since the project is a few years old and the code has evolved, the exact application has changed, but the lesson remains, the benefit of these transformations holds up, even with minor variations in parameter preferences.

Moving on to the synthetic images and results by dataset. One of the major tests for the project was the BRESSAY dataset. The challenge here was the noise of the real images (typical of essay sheets) and the labels for strike-throughs. Even by focusing only on the text and ignoring all strike-through labels, the result was still positive.

Examples of Generated Images and Comparison with Related Works on BRESSAY (Lower Is Better)

Another challenge was the CVL-Database, which tested the optical models against diverse and varied handwriting styles. The synthesis generator felt the weight of that diversity when creating new samples, but even so, the use of synthetic data during training still helped improve accuracy.

Examples of Generated Images and Comparison with Related Works on CVL-Database (Lower Is Better)

For IAM, which is the most referenced dataset in the field, the challenge was to reproduce punctuation and spacing patterns. Transfer learning once again stood out, and at this point, you may be wondering “Will transfer learning alone yield the best results?” Not always. When only a minimal amount of training data is available, transfer learning-based models may show less stable convergence. In this scenario, data synthesis proved to be a valuable complement, leading to more consistent results.

Examples of Generated Images and Comparison with Related Works on IAM (Lower Is Better)

In the case of the Parzival dataset, the main challenge was dealing with a medieval writing style. Since it is a historical corpus in archaic German, it was necessary to remap some characters to support both image generation and recognition. For the generator, the difficulty was reproducing uncommon characters while preserving of the original writer style. In addition, because of the vocabulary and older spelling conventions, the Language module was not applied in this context.

Examples of Generated Images and Comparison with Related Works on Parzival (Lower Is Better)

The RIMES dataset presents a large variety of French handwriting styles, but with relatively clean and uniform image quality. This consistency made the generation of synthetic data easier, which directly contributed to recognition performance. As the second most commonly used dataset in handwritten text recognition research, RIMES served as an important benchmark for validating the system.

Examples of Generated Images and Comparison with Related Works on RIMES (Lower Is Better)

For the Saint Gall dataset, the context involved Latin texts with uniform handwriting and a limited character set, without punctuation or accents. This structural simplicity made it easier for the generator to create synthetic images with consistent variations. However, the archaic language still posed a challenge when generating unseen characters while following the same cursive style as the original images.

Examples of Generated Images and Comparison with Related Works on Saint Gall (Lower Is Better)

To close the dataset analysis, the Washington dataset presented a particular challenge because it had the smallest amount of data among all the evaluated datasets. This kind of scarcity is a natural obstacle for deep learning models, which required a stronger reliance on data augmentation and data synthesis techniques to obtain consistent improvements.

Examples of Generated Images and Comparison with Related Works on Washington (Lower Is Better)

Looking at everything that was tested, it is clear that each technique plays a specific role depending on the scenario. Data augmentation helped where a reasonable volume of data already existed by providing the necessary visual variability, though this approach requires adjustments for each new context. On the other hand, transfer learning showed improvements by adapting knowledge from pre-trained models to smaller databases. However, this technique faced convergence difficulties in cases of extreme data scarcity, which is a common challenge in deeper architectures that still require a minimum amount of information to learn with stability.

Data synthesis was a positive highlight, especially with historical manuscripts. By generating additional samples, it was possible to stabilize training from the start and improve generalization where real data was limited. It is true that some generated images show certain visual inconsistencies, but this lack of perfection actually helped the recognition model become more resilient, forcing it to focus on the textual content. In the end, line-level synthesis proved to be not just a valid strategy, but a competitive approach for bypassing the lack of real-world examples.

Finally, it is worth mentioning that the generator model was trained for only 50 epochs due to time constraints, suggesting that a longer exploration could lead to even better results. In fact, since the conclusion of the project, several adjustments and improvements have been implemented in both the synthesis and recognition modules. Thus, although the current results already surpass those documented here, I chose to keep the original records to preserve the original analysis. Enough experiments!

So, can an AI learn to write before it reads? Well, not exactly. In practice, the recognition model (reading) must first learn enough in order to guide the generator model (writing), even though both can evolve together.

Closing Thoughts

If you have followed along this far, whether by reading every technical detail or jumping straight to the conclusions, my sincere thanks. To be honest, there were many moments when I didn't believe this material would ever see the light of day. My last major public contribution was in 2019, about my master's project that began back in 2018. Today, I'm finally publishing this work, which started in 2021, and closing a cycle that demanded as much as any other great challenge I have ever faced.

Looking back, this project was built through many important moments and creative breakthroughs, sometimes with a level of tiredness that felt close to exhaustion. Every small win in the code happened alongside real life. There were hard moments, excitement, boredom, and satisfaction, all mixed into the journey. A lot has happened since the first line of code, and each technical step forward also became a personal milestone.

None of this would have been possible without the support of the people who were with me. I'm deeply grateful to my wife, Sarah, for her support during all these years. To my advisors, Byron and Alejandro, who not only guided me, but also trusted me enough to follow these crazy ideas. Lastly, thank you to Cubricks; starting to work with them at the beginning of my doctorate gave me experience that went far beyond borders.