Online Hand Writing Recognition using VLMs

Photo by Steve John on Unsplash

Online handwriting recognition plays a crucial role in various applications, from digitizing handwritten notes to enabling gesture-based input on digital devices. Recent advancements in machine learning, particularly with models like transformers, have significantly enhanced the accuracy and efficiency of this technology. In this article, we delve into key findings and techniques highlighted in recent research, shedding light on how different factors impact the recognition of online handwriting.

Impact of Time Sampling on Ink Representation

Figure-6

One of the critical factors influencing the accuracy of online handwriting recognition is time sampling delta. As illustrated in Figure 6, the choice of time intervals (delta) significantly affects the representation of handwriting ink. In experiments using a fixed 20ms delta across datasets, researchers observed that larger deltas result in fewer points being captured in the ink trajectory. While this reduces computational load, it also risks missing important details in handwriting, potentially impacting recognition accuracy. Thus, finer time sampling is crucial for preserving intricate writing nuances.

Training Strategies: Fine-tuning on Mixed Datasets

Table-10

Table 10 presents insightful comparisons between fine-tuning models on individual datasets versus a mix of datasets comprising Math Writing, Deep Writing, and VNOnDB. The study reveals varying impacts on different models:

• CTC Transformer (trained on public dataset of 100k steps to advance hand writing recognition): Demonstrated no improvement when trained on a mix of datasets, indicating that this model type might not benefit from combined training strategies.

PaLI (Parallel Latent Encodings): - PaLI is a model designed for handwriting recognition tasks, utilizing parallel latent encodings to process handwriting data efficiently. It aims to improve recognition accuracy by leveraging latent representations of input sequences. PaLi is both Encoder and Decoder.

***PaLM-E (Path-Level Modeling with Encoder):

• *PaLM-E is a model framework specifically tailored for handling online handwriting recognition tasks. It emphasizes path-level modeling through an encoder architecture to capture intricate details of handwriting strokes effectively. PaLi is mainly Encoder.
• PaLI utilizes latent encodings to process input sequences, making it both an encoder and decoder in the context of transforming raw handwriting data into meaningful representations for recognition tasks.
• PaLM-E focuses on path-level modeling through its encoder architecture, responsible for converting raw handwriting stroke data into latent representations. It doesn’t involve decoding the data back into its original form but rather focuses on feature extraction and representation learning.

Leveraging Time and Distance Information in Image Rendering

Figure-7

In Figure-7, an intriguing example highlights the importance of capturing the order of writing in online handwriting recognition. The illustration depicts an ambiguous initial letter scenario where the correct prediction shifts from “R” to “K” when additional time and distance information is considered in the image. This underscores the role of temporal sequencing and spatial context in enhancing the accuracy of handwriting recognition models.

Baseline Architecture: Insights into Transformer Models

Table-11

Table 11 outlines the baseline architecture of the CTC transformer used in the study for each dataset:

• Math Writing: This dataset employs a model with 35 million parameters, featuring 11 layers, 512 embedding size, 8 attention heads, and specific activation and dropout settings tailored for optimal performance.
• VNOnDB and Deep Writing: These datasets utilize models with varying parameters and architectural configurations, reflecting the nuanced requirements of different handwriting recognition tasks.

The baseline model integrates transformer encoders with CTC loss, leveraging Bezier curve encoding of raw ink points. This approach streamlines preprocessing while optimizing model training for diverse handwriting datasets.

Tokenization Techniques: Coordinate vs. Histogram

Table-12

Table 12 compares two tokenization methods employed in the study:

• Coordinate Tokenizer: Utilizes a resizing approach with a vocabulary of 900 tokens and assigns 2 tokens per ink point, emphasizing efficient representation of handwriting trajectories.
• Histogram Tokenizer: Implements a non-resizing strategy, utilizing polar forms of offsets to achieve a vocabulary of 12,000 tokens with 1 token per ink point. This method accommodates detailed capturing of handwriting nuances, particularly beneficial for complex datasets.

The histogram tokenizer’s adaptive binning approach enhances granularity, especially around zero angles and distances, thereby optimizing recognition accuracy and resource efficiency.

Specialized Vocabulary for Math Writing

A specialized vocabulary tailored for the Math Writing dataset includes 142 LaTeX symbols and 87 English characters with punctuation marks. This adaptation acknowledges the diverse representation of mathematical symbols, ensuring comprehensive coverage and accuracy in recognizing complex mathematical notations.

Conclusion

Advancements in online handwriting recognition underscore the interplay between model architecture, training strategies, and tokenization techniques. As highlighted by recent research findings, optimizing time sampling, leveraging mixed dataset training selectively, and employing sophisticated tokenization methods are pivotal in enhancing the accuracy and efficiency of handwriting recognition models. These insights not only contribute to advancing current technologies but also pave the way for future innovations in digital handwriting applications across various domains.

By understanding these nuanced aspects of online handwriting recognition, researchers and developers can further refine algorithms and models, ultimately improving user experience and expanding the utility of digital handwriting in diverse technological landscapes.

Aimanzahid Suresh kumar