AI-Lab at Ruter/TET Digital 2024–2025

This article is the fourth part of a series detailing the work carried out at Ruter’s and TET Digital’s AI-lab. Participating in this year’s AI-lab were Bjarki Thor Norddahl, Leander S. Parton, Mads Saua Balto and Julie Nomeland.

Part 1: Customer inquiries, AI case handler and insights Part 2: Mobility and travel time predictions Part 3: RuterGPT

Training a Norwegian language Text-To-Speech model for use in public transport announcements

Each year, Ruter — the public transportation company in the greater Oslo region — hires a selection of students specializing in programming and AI to work on a three-month summer project, dubbed the AiLab. During the Ruter AiLab 2024–25, the summer project was to develop a high-quality Norwegian language Text-To-Speech-model. In order to achieve this, we also had to develop smaller side results, including a Norwegian grapheme-to-phoneme-model and a text normalizer, all of which we shall cover in this article.

Current State of Norwegian Language TTS

Text-to-speech (TTS) systems are invaluable for promoting accessibility and inclusivity in different domains, not least in conveying public information, such as in public transport systems. Where manual audio recording of text is not feasible, automatic transcription systems like TTS provide the tools to disseminate important information to travellers on public transport, in order to facilitate greater ease of use for these services — this could be through announcing train/bus stops, arriving/departing lines, or service disruption notices.

Despite the importance of these kinds of systems — in the public transport sector, as well as many other domains — the selection is scarce when it comes to Norwegian language TTS models. Among those with substantial quality, all are commercial and not open-source. Microsoft, Google, ElevenLabs and OpenAI all have Norwegian voices in their applications, but the models are closed-source and only available through subscription. This does not provide substantial flexibility or practicality for large-scale applications.

Foto: Ruter As / Redink, Thomas Haugersveen

Procuring training datasets

The TTS landscape is also shaped by what datasets are available to use for ground training or finetuning. Datasets for training of text-to-speech systems usually consist of annotated audio, that is, recordings of speech along with the transcript of what’s being said. Much time was devoted in our project to pull together as much freely licensed Norwegian language annotated audio as possible, as the quality of the finished product would depend heavily on the amount of data it would be trained on.

After uncovering, quality-checking and benchmarking several freely available datasets, we ended up using two datasets from the National Library of Norway, Nasjonalbiblioteket, called NST and NB Tale. As the Norwegian language consists of many dialects, which these datasets reflected, we had to select a subset of these datasets which were most similar to the dialect spoken in the capital region of Oslo. These subsets combined resulted in roughly 125 hours of annotated speech. Additionally, we had to provide an out-of-distribution dataset for model generalization, which merely had to be text-based, without corresponding audio. For this purpose, we chose the nbnn_language_detection dataset from NbAiLab.

Normalization and Phonemization

Normalization

Most text contains symbols, acronyms, punctuation and other special tokens that carry an implicit pronunciation, but which a computer system (such a TTS-model) cannot directly read. For instance, the sentence “On the 14 Feb. 2024, I woke up at 10 AM” is read aloud as “On the fourteenth of February twenty twenty-four, I woke up at ten ay em”. The first sentence has an implicit reading (that requires prior knowledge of English to deduce), while the latter sentence has an explicit reading — even someone who only knows the phonetic rules of English would be able to read it. It is naturally advantageous to transform text into this readable format before feeding it to the TTS model, a process we call text normalization.

However, text normalization is not at all straight-forward. Take the simple example of numbers. While at first one might think to just replace any number tokens into their written equivalent (“94” -> “ninety-four”), one will find that sometimes numbers denote ordinality rather than cardinality (“ninety-fourth”, rather than “ninety-four”). Even having some heuristic for this, cardinal numbers might be read in different ways in different contexts. The number 2024 is usually read as “twenty twenty-four” rather than “two thousand and twenty-four”, because it usually denotes a year — but not if you were talking about “2024 tonnes of gross import”. Also, long numbers are sometimes telephone numbers, whose digits are often read pairwise, and sometimes mere statistics, with the full number being read out loud. You’ll find similar rules, exceptions, and exceptions to exceptions for dates, timestamps, acronyms, fractions, special characters, and much more.

For normalizing Norwegian text, we could not find any premade open-source models available for us to use. We also decided against using LLMs or other AI solutions for normalizing, on the grounds of computation time and quality control. In the end, we decided to create a complex rule-based system to normalize Norwegian text, which we call TET-TextNormalizer.

We created a pipeline in which the text is transformed according to a set list of heuristics in a set order, which has been designed to maximize the probability of normalizing an arbitrary sentence correctly. While it is not foolproof, we find that the TET-TextNormalizer is a relatively robust tool for this purpose. The TET-TextNormalizer handles, among others, cardinal and ordinal numbers, addresses, dates, URLs, currencies, number ranges, phone numbers, and special characters.ndre og femtitre komma syttiåtte kvadratmeter, men ifølge reguleringsplanen fra nitten nittini må tomten deles i to, hver på syttiseks komma åttini kvadratmeter.

Example of TET-TextNormalizer in action:

Original: Huset på Storgata 42A har et areal på 153,78 m², men ifølge reguleringsplanen fra 1999 må tomten deles i to, hver på 76,89 m².

Transformed: Huset på Storgata førtito a har et areal på hundre og femtitre komma syttiåtte kvadratmeter, men ifølge reguleringsplanen fra nitten nittini må tomten deles i to, hver på syttiseks komma åttini kvadratmeter.

Phonemization

After the text has been normalized, it is still not ready to be fed to the TTS model directly. Just about all languages, Norwegian included, has a mismatch between the way words are written, and the way they are pronounced. As an example, the -gh in “cough” is not pronounced the same as the -gh in “through”. Likewise, in Norwegian, the k- in “kirke” is not pronounced the same as the k- in “kilt”. The letters we use in writing are called graphemes, while the sounds they represent are called phonemes.

Since the TTS-model is to learn to pronounce sounds based on text, it is disadvantageous to give it text where the symbols it reads are not consistent with the sounds it should make. It is therefore necessary to transform the text to a new set of symbols, where each symbol is always pronounced the same. Such a transformation we call phonemization — we define a new set of graphemes with a one-to-one mapping to the phonemes of the Norwegian language. The symbols we used for this were fetched from the International Phonetic Alphabet — with this transformation, the word “kirke” is written as “ɕˌiɾkə”, while the word “kilt” is written as “kˈilt” — the initial sound is now different between the two words, to represent different sounds.

An example of the normalization/phonemization pipeline

Using the NB Uttale dataset from Nasjonalbiblioteket, we employed a deep learning-based architecture, using an open-sourced available library from Spring Media to train a phonemization model. The final model, which we named TETPhonemizer, can handle five different dialects of Norwegian and is flexible enough to correctly phonemize tokens it has not seen during training.

Training TET-to-speech

The training architecture used for training our TTS model was StyleTTS 2, an open-source text-to-speech module which employs style diffusion and adversarial training to achieve a natural and flexible voice production. StyleTTS2 is the foundational model for several high-performing TTS models like Kokoro.

StyleTTS2 trains in two stages. It first pretrains the acoustic stack (text encoder, text-audio aligner trained with the Transferable Monotonic Aligner (TMA) objective, style encoder and decoder) on mel reconstruction, establishing reliable phoneme-to-time alignment and a usable prosody space. It then switches to joint end-to-end training that adds a text-conditioned diffusion model to sample the style vector and optimizes everything with adversarial losses, producing natural, controllable speech.

StyleTTS-2 requires three submodules, one of which — PL-BERT — is language specific and needed to be retrained before we could use it. In order to do this, we downloaded the Norwegian shard of the wikipedia dataset, consisting of more than 600 000 rows. In order to ease the processing load, we cleaned up the dataset and split the rows into smaller chunks, giving us a total dataset of 2.5 million rows. With this, we retrained a new PL-BERT for further use in our TTS training. The other two submodules were transferable from English to Norwegian, and thus needed no further work. We could then start the two-stage training of the main TTS system.

The training of the StyleTTS2 model took around two weeks of non-contiguous training using high-end resource infrastructure.

At around 41 epochs, the validation plot sees a sudden increase, as this is where we switched to stage two of the training — despite appearances, the loss in practice continues to fall throughout. In contrast with stage one, stage two only needs about three or four epoch’s worth of training, so we stopped it after epoch 44.

Results

The finished model, which we call TET-to-speech, has the functionality to clone voices, or use one of the existing voices in the training dataset, to read arbitrary Norwegian text with a natural and realistic pronunciation.

Text is first normalized using the TET-normalizer, and then transcribed phonemically using TET-phonemizer. The phonemes are fed into the TTS-model, and a voice from the training dataset is selected, which produces an audio file of the text being read in believable Norwegian in that voice. We specifically selected Oslo-proximate dialects when curating our training dataset, meaning that all outputted speech uses an Oslo dialect.

One can optionally also provide an audio file of real speech as input, which will result in the voice being clones, and the text thus being read in this voice. Having trained on multi-voice datasets, the transposition to a new voice is quite good, but depending on the source voice the outputted speech may be noticably different from the original voice. Due to an overrepresentation of female voices in our training datasets, we note that using female voices — both cloning and pre-existing — give better overall results.

Examples

Sentence: “Avgangen fra Ahus kl. 15:00 i retning Økern T er ca. 20 minutter forsinket. Forsinkelsen skyldes mye trafikk og kø.”

[embed]

Sentence: “Linje 17 stopper ikke mellom Heimdalsgata og Storgata på grunn av vedlikeholdsarbeid på infrastrukturen. Det kjøres buss for trikk mellom Heimdalsgata og Jernbanetorget.”

[embed]

Our model also makes occasional mistakes, especially with respect to intonation or rhythm.

In the following sentence, the intonation on “hytten min” sounds unnatural, with more stress on the word “min” than would normally be pronouned.

Sentence: “Hver sommer bader jeg i sjøen utenfor hytten min.”

[embed]

In the following sentence, the pronunciation of the word “fuglene” is unnatural for the Oslo-approximate dialect; a speaker would usually elide the “g” in the word.

Sentence: “Jeg så ut av vinduet på fuglene som flydde sørover i flukt fra vinteren.”

[embed]

However, our G2P-model and normalization tools are manually configurable and tweakable, with which we can continuously patch many of these mistakes, and which gives us more control over the generated output than we would get in commercially available models.

Next steps

Our model is so far only capable of providing speech in the Oslo/eastern Norwegian dialect. Our phonemization model can do five different broad dialects, so expanding the capabilities of the TTS-model itself would be dependent on a large enough annotated audio dataset in the different dialects. This is one area of improvement we might follow next.

Additionally, there’s always room for tweaking and tuning the current model — there are some words for which we can and do alter the phonemization for a more natural sounds, especially placenames which might not be in the training dataset for the phonemizer. As some words may be pronounced differently with the same spelling, we might also benefit from some additional contextual step, like part-of-speech tagging.

Furthermore, we believe that the horizon of Norwegian language TTS can be expanded by the creation of bigger and better datasets, as scarcity of high quality annotated Norwegian language audio was a major challenge in the development of TET-to-speech. We also see the potential of annotated datasets with emotions to prodcue even more flexible and natural speech.

Conclusion

In the Ruter/TET Digital AiLab summer project of 2024/2025, we developed a comprehensive Norwegian language text-to-speech system using the StyleTTS 2 architecture. We created a complete pipeline including TET-TextNormalizer for text preprocessing, TETPhonemizer for grapheme-to-phoneme conversion, and the main TET-to-speech model. Using datasets from Nasjonalbiblioteket, we trained the model non-consecutively for two weeks, resulting in a natural and flexible text-to-speech model for the Oslo dialect of Norwegian.

Acknowledgements

We extend our sincere gratitude to Alexander Amiri, Jawad Saleemi, Umair Mehmood Imam, Jonathan Feinberg and the rest of the AI/ML team at Tet Digital for their support and help during the development of this project.

Written by Bjarki Thor Norddahl, Leander S. Parton, Mads Saua Balto and Julie Nomeland