Measuring pronunciation variation

Voice Activity Detection is a technique used to automatically identify segments of an audio recording that contain speech and distinguish them from silence or background noise. To this aim we use Silero VAD, a neural-network-based Voice Activity Detection system that automatically identifies speech segments in audio recordings. Unlike traditional VAD methods that rely primarily on acoustic features such as signal energy, Silero VAD uses a pretrained deep learning model to distinguish speech from silence and background noise, making it particularly robust under challenging recording conditions. Silero VAD has been trained on speech data covering more than 6,000 languages. It is therefore suitable for detecting speech in both Frisian and Dutch recordings without requiring language-specific training. See https://github.com/snakers4/silero-vad.

Automatic Speech Recognition (ASR) is a technology that automatically converts spoken language in audio recordings into written text. In this step of the pipeline, the speech intervals identified in the preceding step and stored in a Praat TextGrid are orthographically transcribed using ASR. For this purpose, either Whisper or wav2vec models can be employed.

For Frisian, we use a pretrained wav2vec 2.0 model, specifically greenw0lf/wav2vec2-large-xls-r-1b-frisian-cv-8-large-train. For Dutch, we use the pretrained wav2vec 2.0 model jonatasgrosman/wav2vec2-xls-r-1b-dutch.
We also experimented with the multilingual Whisper large-v3-turbo model. Although this model supports a wide range of languages, Frisian is not among those officially supported. Dutch is supported, but we observed that when the model failed to recognize words correctly, it occasionally produced characters from other writing systems, such as Cyrillic. Such transcription errors are particularly problematic for the subsequent stage of our processing pipeline. For these reasons, we currently prefer the language-specific wav2vec models over Whisper.

To run these models, we use a GPU hosted by LeafCloud, a climate-friendly cloud computing provider.

Once the speech intervals have been detected and orthographically transcribed, we use forced alignment to segment the speech recordings into individual speech sounds. Forced alignment is a technique that automatically aligns a known phonetic transcription with its corresponding audio recording, determining the start and end times of individual words and speech sounds. The phonetic transcriptions required for this process are generated from the orthographic transcriptions produced by ASR, using G2P Frysk for Frisian and G2P Nederlands for Dutch.

For forced alignment, we use the Montreal Forced Aligner. MFA was selected because it can be trained for any language provided that suitable training data are available, and it produces relatively good alignments. We trained separate acoustic models for Frisian and Dutch using datasets from the Mozilla Data Collective, specifically the Common Voice Scripted Speech corpora. For Frisian, we used version 22, comprising 9,053 utterances from 1,367 speakers, with a total duration of 12.97 hours. The Dutch dataset comprised 92,942 utterances from 562 speakers, with a total duration of 116.13 hours. After training, the pretrained acoustic model were adjusted to the characteristics of the target recordings using MFA’s built-in adapt functionality.

For each audio recording, MFA produces a Praat TextGrid containing two annotation tiers. The first tier divides the speech into word intervals, while the second tier segments each word into individual speech sounds, labelled with their corresponding phonetic symbols. These time-aligned phonetic segments provide the basis for subsequent acoustic measurements of vowels and consonants.

For measuring the duration, f0 and formants of the vowels, we use either Praat script of New-FAVE, a tool specifically designed for extracting and optimising vowel formant measurements. New-FAVE employs the FastTrack method, which evaluates multiple candidate formant trajectories and selects the most plausible ones based on their acoustic characteristics and the distribution of vowels within and across speakers.

We use PraatSauce in order to measure acoustic properties associated with voice quality and phonation, such as f0, H1H2c, HNR25, and CPP.