ufakzeka-kulak

Speech in, text out

The model reads 80 log-mel features from mono audio at 16,000 Hz and writes out one utterance at a time. Its encoder is a FastConformer of 16 layers and hidden size 256, and it feeds two outputs, the default token-and-duration transducer (TDT) decoder and a CTC head. The vocabulary is a SentencePiece BPE of 1,024 pieces.

Runtimes

The NeMo file is the reference. Every other file was compared with NeMo on the Common Voice and FLEURS development sets, and only files whose normalised WER stayed within 0.0050 of NeMo's on both sets, with no failed utterance, are released. The sherpa-onnx files are used with the samples multiplied by 8, and the CTC file with zeros appended to the audio. The MLX folder ships its own feature function, since parakeet-mlx computes its features differently from NeMo. The model card has the details and the examples for the other runtimes.

from nemo.collections.asr.models import ASRModel

model = ASRModel.from_pretrained("ufakai/ufakzeka-kulak")
print(model.transcribe(["speech.wav"])[0].text)

model.change_decoding_strategy(decoder_type="ctc")  # the CTC head
print(model.transcribe(["speech.wav"])[0].text)

import soundfile as sf
import torch
from huggingface_hub import snapshot_download
from transformers import ParakeetForTDT, ParakeetProcessor

root = snapshot_download("ufakai/ufakzeka-kulak", allow_patterns=["transformers-tdt/*"])
folder = f"{root}/transformers-tdt"
processor = ParakeetProcessor.from_pretrained(folder)
model = ParakeetForTDT.from_pretrained(folder, dtype=torch.float32).eval()
audio, rate = sf.read("speech.wav", dtype="float32")
inputs = processor(audio=audio, sampling_rate=rate, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs)
print(processor.decode(out.sequences[0], skip_special_tokens=True))

Test results

The training recipe, the checkpoint and the released files were chosen on development sets, and the test sets were scored once, for one model. The model was to ship only if it beat Whisper base on both FLEURS Turkish test and Common Voice 27.0 Turkish test, each by a paired interval lying wholly below zero. It passed on both.

Values in square brackets are 95 percent intervals from a blockwise bootstrap; the interval of a difference between two models is paired. A difference is ufakzeka-kulak's WER minus the Whisper model's, so a negative value means fewer errors. Decoding is greedy, with the TDT decoder. Whisper base and Whisper small (MIT; their maker does not fully disclose their training data) were scored by us with the same Turkish normaliser, on each set with the better of greedy decoding and the Whisper package's decoding settings (beam search of 5 with temperature fallback). Whisper large-v3 (greedy) scores 0.0604 on FLEURS test and 0.0767 on Common Voice test, lower than ufakzeka-kulak; its difference was not tested. Real-TurnTurk has only 6 speakers, so its interval is wide.

The released int8 sherpa-onnx TDT file reaches a WER of 0.0921 on FLEURS test (difference to NeMo 0.0000) and 0.0932 on Common Voice test (+0.0016); the int8 CTC file reaches 0.1128 (+0.0207) and 0.1200 (+0.0284).

Training data

The licence of every training source allows commercial use. The sources are Mozilla Common Voice, antalia-voice-corpus and ISSAI's Turkish Speech Corpus, and 4,696 of the 4,977 hours are YouTube audio collected by ESPnet's YODAS and YODAS v3, whose authors say they took only videos published under a Creative Commons licence; they release the audio under CC BY 3.0. On YouTube the one Creative Commons licence an uploader can choose is the Creative Commons Attribution licence (CC BY). The YODAS v3 audio was cut along the spans of Alania Turkish Speech Style Captions (CC BY 4.0). Automatic captions were not used as training text. The training text of each YODAS utterance is the uploader's subtitle where it agrees with the Whisper large-v3-turbo transcript; otherwise it is the Whisper large-v3-turbo transcript, checked by Omnilingual ASR CTC 7B v2. Whisper large-v3-turbo is under MIT and Omnilingual ASR CTC 7B v2 under Apache-2.0; their makers do not fully disclose their training data. 432 training utterances that matched the evaluation sets by text or audio were removed, and the 2 MediaSpeech clips that matched training audio were left out of its evaluation. The full list of sources and their attribution is in the ATTRIBUTION.md file of the model repository.

Where it is weak

• It is trained for Turkish only; other languages, and Turkish mixed with them, were not measured.
• Its word error rate was not measured by speaker gender, age or regional accent.
• It is weak on conversational speech. On Real-TurnTurk, conversations of 6 speakers, its WER is 0.3807.
• It does not tell speakers apart and gives no word timestamps that were checked.
• For 4,327 of the 4,696 hours of YODAS recordings, no uploader subtitle agreed with the transcript, so one speech recognition model wrote the training text and a second checked it; errors on which the two agreed remain in the training text.
• The input gain used with the sherpa-onnx files is a workaround that holds at normal recording levels; on the FLEURS copy made 30 dB quieter the error rate rises more than with the NeMo file.
• The MLX files were checked on mlx’s CPU backend under Linux only, not on Apple silicon.

Files and code

• Demokulak.ufakzeka.com/en
• Hugging Faceufakai/ufakzeka-kulak
• GitHubufakai/ufakzeka-kulak
• Research postMethod, measurements and limits