The video evaluates two open-source AI models for speech synthesis and transcription, finding that the lightweight ZONOS3-Mini runs efficiently on standard laptops, while the large SOV-Transcribe-1 requires expensive high-memory GPUs and struggles with accurately transcribing complex entities like email addresses. Despite these limitations, the combined system shows promise for private, offline voice applications, with future improvements aimed at enhancing transcription accuracy and usability.
The video explores the feasibility of building a complete speech loop using two open-source AI models: ZONOS3-Mini for speech synthesis and SOV-Transcribe-1 for speech-to-text transcription. The presenter tests these models on a laptop with an RTX 4060 GPU and a rented high-memory GPU, measuring performance, accuracy, and resource requirements. The speaking model requires about 1.7 GB of video memory and runs efficiently on an 8 GB GPU laptop, producing speech with low latency and real-time speed. In contrast, the listening model demands around 48 GB of GPU memory, necessitating a powerful rented GPU, making it impractical for typical laptops.
The testing methodology involved generating 56 audio clips from four synthetic voices reading 14 holiday-themed sentences, including plain text, technical terms, email addresses, and numbers. Each clip was also tested with added white noise to simulate challenging listening conditions. The transcriptions were compared word-for-word against the original text to calculate word error rates and entity accuracy, focusing particularly on critical elements like email addresses and technical terms. The speaking model demonstrated fast response times and low memory usage, while the transcription model showed strong accuracy on clean speech but struggled with certain entities, especially email addresses.
The transcription model, SOV-Transcribe-1, is a large, complex model with 26.6 billion parameters and requires a high-end GPU with at least 48 GB of memory to run. The presenter rented an 80 GB A100 GPU in the cloud to run it, incurring costs of about $2.09 per hour. Despite the high resource demands, the model achieved a low overall word error rate of around 5% and high entity accuracy for technical terms and numbers. However, email addresses proved challenging, with only about 57.5% accuracy, as the model often defaulted to more common spellings rather than the exact ones spoken, highlighting the difficulty of transcribing spelled-out entities accurately.
Noise robustness was tested by adding white noise at 10 decibels below the voice level, which slightly increased the word error rate but did not cause a significant drop in transcription quality. The choice of synthetic voice had minimal impact on accuracy, with similar error rates across different voices. The presenter cautions that these results are based on synthetic speech and limited noise types, so real-world performance with diverse accents, background noises, or live recordings may vary. The video also notes the importance of providing spelling context to improve transcription accuracy for critical entities like email addresses.
In conclusion, the video demonstrates that while the open-source speech synthesis model is practical for personal use on standard laptops, the transcription model currently requires expensive, high-memory GPUs typically available only in data centers. The combined speech loop is functional but has limitations, especially in accurately transcribing complex entities such as email addresses. The presenter suggests future work to improve entity accuracy by passing spelling context to the transcription model and plans to test voice cloning with personal recordings. Overall, the open models offer a promising foundation for private, offline voice applications, but users should be aware of current hardware and accuracy constraints.
Useful Links
- Zyphra/ZONOS3-Mini model repository on Hugging Face — Directly provides the speech synthesis model used in the video for replication and further testing.
- Zyphra/ZONOS3-Mini GitHub repository — Contains source code, build instructions, and model files essential for reproducing the speech synthesis setup.
- ArXiv paper 2611.16525 — Contains original research and technical details about the transcription model used in the video.