STORM ASR 2.0 benchmark
Our most accurate speech recognition for Nigerian languages yet, tested against the leading Nigerian speech recognition models on the same 22,921 recordings in Yoruba, Hausa, Igbo, Nigerian Pidgin, Nigerian English and code-mixed speech.
- words wrong per 100 across all languages, the lowest of every model tested
- 15.6
- fewer word errors than the next best model (22.6)
- 31%
- test groups (five languages and code-mixed) where STORM ASR 2.0 is most accurate
- 6 of 6
- letters wrong per 100, with tone marks counted
- 5.6
Words wrong per 100 words spoken
Lower is better. Same 22,921 recordings for every model.
- STORM ASR 2.0
- SBPN Large
- Meta MMS-1B
Word error rate
Words wrong per 100 words spoken. A wrong, missing or extra word each count as one error. Best result in each row highlighted.
| Language | Recordings | STORM ASR 2.0 | SBPN Large | Meta MMS-1B | NCAIR |
|---|---|---|---|---|---|
| Nigerian English | 1,188 | 4.5 | 9.0 | 19.3 | 12.1 |
| Hausa | 3,615 | 11.9 | 18.1 | 28.2 | n/a |
| Igbo | 4,662 | 22.2 | 28.7 | 52.3 | n/a |
| Nigerian Pidgin | 1,369 | 14.2 | 15.5 | 50.4 | n/a |
| Yoruba | 2,499 | 19.4 | 24.5 | 51.6 | 44.3 |
| Code-mixed (Yoruba + English) | 9,588 | 14.8 | 25.1 | 69.0 | 58.0 |
| All languages | 22,921 | 15.6 | 22.6 | 51.1 | n/a |
Letter error rate
Letters wrong per 100 letters. A missing or wrong tone mark counts as a wrong letter, so this shows how well each model writes Yoruba, Igbo and Hausa as they are really written.
| Language | Recordings | STORM ASR 2.0 | SBPN Large | Meta MMS-1B | NCAIR |
|---|---|---|---|---|---|
| Nigerian English | 1,188 | 1.9 | 3.5 | 6.8 | 6.1 |
| Hausa | 3,615 | 3.2 | 4.7 | 7.4 | n/a |
| Igbo | 4,662 | 8.0 | 10.1 | 17.6 | n/a |
| Nigerian Pidgin | 1,369 | 7.4 | 7.6 | 21.4 | n/a |
| Yoruba | 2,499 | 7.7 | 9.8 | 18.8 | 20.1 |
| Code-mixed (Yoruba + English) | 9,588 | 5.3 | 11.1 | 27.4 | 36.4 |
| All languages | 22,921 | 5.6 | 8.7 | 18.7 | n/a |
Examples
Recordings from the test set, as each model wrote them.
How we tested
- Test set. 22,921 recordings, about 34 hours, in Nigerian English, Hausa, Igbo, Nigerian Pidgin, Yoruba and Yoruba mixed with English. Every model transcribed exactly the same recordings.
- STORM ASR 2.0 was measured through the live production service, exactly as customers receive it, with no language given: it recognises the language on its own. Meta MMS-1B and the NCAIR models were given the correct language for every recording.
- Scoring. Capitals and punctuation are ignored; tone marks count, so a missing or wrong tone mark makes a word wrong. Recordings whose reference text contains digits were left out, because models write numbers differently. All counts were made by our own scoring code from each model's raw transcripts.
Models compared
- SBPN Large (ogunlao/SBPN_multilingual_large), a multilingual model for Nigerian languages, licensed for research use only.
- Meta MMS-1B (facebook/mms-1b-all) with its Yoruba, Hausa, Igbo, Pidgin and English adapters.
- NCAIR Yoruba-ASR and NigerianAccentedEnglish, which cover Yoruba and Nigerian English only.
Each was run from its published weights with its default settings.
STORM ASR 2.0 is available now.
Files and real-time streams, no language setting needed, one API key.