1. Update eval-d1, eval-whatsapp results to align with dataset modifications.

2. Update about page to include informaiton about eval-whatsapp.
3. Results for 2025.05.13 Whisper v3 and Whisper v3 turbo models.

benchmark.csv

@@ -1,16 +1,18 @@
 engine,model,ivrit-ai/eval-d1,ivrit-ai/eval-whatsapp,ivrit-ai/saspeech,google/fleurs/he,mozilla-foundation/common_voice_17_0/he,imvladikon/hebrew_speech_kan
-faster-whisper,ivrit-ai/whisper-large-v3-ct2-20250403,0.056,0.085,0.071,0.200,0.171,0.095
-faster-whisper,ivrit-ai/whisper-large-v3-ct2-20250209,0.060,-,0.074,0.208,0.172,0.094
-faster-whisper,ivrit-ai/faster-whisper-v2-d4,0.062,-,0.080,0.241,0.207,0.113
-faster-whisper,ivrit-ai/faster-whisper-v2-d3-e3,0.070,-,0.086,0.255,0.214,0.139
-amazon-transcribe,batch,0.068,0.110,0.085,0.230,0.141,0.090
-faster-whisper,large-v2,0.080,0.133,0.098,0.266,0.233,0.164
-faster-whisper,large-v3-turbo,0.085,0.131,0.104,0.289,0.280,0.156
-faster-whisper,ivrit-ai/whisper-large-v3-turbo-ct2-20250403,0.058,0.069,0.074,0.208,0.183,0.100
-faster-whisper,ivrit-ai/whisper-large-v3-turbo-ct2-20250209,0.067,-,0.077,0.245,0.209,0.100
-amazon-transcribe,stream,0.081,0.140,0.090,0.287,0.200,0.131
-faster-whisper,large-v3,0.096,0.147,0.094,0.262,0.231,0.134
-google-speech,google-speech,0.212,0.356,0.189,0.385,0.380,0.292
-openai,gpt-4o-transcribe,0.074,0.129,0.109,0.210,0.169,0.394
-openai,gpt-4o-mini-transcribe,0.093,0.156,0.150,0.300,0.237,0.468
-elevenlabs,scribe_v1,0.201,0.262,0.068,0.181,0.156,0.109
+faster-whisper,ivrit-ai/whisper-large-v3-ct2-20250513,0.051,0.072,0.064,0.174,0.149,0.081
+faster-whisper,ivrit-ai/whisper-large-v3-ct2-20250403,0.055,0.075,0.071,0.200,0.171,0.095
+faster-whisper,ivrit-ai/whisper-large-v3-ct2-20250209,0.059,0.082,0.074,0.208,0.172,0.094
+faster-whisper,ivrit-ai/faster-whisper-v2-d4,0.061,0.098,0.080,0.241,0.207,0.113
+faster-whisper,ivrit-ai/faster-whisper-v2-d3-e3,0.068,0.104,0.086,0.255,0.214,0.139
+amazon-transcribe,batch,0.066,0.104,0.085,0.230,0.141,0.090
+faster-whisper,large-v2,0.077,0.121,0.098,0.266,0.233,0.164
+faster-whisper,large-v3-turbo,0.084,0.128,0.104,0.289,0.280,0.156
+faster-whisper,ivrit-ai/whisper-large-v3-turbo-ct2-20250513,0.053,0.071,0.066,0.181,0.151,0.082
+faster-whisper,ivrit-ai/whisper-large-v3-turbo-ct2-20250403,0.055,0.061,0.074,0.208,0.183,0.100
+faster-whisper,ivrit-ai/whisper-large-v3-turbo-ct2-20250209,0.071,0.104,0.077,0.245,0.209,0.100
+amazon-transcribe,stream,0.079,0.129,0.090,0.287,0.200,0.131
+faster-whisper,large-v3,0.098,0.132,0.094,0.262,0.231,0.134
+google-speech,google-speech,0.211,0.352,0.189,0.385,0.380,0.292
+openai,gpt-4o-transcribe,0.073,0.126,0.109,0.210,0.169,0.394
+openai,gpt-4o-mini-transcribe,0.090,0.158,0.150,0.300,0.237,0.468
+elevenlabs,scribe_v1,0.200,0.264,0.068,0.181,0.156,0.109

src/about.py

@@ -47,6 +47,11 @@ The following datasets are used in our evaluation:
 - **Domain**: Manual transcription of a single podcast episode featuring an informal conversation between two speakers (male and female). Audio is segmented into approximately 5-minute chunks.
 - **Source**: Part of the ivrit.ai corpus. Selected episode has been manually transcribed to golden standard quality to serve as a high-quality evaluation benchmark.
 
+### [ivrit-ai/eval-whatsapp](https://huggingface.co/datasets/ivrit-ai/eval-whatsapp)
+- **Size**: 1:10 hours
+- **Domain**: Freestyle WhatsApp recordings made by volunteers.
+- **Source**: ivrit.ai volunteers. Manually transcribed by an expert.
+
 ### [SASpeech](https://huggingface.co/datasets/upai-inc/saspeech)
 - **Size**: 4 hours (manually corrected portion of the corpus)
 - **Domain**: Economic and political podcast content, containing both read speech and conversational segments. Segments are several seconds in length.