🎙️ Bengali ASR Leaderboard

Welcome to the Bengali Automatic Speech Recognition (ASR) Leaderboard! This leaderboard tracks the performance of ASR models on Bengali speech recognition tasks.

📖 Introduction

This leaderboard evaluates ASR models on their ability to transcribe Bengali speech accurately. Models are evaluated on a practical shared Bengali audio pack using common metrics like Word Error Rate (WER) and Character Error Rate (CER). When available, a local Real-Time Factor (RTF) benchmark can also be displayed.

Metrics Explained

  • WER (Word Error Rate): The percentage of words that are incorrectly transcribed. Lower is better.
  • CER (Character Error Rate): The percentage of characters that are incorrectly transcribed. Lower is better.
  • RTF (Real-Time Factor): End-to-end runtime divided by audio duration. Lower is better.

Evaluation Datasets

The current board is centered on one practical benchmark:

  1. Mixed 10-Audio Bengali Pack: A small shared pack assembled from Mozilla/Common Voice mirror clips and streamed samples from zarifmahir21/banglaASRFiltered

🚀 How to Submit

To submit a model for evaluation:

  1. Ensure your model is uploaded to the Hugging Face Hub
  2. Your model should be compatible with the Hugging Face transformers library
  3. Click the "Submit" button and provide your model name
  4. Wait for the evaluation to complete (this may take some time)

📊 Evaluation Details

Models are evaluated using:

  • Greedy decoding (no language model)
  • Standard preprocessing and normalization
  • Metrics calculated using jiwer library
  • Optional local RTF timing runs for machine-specific speed comparisons

🏆 Leaderboard Rankings

Models are currently ranked by WER on the shared mixed 10-audio pack. CER can be used as a secondary quality signal, while RTF is useful for local speed/accuracy tradeoff analysis.

📊 Leaderboard Notes

  • Models are ranked by average WER from the local evaluation result file
  • WER: Word Error Rate - percentage of words incorrectly transcribed
  • CER: Character Error Rate - percentage of characters incorrectly transcribed
  • RTF: Real-time factor from the optional local timing benchmark file when present
  • Mixed 10-Audio WER reflects the current custom 10-file benchmark pack used by the local evaluator
  • Refresh reloads the canonical local results file: evaluation_results_mixed_10_samples_latest.json