Fine-Tuning Whisper Large v3 for Serbian Speech Workflows

Why Fine-Tune for Serbian?

Whisper Large v3 handles major European languages well, but Serbian creates distinct obstacles. The language exhibits morphological complexity, limited presence in training data, and regional pronunciation variations between Ekavian and Ijekavian dialects. While general transcription performed acceptably, domain-specific audio — legal, medical, and government recordings — required word-level precision that the base model couldn't consistently deliver. Fine-tuning provided a path forward without complete retraining.

Dataset Preparation Is the Real Work

The most revealing finding: model configuration matters far less than data quality. Approximately 70% of project time went toward dataset work:

• Sourcing diverse audio from broadcast, interviews, and read speech corpora
• Manual transcription validation for unclear segments
• Normalization pipeline addressing punctuation, number formatting, and dialect decisions
• Deduplication and train/eval splitting

Results showed that 200 hours of clean, curated audio consistently outperformed 500 hours of mixed-quality recordings across all evaluations.

Training Strategy: Conservative and Iterative

The approach remained measured:

• Frozen encoder layers initially (decoder-only fine-tuning)
• Small learning rate with warmup: 1e-5 to 5e-5
• Gradient accumulation for hardware constraints
• Early stopping based on validation WER

Critically, models trained beyond the point of validation WER improvement showed worse real-world performance — they overfitted to training distribution rather than generalizing to live conditions.

Evaluation in Real Context

Benchmark WER testing proved necessary but insufficient. Parallel evaluations covered:

• Clean studio recordings
• Telephone-quality audio (8kHz, compressed)
• Noisy environments with background noise and multiple speakers
• Domain-specific vocabulary density tests

Real-world performance gaps appeared notably in telephone and noisy categories, influencing preprocessing decisions for production deployment.

Production Considerations

Preprocessing: Apply gain normalization and silence trimming. Inconsistent input levels represent the primary source of accuracy degradation in production.

Post-processing: Domain-specific vocabulary correction layers using dictionary lookups for proper nouns and technical terms provide meaningful gains at minimal cost.

Monitoring: Track per-segment confidence scores and flag low-confidence outputs for human review.

Latency: Large v3 lacks speed for real-time applications; consider distilled variants for synchronous workflows. Asynchronous pipelines achieve acceptable throughput via batching.

Conclusion

Optimal results emerged from iterative experiments using realistic audio samples. Success came from the combination of clean data, conservative training, domain-aware evaluation, and production-ready preprocessing. Fine-tuning Whisper for lower-resource languages remains achievable — the investment centers on data curation rather than model complexity.