Fine-Tuning ARK-ASR-3B
on Custom Dataset for Speech Recognition
Introduction
Fine-Tuning ARK-ASR-3B on Custom Dataset is one of the most effective ways to improve speech recognition accuracy for specialized industries like healthcare. While modern Automatic Speech Recognition (ASR) models perform exceptionally well on general conversations, they often struggle with medical terminology, drug names, clinical abbreviations and diverse speaking styles. Instead of building a speech recognition model from scratch, organizations can fine-tune powerful open-source models such as ARK-ASR-3B using domain-specific datasets to achieve significantly better transcription quality.
In this blog, we demonstrate how AI India Innovations fine-tuned ARK-ASR-3B using a healthcare speech dataset, covering the complete workflow from data preprocessing and model training to evaluation and practical enterprise applications. Whether you're developing AI-powered healthcare solutions, voice assistants, or medical transcription systems, this guide provides a practical roadmap for building domain-specific Speech AI solutions.
Why General ASR Models Struggle with Medical Speech
Medical speech is not simply "normal speech, but harder" - it is a genuinely different problem, shaped by several challenges that rarely show up together anywhere else:
– Complex terminology: words like "cholecystectomy" or "thrombocytopenia" almost never appear in the everyday audio most ASR models are trained on, so the model has no learned pattern to fall back on.
– Drug names: many medications sound alarmingly similar to one another and a mistaken substitution here is a patient-safety issue, not a typo.
– Clinical abbreviations: shorthand like "BP," "NPO," or "q.i.d." carries specific meaning that a general model was never taught to expect.0
– Varied speaking styles: a dictated discharge summary sounds nothing like a doctor casually reassuring a nervous patient and the model has to handle both.
– Accents: healthcare is delivered by professionals from a huge range of linguistic backgrounds and models trained mainly on American or British audio lose accuracy quickly outside that range.
– Background noise and long, uninterrupted conversations: clinics are rarely quiet and clinical dictation can run for minutes without a natural pause.
– Near-homophones: medicine is full of words that sound alike but mean very different things and only domain exposure teaches a model which one is medically plausible.
None of this means general ASR models are bad - it means they were optimized for a different distribution of speech. Fine-tuning is how we take a strong, general foundation and continue training it specifically on medical language, so it learns the vocabulary and phrasing that matter most in this domain, without having to build a new model from scratch.
ARK-ASR-3B
ARK-ASR-3B is an open-source, multilingual speech recognition model released on Hugging Face by the AutoArk research team. It is a 3B-scale, audio-capable autoregressive Transformer built specifically for automatic speech recognition - it listens to audio and generates the matching transcript, the way a human transcriptionist would, but automatically and at scale. The model was introduced alongside a paper on data-efficient on-policy distillation (OPD), a training approach where a smaller student model learns from a stronger teacher model's live feedback on its own generated transcripts rather than only from static transcripts - an approach well suited to specialized domains where large labeled datasets are hard to come by.
Architecture
– A Whisper-style audio encoder with Rotary Position Embeddings (RoPE), which converts raw audio into a rich numerical representation.
– An MLP adapter, a lightweight bridge that translates the encoder's audio representation into a format the language model can read.
– A Qwen decoder, which generates the transcript token by token, the same way a text-generation model writes a sentence.
Key specifications
– Model size: ~3B–4B parameters across the encoder, adapter and decoder combined; checkpoint format is safetensors.
Requires trust_remote_code=True in Hugging Face Transformers because of its custom "arkasr" model type.
– Sampling rate: audio inputs are expected at 16 kHz.
– Tokenizer: a Qwen-derived tokenizer with special audio placeholder tokens marking where audio embeddings are inserted.
– Supported languages: 19, including English, Chinese, German, Japanese, French, Korean, Spanish and several other European languages.
– Reported accuracy: an average Word Error Rate of 5.04% on the Hugging Face Open ASR Leaderboard's English short-form benchmark, currently among the strongest published results on that leaderboard, alongside a high real-time processing factor.
Released under the Apache-2.0 license, with full Hugging Face Transformers compatibility, which is what makes fine-tuning it approachable with familiar tools.
ARK-ASR-3B is a good fit for this project because it strikes a workable middle ground: large enough to bring strong general language understanding to the table, but still practical to fine-tune on a single modern GPU, with open weights and a strong baseline on noisy, long-form audio - the same qualities a real clinical recording environment tends to demand.
Our Custom Healthcare Dataset
For fine-tuning, we used the publicly available "Medical Speech, Transcription and Intent" dataset from Kaggle, contributed by Paul Timothy Mooney. It contains short spoken audio clips of people describing common medical symptoms - the kind of phrases a patient might say when explaining what is wrong to a doctor or a symptom-checker tool - each paired with a text transcription and an intent/symptom label.
– Total audio duration: approximately 8.5 hours
– Language: English
– Audio format: WAV, standardized to 16 kHz mono during preprocessing
– Structure: a CSV metadata file (overview.csv) linking each audio file to its transcript, plus columns for speaker ID, the original symptom prompt and audio-quality flags such as clipping and background noise
– Scale: one widely used repackaging of this dataset on Hugging Face reports 6,276 total audio-transcript rows (381 in one split and 5,895 in another), giving a reasonable sample size for fine-tuning rather than training from scratch
Multiple speakers and speaking styles, from clearly read phrases to more spontaneous, conversational descriptions
Like most real-world speech data, it is not perfectly clean - a portion of clips have background noise, a few labels do not perfectly match the audio and some recordings include leading or trailing silence. None of this is unusual and a preprocessing pipeline (below) is what turns this raw, imperfect data into something reliable to train on. Even at 8.5 hours, the dataset is a reasonable size for fine-tuning, because ARK-ASR-3B does not need to relearn English from scratch - it already understands the language well. What it needs is focused exposure to medical vocabulary and symptom phrasing, which is exactly what this dataset provides.
Dataset snapshot
| Attribute | Value |
|---|---|
Source |
Kaggle – Medical Speech, Transcription and Intent |
Total duration |
~8.5 hours |
Total rows (one common split) |
6,276 (381 train / 5,895 test in that split) |
Audio format / rate |
WAV, resampled to 16 kHz mono |
Metadata format |
CSV (overview.csv) with transcript, speaker ID, intent, quality flags |
Content type |
Patient-style spoken symptom descriptions |
Data Pre-processing Pipeline
– Validation: every WAV file was checked to confirm it opened correctly and had a valid, non-zero duration.
– Noise handling: clips with excessive background noise were flagged, mildly cleaned where possible and excluded only when too degraded to be useful.
– Resampling: all audio was standardized to 16 kHz mono, matching what ARK-ASR-3B's encoder expects - mismatched sample rates are one of the most common, avoidable causes of poor ASR accuracy.
– Transcript cleaning: consistent casing and punctuation, whitespace trimming and correction or removal of a small number of mismatched transcript-audio pairs found during spot checks.
– Final packaging: the cleaned audio-transcript pairs were split into training and validation subsets and loaded into a Hugging Face Dataset object, ready for fine-tuning.
Fine-Tuning ARK-ASR-3B
This is the core of the project - the practical, end-to-end workflow used to fine-tune ARK-ASR-3B on the prepared dataset with Hugging Face Transformers.
1. Environment setup
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate datasets soundfile librosa jiwer
2. Loading the dataset
from datasets import load_dataset, Audio
# Each record: {"audio": "path/to/clip.wav", "text": "transcript here"}
dataset = load_dataset("json", data_files={
"train": "data/train/train.jsonl",
"validation": "data/val/val.jsonl",
})
dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
3. Loading the base model
import torch
from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer
model_path = "AutoArk-AI/ARK-ASR-3B"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_path, trust_remote_code=True,
torch_dtype=torch.bfloat16, attn_implementation="sdpa",
)
4. Preparing training examples
def prepare_example(example):
conversation = [
{"role": "user", "content": [
{"type": "audio", "array": example["audio"]["array"],
"sampling_rate": example["audio"]["sampling_rate"]},
{"type": "text", "text": "Please transcribe this audio."},
]},
{"role": "assistant", "content": example["text"]},
]
return processor.apply_chat_template(
conversation, tokenize=True, return_tensors="pt", sampling_rate=16000
)
processed_dataset = dataset.map(prepare_example, remove_columns=dataset["train"].column_names)
5. Training configuration
from transformers import TrainingArguments, Trainer
training_args = TrainingArguments(
output_dir="checkpoints/ark-asr-3b-medical",
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # effective batch size = 16
learning_rate=1e-5,
num_train_epochs=3,
bf16=True,
evaluation_strategy="steps", eval_steps=200,
save_strategy="steps", save_steps=200, save_total_limit=3,
logging_steps=20, report_to="tensorboard",
)
trainer = Trainer(
model=model, args=training_args,
train_dataset=processed_dataset["train"],
eval_dataset=processed_dataset["validation"],
tokenizer=tokenizer,
)
trainer.train()
trainer.save_model("checkpoints/ark-asr-3b-medical/final")
6. Running inference
def transcribe(audio_path):
conversation = [{"role": "user", "content": [
{"type": "audio", "path": audio_path},
{"type": "text", "text": "Please transcribe this audio."},
]}]
inputs = processor.apply_chat_template(
conversation, add_generation_prompt=True,
return_tensors="pt", sampling_rate=16000).to("cuda")
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
return tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
Model Evaluation and Performance Analysis
Fine-tuning a model is only valuable if it leads to better transcription accuracy. To evaluate the performance of our fine-tuned ARK-ASR-3B model, we compared its predictions with the actual transcripts from the validation dataset using two widely accepted Automatic Speech Recognition (ASR) metrics: Word Error Rate (WER) and Character Error Rate (CER).
Word Error Rate (WER): Measures how many words the model transcribed incorrectly by counting substitutions, deletions and insertions. A lower WER indicates better transcription accuracy.
Character Error Rate (CER): Measures errors at the character level, making it especially useful for identifying mistakes in long medical terms, drug names and abbreviations where even a single incorrect character can change the meaning.
Challenges During Fine-Tuning
A few practical, expected engineering challenges came up during the project - none of them unusual for a domain-adaptation task of this size and each was manageable:
– Dataset imbalance across symptom categories, addressed by monitoring per-category performance rather than relying on a single overall average.
– GPU memory limits, managed with bfloat16 mixed precision and gradient accumulation to simulate a larger effective batch size.
– A handful of noisy or mislabeled clips, caught through spot-checking during preprocessing rather than exhaustive manual review.
– Finding a learning rate and epoch count that improved medical-term recognition without eroding the model's original general-purpose accuracy - a conservative learning rate with close validation monitoring worked well.
Future Improvements
– Larger and more diverse healthcare datasets, covering more speakers, accents and clinical scenarios.
– Coverage of more medical specialties beyond general symptom descriptions - radiology, cardiology, surgical dictation.
– Better noise robustness through deliberate exposure to real hospital and clinic background sound.
– Multilingual fine-tuning, given ARK-ASR-3B already supports 19 languages - valuable for multilingual healthcare settings like India.
– Full clinical conversations rather than short symptom statements, including turn-taking and follow-up questions.
– Integration with Electronic Health Records (EHR), Retrieval-Augmented Generation (RAG) for clinical context and broader healthcare AI agents and assistants.
Conclusion
Fine-tuning foundation speech models like ARK-ASR-3B demonstrates how organizations can transform general-purpose AI into highly accurate, domain-specific solutions. By adapting powerful open-source models with industry-specific datasets, businesses can significantly improve transcription accuracy, automate speech-driven workflows and enhance operational efficiency. Whether it's understanding complex medical terminology, customer conversations, technical discussions, or multilingual interactions, customized Speech AI delivers far greater accuracy and reliability than generic speech recognition systems.
At AI India Innovations, we specialize in designing and deploying enterprise-grade Speech AI solutions tailored to unique business needs. From custom ASR model fine-tuning, multilingual speech recognition, Voice AI assistants and on-premise deployments to seamless enterprise AI integration, we help organizations build scalable, production-ready speech applications. Our expertise extends across a wide range of industries, including Healthcare, Manufacturing, Defence, Banking & Financial Services (BFSI), Retail & E-commerce, Customer Support & Contact Centers, Logistics & Supply Chain and Education. Whether you're looking to build an intelligent voice assistant, automate speech transcription, enable multilingual communication, or develop a custom Speech AI solution powered by models like ARK-ASR-3B, AI India Innovations has the expertise to turn your vision into a reliable, enterprise-ready reality.
