How to Evaluate AI Speech-to-Text APIs Before You Integrate

Picking a speech-to-text (STT) API from a leaderboard screenshot is how products inherit the wrong bill and the wrong failure mode. Before you integrate, run a pre-commit evaluation on your audio—accents, product names, HVAC noise—not on a vendor’s clean demo reel.

This guide is a five-pillar checklist for STT APIs. Concrete examples use Deepgram, AssemblyAI, OpenAI Platform transcription models, and Azure Speech. Meeting-note products such as Otter.ai and editors like Descript appear only as productized alternatives when you do not need an API. ElevenLabs is adjacent for TTS, not STT. This is not a meeting-bot comparison.

Bottom line: Freeze 3–5 real clips (~3 minutes each), spot-check proper nouns and numbers, then decide streaming vs batch by product latency—not by homepage FPS GIFs. Map base $/min or $/hr plus add-ons (diarization, redaction, medical). Read training / retention / region language your counsel can live with. Prefer Deepgram or AssemblyAI when you want speech-native APIs with clear add-on menus; OpenAI when you already standardize on that platform; Azure when you are committed to Microsoft cloud + regional controls.

Findings: public docs + limited hands-on framing (verification 2026-11-04, Asia/Shanghai). No invented WER leaderboards. Re-check official pricing calculators before you forecast COGS.

Who this is for

• Product/engineering leads adding captions, call notes, or voice agents
• Platforms choosing between batch post-processing and real-time captions
• Teams comparing “buy Otter/Descript” vs “build on an STT API”

Out of scope: courtroom stenography certification, and offline-only embedded DSP chips.

The five evaluation pillars

Pillar 1 — Same 3-minute audio, WER-style spot-check

Protocol: Freeze three clips: (A) quiet native speaker reading product names, (B) two-speaker overlap, (C) noisy / accented. Transcribe with each vendor’s default recommended pre-recorded model. Do not publish a fake overall WER—spot-check 20 critical tokens (SKUs, amounts, names).

Pass if critical tokens survive or are systematically fixable with keyterm boosting. Fail if the model invents confident wrong drug/finance terms.

Meeting apps (Otter.ai, Descript) can win if you need a UI tomorrow and not an API—still run the same clips before you standardize.

Pillar 2 — Diarization and timestamps

Protocol: On clip B, require speaker labels and word-level or turn-level timestamps sufficient for your player. Count speaker swap errors qualitatively.

Public pricing reminders (snapshot 2026-11-04):

• Deepgram: Speaker diarization included on pre-recorded; streaming diarization listed about $0.0020/min pay-as-you-go (Growth lower).
• AssemblyAI: Async diarization about +$0.02/hr standard (+$0.065 experimental); streaming diarization about +$0.12/hr.
• OpenAI: Prefer current diarize-capable transcription SKUs on the pricing page; older whisper-1 / GPT-4o-transcribe paths are on deprecation clocks—read the migration notes.
• Azure: Diarization often appears as an enhanced add-on (commonly cited near $0.30/hr on retail mirrors—confirm Azure price sheet for your region).

Pass: Labels are stable enough for your UX. Fail: You need courtroom-grade diarization and the API is “best effort” only—budget human review.

Pillar 3 — Streaming vs batch latency

Protocol: Define the product need first: live captions / voice agent (streaming) vs post-call summary (batch). Measure end-to-end seconds on a 30s utterance for streaming, and queue→result time for batch.

Billing traps to document in the design doc:

• AssemblyAI streaming bills WebSocket open time, including idle—close sockets when calls end.
• Deepgram markets per-second audio billing without block rounding for normal requests—still watch multichannel (stereo can double).
• Azure real-time vs batch vs “fast transcription” are different meters (see Pillar 4).
• OpenAI live transcription SKUs are priced far above file transcription per minute—don’t stream “because we can.”

Pillar 4 — Pricing units and hidden fees (snapshot 2026-11-04, Asia/Shanghai)

Figures are public list hints, not quotes. Regions and commits change numbers.

Deepgram (deepgram.com/pricing)

AssemblyAI (assemblyai.com/pricing)

OpenAI Platform (developers.openai.com pricing)

Azure Speech (azure.microsoft.com pricing/details/speech)

East US retail mirrors commonly list (confirm calculator for your subscription):

Hidden fee checklist: multichannel × channels, diarization/PII add-ons, medical domain packs, idle sockets, endpoint hosting for custom models, data-residency uplifts.

Pillar 5 — Data training, retention, region

Protocol: Paste vendor answers into your security questionnaire: default training on customer audio? Retention TTL? Zero-data-retention options? EU/US endpoints? BAA/HIPAA?

Public posture examples (always re-read DPAs):

• Deepgram: SOC2, GDPR, EU endpoint (api.eu.deepgram.com), HIPAA BAA on Enterprise messaging.
• AssemblyAI: SOC2/ISO/GDPR called out; HIPAA BAA positioned as signable without sales premium.
• OpenAI Platform: enterprise data controls differ from consumer ChatGPT—use the API/enterprise docs, not ChatGPT Free assumptions.
• Azure: Microsoft Products and Services DPA + regional deployment are often the buying reason.

Pass: Legal can quote a URL and a date. Fail: “We’ll anonymize later.”

Build vs buy (Otter / Descript)

If your users only need meeting notes with a UI, evaluate Otter.ai or Descript as products first. Integrate raw STT when you need embedded captions, telephony, or a voice agent. Pair TTS separately via ElevenLabs or Azure/Deepgram TTS if you are closing a talk-loop.

Decision cheat sheet

Method notes

• Limited hands-on: your three clips beat any marketing WER chart.
• Pricing dated 2026-11-04, Asia/Shanghai from official pricing pages / docs.
• Pin model IDs in code; “default” is not a contract.

List your AI tool

Shipping speech or voice AI? List your AI tool on AINav.