The Speech SDK — config, recognizer, synthesizer, SSML

If card 27 was about choosing the speech stack, this one is about using the Speech service properly. Almost all of it hangs off one distinction, so start there.

Two config objects, and the classic swap

SpeechConfig is the connection. It holds the credential and the voice. You can build it from an endpoint plus a key, or an endpoint plus an Entra credential — both work. The voice name is set here.

AudioConfig is the audio source or destination. A microphone, a file, a speaker.

The intended error is swapping them. If a question is phrased around where the audio comes from or goes to, it is asking for AudioConfig. "The input to transcribe is an audio file" is an AudioConfig statement, not a SpeechConfig one.

Hold the sentence: SpeechConfig is the connection; AudioConfig is where the audio lives.

Two clients

Speech to text uses SpeechRecognizer, then recognize_once(). Check result.reason, which comes back as RecognizedSpeech, NoMatch or Canceled.

That middle value is worth noticing. NoMatch means the call succeeded and nothing recognisable was heard — a success response with no content, in the same family as the truncated completions on card 5 and the empty tool results on card 15. Checking only for failure will not catch it.

Text to speech uses SpeechSynthesizer, with speak_text_async() for plain text and speak_ssml_async() for marked-up text.

SSML controls

Speech Synthesis Markup Language gives you control over delivery: style, breaks and silence, phonemes, prosody — pitch and rate — say-as, and background audio.

One precedence rule that gets tested: a <voice> element in the markup overrides the voice set in SpeechConfig. So a question describing the wrong voice coming out, when SpeechConfig clearly names the right one, is pointing at the markup.

The Azure Speech MCP server

The agent route into the same capabilities. It exposes exactly two: speech-to-text and text-to-speech.

It requires an Azure Storage account, because it must store both the input audio and the generated output audio. You connect it with the Foundry resource key plus a shared access signature URL for the blob container, with read and write.

Its constraints:

• The agent's model must support tool calling — GPT-4o or later.
• MCP tools need the Enterprise tier.
• It does not work with a network-secured Foundry.
• The client application just references the agent by name and type.

That third constraint is the one to carry into design work. It joins the list from card 6 of things that do not survive network isolation — another case where a capability and a security posture are simply incompatible, and you have to choose.