How Do LLMs Turn Speech into Structured Commands?

• User request
• Model producing command data
• Packet with fields to validate

What you will learn

• Follow an utterance from audio through transcript and structured fields.
• Understand what schema-guided output guarantees and what it does not.
• Handle ambiguity, invalid values and system-state checks before forwarding a proposal.

Before you start

You need only know that JSON is a text format with named fields and values.

How does speech become something a model can interpret?

A microphone captures a waveform, but a text language model ordinarily receives words rather than raw microphone sound. An automatic speech recognizer (ASR) first produces a transcript. For example, “move toward the north marker for about two hundred metres” becomes text. If recognition returns “north market,” the later model may interpret the wrong phrase fluently, so the pipeline should retain the transcript and allow correction.

The application then sends the transcript with instructions defining the allowed task and field meanings. In an illustrative design, the prompt says to extract a movement intent, a direction and a distance, and to return an explicit clarification state when a value is absent or unclear. The model uses the sentence and instructions to produce a candidate object. This is an application design, not a claim that all language models internally use the same extraction algorithm.

How does a schema shape the result?

JSON Schema is a vocabulary for annotating and validating JSON documents. A structured-output interface can constrain the response format using a schema. Gemini’s official documentation describes JSON output configured with a JSON Schema subset and recommends clear descriptions and strong field types. For this example, a proposal might look like {"intent":"approach","target":"north marker","distance_m":example_value}. The number is an illustrative value, not a measured or tested command.

The schema can require fields and restrict types or enumerated choices. It helps prevent malformed JSON or a distance returned as text. It cannot decide that a place exists, that the transcript was heard correctly, or that the request fits current application state. The schema subset can also omit or ignore some constraints; the API documentation specifically advises application-side validation for unsupported constraints and semantically incorrect outputs.

Treat the model response as untrusted input. Parse it, reject unexpected fields, check required values and units, and enforce the application’s own numeric and vocabulary rules. Then compare the target with available map or object data and check that those data are current. Only a proposal that passes these separate checks can be shown to a person or passed to an approved coordinator.

What should happen when the wording is unclear?

Suppose the user says “go to the north one.” If the visible choices include multiple northern markers, the model should not invent which one was meant. The schema can represent a needs_clarification outcome or a missing target, and the interface can ask the user to identify the marker. If the distance is omitted, ask whether the application has a defined default; do not let the model silently choose one. If no valid answer can be produced, return a controlled error rather than fabricated fields.

This separation makes failures diagnosable. Compare the audio-derived transcript, model proposal, schema result, application validation and final decision. A recognition error points to audio or ASR; a wrong field points to interpretation; a schema failure points to output shape; a state mismatch points to application data. A syntactically valid object can still be semantically wrong, which is why the structured-output documentation explicitly recommends validating values in the application.

Walkthrough: one utterance to a reviewable proposal

Illustrative software design, not a benchmark or flight test.

The ASR component converts the spoken request into a transcript. The application adds the allowed intent and field definitions, then asks the model for structured output. The model returns an intent, target phrase and distance. A JSON parser checks that the response is an object; the schema checks required keys and types; application logic checks the direction vocabulary, unit, permitted range and whether the target appears in current observations. If the transcript contains a competing target or the state data are stale, the interface asks for clarification or refuses to create a task. Otherwise, it presents the parsed request and checks to an authorized human or a separate coordinator. The record keeps the transcript, proposal and reason for the final branch so a developer can locate which stage failed.

The NIST AI RMF offers a useful risk-management frame around this pipeline: define roles, map how outputs will be used, measure the system in conditions resembling use, and manage incidents and updates. It does not prescribe this drone schema; it helps teams make responsibilities and evaluation visible.

Common misunderstanding

Valid JSON means the model understood the request. JSON validity concerns representation; it does not prove the transcript, target or intent is correct.

Frequently asked questions

Does the language model listen to the microphone directly?

In this design, a speech recognizer first converts audio to text. A model that accepts audio would need its own compatible input path and still needs output checks.

What if a required field is missing?

Reject the incomplete proposal or ask a focused question. Do not invent a value unless the application has an explicit, reviewed rule for it.

Can a schema guarantee a safe action?

No. It can constrain data structure and some values; the application must check context and a person or approved controller must decide what happens next.

Summary

• Audio becomes text through speech recognition; the language model interprets the transcript into candidate fields.
• Schema-guided JSON helps constrain structure, while application checks must validate meaning and state.
• Ambiguous, invalid or stale inputs should trigger clarification or rejection before any task is created.