How Can Voice Command Errors Be Measured?

• Test-set audio
• Compared outcomes
• Aggregate metric, with no fabricated numeric data

Build a reference set before running the system

Write the phrases testers will say in advance and label the expected answer: reference transcript, intent, required parameters, or “no valid command”. Record speech under intended conditions such as different speakers, noise and pronunciation. Run the system on those recordings and save its transcript, parsed intent, parameters and final decision. Do not edit reference phrases after seeing the system's errors.

Calculate word-level errors

For a transcript, align the recognised text with the reference text to find the smallest number of substitutions (S), deletions (D) and insertions (I). WER = (S + D + I) / N, where N is the number of words in the reference. WER percent is WER multiplied by one hundred. A substitution replaces a word with another; a deletion is a reference word missing from the output; an insertion is an output word that was not spoken. WER can exceed one if there are many inserted words, so it is not a probability capped at one.

Hypothetical example: a set of ten reference words has one substitution error, one deletion error and one insertion error. Then WER = (1 + 1 + 1) / 10 = 0.3, or thirty percent by word. The denominator is the reference word count, not the number of sentences or output words. For Vietnamese, state how words are tokenised because spaces do not always correspond to a lexical unit in the same way as in English.

Calculate command-level errors

For drone commands, make a per-utterance table: correct command, wrong command, missing parameter, rejected, or no response. Choose the success rule in advance: for instance, a command counts as correct only when its intent and every required parameter match. Then report each category as a proportion of all valid attempts. Do not mix this number with WER: a transcript with one wrong word may preserve the intent, while a short phrase can become a completely different action.

Hypothetical example with ten attempts: seven are correct, one becomes another command, one is rejected, and one gets no response. Correct-command rate is 7/10; wrong-command rate is 1/10; rejection rate is 1/10; no-response rate is 1/10. A rejection is not necessarily an unsafe error: it may be the correct response when the system is uncertain. Report rejection and misrecognition separately to show the tradeoff between ignoring a request and acting incorrectly.

Measurement boundary and interpretation

Record whether timing starts when the speaker begins or when audio reaches the microphone; record whether it ends at transcript, intent or final decision. State whether each denominator is reference words, spoken phrases or command attempts. Keep the phrase set, noise conditions, model version, reject threshold and text-normalisation rules fixed when comparing runs. The AWS evaluation guide emphasises choosing a method for the use case, normalising transcripts consistently and sampling suitable speech; one aggregate score does not replace analysis of specific errors.

Summary

• WER counts substitutions, deletions and insertions relative to reference words.
• Command rates use attempts as the denominator and need an explicit definition of a correct command.
• Rejection, silence and a wrong command are distinct outcomes; report them separately to understand risk and usability.