feat: add Qwen3-ASR speech-to-text by msluszniak · Pull Request #1554 · software-mansion/react-native-executorch

msluszniak · 2026-10-21T10:37:31Z

Description

New createQwen3AsrSpeechToText task for Qwen3-ASR 0.6B, same transcribe/stream API as Whisper; language is auto-detected or forced via QWEN3_ASR_LANGUAGES.
useSpeechToText accepts Qwen3-ASR models; registry entry models.speechToText.QWEN3_ASR.SIZE_0_6B (XNNPACK 8da4w; iOS default Core ML audio tower + MLX decoder; HF react-native-executorch-qwen-3-asr @ v0.11.0).
A variant key naming two backends (MLX_COREML_INT4) is the default only when both are linked; loading a method whose backend is not linked now names it.
Speech demo screens list the model; docs section added.

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

apps/speech: pick "Qwen3-ASR 0.6B (CPU)" on the audio file or microphone screen and transcribe.

Screenshots

Related issues

#769

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

WER on 100 LibriSpeech test-clean clips: 2.46% (fp32 2.14%). Decoder layer 0 must stay 8-bit or transcripts derail.
iPhone 16: Core ML + MLX 293 ms per clip vs XNNPACK 468, WER 2.51%.
No Vulkan: on a Galaxy S26 Ultra it wins prefill but loses encoder and decode (whole clip 451 vs 360 ms).
Exporters and findings: export-scripts !41. ExecuTorch fixes found on the way: Vulkan: keep a reshape that follows a dynamically quantized linear pytorch/executorch#24055, #24056.

msluszniak · 2026-10-21T16:07:55Z

@barhanc I forgot that I need to rebuild libs for this one to cover all backends, will let you know when this will be ready since I want to batch this with Szymon's changes to MIGAN.

Audio tower on Core ML, decoder on MLX. A variant key naming two backends resolves only when both are linked.

Fixes the Qwen3-ASR docs anchor and lets the feature-map test read backends from file names.

barhanc · 2026-10-28T19:10:26Z

+## Qwen3-ASR
+
+[Qwen3-ASR](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) works with the same hook and the same `transcribe` / `stream` API. It detects the language by itself; pass `language` (one of [`QWEN3_ASR_LANGUAGES`](../../06-api-reference/variables/QWEN3_ASR_LANGUAGES.md)) only to force one:
+
+```typescript
+const stt = useSpeechToText(models.speechToText.QWEN3_ASR.SIZE_0_6B.DEFAULT);
+
+const transcript = await stt.transcribe(audioData); // auto-detect
+const french = await stt.transcribe(audioData, { language: 'fr' });
+```
+


Not sure a separate subsection in docs for a model is needed.

barhanc · 2026-10-28T19:12:09Z

+): {
+  /** Whether the model is loaded and ready to transcribe. */
+  isReady: boolean;
+  /** The download or load error, if any. */
+  error: Error | undefined;
+  /** Download progress across every asset, in percent. */
+  downloadProgress: number;
+  /** The config with every remote URL resolved to a local path. */
+  resource: WhisperSttModel<L> | Qwen3AsrSttModel | undefined;
+  transcribe: (WhisperSpeechToText<L> | Qwen3AsrSpeechToText)['transcribe'] | undefined;
+  transcribeWorklet:
+    | (WhisperSpeechToText<L> | Qwen3AsrSpeechToText)['transcribeWorklet']
+    | undefined;
+  transcribeStop: (WhisperSpeechToText<L> | Qwen3AsrSpeechToText)['transcribeStop'] | undefined;
+  stream: (WhisperSpeechToText<L> | Qwen3AsrSpeechToText)['stream'] | undefined;
+  streamInsert: (WhisperSpeechToText<L> | Qwen3AsrSpeechToText)['streamInsert'] | undefined;
+  streamStop: (WhisperSpeechToText<L> | Qwen3AsrSpeechToText)['streamStop'] | undefined;
+};


This is not needed, see TTS example.

barhanc · 2026-10-28T19:12:33Z

+    'name' in sttConfig && sttConfig.name === 'qwen3-asr'
+      ? createQwen3AsrSpeechToText(sttConfig)
+      : createWhisperSpeechToText(sttConfig as WhisperSttModel<L>);


Use the same pattern as TTS example, adding name field to existing Whisper config is a non-destructive change.

barhanc · 2026-10-28T19:13:51Z

I'm not sure about the name qwen3AsrSpeechToText as ASR means the same as speech to text, so it reads a bit weird maybe qwen3SpeechToText or qwen3AutomaticSpeechRecognition / Recognizer would be better.

barhanc · 2026-10-28T19:15:28Z

+  resource: WhisperSttModel<L> | undefined;
+  transcribe: WhisperSpeechToText<L>['transcribe'] | undefined;
+  transcribeWorklet: WhisperSpeechToText<L>['transcribeWorklet'] | undefined;
+  transcribeStop: WhisperSpeechToText<L>['transcribeStop'] | undefined;
+  stream: WhisperSpeechToText<L>['stream'] | undefined;
+  streamInsert: WhisperSpeechToText<L>['streamInsert'] | undefined;
+  streamStop: WhisperSpeechToText<L>['streamStop'] | undefined;


Suggested change

resource: WhisperSttModel<L> | undefined;

transcribe: WhisperSpeechToText<L>['transcribe'] | undefined;

transcribeWorklet: WhisperSpeechToText<L>['transcribeWorklet'] | undefined;

transcribeStop: WhisperSpeechToText<L>['transcribeStop'] | undefined;

stream: WhisperSpeechToText<L>['stream'] | undefined;

streamInsert: WhisperSpeechToText<L>['streamInsert'] | undefined;

streamStop: WhisperSpeechToText<L>['streamStop'] | undefined;

resource: WhisperSttModel<L> | undefined;

// Offline transcription methods

transcribe: WhisperSpeechToText<L>['transcribe'] | undefined;

transcribeStop: WhisperSpeechToText<L>['transcribeStop'] | undefined;

transcribeWorklet: WhisperSpeechToText<L>['transcribeWorklet'] | undefined;

// Live-streaming transcription methods

stream: WhisperSpeechToText<L>['stream'] | undefined;

streamStop: WhisperSpeechToText<L>['streamStop'] | undefined;

streamInsert: WhisperSpeechToText<L>['streamInsert'] | undefined;

barhanc · 2026-10-28T19:18:45Z

+            requireRegisteredBackends(modelPath_, methodName, backends_[methodName]);
+            auto loadCtx = std::format("Load method '{}' failed.\n"


I'm wondering if this is needed if the next line describes the first action on error is to check if required backends are registered.

barhanc · 2026-10-28T19:24:27Z

+  ...method('preprocess', [f32(Dyn('Samples'))], [f32(1, MEL_BINS, Dyn('Frames'))]),
+  ...method(
+    'audio_encoder', // One 8 s attention window: whole 100-frame chunks, plus the valid token count.
+    [f32(1, MEL_BINS, melFrames), i64(1)],
+    [f32(1, audioTokens, 'Dim')]
+  ),
+  ...method('token_embedding', [i64(1, Dyn('Tokens'))], [f32(1, Dyn('Tokens'), 'Dim')]),
+  ...method('text_decoder', [f32(1, Dyn('PrefillLen'), 'Dim'), i64(1)], [f32(1, 'VocabSize')]),


Please add explicit new lines via // prettier-ignore so that the spec is easier to read.

barhanc · 2026-10-28T19:26:50Z

+    const withTensors = <R>(created: Tensor[], fn: () => R): R => {
+      'worklet';
+      try {
+        return fn();
+      } finally {
+        created.forEach((t) => t.dispose());
+      }
+    };


Explicit try finally is imo more clear.

barhanc · 2026-10-28T19:32:20Z

+      return !inTranscript && transcript.startsWith('language ') ? '' : transcript;
+    };
+
+    const resolveLanguage = (options?: Qwen3AsrSttOptions): string | undefined => {


If this takes options explicitly, can't it be a top level helper?

barhanc · 2026-10-28T19:32:59Z

+      const language = resolveLanguage(options);
+
+      const parts: string[] = [];
+      for (const [start, end] of splitAtQuietPoints(audio, maxSamples)) {


Why not use here the VAD instead of heuristic?

msluszniak self-assigned this Oct 21, 2026

msluszniak added the feature PRs that implement a new feature label Oct 21, 2026

msluszniak marked this pull request as ready for review October 21, 2026 13:56

msluszniak linked an issue Oct 21, 2026 that may be closed by this pull request

support qwen3-asr in speech to text #769

Open

msluszniak requested a review from barhanc October 21, 2026 14:01

msluszniak marked this pull request as draft October 21, 2026 16:07

msluszniak marked this pull request as ready for review October 22, 2026 22:13

msluszniak added 5 commits October 28, 2026 17:44

feat: add Qwen3-ASR speech-to-text

f2d429f

fix: encode forced Qwen3-ASR language on the worklet runtime

d6d47d0

feat: add Qwen3-ASR Core ML + MLX variant for iOS

d4c73a9

Audio tower on Core ML, decoder on MLX. A variant key naming two backends resolves only when both are linked.

fix: name an unlinked backend when a model method loads

71b7ba7

Fixes the Qwen3-ASR docs anchor and lets the feature-map test read backends from file names.

refactor: point Qwen3-ASR at its renamed repo and current VERSION_TAG

e2fc471

msluszniak force-pushed the @ms/qwen3-asr branch from e315b13 to e2fc471 Compare October 28, 2026 16:45

barhanc reviewed Oct 28, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat: add Qwen3-ASR speech-to-text - #1554

feat: add Qwen3-ASR speech-to-text#1554
msluszniak wants to merge 5 commits into
mainfrom
@ms/qwen3-asr

msluszniak commented Oct 21, 2026 •

edited

Loading

Uh oh!

msluszniak commented Oct 21, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026 •

edited

Loading

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

barhanc Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		requireRegisteredBackends(modelPath_, methodName, backends_[methodName]);
		auto loadCtx = std::format("Load method '{}' failed.\n"

Conversation

msluszniak commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

msluszniak commented Oct 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barhanc Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msluszniak commented Oct 21, 2026 •

edited

Loading

barhanc Oct 28, 2026 •

edited

Loading