ICASSP 2027 · Audio demo

Speaker-Consistent Grounding for Confusion-Resilient LLM-Based Target Speech Extraction

Repair target-speaker evidence, then ground LLM generation.

Audio examples Paper results Whisper audit Code

Status: Frozen evaluations and audio examples complete.

Overview

TSE can extract the wrong speaker; LLM generation can also change words or identity. We repair target-speaker evidence before grounding generation.

Repair before grounding. CDCS-5 reduces clean wrong-speaker errors from 6.87% to 0.75%. On Noisy TEST, fixed CSG reduces content switching from 12.79% to 9.10%. See paper results ↓

**Problem illustration.** TSE can follow the wrong speaker; generation can drift; grounding can preserve a wrong-speaker anchor. CDCS-5 repairs evidence before fixed or adaptive CSG constrains generation. Waveforms are amplitude-normalized traces from released examples, not necessarily the same trial.

Design rule Repair target identity first; ground autoregressive generation second.

Paper Figure 1: CDCS-5 selects target-consistent evidence from five frozen TSE estimates; evidence tokens, mixture features and target-speaker conditioning guide Qwen-TSE, with CSG constraining autoregressive decoding. — **Paper Figure 1.** Repair before grounding: (a) CDCS-5 selects target-speaker-consistent evidence, (b) the selected waveform is returned directly or tokenized for Qwen-TSE, and (c) CSG constrains autoregressive decoding. Full resolution ↗

Method

CDCS-5 selects the most enrollment-consistent of five frozen TSE candidates. Its waveform is returned directly or used to ground Qwen-TSE. Fixed CSG uses $(\lambda,w)=(1,0)$ . Adaptive CSG uses $\lambda=g(\xi)$ and $w=1$ , with a DEV-frozen decreasing map and one strength per utterance. Both use no clean references, transcripts, oracle labels, or model updates. Grounding constrains drift; it cannot repair a wrong-speaker anchor.

Token-level grounding intuition

Each acoustic token has eight ternary FSQ coordinates. CSG changes Qwen-TSE token selection, not the CDCS-5 selector.

Qwen-TSE proposes

High-scoring next tokens can drift from the evidence.

EvidenceRaw proposal

02110201

22012200

CSG constrains

Penalize tokens with greater Hamming distance from the evidence.

s_{\mathrm{CSG}}(v,t)=s_{\mathrm{LLM}}(v,t)-\lambda\min_{|\delta|\le w}d_H\!\left(v,z^e_{t+\delta}\right)

Fixed

\lambda=1,\;w=0

Adaptive

\lambda=g(\xi),\;w=1

near: small penaltyfar: large penalty

Adaptive CSG sets strength

A mixture–evidence projection score sets one grounding strength per utterance through a DEV-frozen decreasing map.

\xi(y,c^\star)=10\log_{10}\!\frac{\lVert\alpha^\star c^\star\rVert_2^2+\epsilon}{\lVert y-\alpha^\star c^\star\rVert_2^2+\epsilon}

lower

\xi

stronger grounding higher

\xi

weaker grounding

One

\lambda

per utterance · no token-level adaptation

Grounding intuition. Fixed CSG uses

\lambda=1,\;w=0

; adaptive CSG uses

\lambda=g(\xi),\;w=1

. Strength stays constant during each decode. The mapping is selected on DEV and kept fixed for TEST.

Audio examples

One clean and four noisy examples from frozen TEST evaluations. Each includes all five systems in Table 1, in the same order. Clean targets and interferers are for evaluation only, never system inputs.

Listen for: speaker identity, word accuracy, and the fidelity–naturalness trade-off.

Clean example — BSRNN-TSE wrong-speaker failure

BSRNN-TSE follows the interferer. A two-second enrollment view recovers the target: $\operatorname{WER}:100\%\rightarrow0\%$ , $m_{\mathrm{spk}}:-0.285\rightarrow+0.520$ , and $\mathrm{A\text{-}sw.}:1\rightarrow0$ .

Input, enrollment & interferer

Clean mixturesystem input

Target enrollmentspeaker condition

Clean interfererevaluation only

Clean targetevaluation only

BSRNN-TSEwrong speaker · WER 100%

BSRNN-TSE-CDCS-5middle view · WER 0%

Qwen-TSE-UDWER 8.33%

Qwen-TSE-fixed-CSGWER 8.33%

Qwen-TSE-adaptive-CSGWER 8.33%

Whisper large-v3 · greedy English · repetition penalty 1.10 · ASR target reference

Target · large-v3: this attitude springs from a false conception of sin the conception that sin is a small matter easily taken care of by good works
BSRNN-TSE · large-v3: he obtained the desired speed and load with a friction brake also regulated speed but waited for an indicator to verify it WER 100%
BSRNN-TSE-CDCS-5 · large-v3: this attitude springs from a false conception of sin the conception that sin is a small matter easily taken care of by good works WER 0%
Qwen-TSE-UD · large-v3: this attitude springs from a false conception of sin the conception that sin is a small matter easily taken care of by good works that we WER 8.33%
Qwen-TSE-fixed-CSG · large-v3: this attitude springs from a false conception of sin the conception that sin is a small matter easily taken care of by good works that we WER 8.33%
Qwen-TSE-adaptive-CSG · large-v3: this attitude springs from a false conception of sin the conception that sin is a small matter easily taken care of by good works that we WER 8.33%

Speaker confusion: the BSRNN-TSE extraction follows the interferer even without noise. All three Qwen transcripts add “that we” after the target text (WER 8.33%).

Spectrogram

Illustrative cases, not aggregate results. Case identities and existing waveforms are unchanged. Missing Qwen outputs were retrieved from the same completed TEST evaluations; displayed transcripts and WER use one large-v3 configuration. Full selection rules and results: clean record · noisy record.

Results from the paper

Evidence repair and grounded generation

CDCS-5 reduces speaker confusion. CSG further stabilizes the generative branch while retaining its predicted-quality advantage.

Evidence repair · Clean TEST6.87 → 0.75%Wrong-speaker errors · Full → CDCS-5

Grounding · Noisy TEST12.79 → 9.10%Content switching · UD → fixed CSG

Grounding · Noisy TEST1.68 → 0.51%Acoustic switching · UD → fixed CSG

System comparison

6,000 natural-noise + 2,400 controlled-SNR trials. All systems use the same evaluation pool.

Noisy TEST · Table 1
System	WER ↓%	C-sw. ↓%	A-sw. ↓%	Spk. margin ↑	DNSMOS ↑P.808	UTMOS ↑
BSRNN-TSE	39.85	18.79	18.23	0.266	3.031	2.001
BSRNN-TSE-CDCS-5	30.67	11.02	10.39	0.327	2.980	1.948
Qwen-TSE-UD	46.70	12.79	1.68	0.377	3.494	3.216
Qwen-TSE-fixed-CSG	42.11	9.10	0.51	0.386	3.498	3.147
Qwen-TSE-adaptive-CSG	42.73	9.37	0.53	0.388	3.503	3.253

3,000 clean mixtures, evaluated in both target-speaker directions.

Clean TEST · Table 1
System	WER ↓%	C-sw. ↓%	A-sw. ↓%	Spk. margin ↑	DNSMOS ↑P.808	UTMOS ↑
BSRNN-TSE	11.41	7.00	6.98	0.449	3.688	3.606
BSRNN-TSE-CDCS-5	7.02	0.88	0.78	0.506	3.699	3.647
Qwen-TSE-UD	8.99	0.92	0.43	0.421	3.705	3.913
Qwen-TSE-fixed-CSG	8.04	0.92	0.23	0.422	3.699	3.910
Qwen-TSE-adaptive-CSG	8.12	0.90	0.20	0.421	3.711	3.921

Bold: best value per column and condition. CDCS-5 returns the repaired waveform; fixed/adaptive CSG ground the same Qwen-TSE checkpoint. WER measures agreement with automatic target transcripts. DNSMOS and UTMOS predict quality.

Paper Figure 2: acoustic-switch rate and DNSMOS P.808 versus SNR from minus 5 to 15 dB, comparing five TSE systems. — **Paper Figure 2.** Controlled-noise TEST, 480 trials per SNR. Acoustic-switch rate uses a logarithmic axis; quality is DNSMOS P.808. Full resolution ↗

CDCS-5 ablation & selection safety Table 2

Full/TF use the complete enrollment; E/M/L use early, middle and late local views. Wrong-speaker errors use the SI-SDR margin, not the CAMPPlus acoustic-switch definition in Table 1.

Table 2(a) · Wrong-speaker errors (%, ↓)
Configuration	Clean DEV	Clean TEST	Noisy DEV	Noisy TEST
Full (no CDCS)	6.90	6.87	19.85	18.95
Early (E)	8.52	8.57	20.61	19.96
Middle (M)	8.27	8.90	20.92	20.73
Late (L)	8.52	8.50	20.71	20.18
TF-context (TF)	2.35	2.53	14.32	14.37
Full + E/M/L	3.70	3.60	15.96	15.00
Full + TF	1.17	1.18	12.00	11.30
CDCS-5	0.88	0.75	11.02	10.46

Table 2(b) · High-confidence confusion recovery & new errors
Split	CDCS-5 recovery	Oracle availability	New errors
Clean TEST	355/398 (89.20%)	360/398 (90.45%)	2/5,588 (0.04%)
Noisy TEST	374/646 (57.89%)	431/646 (66.72%)	8/4,709 (0.17%)

Recovery and new-error rates use their respective high-confidence subsets, shown in the denominators. Oracle availability uses clean references for analysis only; CDCS never receives these references.

Source: manuscript Tables 1–2 and Figures 1–2, checked 25 October 2026. Download table values & protocol · Metric definitions

Listening audit · selected Natural Noisy TEST example

When Whisper-based WER becomes misleading

A generated waveform and its ASR transcript can fail in different ways. This example shows why we interpret WER together with audio, speaker consistency and predicted quality.

7.68 seconds of audio. 419 insertion errors.

ASR transcript repetition

Whisper large-v3 transcribes the UD output as “out zander paced up and down the hallway f f f …”. The repeated transcript dominates WER; it does not establish that the waveform contains hundreds of spoken repetitions.

17 substitutions+0 deletions+419 insertions

23 reference words

× 100% =1,895.65% WER

WER = (S + D + I) / reference words × 100%. Insertions can push it above 100%.

Clean targetReference audio · evaluation only

Qwen-TSE-UD7.68 s · WER 1,895.65% · speaker margin +0.467

Qwen-TSE-fixed-CSG7.68 s · WER 34.78% · speaker margin +0.485

BSRNN-TSE-CDCS-57.67 s · WER 8.70%

Read the target, UD and grounded ASR transcripts

Clean target · Whisper large-v3: alexander paced up and down the hallway buttoning and unbuttoning his overcoat until she returned and took him up to hildas living room
UD · full automatic transcript: out zander paced up and down the hallway f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f
Fixed CSG · Whisper large-v3: alexander paced up and down the hallway buttoning and unbuttoning his overcoat until she returned into the hotel of his

What is measured: an insertion-heavy ASR transcript, against a 23-word automatic reference.
What to listen for: actual words and repetitions in the waveform. A positive speaker margin confirms neither transcription accuracy nor correct content.
What remains unverified: the exact split between speech-generation and recognition errors; this audit has no human-adjudicated transcript or second-recognizer check.

Post-hoc example, selected for the largest large-v3 insertion count in the complete Natural Noisy TEST audit. English transcription, greedy decoding, repetition penalty 1.10. Clean-target reference text is also automatic. This is not a failure-rate estimate. Full audit & audio hashes.

Evaluation metrics

Introduced in this work

Two operational diagnostics for speaker confusion

WER alone does not identify speaker confusion. We measure both lexical and acoustic switches.

C-sw.↓ lower is better

Content-switch rate

The fixed-ASR output transcript has lower WER against the clean interferer transcript than the target transcript. Ties do not count.

c_{\mathrm{sw},n}=\mathbf{1}\!\left[\operatorname{WER}(\tau_{I,n},\hat\tau_n)<\operatorname{WER}(\tau_{T,n},\hat\tau_n)\right]

A-sw.↓ lower is better

Acoustic-switch rate

Frozen CosyVoice CAMPPlus places the output closer to the clean interferer than to the clean target in cosine similarity. Ties do not count.

\begin{aligned}m_{\mathrm{spk},n}&=\cos\!\left(f(s_{T,n}),f(\hat s_n)\right)\\&\quad-\cos\!\left(f(s_{I,n}),f(\hat s_n)\right)\end{aligned}

a_{\mathrm{sw},n}=\mathbf{1}\!\left[m_{\mathrm{spk},n}<0\right]

\mathrm{C\text{-}sw.}=\frac{100\%}{N}\sum_{n=1}^{N}c_{\mathrm{sw},n}

\mathrm{A\text{-}sw.}=\frac{100\%}{N}\sum_{n=1}^{N}a_{\mathrm{sw},n}

Study-defined diagnostics, not perceptual ground truth. Clean references are for evaluation only. CAMPPlus evaluates identity independently of the ECAPA selector.

WER ↓: $\operatorname{WER}=\frac{S+D+I}{N_{\mathrm{ref}}}\times100\%$ . Reported as the uncapped utterance mean against a same-ASR clean-target transcript.
Spk. margin ↑: Target-minus-interferer CAMPPlus cosine similarity; a negative value triggers A-sw.
DNSMOS P.808 ↑: Predicted P.808 speech quality for both clean and noisy conditions.
UTMOS ↑: Reference-free predicted naturalness; it does not establish content or speaker identity.

Speaker-Consistent Grounding for Confusion-Resilient LLM-Based Target Speech Extraction

Overview

(a) Deterministic waveform TSE

(b) Ungrounded generation

(c) Grounding a bad anchor

(d) Our principle