[Bug] SenseVoice Chinese text clipping misses multi-character queries and uses incorrect timestamp indices

[LauraGPT]

Summary

SenseVoice can produce Chinese transcription, token timestamps and SRT successfully, yet selecting that same Chinese text for audio clipping fails. The saved raw transcript is contiguous Chinese, while the query is converted to space-separated characters; single-character matches also use the wrong timestamp index.

Environment

• FunClip: current main 9e720617949e32da4c0a564d10cf5a64337fe2c2, with relevant source files verified unchanged.
• FunASR: 1.4.15.
• Linux x86-64; Python 3.12.3; isolated venv with the repository requirements.
• torch/torchaudio 2.10.0+cpu, NumPy 1.26.4, librosa 0.11.0, soundfile 0.14.0, ModelScope 1.40.0.
• Gradio 4.44.1 installed, but this reproduction calls the Python API, not a browser.
• Actual CPU model loading and inference; no ASR stand-in in the original acceptance run.

Audio or video input

The original acceptance used an 18-second, 16 kHz mono Chinese WAV with actual create_asr_model("sensevoice", "zh", ...), SenseVoiceSmall, FSMN VAD and CAM++. No hotwords or LLM. All 40 cached model files matched public ModelScope metadata size/SHA-256.

The small transcript/timestamp excerpt below is sufficient to reproduce the matching defect without downloading models or sharing the audio. It is an excerpt of saved real recognition state, not an acoustic accuracy reference.

Steps to reproduce

From the repository root in an installed environment:

import sys
sys.path.insert(0, "funclip")
from utils.trans_utils import pre_proc, proc

raw = "嗯那么今天我们就简"
timestamps = [
    [230, 290], [590, 650], [710, 770],
    [890, 950], [1010, 1070], [1250, 1310],
    [1430, 1490], [1610, 1670], [1970, 2030],
]
for query in (raw[:8], "简"):
    print(repr(query), repr(pre_proc(query)),
          proc(raw, timestamps, pre_proc(query)))

# Diagnostic control only: not a production fix.
for query in (raw[:8], "简"):
    print("token-spaced control",
          proc(pre_proc(raw), timestamps, pre_proc(query)))

In the original real-model run, VideoClipper.recog((16000, data), sd_switch="No") returned 76 characters, 76 timestamps and three SRT entries. Calling clipper.clip("嗯那么今天我们就", 0, 0, state) with that untouched state returned the full input instead of a selected excerpt.

Expected behavior

Queries copied from the recognized text should map to their corresponding timestamps regardless of whether the model's raw Chinese text includes token-separating spaces.

For the excerpt above, the first eight characters map to 230-1670 ms, or sample indices [3680, 26720] at 16 kHz. The ninth character maps to 1970-2030 ms, or [31520, 32480]. These are expected mappings to the model-returned timestamps, not a claim that the timestamps are acoustically perfect.

Actual behavior

The multi-character query returns no ranges. The single-character query finds the character but maps it to the first timestamp, 230-290 ms.

The actual full audio-API run returned:

No period found in the speech, return raw speech. You may check the recognition result and try other destination text.

The serialized PCM16 WAV was still 288,000 samples / 18 seconds at 16,000 Hz, and the clip SRT was empty. Recognition/SRT generation succeeded; end-to-end text clipping did not.

Logs or traceback

Minimal matching output:

'嗯那么今天我们就' '嗯 那 么 今 天 我 们 就' []
'简' '简' [[3680, 4640]]
token-spaced control [[3680, 26720]]
token-spaced control [[31520, 32480]]

There is no matching exception. The real acceptance harness exited with an AssertionError because the clip message reported no match and the output was the original audio.

The mismatch is in the contract between _normalize_recognition_result, pre_proc and proc: normalized raw text can fall back to continuous text/text_tn; query preprocessing inserts spaces; matching searches the un-tokenized raw string, and its timestamp index counts raw spaces. Inserting spaces into the saved raw text explains this particular failure, but is only a diagnostic control. A repair must preserve token/timestamp alignment, punctuation, mixed-language text, repeated matches and the existing ASCII case behavior.

Screenshots or clips

The returned transcript excerpt, model timestamps, matcher outputs and serialized-WAV measurements above demonstrate the issue. No browser/video-export or speaker-diarization acceptance is claimed; the speaker-enabled case was not reached after the plain clipping assertion failed.

Related issues were checked before filing: #161 resolved rich-tag/subtitle-return handling; #88 requested SenseVoice support; #198 fixed English case matching and has reporter confirmation; #60 discussed long multi-selection input without this concrete model-output reproduction. #214 is a separate non-16 kHz normalization defect; this run uses 16 kHz throughout. Those issues are not being reopened.

This report was prepared with Codex assistance. No production repair or release is claimed.

[modelpath-dev]

I will take this issue. Please assign it to me.

The problem seems to be with the way Chinese text is processed for audio clipping. The issue arises when the raw transcript is contiguous, but the query is space-separated. I would first inspect the pre_proc function in trans_utils.py to understand how it processes Chinese text. I suspect the mismatch in tokenization is causing incorrect timestamp mapping. I will adjust the tokenization logic to ensure it aligns with the raw transcript format. I will verify the fix by checking if the queries map correctly to their timestamps.

[LauraGPT]

Thanks for volunteering, @modelpath-dev. A focused draft PR is welcome. GitHub currently reports that your account is not assignable in this repository, so I have not added an assignee.

Please preserve an explicit token-to-timestamp mapping, rather than only inserting/removing spaces in pre_proc: the multi-character miss and the single-character wrong-index result both need coverage. The report's saved-state example is sufficient for a small regression without downloading models. Please also cover punctuation, mixed Chinese/English, existing English case handling, repeated matches, and matching against both contiguous and token-spaced raw transcripts. Keep #214's sample-rate normalization issue separate, and retain unchanged behavior for already-working inputs.

This is coordination on an open bug, not confirmation that a repair or design has been accepted. Please outline the mapping approach in the draft PR so it can be reviewed before merging. No need to repeat the original full model download/inference solely to reproduce this matcher defect.

Prepared with Codex assistance.

[modelpath-dev]

Got it, Laura. I'll focus on creating a token-to-timestamp mapping and ensure coverage for the scenarios you mentioned. I'll outline the approach in the draft PR for review. Thanks for the guidance!