2026-11-13 15:49

Introduction: Edit individual words or emotions without re-recording, preserving original voice quality. What is dots.tts.edit V1.1? dots.tts.edit V1.1 is

What is dots.tts.edit V1.1?

dots.tts.edit V1.1 is an end-to-end text-to-speech (TTS) editing model open-sourced by Xiaohongshu, supporting localized style description-based editing. dots.tts.edit V1.1 enables direct modification of existing audio recordings—replacing specific words, altering sentence emotion (e.g., transforming to sadness), adjusting segment speed—while preserving content and voice characteristics unchanged elsewhere. Multiple edits can be executed in a single operation, yielding natural and fluent output. Furthermore, prosody and emotional editing performance has been significantly enhanced, establishing state-of-the-art results among open-source models.

Key Features of dots.tts.edit V1.1

• Text Editing: Replace, insert, or delete individual words in the recording (e.g., changing “brave” to “small”), keeping all other content intact.
• Emotion Editing: Transform a sentence into a specified emotional state, such as shifting tone to sadness.
• Prosody Editing: Adjust local pitch and speech rate—for example, slowing down the delivery speed of a particular segment.
• Pause Editing: Add or remove pauses at designated text boundaries to control rhythm and sentence segmentation.
• V1.1 New Feature: Local Style Description Editing: Supports free-form natural language descriptions for editing localized segments, with markedly improved prosody and emotion editing performance.
• Composite Editing: Multiple edits across different types can be completed in one request while maintaining overall audio fluency and naturalness.

Technical Principles Behind dots.tts.edit V1.1

• Transcript-Aware Semi-Formalized Edit Instructions: The model employs a “semantic timeline”-style interface using structured operation tags (e.g., <sub>, <del>, <ins>, <emo>, <pitch>, <rate>, <pause/>) to bind edit operations and parameters to specific text fragments or boundaries within the transcript. This avoids explicit time-stamp alignment while enabling composite edits as externally inspectable, verifiable “contracts”—ambiguous instructions are rejected outright.
• Continuous Autoregressive Architecture-Based Editing Model: dots.tts.edit is built upon the continuous autoregressive TTS architecture of dots.tts: a frozen AudioVAE first encodes 48 kHz audio into continuous latent variables; a semantic encoder compresses newly generated audio blocks into compact embeddings, which are fed into a LLM initialized from Qwen2.5-1.5B; at each audio step, the LLM outputs a hidden state that drives a flow-matching DiT acoustic head to denoise the next audio block, using a frozen CAM++ speaker x-vector as side input.
• Task-Driven Data Construction Pipeline: Specialized data pipelines are designed for each edit type, generating “operation-range” controlled training sample pairs: the target region is modified per instruction, while content outside remains fully preserved from the source audio’s original context, teaching the model precise editing with local consistency maintained in unedited regions.
• Bilingual Benchmark doteBench & Performance Validation: Accompanying release of doteBench benchmark includes 2,081 bilingual test cases (Chinese and English), covering text, emotion, prosody, pause, and composite editing. Evaluation spans three dimensions: instruction adherence, local preservation, and audio quality. Experiments show the model outperforms existing open-source solutions in both instruction adherence and local preservation across five edit tasks, with audio quality comparable to current open-source systems; on Seed-TTS-Eval, recognition error rate and speaker similarity differences versus the base TTS model are negligible, confirming no degradation in synthesis quality due to editing capabilities.

Follow WeChat and reply “open source” to join the AI Open Source Project Community Group

How to Use dots.tts.edit V1.1

• Environment Setup: Create a Python 3.10–3.12 conda environment and activate it (conda create -n dots_tts python=3.10 -y && conda activate dots_tts).
• Install Package: Execute pip install dots.tts to install from PyPI, or clone the repository and run pip install -e . -c constraints/recommended.txt for local installation.
• Prepare Source Audio: Prepare a recording file needing editing. Transcribed text is optional; if omitted, the model automatically infers it from the instruction.
• Write Edit Instruction: Compose an edit instruction using structured tags, e.g., replace “brave” with “small” via 'Hello <sub targ="small">brave</sub> world.'. Tags also support <del>, <ins>, <emo>, <pitch>, <rate>, <enhance>, <pause/>, <spk_transfer/>.
• Run via Command Line: Execute dots.tts.edit --model-name-or-path dots-studio/dots.tts.edit --source-audio /path/to/source.wav --instruction '...' --output edited.wav; the model will automatically download and cache the weights on first use.
• Use Python API: Load with DotsTtsEditRuntime.from_pretrained("dots-studio/dots.tts.edit", precision="bfloat16") and call generate_edit(source_audio_path=..., instruction=...) to generate the edited result.
• Use Web Interface Playground: Navigate to apps/edit_playground/frontend, run npm ci && npm run build, then execute python apps/edit_playground/app.py --model-name-or-path dots-studio/dots.tts.edit --optimize and access the visual interface in your browser.
• Speaker Guidance Control: When edit instructions involve pure emotion/background operations, x-vector speaker guidance is disabled by default; text/pitch/speed edits enable it by default. Use --use-xvector on/off to force override.
• Sampling Settings: Default num_steps=10, guidance_scale=1.2; if quality is unsatisfactory, increase steps to 10–32 for retry.
• Online Quick Try: Directly use Hugging Face Space (dots-studio-dots-tts-edit.hf.space) to upload audio and input instructions for real-time playback.

Core Advantages of dots.tts.edit V1.1

• Precision and Control: Structured edit instructions anchored to the transcript precisely bind modifications to target text segments.
• Strong Local Preservation: Edits affect only target regions; remaining content, voice quality, and prosodic features are preserved exactly as-is, achieving top-tier open-source performance.
• Multi-Dimensional Editing Coverage: Supports five edit types: text replacement/insertion/deletion, emotion, pitch/speed, pause, and speaker transfer. V1.1 adds free-form localized style description editing.
• Single-Step Composite Editing: Multiple edits of different types can be executed in parallel within a single inference pass, preserving full audio fluency and naturalness.
• Continuous Autoregressive Architecture with Lossless Audio Quality: Built on a discrete-token-free continuous generation framework (48 kHz AudioVAE + LLM + flow-matching acoustic head), resulting in post-edit audio quality equivalent to the base TTS model.

Project Links for dots.tts.edit V1.1

• GitHub Repository: https://github.com/studio-dots-ai/dots.tts
• Hugging Face Model Hub: https://huggingface.co/dots-studio/dots.tts.edit
• arXiv Technical Paper: https://arxiv.org/pdf/2608.02673

Competitive Comparison: dots.tts.edit V1.1 vs. Alternatives

Use Cases for dots.tts.edit V1.1

• Podcast & Audiobook Post-Production: Correct misread words or unsatisfactory tones without re-recording—edit locally to drastically reduce rework costs.
• Voice Dubbing & Film Production: Rapidly generate multiple emotional versions (e.g., sad, excited) of the same line for director comparison and selection.
• Short Video & On-Camera Content: Creators fix verbal slips or adjust pacing and rhythm while preserving original voice quality, enabling reuse of a single asset across multiple projects.
• Online Education & Lecture Recording: Revise incorrect statements or update outdated content without re-recording entire lessons.
• Voice Assistants & Customer Service Systems: Quickly correct prompt audio or adjust response tone without re-recording entire voice libraries.

Source: AI Tools Collection