What is MuScriptor

MuScriptor is an open-source, multi-instrument music transcription model jointly developed by Kyutai and Mirelo, capable of automatically transcribing real-world music audio across diverse genres into MIDI format. As the first model of its kind trained on a large-scale real-world dataset comprising 170,000 songs, MuScriptor employs a pure decoder Transformer architecture with four parameter scales ranging from 60M to 1.4B, supports instrument-conditioned control, and is released under the MIT license.

Main Features of MuScriptor

• Multi-Instrument Audio Transcription: Automatically converts real-world music audio containing multiple instruments into standard MIDI format, supporting 36 instrument classes including piano, guitar, drums, and bass.
• Multi-Genre Music Support: Covers pop, classical, rock, heavy metal, and other genres, optimized for complex real-world mixing scenarios.
• Instrument-Conditioned Control: Allows users to specify target instrument sets for transcription, enabling customized outputs and consistent instrument recognition across segments.
• Multiple Model Sizes: Offers four parameter variants—60M, 103M, 307M, and 1.4B—balancing lightweight deployment and high-precision requirements.
• End-to-End Automated Processing: Processes input in 5-second audio chunks, automatically performing frame segmentation, feature extraction, note detection, and MIDI generation.

Technical Principles of MuScriptor

• Pure Decoder Transformer Architecture: The model takes 5-second audio clips as input, converts 16kHz mono audio into mel-spectrograms, and predicts MIDI-like token sequences via autoregressive generation, enabling end-to-end multi-instrument transcription.
• Three-Stage Training Strategy: First pre-trained on 1.5 million synthetic MIDI datasets, then fine-tuned on 170,000 real-world music tracks, followed by post-training using 300 high-quality annotated tracks via GRPO-style reinforcement learning to align output quality.
• Instrument-Conditioned Control: Supports specifying target instrument sets via prefix embeddings; users can define desired instruments, while classifier-free guidance ensures consistent instrument assignment across segments.
• Data Synthesis and Augmentation Pipeline: During pre-training, dynamic synthesis workflows apply pitch shifting, tempo variation, and random instrumentization at the symbolic level; audio is synthesized using over 250 SoundFont presets with random detuning to enhance generalization.

WeChat follow-up reply "Open Source" to join the AI Open Source Project Community

How to Use MuScriptor

• Environment Setup: Clone the GitHub repository and install dependencies; ensure system compatibility with Python and deep learning frameworks such as PyTorch.
• Download Weights: Obtain model weight files corresponding to specific parameter sizes (103M, 307M, or 1.4B) from official sources.
• Load Model: Initialize the model using provided inference code, with optional enablement of instrument-conditioned control.
• Input Audio: Feed the audio file (supporting multiple formats) into the model, which automatically splits it into 5-second segments for processing.
• Retrieve MIDI: The model outputs standard MIDI files directly importable into DAWs or notation software for further editing.

Core Advantages of MuScriptor

• Strongest Generalization on Real-World Data: Unlike models relying solely on synthetic data (e.g., MT3), MuScriptor is trained on 170,000 real multi-instrument tracks, significantly reducing error rates in complex real-world mix scenarios. Officially recognized as the best open-source multi-instrument transcription model to date.
• Reinforcement Learning Alignment for High-Quality Output: Through GRPO-style reinforcement learning in the post-training phase, the model achieves improved F1 scores at note onset, offset, and frame levels, minimizing missed detections and false positives.
• Flexible Instrument-Conditioned Control: Users can specify desired instrument types, enabling refined output and consistent instrument recognition across audio segments, avoiding fluctuations in instrument assignment between clips.
• Multi-Scale Open Source: Offers four model variants from 60M to 1.4B, catering to both edge-device deployment and high-accuracy needs; code is MIT-licensed, weights available under CC BY-NC 4.0.

Project Links for MuScriptor

• GitHub Repository: https://github.com/muscriptor/muscriptor
• HuggingFace Model Hub: https://huggingface.co/MuScriptor
• arXiv Technical Paper: https://arxiv.org/pdf/2607.08168v1

Comparison with Competing Models

Application Scenarios of MuScriptor

• Music Production Assistance: Producers can quickly convert existing audio without MIDI into editable MIDI tracks for re-arrangement, remixing, or sampling.
• Automatic Sheet Music Generation: Music education institutions and publishers can auto-convert recordings into sheet music, drastically reducing manual transcription costs.
• Music Information Retrieval: Provides high-quality symbolic input for downstream tasks such as chord recognition, key analysis, and genre classification.
• Generative Music Modeling: Supplies large-scale, high-quality MIDI training data for music generation models, advancing research in symbolic music generation.
• Music Archaeology and Archival: Converts unstructured audio from historical recordings and live performances into structured MIDI data for digital preservation and analytical study.