Local music generation
YuE2 local music generation: make vocal songs and arrangements from lyrics
Follow the process of turning lyrics into a song with vocals and accompaniment, then editing its score in YuE2. The documented starting point is Linux, an NVIDIA GPU1 with BF162 support, and 24 GB of VRAM3. Test Korean singing with a short lyric first.
First distinguish YuE v1 from current YuE2
The repository URL is unchanged, but its main-branch README now describes YuE2. It says the earlier YuE code, docs, and license remain on the YuE-v1 branch.
Do not combine dependency instructions, GPU memory claims, or checkpoint4 names from older YuE tutorials with this setup. The commands and interfaces on this page target YuE2.
If you specifically need to reproduce original YuE, consult the YuE-v1 branch, its Japanese/Korean checkpoint family, and the terms for those weights separately.
YuE2 plans a score from lyrics and style, then builds a full song around it. The default output is 48 kHz stereo audio with vocals and accompaniment. The first run downloads model files, so allow for disk space and network time. This guide also covers revising the generated score yourself.
Check your GPU and operating system
The official quick start uses Linux, Python 3.12, an NVIDIA GPU with BF16 support, and 24 GB of VRAM. The README says it generates 48 kHz stereo without quantization5.
Check your GPU model and VRAM before installing, and leave headroom in system RAM6 and disk space too. A card with less than 24 GB is outside the documented starting configuration.
An out-of-memory failure there is not a benchmark of the model's quality or speed.
The official README and Hugging Face quick start document the NVIDIA CUDA7 path. They do not list Apple silicon Metal, AMD ROCm, or native Windows as supported installation targets. Check current upstream documentation before trying those devices, and treat any successful run as a separate configuration.
Install in an isolated Python environment and make a first song
Prepare Python 3.12 and install the official repository in a fresh virtual environment8. Its PyTorch9 and CUDA dependencies may conflict with packages used by other music tools, so a separate environment is easier to manage.
The commands below follow the repository README's quick start. Once installation finishes, use the included example request for a first generation.
Model weights are not bundled in the package; they download on first use.
Without a request file, the basic CLI10 uses the repository's default request. Choose a new output path so you do not overwrite a previous run.
After the first download finishes, listen to audio.flac. The output folder also contains score.abc11, settings, and intermediate artifacts.
If generation stops early or reports a truncated result, record that state instead of treating it as a complete song.
git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .python examples/generate.py --output outputs/first-songLyrics, vocals, and what instrumental generation means
YuE2's main generation example takes lyrics and a style description to create a complete song with vocals and accompaniment. Keep the style brief focused on genre, instruments, vocal character, language, and tempo; mark lyric sections with tags such as [Verse] and [Chorus].
Compatible cues such as 'soulful low-register lead, slow neo-soul, Rhodes and bass, 82 BPM12' make the request easier to inspect. The model is not guaranteed to follow every requested voice trait or tempo exactly, or sing every lyric word without omission.
If you need accompaniment without vocals, account for this distinction first. The README and default example focus on vocal songs generated from lyrics.
The README does not explicitly establish that an 'instrumental' style phrase always removes vocals or that a dedicated lyric-free mode is supported. If a strictly instrumental result matters, test a short piece and listen for vocal leakage.
For a production that requires guaranteed absence of vocals, prepare a vocal-separation or another dedicated generation path.
Separate Korean input acceptance from Korean singing quality
The YuE2 README headline says “All languages,” and its public examples include English and Chinese songs. The model card, however, tags Chinese and English and does not show a Korean-specific checkpoint or quality evaluation.
Those broad language claims and tags do not establish Korean singing quality. Original YuE v1 has Japanese/Korean checkpoints; keep the versions distinct and audition a short YuE2 lyric for pronunciation and syllable timing.
Start with one or two Korean lyric lines rather than a full song. Note whether the model drops or substitutes characters, how it handles final consonants and liaison, and whether it stretches vowels or merges syllables to fit the beat.
Mixing English or Chinese examples into the same test can obscure what you are evaluating. If Korean is a core requirement, listen across several vocal registers and tempos, and get human review and consent before using the result.
Understand full, melody, off, and the generation stages
The default full mode first creates an editable melody-and-chord plan, then generates the song from that plan. Melody mode plans only the melody; the docs recommend it for covers where the accompaniment should adapt to a new style.
Off generates directly from lyrics and style without a score plan. You can also provide your own composition through the abc input, but it must use the ABC form supported by the repository.
Other ABC dialects may need conversion.
The staged API13 is plan() → generate_semantic() → synthesize() → decode14(). Planning saves a score.abc and plan files; the semantic stage generates tokens15 representing the music.
Synthesis computes acoustic latents16, and the YuE2-Vae17 decoder reconstructs the final stereo waveform. YuE2 does not use YuE v1's separately named Stage 1 / Stage 2 checkpoints or its XCodec Mini reconstruction/vocoder path.
Keep those terms with the legacy version: YuE v1 uses a 7B Stage 1 model for lyric/audio prompting, a 1B Stage 2 model to generate audio-code sequences, and XCodec Mini to turn codes into waveform output. This version distinction matters when comparing memory and timing claims.
Neither pipeline18 separation is the same as exporting isolated vocal and accompaniment stems.
Revise a song by editing a copy of its score
Keep the generated score.abc, then edit a copy. If you want to change melody pitches or rhythm, harmony, or section length, start with one change and listen.
YuE2's editable score makes the composition inspectable, but it is not an audio editor that paints over part of a waveform while preserving the rest. Supplying a revised score generates a new full song.
Keep the original audio, original score, and revised version in separate folders.
You can call the stages separately in Python. The official generation guide saves a plan, reloads it, and continues from semantic token generation.
When editing, do not modify the saved plan in place: revise a copy of score.abc and pass it as the abc argument of a new request. Keeping the configuration, model identities, tokens, latents, and truncation status from save_artifacts() makes it easier to trace how a result was created.
import json
from pathlib import Path
from yue2 import YuE2Pipeline
request = json.loads(Path("examples/song.json").read_text(encoding="utf-8"))
with YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", device="cuda") as pipe:
song = pipe(**request)
song.save_artifacts("outputs/my-song")
print(song.truncated)Read timing and memory figures with their test conditions
The official quick start recommends an NVIDIA GPU with 24 GB VRAM and 24 GB available host RAM. The card's roughly 11 GiB peak VRAM figure does not mean an 11 GB card is sufficient: maximum-context testing reached 14.08 GiB, and results vary by GPU and runtime19. See the hardware guide when comparing other devices.
The Hugging Face model card reports official RTX 4090 measurements. With PyTorch 2.10, Transformers 4.57.6, CUDA graphs20 and FlashAttention21, no quantization, BF16 AR/NAR22, FP3223 VAE, and the default YuE2-Vae, full mode generated 214.85 seconds of audio in 71.04 seconds at 11.18 GiB peak VRAM; melody generated 214.67 seconds in 68.68 seconds (11.02 GiB); off generated 196.88 seconds in 57.91 seconds (11.09 GiB).
The 4090 figures average 32 warm requests per mode and exclude initial model loading and saving. These are model-card results, not measurements from this site.
Check the weight license and output rights separately
YuE2 code and documentation are under Apache 2.0, while the model weights are under CC BY-NC 4.0 with additional permission. The official terms separately let individual creators and musicians use the model and publish, distribute, sell, license, or monetize their outputs.
Companies that want to use the model weights commercially must ask about a separate commercial license. Do not read the code license as permission for the weights.
Checkpoint files, third-party tools, and datasets can have their own terms.
Permission to use the model does not grant rights to another person's lyrics, composition, recording, or a singer's voice or identity. For a cover, separately check rights in the source score, lyrics, recording, and any vocal sample or consent.
Whether an output qualifies for copyright protection or can be distributed on a platform depends on local law and that service's policies. This is not a legal determination; check with the relevant rights holders and qualified advisers before a commercial release.
Terminology notes
GPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textBF16 — A 16-bit floating-point format for storing and computing model values. Support depends on the hardware and runtime.
Back to the textVRAM — Memory used by a graphics card’s GPU for model weights and intermediate values. It is distinct from system RAM.
Back to the textCheckpoint — A file containing saved model weights and related state. Versions or tasks in one model family may use different checkpoints.
Back to the textQuantization — Representing model values with fewer bits. Memory use, accuracy, or execution speed may change; the effects depend on the format and implementation.
Back to the textSystem RAM — System memory that temporarily holds data while programs run. It differs from storage and from a discrete GPU’s VRAM.
Back to the textCUDA — A software platform for general-purpose computing on NVIDIA GPUs. Programs built for CUDA are not guaranteed to run unchanged on other GPUs.
Back to the textPython virtual environment — An isolated space for installing Python packages per project. It helps reduce version conflicts and is not a virtual machine.
Back to the textPyTorch — A software framework for building and running AI models. Check the compatible PyTorch version and hardware support along with the model.
Back to the textCLI — Short for Command-Line Interface: operating a program by entering commands in a terminal.
Back to the textABC notation — A text notation for score information such as pitch and rhythm. It is not an audio file, and tool support can vary.
Back to the textBPM — Short for Beats Per Minute, a tempo measure indicating the number of beats per minute.
Back to the textAPI — A defined interface that lets other code call a program’s functions. The term API alone does not imply sending data to an external server.
Back to the textDecode — For an LLM, this is the stage that generates output tokens after input processing. For a VAE or audio codec, decoding can mean reconstructing the original form from a compressed representation or encoded data.
Back to the textToken — A unit into which a model divides input or output for processing. One token does not equal one character or a fixed duration.
Back to the textLatent representation — An intermediate representation in which a model encodes input information compactly. A decoder is usually needed to reconstruct an image, audio, or other output.
Back to the textVAE — Short for Variational Autoencoder. It can encode input into a compact latent representation or decode that representation into an output.
Back to the textPipeline — A sequence of processing stages from input to output. Different models or tools may be used at each stage.
Back to the textRuntime — Software that loads model files and runs their computations. Supported formats, hardware, and optimizations vary by runtime.
Back to the textCUDA Graphs — A CUDA feature that records and reuses a sequence of GPU operations. It can reduce submission overhead but does not shrink the model.
Back to the textFlashAttention — An implementation that improves memory access in attention computation. Availability and effects depend on hardware, model, and runtime.
Back to the textAR / NAR — AR (autoregressive) generation proceeds in order, conditioned on prior outputs. NAR (non-autoregressive) generation is less sequential.
Back to the textFP32 — A 32-bit floating-point format. It uses more memory per value than BF16 and can represent values more precisely.
Back to the text