How to Make a Music Video with Claude Opus 5.5: Frame-by-Frame Code Rendering Tutorial (2026)
TL;DR: Opus 5.5 doesn't use a video generation model. It draws every frame in code, on the beat, then headless Chrome captures each frame and ffmpeg turns them into a 1080p / 24 fps MP4. I gave it a song, an LRC file, a reference video, and one line — "you design the story" — and got back a 59-second watercolor music video with 13 shots and word-by-word karaoke lyrics. Rendering is the bottleneck: about 20 hours for full quality in the cloud, about 50 minutes for a draft, and 0.5–1 hour on a local machine with a GPU.
Intro: How Opus 5.5 "Draws" a Music Video
I kept seeing "music videos made with Opus 5.5" — hand-painted watercolor, small characters moving to the beat, lyrics lighting up one character at a time. The first guess is a video generation model. It isn't.
Opus 5.5 doesn't generate video. It draws every frame in code. Think of it as an animator that writes programs: it writes a web page and tells that page "at 10.5 seconds, two little characters stand under an umbrella and the hero's eyes turn into hearts"; then a headless Chrome browser takes 24 screenshots a second from the first frame to the last, and ffmpeg stitches those photos and the music into an MP4.
So not one frame in this kind of video is generated. Characters, rooms, the moon, transitions — all painted stroke by stroke in code with a watercolor brush. Two things follow: characters look identical in every shot, and changing a detail means changing a few lines of code.
Second 1 of the final cut: a puzzle moon with one corner missing outside the window. Every stroke in this frame was drawn by code Opus wrote.
This tutorial documents how I used Opus 5.5 to make a 59-second chorus video for Wang Erlang's 《把回忆拼好给你》 ("I'll Piece the Memories Back Together for You"). The final cut has 13 shots, word-by-word karaoke lyrics, and a "puzzle moon" storyline running through all of it. I sent three messages and supplied four things: the song, the lyrics, a reference video, and one line — "you design the story."
First, a moving preview, then a few stills:
Animated preview of the opening — several shots cut together.
"Why did everything become like this" — a rainy street corner in memory, the two of us under one umbrella.
Closing line: the moon is whole again, and she looks up at it from far away.
The overall flow is a straight line:
- Prepare the song audio, an LRC lyric file, and a reference video
- Install the painted-animation skill
- Pick the chorus and measure the beat
- Write the storyboard
- Build characters and scenes, then work shot by shot
- Render contact sheets, review, fix
- Render and export
Below, in that order: what I did, what Opus did, and where we got stuck. You won't write a line of code — you follow the prompts and commands.
Table of Contents
- Before You Start
- Step 1: Pick the Section and Measure the Beat
- Step 2: Creative Direction and Storyboard
- Step 3: Build the Characters, Scenes, and Props
- Step 4: Shot-by-Shot Production and Preview Review
- Step 5: Render and Export
- Pitfalls and Tips
- Prompts You Can Copy
- FAQ
- Summary
Before You Start
Three things to prepare, then one skill to install.
| Material | Purpose | Notes |
|---|---|---|
| Song audio (mp3) | Soundtrack, beat detection | Must be the same version as the lyrics |
| LRC lyric file | Timestamps each line for karaoke | With timestamps, e.g. [05:15.898]独自收集… |
| Reference video (optional) | Shows Opus the style and motif you want | Mine was a paper-craft music video built around puzzles |
First message: I sent the video and the LRC together. Opus repeated back the style and scene order it saw, then pointed out that the two sets of lyrics didn't match.
After I sent the mp3: the 59-second section was its own choice, and so were the storyboard and the character designs.
Pitfall: I first sent only the reference video and the LRC. Opus found that the lyrics in the video and the lyrics in the LRC didn't match at all, so it stopped and asked me for the correct audio. Beats are timed only to the audio and LRC you provide; the reference video only sets the style. The two have to match.
Installing the skill. Send this GitHub link to Claude and ask it to install: https://github.com/Joker-chichi/claude-mv-skill. It reads through the scripts first to confirm they're safe, then shows a skill card — click save and it's installed. The skill is called painted-animation, adapted from the open-source project behind a viral music video. It includes:
- A watercolor brush engine (p5.js + p5.brush) and a frame renderer
- The character Clawd: 31 expressions, 5 turn angles, hats, and emoji
- A beat detection script and a word-by-word karaoke lyric module
- An animation guide: no text on screen, every shot needs an event, every seam needs a transition
The skill's built-in character Clawd and its 31 expressions. Both leads in this video are built on it, with new colors and accessories.
With the skill installed, Opus doesn't design a toolchain from scratch — it works on a project a viral music video has already proven out. The more valuable part is that animation guide: it tells Opus which frames read as "AI-looking" (characters too small, no anticipation before a move, an event that flashes past before viewers can parse it), so it avoids those from the start.
Once installed, type /painted-animation in the conversation to summon it.
Where to do this. The whole thing runs inside Claude Cowork on the web: installing the skill, uploading audio and lyrics, writing code, and cloud rendering all happen in the conversation, with no extra software. You only move the project to your own machine when you want full quality faster — that's the local flow in Step 5.
Step 1: Pick the Section and Measure the Beat
The song runs 5:47, so I only did a 40–60 second chorus. Opus picked 4:48–5:47, the last 59 seconds: "collecting the memories between us, alone" twice, landing on the title line at the end, which closes the emotion on its own.
After picking the section it measured the beat with the skill's beat_grid.py:
- 80.1 BPM, 0.749 seconds per beat
- One lyric line ≈ 4 beats ≈ 3 seconds; long lines with a comma ≈ 6 seconds
- The clip starts on the bar before the first lyric line (288.67 seconds), so frame 0 lands exactly on a beat
Then ffmpeg cuts the section with a 0.5-second fade-in and a 4-second fade-out. Subtract 288.67 seconds from each LRC timestamp and you have the moment each line appears in the video.
Why measure the beat: idle bobbing, firework bursts, and cuts are all locked to the beat, which is what makes the picture move with the music. Set bpm: 80.1, offset: 0 in the code and the whole film shares one grid.
There's a practical consequence: a lyric line lasts only about 3 seconds. Each shot can get one idea across. If a single line has the character walking, then crying, then changing scenes, viewers catch none of it. That's why the skill insists on lyrics and beat grid before the storyboard: a guessed runtime usually comes out about twice as long as reality.
Long lines with commas get 6 seconds, which is exactly enough for the heaviest beats: her shattering into puzzle pieces and the clock hand snapping back both land on those lines.
Step 2: Creative Direction and Storyboard
The skill asks one question about direction: what story, what mood, any images you want repeated or avoided. You can leave it blank. My answer was one line: "You design the story; if you're out of ideas, follow the video I gave you."
Opus pulled the puzzle motif from the reference video and wrote this:
- One-line story: After we parted, I sit alone in a small attic piecing memories back together; I cry as I work, and I want to go back but can't. In the end I turn the finished memory into a full moon and hang it in her sky.
- Characters: "Me" is a terracotta cube with a red scarf; "her" is cyan blue (from the lyric 「澄蓝色的你」, "the cyan-blue you"), with a small flower in her hair.
- Palette arc: deep indigo night (loneliness) → warm yellow in memory (sweetness) → cold blue rain (loss) → warm gold full moon (letting go).
- Bookends: it opens on a puzzle moon with one piece missing; the last piece goes in and it becomes a full moon.
One shot per lyric line, 13 in total. Timestamps below start at the clip point; second 0 is 4:48. Lyrics are translated from the Chinese original that appears on screen.
| Time | Lyric | Shot |
|---|---|---|
| 0–3.3 s | (intro tail) | Puzzle moon with a missing corner outside the window; camera tilts down to me alone in the attic |
| 3.3–6.0 s | Collecting the memories between us, alone | I press a memory into the puzzle board; it lights up |
| 6.0–8.9 s | Even so, I cry every time it comes to this | Holding the piece under the umbrella, laughing then crying; rain starts outside |
| 8.9–15.1 s | Why did everything become like this — we can't go back | Into the memory: a rainy street corner, two people under one umbrella; she shatters into puzzle pieces and flies away |
| 15.1–17.9 s | But I'm still glad you gave me love | I dry my tears and press the piece to my chest |
| 17.9–21.0 s | Every view is proof of our love | Pan across three memories: sunset, a potted plant blooming, fireworks |
| 21.0–27.1 s | Even now, worlds apart — may the rest of your life be lovely | A puzzle seam splits the frame in two; my song drifts across to her side |
| 27.1–29.9 s | Collecting the memories between us, alone (reprise) | Notes turn into puzzle pieces and fly home to the board like birds |
| 29.9–33.2 s | Even so, I cry every time it comes to this (reprise) | Crying and still piecing it together; the rain slowly stops |
| 33.2–38.9 s | Why did everything become like this — we can't go back (reprise) | Trying to turn the clock back; the hands snap back twice; he accepts it |
| 38.9–42.0 s | But I'm still glad you gave me love (reprise) | The last piece is heart-shaped; pressing it in lights the whole picture |
| 42.0–44.9 s | Every view is proof of our love (reprise) | Camera pulls back: the finished picture is the two of us under the umbrella |
| 44.9–59 s | Even now, worlds apart — I'll piece the memories back together for you | A beam of light flies up and completes the moon; far away, she looks up and waves |
Tip: when a line repeats, upgrade it instead of re-shooting it. "We can't go back" happens twice — the first time she shatters and flies away (grief), the second time the clock hand flings him off the stool (a wry kind of helplessness).
Storyboard Highlights
Four acts, in story order. You can see how the puzzle motif carries the whole thing.
Act 1: Alone in the attic. The camera tilts down from the moon with the missing corner and lands on me, piecing the puzzle together. Most of the board is still dark gaps; each piece lights up as it goes in.
3.3 s, "collecting the memories between us, alone": a memory flies toward the puzzle board.
6.0 s, "I cry": laughing turns into tears and rain starts outside. The camera then pushes into the memory through a puzzle-shaped iris.
Act 2: The past you can't get back to. The memory is a warm yellow rainy street corner. I turn to take her hand; she shatters into blue puzzle pieces and flies away, and the whole street fades to cold grey.
11.7 s, "we can't go back": she shatters the instant I reach for her. This is the heaviest moment in the film.
17.9 s, "every view is proof of our love": the camera pans across three memories. In this one she waters the plant and it blooms. The detail comes from an earlier line in the song, about tending a pot of dreams.
Act 3: Worlds apart. A puzzle seam cuts the frame in half: two rooftops, two people, one moon above both. The tune I'm humming turns into golden notes and drifts across the seam to her side.
21.0 s, "even now, worlds apart — may the rest of your life be lovely."
27.1 s, the reprise of "collecting the memories, alone": the notes turn back into puzzle pieces and fly in through the window like birds returning to the nest.
Act 4: Finished, then letting go. He tries to wind the clock back, gets flung off by the hands, sits down and sighs, and accepts it. The last piece is heart-shaped; he presses it in and the whole picture lights up.
33.2 s, the second "we can't go back": same line, this time with a wry kind of helplessness.
38.9 s, "I'm still glad you gave me love": the puzzle is complete, and it's the two of us under the umbrella.
49.9 s, "I'll piece the memories back together for you": a beam of light flies out the window and fills the moon's missing corner, mirroring the opening.
Step 3: Build the Characters, Scenes, and Props
The code splits into two files: hj_set.js holds the world, characters, and props; hj_shots.js holds the 13 shots. Build the set first; shots only direct traffic.
No need to draw characters from scratch. Both leads are the skill's Clawd with new colors: "me" gets a red scarf that flutters, "her" is cyan with a small flower. All 31 expressions and their anticipation moves (blink, squash, rebound) work as they are. Write "happy first, sad one second later, crying at 1.9 seconds" and it performs it.
Scenes:
- My attic: slanted roof, small window, the puzzle moon outside, a rain switch, a clock on the wall, a small desk and stool, and a "warmth" parameter that lets the room warm up at the end
- The rainy street corner in memory: buildings, streetlights, puddles reflecting the sky, able to shift from warm yellow to cold grey
- Two rooftops: each can take half the frame for the "worlds apart" split
Props:
- A puzzle piece generator: give it four edges that are convex, concave, or flat and it draws any piece
- A 4×3 puzzle board: a table records which piece goes in at what time, empty slots show as dark recesses, and pieces glow briefly as they go in
- The finished picture: the two characters under the umbrella, the street after rain, the sun
- The puzzle moon with a missing piece, the umbrella, the clock, the notes
With the set built, Opus arranges everything in a single "test shot" to check the characters, scarf, puzzle board, and moon before writing any shots:
The first test shot: attic, clock, puzzle board, and both leads in one frame. It also exposed a problem — the moon's missing corner was a square, not a puzzle piece. It was changed to a real puzzle shape later.
Lock the look first; the 13 shots then only rearrange what's already drawn — camera angle, expression, light. The style never drifts.
A counterintuitive rule: no text on screen. Apart from the karaoke lyrics at the bottom, there's no title, label, or speech bubble anywhere. Emotion comes from expressions and the little symbols above each character's head (❤, ❗, notes, rain clouds).
Step 4: Shot-by-Shot Production and Preview Review
You can't judge animation from code, so after each batch of shots Opus renders a contact sheet: a frame every 0.4–0.6 seconds, tiled into one large image, and then it looks at the image and finds the problems.
A real review contact sheet covering the "three views" and "worlds apart" shots. The time is in the top-left of each cell; read left to right and you're seeing the order the audience sees.
This is the most worthwhile step in the whole pipeline. Opus is "blind" while it writes code; only when the frames come back can it see what's wrong. I went through about ten rounds of contact sheets, 6–24 frames per round.
Problems it found in the contact sheets and fixed:
| Shot | Problem found | Fix |
|---|---|---|
| Three views | The frame was large, the characters inside were tiny, and the hillside ran past the frame | Characters doubled in size; hillside reshaped so it can't overflow |
| Three views | The watering can floated in mid-air instead of being held | Hung it at her side |
| Fireworks | Only one burst landed; the rest of the timings fell outside the shot | Burst interval changed to half a beat |
| Worlds apart | Both characters and the notes were too small to read | Characters enlarged; notes turned gold with a glow |
| Clock | The character couldn't reach the clock, so the "turn it back" action didn't read | Clock lowered, stool added, camera pushed in, "❗" on the snap-back |
| Ending | The final entrance timing was miscalculated; the iris didn't have time to close | Re-timed the last 11 seconds |
| Puzzle board | The heart in the picture drifted outside the frame | Moved between the two characters |
Two examples of the fixes.
Example 1: too much empty space. In the first version the frame was large, the characters were the size of a fingernail, and the hillside spilled outside the frame. Opus enlarged the characters, re-rendered, and the hillside was still outside. Here's that round — in the sunset picture at the top right, the purple hillside runs past the frame:
Before. For the after, compare the potted-plant still in Storyboard Highlights above.
Example 2: the clock action didn't read. In the first version the clock hung too high, so even standing on the stool the character couldn't reach it, and viewers only saw a cube waving under a clock. Opus lowered the clock, shoved the stool over, and pushed the camera in until the character's hand actually touches the hands; it added a "❗" when they snap back, and the second snap knocks him off the stool.
After the fix: climb onto the stool → turn the hands → get flung back → try again → fall off → sit and sigh. A small story inside a 6-second lyric line.
The review asks a few questions:
- Looking only at the contact sheet, can you tell what happens in this shot?
- Are the characters big enough to separate from the background?
- Does what they're holding touch their hand? Do their feet touch the ground?
- Does every seam have a transition? Mine used three: the puzzle-shaped iris, a brush wipe, and a push through the window.
- Does the audience get enough time to read the important information?
Step 5: Render and Export
Watercolor brushes are GPU-hungry, so where you render decides how long you wait. My measured numbers for this 59-second film:
| Method | Quality | Per frame | Whole film |
|---|---|---|---|
| Cloud, full quality | 1080p, 24 fps, watercolor texture | About 50 s | About 20 h |
| Cloud, draft mode | Half resolution, 12 fps, flat fill | About 4 s | About 50 min |
| Local machine with GPU, full quality | 1080p, 24 fps, watercolor texture | About 1–3 s (estimate) | About 0.5–1 h |
The cloud machine has no GPU and renders in software (soft-gl), which is why it needs 50 seconds a frame; a local GPU renders the same frame in 1–3 seconds.
I took both routes: a cloud draft first to confirm pacing and story, then the project on my own machine for full quality.
The difference between draft and final is mostly texture: the draft uses flat fills at 12 fps and looks slightly steppy; full quality adds watercolor bleed, paper grain, and smoother motion. Storyboard, pacing, beat sync, and lyrics are identical, so a draft is enough to judge whether the story works. Every storyboard still in this tutorial is a full-quality frame.
Local render steps:
- Install Node.js, Google Chrome, and ffmpeg
- Unzip the project and run
npm installin the folder - Render one test frame to confirm it runs:
node render.mjs --sheet=10.5,40.4 --cols=2 --w=960 --out=out/check/test.jpg - Render all frames (interruptible — running it again resumes):
node render.mjs --frames --workers=4 - Encode the MP4 with music:
node render.mjs --encode --audio=assets/clip.m4a --out=out/MV.mp4
Note: the skill never starts a full render on its own — it asks whether you want cloud or local first. Full-quality cloud rendering is too slow to recommend.
Pitfalls and Tips
- Audio and lyrics must match. Beats are timed only to the audio and LRC you provide; the reference video only sets the style.
- Start with the chorus. The full song is too long, and both the code and the render time double. Prove the effect with 40–60 seconds first.
- Install problems behind a restricted network. My cloud environment couldn't reach npm or Google Fonts, so Opus pulled p5 and p5.brush from GitHub and switched the lyrics to the system's Source Han Serif. You probably won't hit this on your own machine.
- No GPU? Use draft mode. Opus added two switches to the renderer:
--fast(watercolor fill becomes flat color) and--half(half resolution). Previews get more than ten times faster. - Watch out for shapes scaled to 0. The puzzle iris at the end crashed the renderer when its size hit 0. The fix: once it's below a certain size, cover the whole frame with a solid color.
- Don't put text on screen. To say "I miss you," draw the character crying — don't hand them a sign.
- Upgrade repeated lyrics instead of re-shooting them. A second chorus can tell the same thing from a different angle.
- Ask Opus to review by looking. Rendering the contact sheet and looking at it is how it caught the floating watering can and the unreachable clock — problems you can't see in code.
Prompts You Can Copy
I sent three messages in total. You can copy them as they are:
- Install the skill: "
https://github.com/Joker-chichi/claude-mv-skill— install this skill first." - Start (attach the reference video and LRC): "/painted-animation The first video is a viral music video. With the lyrics LRC I gave you, redesign the shots and make something similar."
- Send the audio and set the scope (attach the mp3): "/painted-animation I only want the 40–60 second chorus. You design the story — if you're out of ideas, follow what I gave you."
For a more personal film, add one line of direction to message 3, for example: "The story is a long-distance couple. Keep it warm, not too sad. I'd like paper planes to recur, and no rain."
FAQ
Do I need a video generation model to make a music video with Claude Opus 5.5?
How long does it take to render a 59-second music video?
What if the audio and the lyrics don't match?
Can I finish the video without a dedicated GPU?
Does this workflow work for other songs?
Summary
To make a music video with Opus 5.5, you supply the song, the lyrics, a reference video, and a direction. It handles the storyboard, writes the code that draws, and reviews its own frames. The real bottleneck is rendering: without a GPU, get a draft out first, then render full quality on your own machine.
To confirm which entry point gets you Opus 5.5, see Claude Opus 5.5 Access Channels; for installing and configuring Claude Code, see the Claude Code Guide.