№ 08 · Toolkit · Intro
Watch: let your AI see the video
An open-source skill that lets Claude Code and Codex look at a video frame by frame, alongside what is said. How it works, how closely it can watch, how to start, and where it falls short.
Dice Gamble & Marcel Patillo · 6 min · Updated Oct 2026
frames + transcript, timestampednot in a regular chat windowPython, FFmpeg, yt-dlp, Deno for YouTubeframes count toward your AI usageChapter 1 of 5 · Reading only hears the words
Reading only hears the words.
Watching sees everything.
A 30-second clip, six moments.
0:00
“This is where we start.”
Wide shot. Filmmaker beside a tripod on a salt flat at dawn.
0:05
“One camera, one lens, no crew.”
Close-up on the camera and tripod head.
0:10
no speech
Aerial pull-back. A tiny figure and a long shadow.
0:15
no speech
Low angle looking up, sun just behind him.
0:20
“Watch what the light does.”
Medium shot as the sun touches the horizon.
0:25
no speech
The same wide at dusk. The sky has gone amber.
Deep diveReading only hears the words
the shot, the cut, the text on screen, the B-roll nobody talks over. Watch sees those too.
Chapter 2 of 5 · How it works
Follow one video through it.
It turns a video into snapshots and a transcript, then hands both to your AI.
Grab the video
Give it a link or a file. It downloads the video, or just reads the captions when that is all it needs.
yt-dlp fetches it. Captions can be read without downloading.
Pick the moments
It grabs a frame each time the shot changes and skips frames that look the same.
Balanced mode: scene changes, up to 100 frames. A 16×16 thumbnail check drops near-duplicates.
Get the words
It uses the captions first. If there are none, it can transcribe the speech.
Fallback: local WhisperX, or Groq or OpenAI with an API key.
Hand it over
Frames and transcript go to your AI together, each stamped with its time.
Frames up to 512 px wide. Timestamps match the source video.
Ask anything
Now your AI can answer about what it saw, not only what was said.
Optional Gemini mode sends the whole video, picture and sound, with a GEMINI_API_KEY.
Under the hood
What’s the hook in the first 5 seconds?
0:00 A wide shot of the filmmaker on the salt flat as he says “This is where we start.” 0:04 Cut to a close-up of the camera. Two shots and one line in under five seconds.
Example answerIllustration
Deep diveHow it works
Every frame carries its timestamp, so the answer can point to the exact moment.
Deep diveChapter 3 of 5 · Pick how closely it watches
More frames, more detail.
Four detail modes and a Gemini engine trade how closely it looks against how much usage it takes.
transcript: words only
No frames at all. Good for talking-head videos and quick summaries.
efficient: keyframes
Up to 50 keyframes. A fast, low-cost first pass.
balanced: every cut
A frame at every shot change, up to 100. The one for hooks, pacing and B-roll.
token-burner: no limit
Every shot change, with no limit, for frame-by-frame detail on long edits. More frames means more usage.
Gemini engine: sound too
Sends the whole video, picture and sound, to Google’s model, so music and sound count too. It needs a free Google AI Studio key.
Illustration
Pick how closely it watches
More frames means more detail and more usage. Here is how they compare.
| Mode | What it grabs | Good for |
|---|---|---|
| transcript | The words only, no frames | Talking-head videos and quick summaries |
| efficient | Keyframes, up to 50 | A fast, low-cost first pass |
| balanced | A frame at every shot change, up to 100 | Hooks, pacing and B-roll |
| token-burner | Every shot change, no limit | Frame-by-frame detail on long edits |
| Gemini engine | The whole video, picture and sound | Music and sound too. Needs a free Google AI Studio key. |
Chapter 4 of 5 · Try it
Install once, then just ask.
Two commands in Claude Code, a few helper tools, then you ask about a video in plain words.
Open Claude Code
Watch runs in Claude Code, including the Code tab in the desktop app. It doesn’t run in a regular chat window.
Install it once
Add the marketplace, then install the plugin. Two commands, one time.
/plugin marketplace add bradautomates/claude-video · /plugin install watch@claude-video
Let your AI add the tools
It also needs Python, FFmpeg and yt-dlp, plus Deno for YouTube links. Ask your AI to install them for you.
Ask in plain words
Pick /watch:watch from the menu, or just ask: use watch on a YouTube link, what’s the hook in the first 5 seconds, and how fast are the cuts?
B-roll, one clip at a time
It watches one video at a time. For a folder of B-roll, ask it to go through the clips one by one, then tag and sort each by what is in the shot.
Under the hood
Runs in
/plugin marketplace add bradautomates/claude-video/plugin install watch@claude-video
Also needs
/watch:watch, or just ask
Use watch on <youtube link>. What’s the hook in the first 5 seconds, and how fast are the cuts?
0:00 Text on screen before anyone speaks. Then three quick cuts before 0:05.
Sorted by shot
Illustration
Deep diveTry it
It also needs Python, FFmpeg and yt-dlp, plus Deno for YouTube links; ask your AI to install them for you.
It watches one video at a time. For a folder of B-roll, ask it to go through the clips one by one, then tag and sort each one by what is in the shot.
Try it
Copy this.
# in Claude Code, one-time install
/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video
# then pick /watch:watch from the menu, or just ask
Use watch on <youtube link>. What's the hook in the first
5 seconds, and how fast are the cuts?Deep diveChapter 5 of 5 · Limits worth knowing
A first pass, not your eye.
Watch is a strong first pass. Four limits to keep in mind before you trust what it saw.
One video per run
It watches one video at a time. For a folder of clips, ask it to go through them one at a time.
Look-alike shots can merge
It skips frames that look the same, so fast cuts between near-identical shots may count as one.
Frames use your plan
More frames means more usage. Start long videos on efficient.
Speech, not sound
Music and sound effects are not analyzed unless you use the Gemini engine. Videos without captions need transcription.
Try it today
Point Watch at the last video that stopped your scroll and ask what happened in the first five seconds.
Frames use your plan
efficienttoken-burner
More frames, more usage. Start long videos on efficient.
- speech: from captions, or transcribed
- music: only with the Gemini engine
- sound effects: only with the Gemini engine
Use watch on the last video that stopped my scroll. What happened in the first five seconds?
A strong first pass, not a replacement for your own eye.
Illustration
Limits worth knowing
Four things to keep in mind.
One video per run
For a folder of clips, ask it to go through them one at a time.
Look-alike shots can merge
It skips frames that look the same, so fast cuts between near-identical shots may count as one.
Frames use your plan
More frames means more usage. Start long videos on efficient.
Speech, not sound
Music and sound effects are not analyzed unless you use the Gemini engine. Videos without captions need transcription.
The takeaway
Your AI stops reading about the video and starts seeing it: the hook, the pacing, and the B-roll nobody talks over.
Resources
Every link this leans on.
Keep going
Take it further.
Do this today
Point Watch at the last video that stopped your scroll and ask what happened in the first five seconds.
- The field guideEvery breakdown
- Next: The creator pipeline, from idea to upload№ 09 · 10 min
- Build your starter kitGuided, for your assistant
Dice Gamble & Marcel Patillo · Details as listed on each project page, Oct 2026.