Steps Reference

Steps are the building blocks of Montaj's editing pipeline. Each step is an executable that follows the output convention (result on stdout, errors on stderr). Steps are composable at the shell level and discoverable via CLI, HTTP, and MCP.

Montaj ships 39 built-in steps, organized into subdirectories by category: audio/, edit/, generate/, lyrics/, media/, render/, speech/, transform/. The CLI resolves step names automatically. You never need to specify the subdirectory.

Inspect Steps

Inspect steps give the agent (or human) an understanding of the source material before editing begins.

probe

Extract metadata from a video file: duration, resolution, fps, codec, audio channels.

montaj step probe --input clip.mp4
# → JSON: duration, resolution, fps, codec, audio channels

The output is JSON on stdout: the agent uses this to understand what it's working with and make informed editing decisions.

snapshot

Generate a contact sheet frame grid: the agent's visual understanding of the clip.

montaj step snapshot --input clip.mp4
# → /path/to/snapshot.png (frame grid contact sheet)

montaj step snapshot --input clip.mp4 --cols 3 --rows 3
# Custom grid size

montaj step snapshot --input clip.mp4 --at 5.0
# Extract a single frame at a specific timestamp

detect_shots

Split a video into its individual shots using ffmpeg scene scores, with per-shot motion statistics. Pure ffmpeg: no model, no network, no credentials. This is the unit the B-roll footage index is built on: a detected shot is the granularity an editor actually cuts at.

montaj step detect_shots --input clip.mp4
# → {"input":..., "duration":..., "shots":[{index, start, end, duration, motion_mean, motion_peak}]}

montaj step detect_shots --input clip.mp4 --threshold 0.15 --min-shot 1.0
# Lower threshold finds more cuts; min-shot merges anything shorter

Motion is reported two ways because they answer different questions. motion_mean tells you whether the camera moves throughout: a pan, a push-in, handheld. motion_peak tells you whether something happened at some point. A locked-off shot in which a subject walks into frame is mean-static but peak-active.

shot_sheet

Sample N frames per detected shot and tile them into contact sheets, with a tile index mapping every tile back to its shot. Pure ffmpeg. Consumes a detect_shots JSON.

montaj step detect_shots --input clip.mp4 --out shots.json
montaj step shot_sheet --input clip.mp4 --shots shots.json --out-dir sheets/

Multiple frames per shot is the point: one frame tells you the subject, three tell you the action. A boot stepping into water, a camera pushing in, and an on-screen element animating are all invisible in a single frame.

The tile index is what makes the sheets machine-readable: each tile records its shot_index, frame_index, timestamp, and grid position, so whoever reads a sheet can name which shot a tile belongs to instead of guessing from position.

filmstrip

Tile a uniform time-grid of frames into filmstrip contact sheets, with an index JSON mapping every tile to its timestamp. Pure ffmpeg. Unlike shot_sheet, which samples per detected shot, filmstrip samples on a fixed time grid across the whole video: the thumbnail strip a timeline scrubber reads. An all-intra proxy is the expected input, so per-tile seeking stays cheap regardless of position.

montaj step filmstrip --input proxy.mp4 --out-dir strip/
# → {"sheets":[{path, cols, rows, tiles:[{t, row, col}]}], interval, tileWidth}

montaj step filmstrip --input proxy.mp4 --out-dir strip/ --max-tiles 200 --cols 12
# Denser grid, more tiles per row

virtual_to_original

Map virtual-timeline timestamps to original-file timestamps (or the reverse). Useful for inspecting and debugging trim specs.

montaj step virtual_to_original --input spec.json 47.32
# → 95.483  (virtual timestamp → original-file timestamp)

montaj step virtual_to_original --input spec.json 47.32 53.23 66.89
# → one result per line

montaj step virtual_to_original --input spec.json --inverse 95.483
# → 47.320  (original-file timestamp → virtual timestamp)

Smart Cut Steps

Smart cut steps analyze audio and produce trim specs: JSON files describing which ranges of the original source to keep. No video encoding happens at this stage.

waveform_trim

Waveform silence analysis, detects silent sections and outputs a trim spec. Near-instant, no encode.

montaj step waveform_trim --input clip.mp4
# → /path/to/clip_spec.json

montaj step waveform_trim --input clip.mp4 --threshold -30 --min-silence 0.3
# Custom sensitivity

Output: a trim spec JSON with input (original file) and keeps (time ranges to keep).

rm_fillers

Remove filler words (um, uh, hmm) from speech. Takes a trim spec as input and outputs a refined trim spec.

montaj step rm_fillers --input clip.mp4
# → /path/to/clip_spec.json

montaj step rm_fillers --input clip.mp4 --model medium.en
# Higher accuracy, slower

Important: rm_fillers can accept either a video file or a trim spec as input. When given a trim spec, it refines the existing keeps list.

rm_nonspeech

Remove all non-speech audio (noisy ambient audio). Takes a trim spec as input and outputs a refined trim spec.

montaj step rm_nonspeech --input clip.mp4

montaj step rm_nonspeech --input clip.mp4 --model base --max-word-gap 0.18 --sentence-edge 0.10

Important: Input should be a trim spec, not a raw video file.

crop_spec

Crop a trim spec to virtual-timeline windows. Outputs a refined trim spec: no encode.

montaj step crop_spec --input spec.json --keep 8.5:14.8
# Crop to a single window

montaj step crop_spec --input spec.json --keep 0:2.4 --keep 13.84:18.33
# Multiple windows — keeps are concatenated in order

montaj step crop_spec --input spec.json --keep 40.28:end
# Open-ended — keep from virtual 40.28s to end of clip

How Smart Cuts Chain

Smart cut steps form a pipeline where each step refines the trim spec:

waveform_trim(clip.MOV)
  → trim spec (silence removed)

rm_fillers(spec.json)
  → refined spec (fillers removed)

rm_nonspeech(spec.json)
  → refined spec (non-speech removed)

crop_spec(spec.json)
  → cropped spec (virtual-timeline windows)

The original source file path is preserved through the entire chain. Only materialize_cut finally encodes anything, video by default, or an audio-only file with --audio.

Edit Steps

Core video editing operations. Most of these are ffmpeg wrappers with careful codec handling.

jump_cut

Remove time ranges from video, similar to cut, but oriented toward jump-cut editing with explicit keeps or cuts.

montaj step jump_cut --input clip.mp4 --cuts '[[2.0, 3.5], [7.0, 8.0]]'
# Remove specified ranges

montaj step jump_cut --input clip.mp4 --keeps '[[0, 2.0], [3.5, 7.0], [8.0, 12.0]]'
# Keep only specified ranges

cross_cut

Interleave A-roll and B-roll segments, creates a cross-cut edit between two video sources.

montaj step cross_cut --input a-roll.mp4 --b-roll b-roll.mp4 --segment-duration 3

montage

Create a rapid montage from multiple clips, cuts them into short segments and assembles.

montaj step montage --inputs clip1.mp4 clip2.mp4 clip3.mp4 --beat-duration 2

materialize_cut

Encode a trim spec or raw segment into a real media file. Used when a subsequent step needs an actual file rather than a trim spec, remove_bg needs a video, and a cleaned voiceover needs an audio file to sit on audio.tracks[].

montaj step materialize_cut spec.json
# Encode a trim spec to video

montaj step materialize_cut clip.mp4 --inpoint 2.0 --outpoint 8.0
# Encode a raw segment

montaj step materialize_cut --inputs clip0.json clip1.json
# Multiple clips — caps at 2 concurrent encodes by default

montaj step materialize_cut vo_spec.json --audio --out clean.wav
# Audio-only — no video stream

Uses input-level seeking (-ss/-t before -i) so only the requested segment is decoded.

Audio mode is cheap. The concurrency ceiling above exists because video mode is a full libx264 encode, audio mode isn't, and isn't subject to it.

resize

Reframe video to a target aspect ratio.

montaj step resize --input clip.mp4 --ratio 9:16    # TikTok / Reels / Shorts
montaj step resize --input clip.mp4 --ratio 1:1     # Instagram
montaj step resize --input clip.mp4 --ratio 16:9    # YouTube

extract_audio

Extract the audio track from a video file.

montaj step extract_audio --input clip.mp4              # default: wav
montaj step extract_audio --input clip.mp4 --format mp3

proxy

Encode a full-source, 720p, all-intra H.264+Opus editing proxy for instant-seek preview scrubbing. Preview-only, render never reads a proxy file. Pass --tonemap for HDR sources that haven't been conformed yet, to compose a scale-first HDR→SDR tonemap chain ahead of the encode; omit it for sources that are already SDR (e.g. a post-normalize conform pass).

montaj step proxy --input clip.mp4 --out proxy.mp4

montaj step proxy --input hdr_clip.mov --tonemap --out proxy.mp4
# HDR source, not yet conformed

Audio Steps

Audio processing steps for stem separation and waveform analysis.

stem_separation

Separate audio into stems (vocals, drums, bass, other) using Demucs.

montaj step stem_separation --input song.mp3 --stems vocals
# Extract vocals only

montaj step stem_separation --input song.mp3 --out-dir ./stems/
# All stems to a directory

Requires montaj install demucs.

waveform_image

Generate a PNG waveform visualization grid from an audio or video file.

montaj step waveform_image --input clip.mp4
# → /path/to/waveform.png

montaj step waveform_image --input clip.mp4 --chunk-duration 30
# One row per 30s chunk

waveform_peaks

Emit min/max audio peak pairs at a fixed samples-per-second resolution, for a scrubbable waveform view. Mono mixdown; total pairs are clamped to 500k per call.

montaj step waveform_peaks --input clip.mp4
# → {"samplesPerSecond":200, "start":0, "duration":12.4, "peaks":[min,max,min,max,...]}

montaj step waveform_peaks --input clip.mp4 --samples-per-second 800 --start 5 --duration 10
# Higher resolution over a specific window

montaj step waveform_peaks --inputs clip1.mp4 clip2.mp4
# Batch mode — returns an array of results, one per input

Enrich Steps

Enrich steps add information (transcripts, captions) or polish (loudness normalization) to clips.

transcribe

Generate SRT and JSON transcripts with word-level timestamps using local whisper.cpp.

montaj step transcribe --input clip.mp4
# → transcript.json + clip.srt

montaj step transcribe --input clip.mp4 --model medium.en
# Higher accuracy, slower

montaj step transcribe --input clip.mp4 --language es
# Non-English

When given a trim spec as input, transcribe extracts audio only at the keep ranges, runs whisper on the joined audio, and remaps word timestamps back to the original timeline.

Output includes:

• JSON transcript with word-level timestamps (start, end, word)
• SRT file for standard subtitle format

caption

Convert a transcript into an animated caption track. Produces data (not pixels), rendered at review/final render time by the UI and render engine.

montaj step caption --input transcript.json
montaj step caption --input transcript.json --style word-by-word
montaj step caption --input transcript.json --style pop
montaj step caption --input transcript.json --style karaoke
montaj step caption --input transcript.json --style subtitle
montaj step caption --input transcript.json --style highlight-box
montaj step caption --input transcript.json --style outline
montaj step caption --input transcript.json --style clean

Caption Styles

Caption data is stored in project.json as a track with type: "caption" and inline segments with word timestamps:

{
  "id": "captions",
  "type": "caption",
  "style": "word-by-word",
  "segments": [
    {
      "text": "Hello world",
      "start": 0.0,
      "end": 1.2,
      "words": [
        { "word": "Hello", "start": 0.0, "end": 0.5 },
        { "word": "world", "start": 0.5, "end": 1.2 }
      ]
    }
  ]
}

generate_captions

Generate a project's caption track server-side: the single-step equivalent of regenerating captions from the editor UI. Orchestrates materialize_cut → transcribe → caption internally and writes the result to project["captions"]. Takes a project id (not a file path); run it async, since transcription with the large model can take minutes. Any prior caption theme (position, color, fontsize) carries forward.

montaj step generate_captions --project-id my-project-id

montaj step generate_captions --project-id my-project-id --model large --language es --style karaoke

normalize

Loudness normalization, adjust audio levels to meet platform standards.

montaj step normalize --input clip.mp4                       # youtube = -14 LUFS
montaj step normalize --input clip.mp4 --target podcast      # -16 LUFS
montaj step normalize --input clip.mp4 --target broadcast    # -23 LUFS
montaj step normalize --input clip.mp4 --target custom --lufs -18

normalize_window

Normalize a windowed segment [inpoint, outpoint) of a video clip to the project's working color space and codec. Uses fast input-level seeking so only the requested window is decoded; output always starts at time 0 and is dense-keyframe conformant. Handles SDR↔HDR conversion. Used by the lazy source-normalization path (the clips workflow) so a multi-GB source is never re-encoded whole.

montaj step normalize_window --input clip.mp4 --inpoint 12.5 --outpoint 45.0 --out segment.mp4

Lyrics Steps

Steps for syncing lyrics to audio and rendering lyrics videos.

lyrics_sync

Sync lyrics text to an audio file using Whisper, produces word-level aligned captions.

montaj step lyrics_sync --lyrics song.txt --input song.mp3
# → /path/to/captions.json

montaj step lyrics_sync --lyrics song.txt --input song.mp3 --model medium.en
# Higher accuracy

lyrics_render

Render synced captions onto video via ffmpeg drawtext filters. Produces a complete lyrics video.

montaj step lyrics_render --captions captions.json --audio song.mp3
# Render with solid color background

montaj step lyrics_render --captions captions.json --audio song.mp3 --input background.mp4
# Render with background video

VFX Steps

remove_bg

Remove video background using RVM (Robust Video Matting). Outputs a ProRes 4444 .mov with alpha channel for final render and a VP9 WebM for browser preview.

montaj step remove_bg --input clip.mp4

montaj step remove_bg --inputs clip0.mp4 clip1.mp4
# Multiple clips

montaj step remove_bg --input clip.mp4 --model rvm_resnet50
# Higher quality model

montaj step remove_bg --input clip.mp4 --downsample 0.5
# Downsample for faster processing

Requirements

Requires montaj install rvm, installs torch, torchvision, av (pip) + RVM model weights.

Output

The step produces two files and updates the project item:

• nobg_src: ProRes 4444 .mov with alpha channel (for final render)
• nobg_preview_src: VP9 WebM (for browser preview: Chrome supports VP9 alpha; ProRes does not play in browsers)
• Sets remove_bg: true on the item

Usage in Workflows

Used in the floating_head workflow:

1. materialize_cut, encode the trim spec to an actual video file
2. remove_bg, remove background from the materialized file
3. Render engine composites the alpha-channel .mov over the layers beneath

Important: remove_bg requires an actual video file, pass the output of materialize_cut, not a trim spec. This step is long-running (minutes per clip), so always run it in the background with --progress so you can monitor status.

Render Behavior

At render time, when a tracks[1+] item has remove_bg: true and nobg_src is set, the render engine uses the ProRes 4444 .mov (with alpha) in place of the original src. The alpha channel is preserved through the ffmpeg filter graph and composited over the layers beneath.

Render Steps

Steps that produce a pixel-accurate preview without running a full render, used by the editor UI and by agents checking their own work.

sample_frame

Render a fully composited project frame at a given timestamp to PNG (video + image items + active overlay JSXs, no audio).

montaj step sample_frame --project /path/to/project.json --at 12.5
# → /path/to/project/render/samples/frame-12.5s.png

montaj step sample_frame --project /path/to/project.json --at 12.5 --out /tmp/check.png

montaj step sample_frame --project /path/to/project.json --at 12.5 --sdr-curve vivid1
# Grade the HDR-to-SDR tonemap with a specific look curve

montaj step sample_frame --project /path/to/project.json --at 12.5 --prefer-proxy
# Decode each clip from its SDR proxy instead of the master — fast, but no per-curve grade

sample_overlay

Render one frame of an overlay JSX to PNG via Puppeteer; optionally return per-element bounding-box and overflow measurements as JSON.

montaj step sample_overlay --overlay caption.jsx --out /tmp/frame.png

montaj step sample_overlay --overlay caption.jsx --frame 15 --width 1080 --height 1920 --out /tmp/frame.png

montaj step sample_overlay --overlay caption.jsx --props '{"text":"hello"}' --google-fonts "Syne:wght@800" --out /tmp/frame.png

montaj step sample_overlay --overlay caption.jsx --measure --out /tmp/frame.png
# stdout is {"pngPath": ..., "measurements": ...} instead of a bare path

Generation Steps

These steps call external APIs (Kling, Gemini, OpenAI) and require montaj install connectors and montaj install credentials.

kling_generate

Generate video via Kling, text-to-video, image-to-video, or reference-guided generation. Supports two models.

montaj kling-generate --prompt "a calico cat walking through a sunlit kitchen, cinematic" --out /tmp/cat.mp4

montaj kling-generate --prompt "slow zoom in" --first-frame frame.png --out /tmp/zoom.mp4

montaj kling-generate --prompt "character walks left" --first-frame start.png --last-frame end.png --out /tmp/walk.mp4

montaj kling-generate --prompt "same style" --ref-image style1.png --ref-image style2.png --out /tmp/styled.mp4

montaj kling-generate --prompt "..." --out /tmp/pro.mp4 --mode pro --duration 10 --aspect-ratio 9:16

montaj kling-generate --multi-shot --shot-type customize --multi-prompt '<json>' --out /tmp/batch.mp4

Models

The step auto-upgrades to kling-video-o1 when duration is 5/10 and sound is off.

analyze_media

Analyze a media file (video, audio, or image) with Gemini Flash. Supports description, timestamps, and structured output.

montaj analyze-media clip.mp4 --prompt "Describe the scene in 2 sentences."

montaj analyze-media song.mp3 --prompt "Transcribe with timestamps."

montaj analyze-media photo.jpg --prompt "Return JSON: {subject, mood, dominant_colors}" --json-output

montaj analyze-media clip.mp4 --prompt "..." --model gemini-2.5-pro
# Override model

montaj analyze-media clip.mp4 --prompt "..." --out analysis.txt
# Write to file

Note: Images under approximately 18 MB take a fast inline path (no Files API round-trip).

generate_image

Generate an image via Gemini or OpenAI, text-to-image or reference-conditioned.

montaj generate-image --prompt "portrait, studio lighting" --out /tmp/portrait.png

montaj generate-image --prompt "same character, profile view" --ref-image /tmp/portrait.png --out /tmp/profile.png

montaj generate-image --prompt "red apple on white table" --provider openai --out /tmp/apple.png

montaj generate-image --prompt "..." --provider gemini --aspect-ratio 9:16 --out /tmp/tall.png

generate_music

Generate music via Gemini Lyria 3 Clip, produces approximately 30 seconds of audio from a text prompt.

montaj generate-music --prompt "upbeat electronic, 120 bpm" --out /tmp/music.wav

montaj generate-music --prompt "acoustic guitar, mellow" --with-vocals --out /tmp/song.wav

generate_voiceover

Generate speech audio via Kling TTS or Gemini TTS.

montaj generate-voiceover --text "Welcome to our farm" --out /tmp/vo.wav

montaj generate-voiceover --text-file script.txt --voice Kore --vendor gemini --out /tmp/vo.wav

montaj generate-voiceover --text "..." --vendor kling --out /tmp/vo.mp3

search_images

Search for images from Wikimedia Commons, TheSportsDB, or the open web (SerpApi/Google Images). Returns {results: [{title, url, width, height, mime, license, artist, source, thumbnail?}]}.

montaj step search_images --query "golden gate bridge"

montaj step search_images --query "barcelona fc" --provider sportsdb

montaj step search_images --query "cityscape at night" --provider web --limit 10

Note: Web-provider results are not license-filtered, editorial use is the caller's responsibility.

search_news

Search Google News RSS for recent articles.

montaj step search_news --query "openai gpt-6 launch"
# → {"items": [{title, link, pubDate, source, snippet}]}

montaj step search_news --query "colombia elections" --limit 10

fetch_image

Download an image from any public HTTPS URL to a local path. SSRF-guarded: private/internal IPs rejected, HTTPS-only, content-type must be image/*, capped at 25 MiB by default.

montaj step fetch_image --url "https://..." --out /tmp/photo.jpg

fetch

Download videos from URLs via yt-dlp.

montaj fetch --url "https://www.tiktok.com/@handle/video/123"

montaj fetch --url "https://www.tiktok.com/@handle" --limit 15 --out ./clips/

Models Used