Scenes and analysis

Indexing produces two layers of analysis:

• Scene level (local). The pipeline cuts the video at shot boundaries into timecoded segments and analyzes each one as a single clip, video and audio together. Everything a scene carries can be judged from that clip alone.
• Video level (global). A summarizer reads every scene's signals in order and produces the whole-video summary, genre, IAB labels and the GARM assessment. These need the full context, so they are never predicted per scene.

What a scene is

A scene is a contiguous segment of one video, addressed by video_id + scene_index (0-based) and bounded by start_ms / end_ms in milliseconds. Scene boundaries come from automatic shot-cut detection; the pipeline normalizes segment length to roughly 5–30 seconds. Search returns matches at the scene level, and ad break-points reference scene boundaries.

Scene fields

The same scene object (SceneLite) is returned by GET /api/v1/videos/{video_id}/scenes, the scene stream, the scenes array on the video detail, and inside search results (scenes[].scene). Field order mirrors the captioning priority groups.

Identity and timing

idstringrequired

Scene UUID.

video_idstringrequired

Parent video.

scene_indexintegerrequired

0-based ordinal inside the video.

start_ms / end_msintegerrequired

Scene start and end, in milliseconds from the start of the video.

Meaning

captionstring | null

Two to three English sentences narrating what happens, combining visuals, audio and on-screen text. This is the narrative description that the caption search modality embeds.

transcriptstring | null

Speech from the audio track only, translated to English. Empty when nobody speaks. Burned-in subtitles belong to text_on_screen, not here. Embedded for the transcript search modality.

text_on_screenstring[]

Text visible in the frame (titles, lower thirds, burned-in captions, signage), in the original language and casing.

actionstring[]

Up to three body-level actions actually visible in the clip, as free-form lowercase snake_case slugs (for example kneading_dough), most prominent first. Empty when no person is visible.

human_activitystring[]

action mapped onto the AVA-80 activity vocabulary. Derived by rule, so it can be shorter than action when an action has no AVA-80 counterpart.

Mood and audio

valenceinteger | null

Emotional polarity the scene conveys, 1 (very negative) to 5 (very positive). Describes the content, not a viewer reaction.

arousalinteger | null

Energy / activation, 1 (very low) to 5 (very high).

music_moodstring | null

Character of the background music: calm, energetic, dramatic, playful, melancholic, tense, or none.

audio_eventsstring[]

Up to five non-speech, non-music sounds as snake_case slugs (applause, laughter, siren, ...).

Brand safety

garm_categoriesstring[]

GARM brand-safety categories actually depicted in this clip. Most scenes have an empty array. Scene-level flags are deliberately sensitive; the video-level garm_video assessment is the gatekept view. See Taxonomies and verdicts.

Entities

brandsstring[]

Up to five brand names explicitly shown (logo, on-screen text, packaging) or spoken. No guessing; original spelling.

product_categories_visiblestring[]

Up to six generic English nouns for product types visible on screen (smartphone, burger, ...).

peoplestring[]

Real names only, written on screen or spoken. Role or demographic labels such as "host" are never used, so this is usually empty. No facial recognition is involved; see Data handling.

locationsstring[]

Zero to two generic English descriptions of the setting (indoor cafe, city street). Never an exact address.

Environment and media

weatherstring | null

Observed weather or sky state from a fixed vocabulary of 20 values. Indoor is used only when the clip is clearly inside a building or vehicle; a sky that is simply out of frame is Unknown / Not visible.

keyframe_urlstring | null

Presigned URL of a representative frame. Valid for about one hour; fetch the scene again for a fresh link rather than storing it.

clip_urlstring | null

Presigned URL of the scene clip. Same one-hour validity as keyframe_url.

info

Scenes do not carry IAB categories, genre or a GARM severity level. Those are video-level fields; the only per-scene severity is garm_level on scene markers (below).

Scene markers

scene_markers on the video detail is a lightweight record for every scene, so you can draw a timeline without paging through the full objects.

Field	Type	Meaning
id, scene_index, start_ms, end_ms		Same as the scene.
garm_level	high | medium | low | none	Highest brand-safety severity touching this scene. Defaults to none.
garm_categories	string[]	The scene's GARM flags.

Video-level fields

Returned by GET /api/v1/videos/{video_id}. The list endpoint returns the subset marked with a check.

Field	Type	In list	Meaning
summary	string | null	yes	Three to five English sentences synthesizing the whole video: topic, narrative arc, mood changes, intent. Not a scene-by-scene recap.
genre	string[]	yes	One to three genre labels from a fixed film-and-TV vocabulary, dominant first.
iab_content_leaf	string[]	yes	What the video is about, as IAB Content Taxonomy leaf labels (up to five).
iab_audience_interest_leaf	string[]		IAB Audience Taxonomy interest segments for the viewers this content suits (up to five).
iab_audience_purchase_intent_leaf	string[]		IAB Audience Taxonomy purchase-intent segments those viewers are likely to have (up to five).
iab_ad_product_leaf	string[]		IAB Ad Product Taxonomy categories of ads that fit next to this content (up to five).
garm_video	GarmEntry[]	yes	Video-level GARM assessment: one entry per category judged genuinely present, each with category, level (high | medium | low), reasoning and optional time_ranges.
transcript_coverage	number | null		Share of the video with speech.
scene_count	integer		Total scenes.
scenes	SceneLite[]		First scene_limit scenes.
scene_markers	SceneMarker[]		All scenes, lightweight.
aggregations	VideoAggregations | null		Roll-ups below.
keyframe_url	string | null	yes (list only)	Representative frame for the video card.
title, duration_ms, source_url, created_at, external_content_id		partly	Metadata. source_url is populated only for http(s) origins.

Lifecycle fields (status, reason_code, indexed_at, index_trigger, job_id, reindex_run_id) are covered in Videos and jobs.

Aggregations

aggregations is computed server-side by walking every scene once, so you get whole-video panels without paginating.

Field	Shape	Source scene field
brands	{name, count, scene_indexes[]}	brands
products	{name, count}	product_categories_visible
locations	{name, count}	locations
people	{name, count}	people
audio_events	{name, count}	audio_events
music_moods	{mood, count}	music_mood
text_on_screen	string[]	text_on_screen
mood_timeline	{scene_index, start_ms, valence, arousal} per scene	valence, arousal
actions	{name, count}	action
human_activities	{name, count}	human_activity
weather_counts	{name, count}	weather (Unknown / Not visible excluded)

Streaming vs paged scenes

Paged

GET /api/v1/videos/{video_id}/scenes?offset=&limit= returns a ScenesPage: items, offset, limit (max 200, default 50) and total. Iterate offset by limit until you reach total.

An unknown video_id returns an empty page with total: 0, not a 404. Check the video exists with GET /api/v1/videos/{video_id} if you need to tell the two apart.

Streamed (NDJSON)

GET /api/v1/videos/{video_id}/scenes/stream pushes one JSON object per line as analysis completes:

Line	Meaning
{"event": "scene", "scene": {...}}	A newly analyzed scene, same shape as a paged item. Never repeated within a connection.
{"event": "keepalive"}	Idle heartbeat. Ignore.
{"event": "done", "job_status": "done"}	Indexing finished; the connection closes. job_status can also be failed, or null for a video without an indexing job.
{"event": "timeout"}	Server-side maximum stream duration reached. Reconnect for a fresh snapshot.

Scene order follows analysis completion, not scene_index. Connecting to an already-indexed video replays all scenes immediately and closes with done. If the connection ends without a terminal line, reconnect; already-analyzed scenes replay first. An unknown video_id returns 404 here.

tip

Use the stream while a job is running to start work on early scenes; use pages for random access once the job is done. Both return the same scene object.

Where else scenes appear

• Search: POST /api/v1/search groups scene hits by video. Each scenes[] entry is {score, scene} with the full scene object.
• Ad matching: break-points reference scene_id with start / end timestamps at scene boundaries.
• Live streams: a burst's scenes[] are analyzed HLS segments with the same analysis fields plus segment metadata (segment_index, segment_url, status). See Live.