Searching scenes

Search runs over the scenes of your indexed videos and returns scene-level matches grouped by their parent video. Three endpoints share one response shape:

Endpoint	Query	When to use	Reference
POST /api/v1/search	Text (JSON)	Default. Scores scenes across caption and, optionally, transcript and multimodal embeddings.	Semantic search
POST /api/v1/search/deep	Text (JSON)	Same request; adds a second-pass multimodal rerank over the top scenes. Higher quality, typically +2–5 s.	Deep search
POST /api/v1/search/multimodal	Image or audio file (multipart)	When text is a poor description of what you are looking for. Results come back in the omni column only.	Multimodal search

note

Every example uses https://video.pyler.ai and an API key in $VUF_API_KEY. A video is searchable once its job reports status: "done"; see Indexing videos.

Request fields

POST /api/v1/search and POST /api/v1/search/deep take the same JSON body.

querystringbodydefault:

Natural-language description of the moment you want.

modalitystringbodydefault: both

Which embeddings drive the search: caption, transcript, omni, both or all. See Modalities.

kintegerbodydefault: 20

Videos returned per modality column. 1–100.

scene_poolintegerbodydefault: 60

Candidate scenes pulled before grouping into videos. 10–200.

iab_t1string[]body

Restrict to videos in these IAB Tier-1 content categories.

garm_min_levelstringbody

Brand-safety floor: low, medium or high.

duration_min_secintegerbody

Minimum video duration in seconds.

duration_max_secintegerbody

Maximum video duration in seconds.

length_bucketstringbody

short, medium or long.

video_idstring (uuid)body

Restrict the search to one video's scenes.

judgestringbodydefault: none

text turns on the AI relevance judge and switches the response to an NDJSON stream. Requires modality: "caption". See AI judge.

min_relevanceintegerbodydefault: 3

With judge: "text", the 0–10 relevance a scene needs to stay in the summary line.

deepbooleanbodydefault: false

Also present on the body. POST /api/v1/search/deep is the documented way to request the rerank pass.

Modalities

modality	Matches against	Columns populated
caption	Narrative description of what happens in the scene	caption
transcript	Verbatim speech (silent scenes excluded)	transcript
omni	Visual/audio embedding	omni
both (default)	Caption and omni, side by side	caption, omni
all	Caption, transcript and omni	all three

Columns that are not part of the chosen modality come back as empty arrays.

A first search

cURL

curl -X POST https://video.pyler.ai/api/v1/search \
  -H "Authorization: Bearer $VUF_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "two people having a coffee shop conversation",
    "modality": "both",
    "k": 10,
    "iab_t1": ["Food & Drink"],
    "garm_min_level": "medium",
    "duration_max_sec": 600
  }'

Python

import os, requests

BASE = "https://video.pyler.ai/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['VUF_API_KEY']}"}

r = requests.post(f"{BASE}/search", headers=HEADERS, json={
    "query": "two people having a coffee shop conversation",
    "modality": "both",
    "k": 10,
    "iab_t1": ["Food & Drink"],
    "garm_min_level": "medium",
    "duration_max_sec": 600,
})
r.raise_for_status()
result = r.json()

for hit in result["caption"]:
    v = hit["video"]
    print(f'{hit["score"]:.2f}  {v["title"]}  ({v["id"]})')
    for s in hit["scenes"]:
        sc = s["scene"]
        print(f'   {s["score"]:.2f}  scene {sc["scene_index"]}  {sc["start_ms"]}–{sc["end_ms"]} ms  {sc["caption"]}')

Result shape

The response has three columns, caption, transcript and omni. Each is a list of video hits (VideoMatch), and each video hit lists the scenes inside it that matched (SceneMatch).

Response (200, truncated)

{
  "caption": [
    {
      "score": 0.82,
      "video": {
        "id": "3f6c1b2e-9a4d-4c7e-8b1f-2d5a6e7f8c90",
        "title": "Morning at the corner cafe",
        "duration_ms": 412000,
        "summary": "Two friends catch up over coffee...",
        "genre": ["drama"],
        "iab_content_leaf": ["Food & Drink"],
        "garm_video": [],
        "keyframe_url": "https://...",
        "created_at": "2026-10-01T10:00:00Z",
        "is_user_upload": true,
        "org_id": "org_...",
        "status": "done",
        "external_content_id": null,
        "done_scenes": 38,
        "total_scenes": 38
      },
      "scenes": [
        {
          "score": 0.83,
          "scene": {
            "id": "9c0d1e2f-3a4b-4c5d-8e6f-7a8b9c0d1e2f",
            "video_id": "3f6c1b2e-9a4d-4c7e-8b1f-2d5a6e7f8c90",
            "scene_index": 7,
            "start_ms": 12000,
            "end_ms": 16500,
            "caption": "Two people sit across a small table with coffee cups, talking.",
            "transcript": "So how did the interview go?",
            "text_on_screen": [],
            "valence": 3,
            "arousal": 2,
            "music_mood": null,
            "audio_events": ["speech"],
            "garm_categories": [],
            "brands": [],
            "product_categories_visible": ["coffee"],
            "people": [],
            "locations": ["cafe"],
            "action": ["talking"],
            "human_activity": ["conversation"],
            "weather": null,
            "keyframe_url": "https://...",
            "clip_url": "https://..."
          }
        }
      ]
    }
  ],
  "transcript": [],
  "omni": [ ]
}

caption / transcript / omniVideoMatch[]

One list per modality column. Empty when the column is not part of the requested modality.

VideoMatch.scorenumber

Aggregated video score. It favours a strong single match while letting multi-match videos surface above borderline single-match ones. Compare scores within one response only; the aggregation can change between releases.

VideoMatch.videoVideoLite

Compact video summary: id, title, duration_ms, summary, genre, iab_content_leaf, garm_video, keyframe_url, created_at, is_user_upload, org_id, status, external_content_id, done_scenes, total_scenes.

VideoMatch.scenesSceneMatch[]

The scenes inside this video that matched, each with its own score and the full scene record.

SceneMatch.sceneSceneLite

The timecode is start_ms / end_ms (milliseconds from the start of the video); scene_index is the 0-based position. The record also carries caption, transcript, text_on_screen, mood (valence, arousal, music_mood), audio_events, garm_categories, brands, product_categories_visible, people, locations, action, human_activity, weather, and presigned keyframe_url / clip_url.

To jump to a hit in a player, seek to scene.start_ms; to show it, render scene.keyframe_url or play scene.clip_url. For the whole video, GET /api/v1/videos/{video_id} with video.id.

Limits and paging

Search has no cursor. One request returns at most k videos per column (1–100, default 20), drawn from a pool of scene_pool candidate scenes (10–200, default 60). To get different results, narrow the request rather than paging it:

• Raise k and scene_pool within their caps when you need more hits.
• Filter with iab_t1, garm_min_level, duration_min_sec / duration_max_sec or length_bucket.
• Set video_id to search inside one video.

Deep search

POST /api/v1/search/deep takes the same body and returns the same shape. It applies a second-pass multimodal rerank over the top scenes, so ordering is more precise at the cost of latency (typically +2–5 s).

cURL

curl -X POST https://video.pyler.ai/api/v1/search/deep \
  -H "Authorization: Bearer $VUF_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "the climactic moment the lead looks straight at the camera", "modality": "both", "k": 20}'

Python

r = requests.post(f"{BASE}/search/deep", headers=HEADERS, json={
    "query": "the climactic moment the lead looks straight at the camera",
    "modality": "both",
    "k": 20,
})

Give deep search a longer client timeout than a plain search; see Timeouts and retries.

Multimodal search

POST /api/v1/search/multimodal takes multipart/form-data instead of JSON. The query is the uploaded file; results come back in the omni column only.

Field	Type	Default	Notes
file	file	required	The image or audio file to query with.
modality	string	image	image or audio, describing file.
k	integer	20	Videos returned.
scene_pool	integer	60	Candidate scenes pulled before grouping.

cURL

curl -X POST https://video.pyler.ai/api/v1/search/multimodal \
  -H "Authorization: Bearer $VUF_API_KEY" \
  -F "file=@query-frame.jpg" \
  -F "modality=image" \
  -F "k=10"

Python

with open("query-frame.jpg", "rb") as f:
    r = requests.post(f"{BASE}/search/multimodal", headers=HEADERS,
                      files={"file": f}, data={"modality": "image", "k": 10})
r.raise_for_status()
hits = r.json()["omni"]

AI judge (NDJSON)

With judge: "text" (requires modality: "caption"), POST /api/v1/search scores each candidate scene for relevance to the query and the response becomes an application/x-ndjson stream, one JSON object per line:

type	Content
meta	The plain search result (query, judge, the caption / transcript / omni columns, verify_scene_ids, total).
result	One line per judged scene: scene_id, video_id, score (0–10 relevance), reason, error.
summary	Only scenes at or above min_relevance (default 3), as videos ordered by best relevance then best cosine, with score unchanged and the top k videos. empty: true means nothing matched; all_errors: true means the judge itself failed; errors counts failed judgements.
done	End of stream.

cURL

curl -N -X POST https://video.pyler.ai/api/v1/search \
  -H "Authorization: Bearer $VUF_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "child playing in snow", "modality": "caption", "k": 10, "judge": "text", "min_relevance": 5}'

Python

import json

body = {"query": "child playing in snow", "modality": "caption", "k": 10,
        "judge": "text", "min_relevance": 5}
with requests.post(f"{BASE}/search", headers=HEADERS, json=body, stream=True) as r:
    r.raise_for_status()
    for line in r.iter_lines():
        if not line:
            continue
        evt = json.loads(line)
        if evt["type"] == "result":
            print(evt["scene_id"], evt["score"], evt["reason"])
        elif evt["type"] == "summary":
            kept = [] if evt["empty"] else evt["videos"]
        elif evt["type"] == "done":
            break

Stream (abridged)

{"type":"meta","query":"child playing in snow","judge":"text","caption":[...],"transcript":[],"omni":[],"verify_scene_ids":[...],"total":14}
{"type":"result","scene_id":"9c0d...","video_id":"3f6c...","score":8,"reason":"A child packs snow in a backyard.","error":false}
{"type":"summary","min_relevance":5,"videos":[...],"empty":false,"errors":0,"all_errors":false}
{"type":"done"}

Recipe: index a video and search it

1. Submit the video

   curl -s -X POST https://video.pyler.ai/api/v1/videos \
     -H "Authorization: Bearer $VUF_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"url": "https://example.com/your-clip.mp4", "title": "demo clip"}'
   # {"video_id": "3f6c...", "job_id": "a1b2...", "created": true}

2. Poll until the job is done

   while true; do
     status=$(curl -s https://video.pyler.ai/api/v1/jobs/$JOB_ID \
       -H "Authorization: Bearer $VUF_API_KEY" | jq -r .status)
     echo "  $status"
     [ "$status" = "done" ]   && break
     [ "$status" = "failed" ] && { echo "job failed"; exit 1; }
     sleep 5
   done

3. Search a moment inside it

   Pass the new video_id to keep the search inside that one video, or leave it out to search your whole library.

   cURL

   curl -X POST https://video.pyler.ai/api/v1/search \
     -H "Authorization: Bearer $VUF_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"query": "child playing in snow", "modality": "both", "k": 10, "video_id": "'"$VIDEO_ID"'"}'

   Python

   r = requests.post(f"{BASE}/search", headers=HEADERS, json={
       "query": "child playing in snow", "modality": "both", "k": 10, "video_id": video_id,
   })
   for hit in r.json()["caption"]:
       for s in hit["scenes"]:
           sc = s["scene"]
           print(sc["start_ms"], sc["end_ms"], sc["clip_url"])

   Each scenes[*].scene exposes keyframe_url and clip_url as presigned URLs, ready to render directly. They expire after about an hour, so render them now and search or fetch the scene again for a fresh link rather than storing them.

Phrasing queries

The modality decides what your words are compared against, so phrase the query the way that index is written.

• caption: describe what is seen and happening, as a caption would: "two people having a coffee shop conversation", "child playing in snow". Subjects, actions and setting work better than abstract labels.
• transcript: use the words likely to have been spoken; it matches verbatim speech and skips silent scenes.
• omni: use it for looks and sounds that are hard to put into words, or switch to multimodal search and upload a reference frame or audio clip.
• both / all: run several columns side by side when you are unsure which index holds the moment, then read whichever column scores higher.
• Add video_id when you already know the video, and iab_t1 / garm_min_level / duration filters to keep the candidate pool relevant.
• Use /search/deep when ordering matters more than latency, and judge: "text" when you need a 0–10 relevance per scene with a reason you can show.

Next steps

Indexing videos

Submission paths, job polling and reading scenes.

Scenes and analysis

What each scene field means.

Taxonomies and verdicts

IAB categories and GARM levels used by the filters.

Usage and limits

Caps on k, scene_pool, timeouts and retries.