video-search-and-summarization Agent工作流
经 AI Skill Hub 精选评估，video-search-and-summarization Agent工作流 获评「强烈推荐」。已获得 1.2k 颗 GitHub Star，这款Agent工作流在功能完整性、社区活跃度和易用性方面表现出色，AI 评分 8.0 分，适合有一定技术背景的用户使用。
📚 深度解析
video-search-and-summarization Agent工作流 工作流的设计遵循"最小配置，最大复用"原则：核心逻辑已经封装好，用户只需配置自己的 API Key 和业务参数即可快速上手。工作流内置错误处理和重试机制，在网络波动或 API 限速等情况下仍能稳定运行，适合作为生产环境的自动化基础设施。
在实际部署时，建议先在测试环境中运行 3-5 次，验证各个环节的输出结果符合预期，再部署到生产环境。AI Skill Hub 评分 8.0 分，是同类 Agent 工作流中的精选推荐。
📋 工具概览
提供GPU加速的视觉智能体参考架构套件，支持视频分析、检索和摘要生成。整合LLM和RAG技术，面向开发者构建高性能视频处理工作流，适合AI应用开发和视频智能分析场景。
video-search-and-summarization Agent工作流 是一套完整的 AI Agent 自动化工作流方案。通过可视化的节点编排，将复杂的多步骤任务拆解为清晰的自动化流程，实现全程无人值守的智能处理。支持与数百种外部服务和 API 无缝集成，适合构建数据处理管线、业务自动化和 AI 辅助决策系统。
📖 中文文档
提供GPU加速的视觉智能体参考架构套件，支持视频分析、检索和摘要生成。整合LLM和RAG技术，面向开发者构建高性能视频处理工作流，适合AI应用开发和视频智能分析场景。
video-search-and-summarization Agent工作流 是一套完整的 AI Agent 自动化工作流方案。通过可视化的节点编排，将复杂的多步骤任务拆解为清晰的自动化流程，实现全程无人值守的智能处理。支持与数百种外部服务和 API 无缝集成，适合构建数据处理管线、业务自动化和 AI 辅助决策系统。
- 可视化 Agent 工作流编排，无需编写复杂代码
- 支持多步骤自动化任务链，实现全流程无人值守
- 与外部 API、数据库和第三方服务无缝集成
- 内置错误处理与自动重试机制，保障稳定运行
- 提供可复用的自动化模板，快速在同类场景部署
- 自动化日常重复性工作，将精力集中于创造性任务
- 构建数据采集 → 处理 → 输出的完整自动化管线
- 实现跨平台、跨系统的数据流转和业务协同
# 方式一：pip 安装（推荐）
pip install video-search-and-summarization
# 方式二：虚拟环境安装（推荐生产环境）
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install video-search-and-summarization
# 方式三：从源码安装（获取最新功能）
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization
cd video-search-and-summarization
pip install -e .
# 验证安装
python -c "import video_search_and_summarization; print('安装成功')"
- 访问 GitHub 仓库获取工作流文件
- 在对应平台（Dify / Flowise / Make 等）中找到「导入工作流」功能
- 上传工作流文件
- 按照提示配置必要的环境变量和 API Key
- 运行测试确认流程正常后投入使用
# 命令行使用
video-search-and-summarization --help
# 基本用法
video-search-and-summarization input_file -o output_file
# Python 代码中调用
import video_search_and_summarization
# 示例
result = video_search_and_summarization.process("input")
print(result)
# video-search-and-summarization 配置文件示例（config.yml） app: name: "video-search-and-summarization" debug: false log_level: "INFO" # 运行时指定配置文件 video-search-and-summarization --config config.yml # 或通过环境变量配置 export VIDEO_SEARCH_AND_SUMMARIZATION_API_KEY="your-key" export VIDEO_SEARCH_AND_SUMMARIZATION_OUTPUT_DIR="./output"
简介
NVIDIA AI Blueprint: Video Search and Summarization (VSS)
Overview
The NVIDIA Blueprint for Video Search and Summarization (VSS) provides a suite of reference architectures for building vision agents and AI-powered video analytics applications. Those architectures bring together accelerated vision microservices, vision language models (VLMs), and large language models (LLMs) so you can use them in existing applications, as standalone microservices, or as part of a larger vision agent.
VSS is organized into three areas of processing and analysis: real-time video intelligence (feature extraction, embeddings, and stream understanding with results published to a message broker), downstream analytics (enrichment of metadata into trajectories, incidents, and verified alerts), and agentic and offline processing (orchestrated tools for search, Q&A, summarization, and clip retrieval, including via the Model Context Protocol).
This repository implements the blueprint and powers the NVIDIA build experience for natural-language video agents—search, summarization, visual Q&A, and related workflows—backed by generative AI, VLMs and LLMs, and NVIDIA NIM microservices as configured in the stacks below.
Repository Structure Overview
| Directory | Description |
|---|---|
services/agent/ | Video search and summarization agent (Python). Contains src/vss_agents/ (tools, agents, APIs, embeddings, evaluators, video analytics), tests/, stubs/, docker/, and 3rdparty/. See [services/agent/README.md](services/agent/README.md). |
services/ui/ | Frontend monorepo (Next.js, Turbo): apps/ (nemo-agent-toolkit-ui, nv-metropolis-bp-vss-ui) and shared packages/. See [services/ui/README.md](services/ui/README.md). |
services/analytics/ | Downstream analytics services for processing real-time video intelligence metadata. Contains behavior analytics stream processing and REST APIs for querying analytics results. |
services/analytics/behavior-analytics/ | Python streaming pipeline for spatial AI analytics, incident detection, Smart City, warehouse, playback, and other behavior analytics applications. Includes app entry points, configs, Docker support, tests, and detailed guides. See [services/analytics/behavior-analytics/README.md](services/analytics/behavior-analytics/README.md). |
services/analytics/video-analytics-api/ | Node.js and Express REST API service for VSS Video Analytics data. Exposes metrics, tracker, frames, behavior, clustering, events, sensor, config, alerts, and incidents endpoints backed by Elasticsearch. See [services/analytics/video-analytics-api/README.md](services/analytics/video-analytics-api/README.md). |
deploy/ | Deployment configs, Docker Compose, and Helm charts: NIM model configs, developer profiles (dev-profile-base, dev-profile-search, dev-profile-alerts, dev-profile-lvs), foundational services, LVS, RTVI, VLM-as-verifier, VST, and root compose.yml. Also contains deploy/docker/scripts/ — the Brev launchable notebook and dev-profile / patch scripts. |
tools/message-broker-consumers/ | Multiprocessing Redis and Kafka consumers that decode VSS protobuf messages from streams/topics and export them as JSON Lines files for inspection, debugging, or offline processing. See [tools/message-broker-consumers/README.md](tools/message-broker-consumers/README.md). |
tools/sdg-postprocessing/ | Dataset post-processing utilities for synthetic data generation workflows: semantic labeling helpers, raw data sanity checks, RGB/depth/video conversion, and ground-truth conversion for MTMC-compatible datasets. See [tools/sdg-postprocessing/README.md](tools/sdg-postprocessing/README.md). |
tools/rtvi-cv-mv3dt-utils/ | Offline utilities for generating MV3DT RTVI-CV configuration artifacts, including per-camera camInfo projection configs and MQTT publish/subscribe topology files for warehouse MV3DT deployments. See [tools/rtvi-cv-mv3dt-utils/README.md](tools/rtvi-cv-mv3dt-utils/README.md). |
skills/ | [agentskills.io](https://agentskills.io/specification)-compatible agent skills for VSS: one self-contained subdirectory per skill with SKILL.md frontmatter. Covers deploy and usage of search, summarization, alerts, VIOS, RT-VLM, LVS, and other related workflows—see the catalog and install notes in [skills/README.md](skills/README.md). |
libs/analytics/spatialai-data-utils/ | Spatial AI Data Utils (SDU): NVSchema / ground-truth / calibration / Sparse4D loaders, camera calibration + grouping (BEV group-origin / per-group fan-out), 3D↔2D geometry, multi-cam 3D-bbox visualization, detection (mAP) + tracking (HOTA, CLEAR, identity, count) evaluators, NVSchema result converters, and video↔frame utilities. See [libs/analytics/spatialai-data-utils/README.md](libs/analytics/spatialai-data-utils/README.md). |
Prerequisites
Hardware Requirements
The platform requirement can vary depending on the configuration and deployment topology used for VSS and dependencies like VLM, LLM, etc. For a list of validated GPU topologies and what configuration to use, see the GPU requirements.
Launchable Deployment
Ideal for: Quickly getting started with your own videos without worrying about hardware and software requirements.
Follow the steps from the documentation and notebook in deploy/docker/scripts directory to complete all pre-requisites and deploy the blueprint using Brev Launchable in a 2xRTX PRO 6000 SE AWS instance. - deploy/docker/scripts/deploy_vss_launchable.ipynb: This notebook is tailored specifically for the AWS CSP which uses Ephemeral storage.
Docker Compose Deployment
Ideal for: Deploying a VSS agent on your own hardware or bare metal cloud instance.
System Requirements
- OS:
- x86 hosts: Ubuntu 22.04 or Ubuntu 24.04
- DGX-SPARK: DGX OS 7.4.0
- IGX-THOR: Jetson Linux BSP (Rel 38.5)
- AGX-THOR: Jetson Linux BSP (Rel 38.4)
- NVIDIA Driver:
- 580.105.08 (x86 hosts with Ubuntu 24.04)
- 580.65.06 (x86 hosts with Ubuntu 22.04)
- 580.95.05 (DGX-SPARK)
- 580.00 (IGX-THOR and AGX-THOR)
- NVIDIA Container Toolkit: 1.17.8+
- Docker Engine: 28.3.3 <= Docker Engine < 29.5.0
- Docker Compose: v2.39.1+
- NGC CLI: 4.10.0+
Docker upper bound: Docker Engine 29.5.0+ may fail pulling NGC-hosted images. Use Docker Engine 28.3.3 or another supported version below 29.5.0.
Please refer to Prerequisites section here for installation details.
Use Case / Problem Description
The NVIDIA AI Blueprint for Video Search and Summarization addresses the challenge of deploying visual agents capable of interacting with large volumes of video data, both stored and streamed. This can be used to create vision AI agents, that can be applied to a multitude of use cases such as monitoring smart spaces, warehouse automation, and SOP validation. This is important where quick and accurate video analysis can lead to better decision-making and enhanced operational efficiency.
Quickstart Guide
Obtain API Key
- NVIDIA AI Enterprise developer licence required to local host NVIDIA NIM.
- API catalog keys:
- NVIDIA API catalog or NGC (steps to generate key)
Agent Workflows
We provide multiple reference Agent Workflows which demonstrate how the individual components can be leveraged by an agent:
| Workflow | Description |
|---|---|
| [Q&A and Report Generation (Quickstart)](https://docs.nvidia.com/vss/latest/quickstart.html) | Video retrieval, VLM-based Q&A, and report generation on short video clips |
| [Alert Verification](https://docs.nvidia.com/vss/latest/agent-workflow-alert-verification.html) | Realtime processing of videos using perception (object detection, tracking) and behavior analytics to generate alerts, which are subsequently verified with VLM to reduce false positives |
| [Real-Time Alerts](https://docs.nvidia.com/vss/latest/agent-workflow-rt-alert.html) | Continuous processing of video streams through VLM for anomaly detection |
| [Video Search](https://docs.nvidia.com/vss/latest/agent-workflow-search.html) | Natural language search across video archives using video embeddings (alpha) |
| [Long Video Summarization](https://docs.nvidia.com/vss/latest/agent-workflow-lvs.html) | Analysis and summarization of extended video recordings through chunking and aggregation of dense captions |
Software Components
- NIM microservices: Here are models used in this blueprint:
- Real-time video intelligence: The Real-Time Video Intelligence layer extracts rich visual features, semantic embeddings, and contextual understanding from video data in real-time, publishing results to a message broker for downstream analytics and agentic workflows. It provides three core microservices for processing video streams.
- Downstream analytics: The Downstream Analytics layer processes and enriches the metadata streams generated by real-time video intelligence microservices, transforming raw detections into actionable insights and verified alerts.
- Agent and offline processing: The top-level agent leverages the Model Context Protocol (MCP) to access video analytics data, incident records, and vision processing capabilities through a unified tool interface. It integrates multiple vision-based tools including video understanding with Vision Language Models (VLMs), semantic video search using embeddings, long video summarization for extended footage analysis, and video snapshot/clip retrieval.
架构先进，集成LLM+视觉能力，参考代码质量高。Star数适中但增长稳健，是构建视频智能应用的优质基础设施。
- 需要让 Claude / Cursor 操作本地工具的 AI 工程师
- 构建多智能体协作系统的 Agent 开发者
- 构建企业知识库 / RAG 检索应用的团队
- 配置 MCP 服务器时建议使用 stdio 传输 + JSON-RPC，避免暴露公网
- 生产部署优先使用 Docker Compose 隔离依赖，并挂载 volume 持久化数据
- 分块大小建议 256-512 tokens，向量库优选 pgvector 或 Qdrant
- Agent 任务先做 dry-run 验证工具调用链，再开启自主执行
- API key 直接提交到 git 仓库（请用 .env 并加入 .gitignore）
- MCP 配置路径拼错或权限不足，重启 Claude Desktop 才生效
- 容器内无法访问宿主机 localhost — 使用 host.docker.internal
- embedding 模型与查询模型不一致导致检索失效
- Python 依赖冲突：建议用 venv / uv 隔离环境
- Docker：video-search-and-summarization 提供官方镜像，docker compose up 一键启动
- CLI：直接 npm install -g / pip install，命令行调用
- 云端托管：可放在 Vercel / Railway / Fly.io 等 PaaS 平台
⚡ 核心功能
- 可视化 Agent 工作流编排，无需编写复杂代码
- 支持多步骤自动化任务链，实现全流程无人值守
- 与外部 API、数据库和第三方服务无缝集成
- 内置错误处理与自动重试机制，保障稳定运行
- 提供可复用的自动化模板，快速在同类场景部署
- 需要让 Claude / Cursor 操作本地工具的 AI 工程师
- 构建多智能体协作系统的 Agent 开发者
- 构建企业知识库 / RAG 检索应用的团队
- 配置 MCP 服务器时建议使用 stdio 传输 + JSON-RPC，避免暴露公网
- 生产部署优先使用 Docker Compose 隔离依赖，并挂载 volume 持久化数据
- 分块大小建议 256-512 tokens，向量库优选 pgvector 或 Qdrant
- Agent 任务先做 dry-run 验证工具调用链，再开启自主执行
- API key 直接提交到 git 仓库（请用 .env 并加入 .gitignore）
- MCP 配置路径拼错或权限不足，重启 Claude Desktop 才生效
- 容器内无法访问宿主机 localhost — 使用 host.docker.internal
- embedding 模型与查询模型不一致导致检索失效
👥 适合人群
🎯 使用场景
- 自动化日常重复性工作，将精力集中于创造性任务
- 构建数据采集 → 处理 → 输出的完整自动化管线
- 实现跨平台、跨系统的数据流转和业务协同
⚖️ 优点与不足
- +大幅减少重复性人工操作
- +可视化流程，清晰直观
- +可扩展性强，支持复杂场景
- −初始配置和调试需投入一定时间
- −强依赖外部服务的稳定性
- −复杂场景需具备一定技术基础
该工具使用 NOASSERTION 协议，商用场景请仔细阅读协议条款，必要时咨询法律意见。
AI Skill Hub 为第三方内容聚合平台，本页面信息基于公开数据整理，不对工具功能和质量作任何法律背书。
建议在沙箱或测试环境中充分验证后，再部署至生产环境，并做好必要的安全评估。
📄 NOASSERTION — 请查阅原始协议条款了解具体使用限制。
🔗 相关工具推荐
❓ 常见问题 FAQ
AI Skill Hub 点评：video-search-and-summarization Agent工作流 的核心功能完整，质量优秀。对于自动化工程师和运维人员来说，这是一个值得纳入个人工具库的选择。建议先在非生产环境试用，再逐步推广。
| 原始名称 | video-search-and-summarization |
| 原始描述 | 开源AI工作流：Suite of reference architectures for building GPU-accelerated vision agents and 。⭐1.2k · Python |
| Topics | 视频分析GPU加速智能体LLMRAG工作流 |
| GitHub | https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization |
| License | NOASSERTION |
| 语言 | Python |
收录时间：2026-05-16 · 更新时间：2026-05-19 · License：NOASSERTION · AI Skill Hub 不对第三方内容的准确性作法律背书。