Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding

Yikai Zheng^1,2, Xin Ding³, Yifan Yang⁵, Shiqi Jiang⁵, Hao Wu⁴, Qianxi Zhang⁵, Weijun Wang¹, Ting Cao¹, Yunxin Liu¹

¹Institute for AI Industry Research (AIR), Tsinghua University ²Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University
³University of Science and Technology of China ⁴Nanjing University ⁵Microsoft Research Asia

Paper Code 🤗 Model

TL;DR

Problem We tackle Proactive Streaming Video Understanding, where the model needs to keep monitoring the video and answer user queries at proper time.
Solution Em-Garde introduces a two-stage system that separates heavy semantic understanding and lightweight streaming perception
Impact We achieve fast, accurate proactive interactions for live video analytics, surveillance, and embodied AI.

Abstract

Recent advances in Streaming Video Understanding has enabled a new interaction paradigm where models respond proactively to user queries. Current proactive VideoLLMs rely on per-frame triggering decision making, which suffers from an efficiency-accuracy dilemma. We propose Em-Garde, a novel framework that decouples semantic understanding from streaming perception. At query time, the Instruction-Guided Proposal Parser (IGPP) transforms user queries into structured, perceptually grounded visual proposals; during streaming, a Lightweight Proposal Matching Module (LPMM) performs efficient embedding-based matching to trigger responses. Experiments on StreamingBench and OVO-Bench demonstrate consistent improvements over prior models in proactive response accuracy and efficiency, validating an effective solution for proactive video understanding under strict computational constraints.

Method

Teaser figure showing Em-Garde's approach

Unlike former Proactive Streaming VideoLLMs which make triggering decisions from scratch at every timestep, Em-Garde decouples semantic understanding from streaming perception, and extracts heavy instruction-related reasoning out of the streaming loop, enabling efficient and generalizable triggering decisions for proactive responses.

Em-Garde consists of two stages: (1) the Instruction-Guided Proposal Parser (IGPP) transforms user queries into structured, perceptually grounded visual proposals; and (2) the Lightweight Proposal Matching Module (LPMM) performs efficient embedding-based matching to trigger responses during streaming.

Experiments

Proactive Response Results

Em-Garde demonstrates superior proactive response abilities across different benchmarks, outperforming prior streaming VideoLLMs by a significant margin.

Online VideoQA Results

While optimized for proactive responses, Em-Garde maintains strong online video question-answering abilities, performing on par with or better than state-of-the-art models.

Efficiency

By decoupling heavy semantic reasoning from the streaming loop, Em-Garde achieves efficient real-time streaming inference. The lightweight LPMM maintains constant per-frame latency even on long videos, while delivering the highest throughput-accuracy trade-off among all compared methods.

Efficiency: streaming inference time over video length

Citation

@misc{zheng2026emgarde, title={Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding}, author={Yikai Zheng and Xin Ding and Yifan Yang and Shiqi Jiang and Hao Wu and Qianxi Zhang and Weijun Wang and Ting Cao and Yunxin Liu}, year={2026}, eprint={2603.19054}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2603.19054}, }