Hard Times: A Step Toward Closing the Multimodal Data Gap

Most video understanding benchmarks are easy to game. Models can often answer correctly by guessing or relying on external general knowledge, without ever watching the video.[1] In some cases, models achieve surprisingly high scores even when the video is not provided at all. When a question does not force the model to consider all of the provided evidence for answering, a high score measures guessing ability, and tells you little about how the model will behave in the real world.

We are excited to introduce HARD-TIME, a benchmark for long-video understanding designed to test whether models can ground answers in visual and audio evidence across time. HARD-TIME focuses on two common failure modes in video-language models: hallucination and sycophancy, and temporal retrieval over long videos.

Frontier models underperform on HARD-TIME

We evaluate Gemini 3.1 Pro, Gemini 3.5 Flash, and Gemini 3.1 Flash-Lite on HARD-TIME. The benchmark targets long-video questions that require evidence from both video and audio, and where the correct response depends on information distributed across time.

Hard times, hard negatives The hard-negative task evaluates whether models can reject plausible but unsupported premises. For each video, we construct questions about objects or states that are semantically close to entities that appear in the video, but are not themselves present. For example, a video containing a notebook may yield a question about a laptop, while a video containing a mug may yield a question about a water bottle. As a result, the questions remain realistic and scene-grounded, but cannot be answered from the video evidence.

Since the prompt’s target entity is absent by construction, the correct response is to mark the question as unanswerable. We therefore treat any attempt to provide a substantive answer as incorrect.

Temporal Retrieval Bench Temporal retrieval targets cross-modal temporal grounding. Given a natural-language description of a moment, the model must return the time interval in the video where the event occurs. The prompts are designed to span modalities, so neither a single frame nor the transcript alone is sufficient. For an example, consider the following query:

"When does she say the nails look beautiful and that she'll add cuticle oil before showing the result?"

Answering this requires locating a spoken statement (audio), the subsequent visual event of the result being shown (vision), and the temporal ordering between them, then returning the bounding interval. We ask the models to report timestamps in mm:ss according to this prompt:

You are solving a temporal video grounding question. Watch the supplied video and identify when the described moment or event happens. Return timestamps in the specified JSON format as strict MM:SS values from the beginning of the video, for example 07:35. If the event is present, provide the tightest start_time and end_time window you can support. If the event is not visible or audible, set found=false and use null timestamps.

To evaluate the model's performance we use the metric Intersection over Union (IoU), which measures the time-window overlap between expected and given answer, divided by their union. As thresholds, we define IoU@{0.3, 0.5, 0.7} and median midpoint error, which is the median distance of the midpoint of the prediction versus ground truth.

Dataset

HARD-TIME is built on a custom dataset of 5,942 unique videos paired with 9,443 questions, split roughly evenly between 4,812 temporal-retrieval and 4,631 hard-negative questions. The full benchmark is available on Hugging Face. Every question is hand-curated. Each video carries at most two questions (mean 1.6), and 3,501 of the videos appear in both tasks, so the two benchmarks measure different skills on a largely shared video pool, not two different distributions.

The distribution is intentionally long-form: median 15.4 minutes (mean 17.5), with 10% of videos under 5.5 minutes and 10% beyond 33 minutes, out to a 45-minute cap.

What the videos are about The corpus is real-world instructional video: people doing things with their hands and their screens, where questions have verifiable answers. About half the titles carry explicit how-to markers ("tutorial," "step by step," "for beginners"), and one in ten is shot first-person.

The same ~29% software share holds within each task, so screen-recorded computer use is a first-class citizen of both benchmarks. About 8% of hard-negative questions target on-screen UI elements (buttons, menus, icons) rather than physical objects.

How hard-negatives are anchored Every hard-negative question is pinned to a specific spoken line in the video (100% carry a transcript anchor), 93% are temporally framed ("When she explains…, what color is the…"), and 80% explicitly reference the spoken moment. The absent entity is chosen to be semantically close to something that is on screen, a laptop for a notebook, a water bottle for a mug, so the question stays realistic while remaining unanswerable.

How temporal moments are shaped Gold moments are short and sparse relative to the videos that contain them: median 10 seconds (mean 14.8s; 10th to 90th percentile 3 to 34s) inside videos with a median length of 15.4 minutes, a needle-to-haystack ratio of roughly 1:90. Moments are spread across the full length of the video (every position decile is populated), so neither "answer near the start" nor "answer in the middle" is a usable prior.

Results

Discussion

We organize our findings around the failure modes the benchmarks were built to expose.

Temporal retrieval and long-context grounding. Two patterns stand out. First, models find the region but miss the boundaries. On the Temporal Retrieval Bench, recall falls fast as the IoU threshold tightens. For all three Gemini models, R@1 at IoU greater than or equal to 0.3 sits around 0.72 to 0.78 but drops to around 0.38 at IoU greater than or equal to 0.7. Models locate roughly where an event happens but rarely bound exactly when it starts and ends.

Second, Gemini 3.1 Flash-Lite exhibits a catastrophic failure-mode: a subset of predictions that land in the wrong region of a long video, far enough to inflate the median midpoint error into the thousands of seconds even while IoU and recall stay competitive. This happens because Gemini 3.1 Flash-Lite ignores the output instructions and shifts minutes to hours (e.g. 00:13:40 → 13:40:00). When excluding these cases, Gemini 3.1 Flash-Lite performs on par with the other models.

Robustness to hard negatives. With weak distractors removed, accuracy falls substantially, indicating that part of headline video-QA performance elsewhere comes from distractor weakness rather than grounding.

When a model gets a hard negative wrong, it almost always makes the same mistake. It grabs a real, nearby object and calls it the missing one. Asked about a water bottle, it described the mug that was actually on the desk. Asked about a mouse, it answered for the keyboard. Here is the breakdown from our run on Gemini 3.1 Pro:

These models rarely invent something from nothing. They refuse to reject a false premise and instead snap to the closest real thing on screen.

Future work

HARD-TIME brings us an important step closer towards models that excel at long-video understanding. It shows two central weaknesses of current models and a scalable approach to fix them. We’re constantly exploring new and exciting architectures to advance the frontier of multimodal AI. If this research is interesting to you, or you want to use our dataset for evaluations, please reach out here.