Memories.ai ships O-MARC: how to make audio-visual AI faster, cheaper, and more accurate at once
- AI that watches video and listens to audio at the same time sits at the frontier of video understanding, but a single second of video eats up a huge number of tokens. The cost of running multiple camera feeds continuously is the main reason this class of model has stayed stuck in the lab.
- Memories.ai's O-MARC does the "smart discarding": it keeps only the key frames and audio anchors relevant to the question and throws the rest out as noise, cutting inference time by 34.6% (1.53x faster) and memory use by 34.7%.
- Compression usually trades accuracy for speed. This method claws the accuracy back with a distillation trick — the same model answers the same question twice, once "open-book" and once "closed-book" — pushing the average across four benchmarks to 45.8, above the uncompressed baseline's 44.1.
- The compression module, OMAC, is training-free and drops in front of an already-deployed model. Getting the higher accuracy score requires an extra round of reinforcement-learning training on top.
- The companion benchmark, UGC-AVQA, covers 1,000 videos and 4,816 questions. Every candidate question is first tested with the audio track deleted; anything the AI can still answer gets cut, leaving only questions that genuinely require hearing.
On July 8, Memories.ai released a piece of research called O-MARC (the companion paper had already gone up on arXiv on May 26). It tackles a very specific problem: how to make AI that processes video and audio together run faster, cheaper, and more accurately all at once.
Today's audio-visual AI is stuck on two problems
1. It's too expensive and too slow — the compute bill doesn't add up
A model doesn't "watch" a video directly. Before it goes in, the video gets chopped into a long string of tokens: each frame is sliced into many chunks, one chunk per token; audio is sliced by time into a run of tokens; then your question gets tacked on. The three pieces are strung together into one long sequence, and the model reads through it from start to finish.
The longer that sequence, the more memory and time it costs. Both video and audio are unusually token-hungry modalities, so stacking them together roughly doubles the sequence length.
Diagram (drawn by this site). Dark orange blocks are frame tokens, lighter blocks are audio tokens, dark gray is text. A 32-frame short clip alone already runs to a few thousand tokens.
A real measurement from the paper: even capped conservatively at one frame per second and 32 frames per clip, running a single clip (the 64-frame setup) on the long-video benchmark WorldSense takes 3.81 seconds and 24.2GB of memory. A single GPU running one stream alone is nearly maxed out.
Multiply those two numbers by the number of camera feeds and hours of runtime, and it's clear why scenarios that need 24/7 multi-feed video — surveillance centers, factory floors — have found the cost prohibitive.
2. Benchmarks that don't actually measure what they claim to
Say you're evaluating two video-analysis systems, both claiming to understand video and audio together. You run them on an off-the-shelf audio-visual benchmark: system A scores 70, system B scores 60. The question is — how much of A's 70 actually came from listening?
Most audio-visual benchmarks can't answer that. A large chunk of their questions are guessable from the picture alone — a car crash, someone falling, a machine smoking — none of which require sound to identify. A model that ignores audio entirely can still post a good score, which turns "joint audio-visual understanding" into something nobody can actually verify at the benchmark level.
Say you want to hire someone who knows English. The test you give them is all multiple choice, and three of the four options in every question are obviously grammatically wrong. Someone who doesn't know English can still score 80 by process of elimination. That test isn't measuring the skill you actually care about.
For anyone buying this kind of system, that means there's no reliable way to tell which vendor genuinely integrated audio and which one just bolted it on for show. The industry has been missing a test that can actually verify "it really understood what it heard."
Memories.ai's two co-founders, Shawn Shen and Ben Zhou, both came out of Meta, where they worked on the AI system behind Meta's Ray-Ban smart glasses. Along the way they ran into an unavoidable problem: the glasses record constantly, but users could never pull anything back out of that footage — the AI saw it but couldn't remember it. That pain point is what pushed them to leave and found Memories.ai in 2024, dedicated to turning video into something that can be searched and recalled. O-MARC is the latest step on that path.