A Unified Framework for Generation-Time Source Attribution in Omni-Modal LLMs
Omni-modal models answer questions about long videos, recordings, and documents, but they do not show where an answer came from. Two ways to find out exist today, and they answer different questions.
Every generated token attends over the whole input, so the generation pass already holds a distribution over the sources for every token. The question is how to read it once the sources are seconds of sound and video rather than sentences.
Use the structure the input already has: about one second per unit for video and audio, following the encoder's own time grid, and one unit per text passage or image. The question, the options, and the template are never candidates.
Average attention over all heads of the last third of the decoder layers. A unit's score for a sentence is the attention mass the sentence's tokens put on it. Scores add up, so one pass gives sentence- and response-level attributions.
sk(Sj) = Σt∈Ck Σi∈Sj at(i)A unit is labelled when its score is ≥ 1.5× the mean of the other units of its own modality, so text cannot crowd out images. Flat scores give no label, and the sentence abstains. Time units merge into intervals.
Where ablation data are available, learn one weight per attention head. It is still a single pass at test time.
Pick a modality and a model, then click a sentence. Cells are source units, and darker cells hold more attention mass. Orange outlines are the units OmniTrace labels. Click a cell to play the clip from there.
These examples were selected (all three models answer correctly). The scores, labels, and causal numbers are the unedited outputs of the runs in the paper. Video: NExT-GQA, frames only. Audio: MMAU. Documents: MMDocRAG, 15 quotes.
We hide units at the encoder input behind content-free placeholders of the same length (black frames, silence, a black image, blank tokens), replay the model's own response, and measure how much its log-probability drops. Each method is compared with hiding the same number of random units. The same operation lets leave-one-out and ContextCite run on audio and video, so this is a direct comparison of single-pass and perturbation attribution on continuous inputs.
On all three models and all three modalities, hiding OmniTrace's units hurts more than hiding random units at every budget. This also holds for single sentences: a sentence's units matter for that sentence more than for the others.
From one pass, OmniTrace reaches 96% of leave-one-out's lift on video, 77% on audio, and 95% on documents, and it has about twice the lift of embedding similarity.
On all three models, the audio seconds a model cites itself matter less than the ones OmniTrace finds. On Qwen2.5-Omni, citations are often missing, sometimes copy the prompt's example span, and asking for them changes the answer.
The setting was fixed on Qwen2.5-Omni video and carried unchanged to MiniCPM-o 4.5 and Qwen3-Omni-30B. Where the default falls short (MiniCPM-o documents), trained head weights recover 96% of leave-one-out's lift, still in one pass.
| Video | Audio | Image–text | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Fwd | Lift | S-lift | Fwd | Lift | S-lift | Fwd | Lift | S-lift |
| Baselines on Qwen2.5-Omni-7B | |||||||||
| Embedding similarity | 0 | 0.008 | 0.009 | 0 | 0.034 | 0.031 | 0 | 0.216 | 0.216 |
| Self-citation | 1 | 0.013 | 0.012 | 1 | 0.040 | 0.036 | 1 | 0.184 | 0.349 |
| Leave-one-out* | 41 | 0.016 | 0.024 | 17 | 0.100 | 0.133 | 16 | 0.368 | 0.404 |
| ContextCite* | 33 | 0.016 | 0.027 | 28 | 0.099 | 0.139 | 33 | 0.389 | 0.397 |
| OmniTrace on Qwen2.5-Omni-7B | |||||||||
| OmniTrace | 0 | 0.016 | 0.015 | 0 | 0.077 | 0.110 | 0 | 0.351 | 0.376 |
| OmniTrace-trained | 0 | 0.014 | 0.016 | 0 | 0.081 | 0.116 | 0 | 0.374 | 0.385 |
| OmniTrace on MiniCPM-o 4.5 | |||||||||
| OmniTrace | 0 | 0.024 | 0.027 | 0 | 0.136 | 0.102 | 0 | 0.105 | 0.159 |
| OmniTrace-trained | 0 | 0.067 | 0.065 | 0 | 0.169 | 0.120 | 0 | 0.432 | 0.706 |
| OmniTrace on Qwen3-Omni-30B-A3B | |||||||||
| OmniTrace | 0 | 0.033 | 0.032 | 0 | 0.112 | 0.095 | 0 | 0.316 | 0.448 |
| Agreement with human evidence (Qwen2.5-Omni-7B) | Video AUROC | Video Hit@1 | Docs AUROC | Docs Hit@1 |
|---|---|---|---|---|
| Random | 0.500 | 0.215 | 0.500 | 0.194 |
| Embedding similarity | 0.604 | 0.288 | 0.763 | 0.527 |
| Leave-one-out | 0.549 | 0.309 | 0.721 | 0.657 |
| ContextCite | 0.566 | 0.342 | 0.743 | 0.642 |
| OmniTrace | 0.637 | 0.230 | 0.829 | 0.698 |
| OmniTrace-trained | 0.626 | 0.321 | 0.815 | 0.695 |
Supports Qwen2.5-Omni, MiniCPM-o 4.5, and Qwen3-Omni-30B-A3B. Any attribution method that ranks the same units can be scored with the same causal test.
# install (one environment per model family) git clone https://github.com/UCSB-AI/OmniTrace.git cd OmniTrace pip install -r requirements.txt python -m spacy download en_core_web_sm # the three examples on this page python scripts/demo.py # paper runs + causal test bash scripts/run_mmau.sh full bash scripts/run_mmau.sh causal
from omnitrace import OmniTrace ot = OmniTrace("qwen2.5-omni-7b") out = ot.trace({ "question": "What sport is the first speaker getting a scholarship for?", "options": ["Basketball", "Soccer", "Baseball", "Softball"], "sources": [{"sid": "a0", "modality": "audio", "path": "examples/media/427e55eb-4ab4-4bb0-8c8f-574f10097087.wav"}], }) for s in out["sentences"]: print(s["text"], s["spans"] or s["sources"])
@misc{yan2026omnitrace,
title = {OmniTrace: A Unified Framework for Generation-Time Source Attribution in Omni-Modal LLMs},
author = {Qianqi Yan and Yichen Guo and Ching-Chen Kuo and Shan Jiang and Hang Yin and Yang Zhao and Xin Eric Wang},
year = {2026},
eprint = {2604.13073},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2604.13073}
}