Who Judges the Frame? Auditing Multimodal LLM Judges for News Framing Across Event

News coverage of major world events is shaped by text, images, and their combinations through framing. Large Language Models (LLMs), including multimodal LLMs, are being used to analyze these multimodal datasets for aspects like framing, sentiment, ideological slant, and perspective differences. This introduces a methodological challenge because LLM outputs may reflect model tendencies, prompt design, and various social or cultural assumptions, not just content properties. The research audits the use of LLMs for large-scale framing and perspective analysis in multimodal news coverage using an event-centered dataset from 2025–2026. Researchers combine embedding-based measures of within-event viewpoint similarity with model-based assessments of framing across different input conditions. The study aims to examine the usefulness and limitations of LLM-based media analysis by developing an audit protocol that mandates reporting modality effects, metadata sensitivity, and prompt artifacts alongside substantive claims about news framing.

The core tension explored is the reliability of LLMs as objective instruments for complex socio-political analysis, specifically in multimodal contexts like news framing. The methodological contribution shifts the focus from treating model outputs or dataset labels as ground truth to auditing the system itself. This implies that scale and complexity in media analysis do not automatically confer accuracy; they introduce systemic artifacts dependent on prompt engineering and inherent model biases. The necessity of reporting modality effects and prompt artifacts suggests that any claim about ideological viewpoint differences derived from LLMs must be contextualized by the specific operational context—the model, the input modality, and the prompting strategy. This challenges the implicit trust placed in automated analysis systems, demanding transparency regarding the non-content variables influencing results. The subsequent research steps should focus on developing standardized, reproducible auditing benchmarks for multimodal framing assessment, testing how different modalities interact with ideological concepts, and establishing causal links between prompt design choices and resulting analytical distributions. What operational criteria must be established to determine when LLM output moves from being a useful analytical tool to an opaque source of mediated interpretation?