EmoHarness

EmoHarness

Boosting Multimodal Emotion Understanding
with Meta-Harness Evolving

Fenghai Li1 Bingxuan Li1 Jiawen Zhang2 Yifan Shen1 Jiateng Liu1 Yutong Zhang3 Yue Guo1

1University of Illinois Urbana-Champaign 2University of California, Davis 3Stanford University

Paper Code BibTeX

Frozen student, evolving harness

Better emotion reading, with the model’s weights left untouched

Small multimodal models misjudge emotion when words, voice and visual cues conflict. EmoHarness runs such a model, the student, inside a harness: the prompts, settings and verification code around its calls. What the student sees, what is said and how it sounds become numbered evidence that every answer must cite.

Offline, a stronger LLM, the Builder, reads the student’s errors and execution traces and edits the harness. Validation gates decide which edits are kept.

What stays out of the loop

  • No weight updates
  • No Builder calls at inference
  • No evaluation items shown to the Builder
  • No raw audio: the student reads transcript and prosody text

One clip, conflicting cues

Selected case

Gemma-4-E4B on a fine-grained sentiment clip, a task type held out from harness construction

Identify the most prominent fine-grained sentiment displayed in the video.

One of 9 labels, from very strong negative to very strong positive reference: negative

Speaker at 0.40 seconds, hands clasped; box A marks the clasped hands. Subtitle: It's.
0.40 s
Speaker at 0.93 seconds with a serious face; box B marks the face. Subtitle: my form.
0.93 s
Speaker at 1.17 seconds. Subtitle: form so.
1.17 s

Seenv2v3Upright posture, clasped handssuggests neutral

Saida1“It’s not my form so”suggests negative

+ edits evolved by GPT-5.6 Luna4 calls

  • Posture backs a “neutral” rival
  • Speech [a1] outweighs it
  • GPT-6.1 Sol’s harness also answers negative

Selected, not representative. The student saw the clip at 2 fps; frames are subtitled with the words spoken near each one.

Overview

On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6 to 13.4 points over a single video-only call, for all four students and both Builders. Most of the boost comes from supplying and organizing audio-derived evidence.

Where the boost comes from

Range across the four students (both Builders for evolved edits), task-macro Recognition, percentage points.

  1. Add transcript and prosody textAudio − Video +5.38 to +10.71
  2. Add the four-call scaffoldShared − Audio −0.79 to +2.54
  3. Add evolved editsMHE − Shared −0.07 to +0.95
  4. EmoHarness in totalMHE − Video +7.61 to +13.36

No evolved increment survives Holm correction over eight tests.

Read the abstractHide the abstract

Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student’s visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student’s errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6–13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by −0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness’s emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.

EmoHarness at a glance. (a) A selected held-out clip. (b) Meta-Harness Evolving: the Builder edits the harness offline; checks and validation gates promote edits over four generations. (c) Task-macro Recognition contrasts per student; whiskers are 95% intervals and the two scales differ.

How EmoHarness works

A harness is a bundle of prompts, settings files and verification code. Meta-Harness Evolving builds one offline from the student’s failures. At inference the frozen student runs the frozen harness, with no Builder in the loop.

Building a harness and running it. (a) Offline, the Builder turns construction failures and traces into at most four candidate edits per generation; checks and gates promote the best one or keep the incumbent. (b) At inference the frozen student makes four calls, at most six with retries, around deterministic verification and a fixed controller. Click to enlarge.

At inference: four student calls around fixed code

  1. Observe extracts facial, bodily, scene and temporal cues from the video at 2 fps. Code indexes them as v1, v2, … next to transcript sentences and prosody phrases, a1, a2, …; later calls see only this text record.
  2. Propose lists candidate labels with confidences and the IDs of supporting and contradicting evidence. Code checks every ID against the record, then ranks the candidates.
  3. Review re-examines the ranked candidates and is verified the same way. A fixed controller then picks at most three labels from the allowed vocabulary.
  4. Explain writes an explanation that cannot change the labels.

Audio reaches the student only as text: Whisper large-v3 transcripts and openSMILE eGeMAPSv02 features turned into descriptions such as “high pitch”, never into emotion labels.

Offline: Meta-Harness Evolving

Over four generations, the Builder (GPT-5.6 Luna or GPT-6.1 Sol) reads the incumbent’s failures on a construction split, clustered by gold–prediction confusion and by cross-modal conflict, together with execution traces. It proposes up to four candidate harnesses per generation, each aimed at one failure cluster.

Candidates pass file-scope, schema, loadability and static-code checks and a proposal reviewer that rejects shortcuts such as forcing a label. Survivors run on a separate validation split, and one replaces the incumbent only if it clears every gate:

Recognition
higher than the incumbent’s
Regression
loses at most 5% of validation items already solved
Mean calls
at most 4.5 student calls per item
Reasoning proxy
within 0.02 of the incumbent

The Builder sees neither validation scores nor gate decisions, only the resulting harness. If no candidate is ever promoted, the evolved harness is the scaffold itself.

The Builder may edit

  • Prompts for the four calls
  • Settings, such as how the transcript is presented
  • Restricted verification code

Fixed throughout

  • The student’s weights
  • Perception tools and their cached outputs
  • Evidence and output schemas, and the output controller

Results on MME-Emotion

Four students, served with vLLM at temperature zero: Qwen3-VL-8B, Qwen3-VL-4B, Qwen3.5-4B and Gemma-4-E4B. Evaluation covers 5,507 items, 2,963 in the three task types seen during construction and 2,544 in five held-out types. Recognition is scored locally against gold labels, without an LLM judge.

Task-macro Recognition (%)

Show every task typeHide task types
Recognition (%) per task type for the selected student. Bold marks the unique best value in each column.

Seen types: ER-Lab, ER-Wild (lab and in-the-wild emotion recognition) and SA (sentiment analysis). Held-out types: Noise-ER, FG-ER, ML-ER (noisy, fine-grained, multi-label emotion recognition), FG-SA (fine-grained sentiment) and IR (intent recognition). FG-ER and ML-ER use recall-like Recognition. † No candidate passed the gates, so the harness equals Shared.

Added evidence does most of the work

Transcript and prosody text give every student its largest gain. Audio − Video ranges from +5.38 (Qwen3.5-4B) to +10.71 points (Gemma-4-E4B), and every interval excludes zero.

The scaffold helps two students

Shared − Audio is +2.23 for Qwen3.5-4B and +2.54 for Gemma-4-E4B, partly because the scaffold predicts more labels on multi-label tasks, which recall-like scoring rewards. For the Qwen3-VL students it is +1.32 and −0.79, with intervals that include zero.

Evolution adds little on top

MHE − Shared ranges from −0.07 to +0.95 points. The largest, GPT-5.6 Luna on Qwen3-VL-4B (+0.95, 95% CI [+0.12, +1.82]), does not survive Holm correction (adjusted p = .192), and neither does any other contrast.

Transfer to held-out task types

With one exception, held-out changes are as small as seen-type changes. MHE − Shared ranges from −0.16 to +0.47 points on seen types and, for seven of the eight student–Builder pairs, from −0.14 to +0.28 on held-out types.

The exception is again GPT-5.6 Luna on Qwen3-VL-4B: +1.24 on held-out types, lifting held-out Recognition to 36.65, still below single-call Audio at 37.55. With seen-type gains this small, the held-out results say little about transfer beyond this pair.

Recognition paths per student. (a) Seen versus held-out macro Recognition, from Video through Audio to Shared. (b) MHE − Shared, zoomed ten times; violet: GPT-5.6 Luna, blue: GPT-6.1 Sol.

What the evaluation clips look like

Eight MME-Emotion evaluation items, most two to three seconds long. Each asks for the most prominent emotion, sentiment or intent from a fixed label set; play with sound, since the spoken words often decide the answer.

ER-Labseen2.1 s

Reference Excited, one of 9 labels

  • Anger
  • Sadness
  • Surprise
  • Happiness
  • Excited
  • Fear
  • Frustration
  • Neutral
  • Other

Seen types were used during harness construction; held-out types were not. Clips come from the public MME-Emotion benchmark and are shown as distributed.

What an evolved edit changes

Two Gemma-4-E4B items, one where an edit evolved by GPT-5.6 Luna fixes the scaffold’s error and one where it does not. Over all 5,507 items, this harness changes Shared’s label set on 169 items: 46 improve, 45 worsen and 78 score the same. Cases are selected, not representative.

Shared rates a mildly negative remark as strong negative because a confidence score, not an observed cue, ranks that label first. With the evolved edit, the review call treats the composed posture as evidence for a neutral reading, and the harness answers the reference, negative.

Each card gives one condition’s answer (green: correct, red: wrong); lettered boxes mark the regions the cards cite; amber notes are audio-derived. The edit panel quotes the rule GPT-5.6 Luna added to the review prompt.
Show two more casesHide the extra cases
(a) Qwen3-VL-4B, fine-grained sentiment: the GPT-5.6 Luna harness reaches the reference negative; GPT-6.1 Sol’s does not. (b) Qwen3-VL-8B, lab emotion recognition: the correct Excited fits the evolved ranking code, while the written explanation favors Happiness.

Emotion context on IntentQA

Each evolved harness writes one emotion context per video, with the predicted emotion, alternatives and a four-step analysis, without seeing any question. The bare student then answers 2,134 IntentQA test questions on 567 videos from the frames, the frames plus transcript and prosody, or those inputs plus the emotion context.

Question-answering accuracy (%). Bold: best per student.
InputVL-8BVL-4BQ3.5Gemma
Frames70.9967.4879.9057.31
Audio72.6368.4277.8460.59
+ emotion, Luna73.1070.2976.34†61.62
+ emotion, Sol72.8769.9276.4361.95

VL-8B and VL-4B: Qwen3-VL-8B and 4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B. Luna: GPT-5.6 Luna; Sol: GPT-6.1 Sol. † Context written by the retained Shared scaffold.

Adding the emotion context raises accuracy in all six pairs for Qwen3-VL-8B, Qwen3-VL-4B and Gemma-4-E4B, by +0.23 to +1.87 points. Qwen3.5-4B loses 1.50 and 1.41 points; it also loses 2.06 points when only transcript and prosody are added.

None of the eight contrasts survives Holm correction (smallest adjusted p = .085). There is no length-matched control, so longer inputs rather than emotion content could explain these changes.

Change from adding emotion context, by question type. Bars: mean over 4 students and 2 Builders; dots: the 8 individual changes. The mean is positive for every type; none survives Holm correction.

Why evolved gains stay small

The gates promoted six GPT-5.6 Luna edits and five GPT-6.1 Sol edits across the eight runs. What the analysis points to:

Validation gains shrink on evaluation

Promoted edits raised validation Recognition by up to 2.54 points. Scored the same way on the 2,963 seen-type evaluation items, the seven promoted harnesses change by −0.20 to +0.30. Each promotion picks the best of up to four candidates on 394 items, and three of the eleven rest on a single validation item.

The regression gate blocks broad edits

All twelve candidates that gained at least eight validation items (2.03 points) exceeded the 5% regression budget. The gate protects solved cases at the cost of broad edits; whether a looser gate generalizes better needs a separate test set.

Most evolved harnesses are conservative

Apart from GPT-5.6 Luna on Qwen3-VL-4B, which changes the label set on 21.6% of evaluation items, each evolved harness changes it on fewer than 4%.

Edits are instructions the student may not follow

In the failure case, the rule written for the Neutral-to-Frustration confusion is ignored. The student’s instruction following, not only the Builder’s proposals, can limit what edits achieve. Future gates could check that traces apply edits and that citations match the media.

Trajectories on the validation set. Filled markers were promoted, open ones rejected by the gates; crosses were screened out before scoring.
Validation gains overstate evaluation gains. Each evolved harness’s gain over Shared on validation (394 items) and on evaluation (5,507 items).

Citation

@misc{li2026emoharness,
  title         = {EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving},
  author        = {Li, Fenghai and Li, Bingxuan and Zhang, Jiawen and Shen, Yifan and Liu, Jiateng and Zhang, Yutong and Guo, Yue},
  year          = {2026},
  eprint        = {2611.11994},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2611.11994}
}

Open in new tab