An avatar that
keeps learning.
Generate an avatar’s speaking and listening motion from user face video and both voices, while learning from the conversation as it unfolds.
EvolvingAvatar
Interactive 3D Head Generation That Adapts as Conversations Unfold
01 The task
Speaking and listening,
in one interaction.
Psychological support, tutoring, and companionship all involve more than spoken words. An interactive avatar also needs head motion that follows the exchange.
Psychological support
An illustrative psychological support dialogue alongside a human clinician.
UserToday felt overwhelming.
AvatarI’m here. Take your time.
Illustrative exchange · Avatar speech is supplied by an external dialogue system.
Motion focusCoordinate a listener’s expression and neck motion with the speaker’s pauses.
Interactive 3D head generation
Give the conversation a visible response.
Generate the avatar’s head motion during both speaking and listening, in coordination with the ongoing exchange.
Input
The user’s face video and the audio of both the user and avatar, using only information that has arrived.
Output
Coordinated expression, neck, and jaw motion for the avatar. A separate dialogue system provides its speech.
Coordinated motion depends on context that changes throughout a conversation.
Why adapt02 Why adapt
From responding to context
to learning from it.
A fixed-parameter generator can respond to changing inputs, yet it does not learn from the interaction. We use test-time training (TTT) to adapt from the context arriving within each conversation.
Current context
Start from what has arrived.
Both generators can respond to current video and audio. EvolvingAvatar also begins adapting from this first context, without waiting for a long conversation history.
Current context · fixed parameters
Current context · adaptation begins
Responding to new input and learning from it are different capabilities.
EvolvingAvatar turns this opportunity into an online learning process.
Method03 The method
Observe. Learn.
Generate the response.
EvolvingAvatar fuses arriving audiovisual context, learns from its relationships, and uses the adapted context to generate coordinated head motion.
Arriving context, available now.
See the partner. Hear both voices.
User face video and dyadic audio are fused into a causal conversational context. The generator observes only what has arrived so far.
Learning happens inside the conversation. Persistent adaptation carries context forward, while transient jaw adaptation responds to current articulation. Target motion labels are not required at deployment.
Evaluating this behavior requires aligned conversational data and tests beyond familiar recordings.
Benchmark04 InterHead-Bench
Two recording formats.
One shared benchmark.
InterHead-Bench combines single-view and dual-view conversation videos into aligned multimodal data. It supports evaluation during speaking and listening, from familiar recording conditions to a new domain.
One view, two participants
Recover each side of the conversation.
A shared recording requires identifying participants and associating speech with the corresponding face. The framework aligns these signals into a common conversational timeline.
Three evaluation settings distinguish overlap with training from unfamiliar participants and recording domains.
Held-out samples from the training recording domain. Participants or recorded interactions may overlap with training.
Participants and interactions are held out from training, within the same recording domain.
Conversations from a different recording domain test a broader distribution shift.
The following comparisons use these three evaluation settings.
Examples05 Motion and perception
Same conversation.
Different responses.
Compare synchronized motion from each method across ID, OOD, and OOD-Hard. Human judgments complement the selected examples below.
A conversation, several perspectives.
Curated examples from our case studies.
Synchronized comparisons · Same conversation
Selected qualitative examples, not an aggregate evaluation.
The human perspective
How does the motion
feel in conversation?
Human evaluation asks which generated motion looks more realistic in conversation, complementing the visual comparisons above.
Preference for EvolvingAvatar over UniLS
Human preferences on the evaluated examples. These comparisons do not by themselves isolate the effect of test-time training.
Conversation-level statistics complement what we see in individual examples.
Results06 Across the conversation
Follow the change
across the conversation.
Measure how closely generated expression interactions match recorded ones across five equal conversation intervals. Switch distributions to examine the change relative to each method’s starting point.
Expression interaction distance · lower is better
Each method is normalized to its own first interval (100). Lower values indicate a closer match to recorded user–avatar expression statistics.
less mismatch on OOD-Hard
Improvement varies over the conversation. This is the best later interval, not a claim of monotonic progress.
The paper details the model, evaluation protocol, and full experimental results.
PaperAbstract
Interactive 3D head generation requires coordinated speaking and listening motion that responds to a conversational partner. EvolvingAvatar uses self-supervised test-time training to learn from user face video and dyadic audio as conversations unfold. Dyadic Context Prediction supplies an adaptation objective without target motion labels at deployment. Persistent adaptation learns within the conversation, while transient jaw adaptation responds to current articulation.
Build on this work
Citation