GENEA Leaderboard Submission Page

Our official webpage: https://genea-workshop.github.io/leaderboard

What is GENEA Leaderboard?

GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Leaderboard is a leaderboard for benchmarking gesture generation models with human evaluation. The GENEA Leaderboard is the evolution of the GENEA challenge, which was held at the GENEA Workshop at ICMI 2020, 2021, 2022, 2023, and 2024.

Why we need GENEA Leaderboard?

We standardize the input, output and evaluation process of gesture generation models with human evaluation.

How we evalution in GENEA Leaderboard?

We provide a input to teams, and each team's model will generate gestures output in .npy format.

Team ATeam BTeam Z
Input 1
Input 1
Input 2
Input 2
Input 3
Input 3
Input 4
Input 4Model SAModel SBModel SZMotion SA_1Motion SA_2Motion SA_3Motion SA_4Motion SB_1Motion SB_2Motion SB_3Motion SB_4Motion SZ_1Motion SZ_2Motion SZ_3Motion SZ_4Ground TruthBaseline BA
$$\cdo...
Video SA_1Video SA_2Video SA_3Video SA_4Video SB_1Video SB_2Video SB_3Video SB_4Video SZ_1Video SZ_2Video SZ_3Video SZ_4Video BA_1Video BA_2Video BA_3Video BA_4Video NA_1Video NA_2Video NA_3Video NA_4Motion BA_1Motion BA_2Motion BA_3Motion BA_4Motion NA_1Motion NA_2Motion NA_3Motion NA_4

Then, we convert .npy to video .mp4 and recruit Prolific to start user studies. Using pairwise comparison, we evaluate the generated gestures with human evaluation

Video SA_1Video SA_2Video SA_3Video SA_4Video SB_1Video SB_2Video SB_3Video SB_4Video SZ_1Video SZ_2Video SZ_3Video SZ_4Video BA_1Video BA_2Video BA_3Video BA_4Video NA_1Video NA_2Video NA_3Video NA_4Participant 1Participant 2Participant 3Participant 4ScreenSA_3 - SB_3ScreenNA_2 - SZ_2ScreenSA_4 - BA_4
emotion mismatching studies
emotion mismatching studiesScreenSZ_3 - BA_3ScreenSA_4 - SZ_4ScreenSA_2 - NA_2
speech appropriateness studies
speech appropriateness studiesScreenSA_1 - BA_1ScreenSA_2 - SB_2ScreenSC_3 - NA_3
pairwise emotion studies
pairwise emotion studiesScreenBA_4 - SB_4ScreenBA_2 - SZ_2ScreenNA_1 - BA_1
pairwise human-likeness studies
pairwise human-likeness studiesResult Study 1Result Study 2Result Study 3Result Study n
Score SAScore SBScore SZ

Announcement

  • The GENEA team is preparing an online leaderboard for benchmarking gesture generation models with human evaluation. This project is the evolution of the GENEA challenge, so stay tuned!

  • If you would like to stay updated on major developments in theleaderboard project, sign up with your e-mail address on this link!

Workshop

workshop

Workshop
Official GENEA Workshop at ICMI 2024

Links to previous years


avatar

Follow us on Twitter

For updates, follow us: GENEA Workshop