Data generation and augmentation for improved sign language recognition

Thesis proposal for students at:

1. Saarland University, Saarbrücken, Germany.

2. Technische Universität Berlin, Germany

Published: 27.05.2026

Proposed by the SCAAI Group (Social, Cognitive and Affective AI, https://scaai.dfki.de) at the German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany

## Introduction/Problem

Sign Language translation is much harder than text translation.

Translating sign language into text includes a computer vision process: analyzing a video stream to convert it into a sequence of characters.

In addition, sign language is a low-resource language. The amount of available parallel corpora (text sentences with the corresponding sign language videos) is only a tiny fraction of the size of corpora.

To mitigate resource scarcity, one possible approach is to rely on synthetic data generation and data augmentation.

To generate data, one could rely on the animation of 3D avatars, for example by using our internal sign language synthesis tool, the MMS-Player (Nunnari et al., 2025), available at https://github.com/DFKI-SignLanguage/MMS-Player.

To augment such data, it would be possible to perturb the reference generation through the alteration of environmental parameters (background, light, camera position) or avatar visual features (body sizes and proportions, skin tone, clothing), and avatar motion (speeds and acceleration, body relocations).

This approach has been preliminarily investigated by Barberà et al. (2026).

## Goals

The goals of this thesis are to:

* Investigate data synthesis and augmentation techniques for sign language recognition.

* Take control of the MMS-Player and use it for data synthesis.

* Implement augmentation techniques.

* Reproduce an existing sign-to-text translation pipeline and test it in several augmentation conditions.

In addition to the written manuscript, the thesis must produce a demonstrable working system, released as open-source software in the “DFKI Team on Sign Language” public repository: https://github.com/DFKI-SignLanguage.

Satisfactory results may lead to a scientific publication.

## Requirements

* Passion for 3D graphics and human characters animation.

* Strong Python programming skills.

* Software engineering skills for organizing large code repositories into modules, classes, and scripts, not only plain sequential notebooks.

* Experience with the Blender 3D editor and APIs (https://www.blender.org).

* Proactive and propositive attitude.

* Knowledge of any sign language is not required but would be considered an asset.

* For Uni Saarland students, the thesis will follow the guidelines of the UMTL department: https://umtl.cs.uni-saarland.de/teaching/thesis.html

## Contacts

If interested, send a CV, transcript of your grades, and possibly links to your selected existing open-source software repositories to Fabrizio Nunnari <fabrizio.nunnari@dfki.de>

## References

Nunnari, F., Mishra, S. and Gebhard, P. (2025) “MMS Player: an open source software for parametric data-driven animation of Sign Language avatars,” Adjunct Proceedings of the 25th ACM International Conference on Intelligent Virtual Agents. 9th Workshop on Sign Language Translation and Avatar Technology, Berlin Germany: ACM, pp. 1–8. Available at: https://doi.org/10.1145/3742886.3756710.

Barberà, G. et al. (2026) “Capturing Methodology for Generating Synthetic and 3D Training Data in Catalan Sign Language (LSC): The Case of Verbal Agreement,” 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion. LREC, ELRA, pp. 1–9. Available at: https://www.sign-lang.uni-hamburg.de/lrec/pub/26041.pdf.