AME.LAB — assistive-technology lab · iCOM
A 3D avatar that signs in LIBRAS, Brazilian sign language, for deaf users — in the browser and in VR — and a camera that reads the signs back.
The problem
LIBRAS is an official language in Brazil (Law 10,436/2002, Decree 5,626/2005), and services are required to communicate in it. In practice, a deaf person at a kiosk, a help desk or a hospital front desk rarely finds an interpreter. Text alone doesn't solve it: for many deaf people Portuguese is a second language. The answer had to sign — and understand signs.
What the house made
An avatar that signs in LIBRAS, on three screens: the browser, with captions under the avatar ("bom dia", "médico, por favor", "preciso, urgente"); a photoreal scene in Unreal; and virtual reality on Meta Quest, where the user signs with their own hands, no controllers. The other way round, a plain webcam reads the person signing and writes the words and numbers in Portuguese.
How it was built
Signing is recorded from interpreters: MediaPipe reads hands, arms and face in the interpreter's video, and those points drive the avatar's rig — in Blender to clean up, in Unreal to play. Recognition uses the same landmarks from the webcam and a classifier over the sequence of movement. Instead of translating "anything", the vocabulary is built per domain — like a visual FAQ for a bus terminal or a clinic — which keeps it small and precise. Latency, accuracy and vocabulary size [to measure].
Screens
Credits
Development: Nilo River. With AME.LAB and iCOM. Pose: MediaPipe (Google). Avatars: MetaHuman (Epic Games) and Ready Player Me. LIBRAS interpreters, deaf-user testing and other credits [to measure]