AI and machine learning read the geometry of the game world in real time and turn it into spatial audio: objects, walls, openings become sound you can navigate by. Shipped open-source and free.
technique
3D audio, real-time geometry reading, ML, open-source plugin
period
2 years of R&D with blind players and AbleGamers → open-source 2024-03
proof
20+ awards: Spikes Asia Grand Prix · D&AD Pencil · One Asia Silver · One Club Merit · One Show / Clio · Cannes Lions 2024 shortlist
EEG and biosensors reading each participant live at Itaú's stand at Web Summit Rio 2024; at the end, a QR code delivered the video generated from each person's brainwaves.
technique
EEG, biosensors, Python, WebView, WebGL shaders, custom Linux on mini PCs
A LIBRAS avatar on three screens: the browser, with captions; a photoreal scene in Unreal; and virtual reality on Meta Quest, with hand tracking. The other way round, the webcam recognises signs and numbers and writes them in Portuguese.
technique
computer vision (MediaPipe), sign classification, interpreter video → avatar, Blender, Unreal Engine (MetaHuman), Meta Quest with hand tracking, Ready Player Me
The student's webcam becomes a 3D avatar that does the exercise with them — and counts the reps.
project
Hiveclass · physical-education avatar
client
Hiveclass
delivered
A 3D avatar driven by the student's pose, inside the platform's task: video explanation, calibration (whole body in frame, right hand up to start) and a demonstration with a rep counter and live feedback. Before that, an MVP comparing the student's skeleton with the correct form, plus student and teacher portals.
technique
MediaPipe (33 landmarks), landmark → rig solver, IK, three.js / React Three Fiber, Ready Player Me, Next.js, socket.io
period
prototype 2025-05 → validation inside the platform 2026-05
proof
16 jumping jacks counted correctly in the 2025-05-27 test · 10/10 in the in-platform validation
AXIS hardware, firmware and protocol: dual-core ESP32, IMU with sensor fusion (Madgwick, quaternions), a Velostat pressure matrix under the griptape, fire-and-forget UDP telemetry and an on-board neural network that recognises the trick. Outside, the Gappin dashboard reads the live stream.
Computer vision, 3D pose, biomechanics, custom ML, and AXIS: the smart skateboard with sensors in the deck and its own firmware. A dashboard with a global, real-time panorama of skateboarding. 55k entities.
technique
computer vision, 3D pose, ML, IoT (AXIS), firmware, data visualization
An augmented-reality app: in the invitation, the mascot appears and talks to the guest. At the party, employees dropped the mascot onto the floor at the backdrop and took a photo or video with the app's own capture function, which shared straight to social media with the company frame.
technique
in-app augmented reality, 3D mascot placed on a flat surface, in-app photo and video, company frame, social sharing
A camera watches a breaker, recognises the move, and a brand avatar dances along, with a live score.
project
Roda 220v · breaking read by a camera
client
personal, non-commercial project · pilot in Cute Crew's brand identity
delivered
A breaking installation: the camera reads the pose, a classifier tells toprock, footwork and power move apart, and a 3D brand b-girl performs the move on screen, with score, combo and freeze-on-the-beat. Solo, Versus, a leaderboard and gesture menus.
technique
RTMPose (ONNX), YOLOX, keypoint classifier, FastAPI + WebSocket, three.js with toon shading, mocap retargeting in Blender, librosa
period
pilot in 2026-10 (first commit 2026-10-06)
proof
RTMPose-m: 65% PCK@0.1 vs 44% for MediaPipe full on BRACE, at 11.7 ms per frame · classifier: 92% per segment · live: matches ground truth 71% of the time
A mobile beat'em up where Indigenous guardians drive out invaders — and every stage carries the fact, the date and the source.
project
GUARDIÕES
client
personal project · non-commercial parody
delivered
A pixel-art browser game built to open inside Instagram: 4 playable guardians, 4 stages on Indigenous lands, bosses, speedboat, motorbike and mine-cart chases, and cards with dated, sourced facts. Nobody dies: the invader gets hit, drops what he brought and runs.
technique
dependency-free JavaScript, hard-pixel 360×640 canvas, art generated in Gemini (Vertex AI) and cleaned by in-house scripts, WebAudio, autopilot, trailer rendered frame by frame, Cloudflare Pages + D1
