Wētā Workshop Unleashed exhibits face-tracking skull

Normally when the eyes of an artwork follow you around the room, we chalk it up to an optical illusion. Not this time! Our Senior Programmer, James Brown has invented an animatronic skull, driven by face tracking software and visitors to Wētā Workshop Unleashed in Auckland now have an opportunity to experience the technology for themselves.

The skull is part of the horror space which takes guests behind the scenes of fictional creature feature – Fauna.

When guests stand in front of the skull they are in the eyeline of a small camera. The camera records a guest's movements and sends the information to a small computer.

In the backend, a computer vision library tracks the features of the guest's face such as their mouth and the corners of their eyes. James’ software then translates these points into the servos (small motors inside the skull). This all happens in the blink of an eye so that they skull mimics each facial expression back to the guest in real time. "It’s live puppetry," says James.

This project was inspired by Wētā Workshop’s previous work with facial and hand tracking software developed for our mixed reality experiences, Dr. Grordbort's Invaders and Dr. Grordbort's Boosters. Building upon the groundwork laid down by James and the team for these projects, James has now built facial tracking for physical objects.

For James, this project started as a hobby horse during his free hours in New Zealand’s March 2020 COVID-19 lockdown. He began the experimentation process with wires, microchips and Lego Technic.

James continues, "It was also handy for using different 3D printers which can vary in their output. As each iteration of the skull changed, I could always rely on the standardised measurements of my Lego Technic."

Once back in the office, Wētā Workshop Co-founder and Creative Lead Richard Taylor, saw the skull and asked James if he would like it showcased it in an upcoming exhibition. The skull began to take shape once again. The actual skull, including the moving parts, such as the jaw and eyes are 3D printed. Wētā Workshop also helped source some high-end servos, enabling the skull to reliably run all day long.

Wētā Workshop’s Interactive division sets aside time each Friday afternoon to share their work and hobbies with one another before heading off to enjoy the weekend. One Friday, James brought his invention in to show the team and had a chat with our Art Director, Steve Lambert. Steve’s background in animation led to greater detail in the skull’s jaw movements, which can now move from side to side as you move your own jaw. Ideally in the future, James would add skin, lips and cameras behind the eyes, giving the skull greater nuance and detail.

James Brown joined the Wētā Workshop’s Interactive division in 2018 with a background in digital interactive and a Master of Electrical & Information Sciences from Cambridge University. Our interactive projects span digital games, immersive experiences, and far beyond. If you would like to get in touch about a collaborative project, please contact us.