The 3D Selfie Arrives, Make One of Your Own

Technology at a very early stage

The technique is far from perfect but this is the breakthrough computer scientists have been looking for.

It has been developed using a Convolutional Neural Network (CNN) – an area of artificial intelligence (AI) which uses machine learning to give computers the ability to learn without being explicitly programmed.

The research team, supervised by Dr Yorgos Tzimiropoulos, trained a CNN on a huge dataset of 2D pictures and 3D facial models. With all this information their CNN is able to reconstruct 3D facial geometry from a single 2D image. It can also take a good guess at the non-visible parts of the face.

Simple idea complex problem

Dr Tzimiropoulos said: “The main novelty is in the simplicity of our approach which bypasses the complex pipelines typically used by other tecniques. We instead came up with the idea of training a big neural network on 80,000 faces to directly learn to output the 3D facial geometry from a single 2D image.”

This is a problem of extraordinary difficulty. Current systems require multiple facial images and face several challenges, such as dense correspondences across large facial poses, expressions and non-uniform illumination.

Aaron Jackson said: “Our CNN uses just a single 2D facial image, and works for arbitrary facial poses (for instance front or profile images) and facial expressions (for instance smiling).”

Adrian Bulat said “The method can be used to reconstruct the whole 3D facial geometry including the non-visible parts of the face.”

Their technique demonstrates some of the advances possible through deep learning – a form of machine learning that uses artificial neural networks to mimic the way the brain makes connections between pieces of information.

Dr Vasileios Argyriou, from Kingston University’s Faculty of Science, Engineering and Computing, said: “What’s really impressive about this technique is how it has made the process of creating a 3D facial model so simple.”

What could the applications be?

Aside from the more standard applications, such as face and emotion recognition, this technology could be used to personalise computer games, improve augmented reality, and let people try on online accessories such as glasses.

It could also have medical applications – such as simulating the results of plastic surgery or helping to understand medical conditions such as autism and depression.

Their video example can be seen here.