* These author names are in alphabetical order due to equal contribution.

Frontalized faces. Top: Input photos; bottom: our frontalizations, obtained without estimating 3D facial shapes.

Abstract

“Frontalization” is the process of synthesizing frontal facing views of faces appearing in single unconstrained photos. Recent reports have suggested that this process may substantially boost the performance of face recognition systems. This, by transforming the challenging problem of recognizing faces viewed from unconstrained viewpoints to the easier problem of recognizing faces in constrained, forward facing poses. Previous frontalization methods did this by attempting to approximate 3D facial shapes for each query image. We observe that 3D face shape estimation from unconstrained photos may be a harder problem than frontalization and can potentially introduce facial misalignments. Instead, we explore the simpler approach of using a single, unmodified, 3D surface as an approximation to the shape of all input faces. We show that this leads to a straightforward, efficient and easy to implement method for frontalization. More importantly, it produces aesthetic new frontal views and is surprisingly effective when used for face recognition and gender estimation.
Camera ready
Extended abstract
CVPR poster
arXiv preprint PDF (early arXiv preprint arXiv:1411.7964, 28 Nov. 2014)
BibTeX

Downloads

1. Frontalization code: Our MATLAB implementation of the method described in the paper is available on the FTP in the file frontalization.0.1.3.zip. Please see the README.txt for details on how to install and run the code. Though our code is written entirely in MATLAB, it has a few dependencies, including the calib function and a facial feature detector. See the README.txt for further details. The entire frontalization project was independently ported to Python by Douglas Souza. This useful port is available directly from his github page. This port uses the DLIB facial feature detector. Thank you Douglas! Yuval Nirkin has provided a MATLAB wrapper for the DLIB face and facial landmark detector used with the frontalization code. Thank you, Yuval!

2. LFW3D: The entire LFW collection, frontalized and stored in the same directory structure as the original LFW, is available on the FTP in the file LFW3D.0.1.1.zip.

3. Adience3D: The entire Adience image collection, frontalized and stored in the same directory structure as the original Adience collection, is available on the FTP in the file Adience3D.0.1.1.zip. Images in the above sets were frontalized using the SDM facial feature detector (Intraface). Though we have found SDM to provide the best results, alternative detectors can easily be used with our code in its stead and we offer a number of other options. Please see documentation in the README.txt and the MATLAB functions themselves.

Note that we are continuing to improve the quality of the frontalized images, and may post updates to both these collections (hence the version numbers on both files).

What’s New

• 6-June-2017 Please see our followup project on face recognition, with more details on rendering and new Python code supporting more rendered views.

• 21-March-2016 To help run frontalization on MATLAB, Yuval Nirkin has provided a MATLAB MEX for detecting faces and facial landmarks using the DLIB library.

• 21-March-2016 Added a link to Python port of the frontalization project, contributed by Douglas Souza.

• 8-December-2015 Demo now allows for frontalization with any standard sparse (five-point) landmark detection method such as the one described in our recent paper here. Choosing this option in demo.m currently allows for manual five-point localization.

• 18-May-2015 frontalization.0.1.2 now includes example usage with the DLIB facial feature detector (based on Kazemi and Sullivan, CVPR’14). We have found it to provide excellent performance, and so offer it as an alternative to the commercial SDM detector and the older detector by Zhu and Ramanan.

Download Instructions

These resources are available on our FTP server: agas.openu.ac.il. Direct access to the download directory is available through http://www.cslab.openu.ac.il/download/.

For the FTP user name and password, please provide the details below. We ask for these details so we can keep in touch in case we find any need for a critical update or in case we would organize a dedicated workshop, etc. When done, you will immediately be able to see the FTP user name.

If you have problems accessing the files through the direct link above please use instead a dedicated FTP client, and connect to the server agas.openu.ac.il. Two free clients are FileZilla and WinSCP.

Other Related Papers / Projects / Code

• Iacopo Masi, Tal Hassner, Anh Tuan Tran, and Gerard Medioni, Rapid Synthesis of Massive Face Sets for Improved Face Recognition, IEEE International Conference on Automatic Face and Gesture Recognition (FG) Washington, DC, May, 2017 (PDF, project and code)

• Iacopo Masi, Anh Tuan Tran, Tal Hassner*, Jatuporn Toy Leksut and Gerard Medioni, Do We Really Need to Collect Millions of Faces for Effective Face Recognition? European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, Oct. 2016 (PDF, project and code)
  * Denotes equal contribution.

• Yue Wu, Tal Hassner, KangGeon Kim, Gerard Medioni and Prem Natarajan, Facial Landmark Detection with Tweaked Convolutional Neural Networks, arXiv preprint arXiv:1511.04031, 21 Mar. 2016 (PDF, project and code)
  * Denotes joint first authorship / equal contribution

• Eran Eidinger, Roee Enbar, and Tal Hassner, Age and Gender Estimation of Unfiltered Faces, Transactions on Information Forensics and Security (IEEE-TIFS), special issue on Facial Biometrics in the Wild, Volume 9, Issue 12, pages 2170 - 2179, Dec. 2014 (Project)

• Tal Hassner, Liav Assif, and Lior Wolf, When Standard RANSAC is Not Enough: Cross-Media Visual Matching with Hypothesis Relevancy, Machine Vision and Applications (MVAP), Volume 25, Issue 4, Page 971-983, 2014 (Project, Springer)

• Tal Hassner, Viewing Real-World Faces in 3D, International Conference on Computer Vision (ICCV), Sydney, Austraila, Dec. 2013 (Code, project, PDF)

Copyright and disclaimer
Copyright 2014, Tal Hassner