Pose-Free Face Swapping Based on a Deformable 3D Shape Morphable Model

Yuan LIN; Shengjin WANG

doi:10.1587/transinf.E97.D.305

Pose-Free Face Swapping Based on a Deformable 3D Shape Morphable Model

Yuan LIN, Shengjin WANG

Full Text Views

0

Cite this

Summary :

Traditional face swapping technologies require that the faces of source images and target images have similar pose and appearance (usually frontal). For overcoming this limit in applications this paper presents a pose-free face swapping method based on personalized 3D face modeling. By using a deformable 3D shape morphable model, a photo-realistic 3D face is reconstructed from a single frontal view image. With the aid of the generated 3D face, a virtual source image of the person with the same pose as the target face can be rendered, which is used as a source image for face swapping. To solve the problem of illumination difference between the target face and the source face, a color transfer merging method is proposed. It outperforms the original color transfer method in dealing with the illumination gap problem. An experiment shows that the proposed face reconstruction method is fast and efficient. In addition, we have conducted experiments of face swapping in a variety of scenarios such as children's story book, role play, and face de-identification stripping facial information used for identification, and promising results have been obtained.

Publication: IEICE TRANSACTIONS on Information Vol.E97-D No.2 pp.305-314

Publication Date: 2014/02/01

Publicized

Online ISSN: 1745-1361

DOI: 10.1587/transinf.E97.D.305

Type of Manuscript: PAPER

Category: Computer Graphics

Authors

Yuan LIN
Tsinghua University
Shengjin WANG
Tsinghua University

Keyword

3D face reconstruction, face swapping, color transfer, multi-resolution spline, human-computer interaction

Cite this

Copy

Yuan LIN, Shengjin WANG, "Pose-Free Face Swapping Based on a Deformable 3D Shape Morphable Model" in IEICE TRANSACTIONS on Information, vol. E97-D, no. 2, pp. 305-314, February 2014, doi: 10.1587/transinf.E97.D.305.
Abstract: Traditional face swapping technologies require that the faces of source images and target images have similar pose and appearance (usually frontal). For overcoming this limit in applications this paper presents a pose-free face swapping method based on personalized 3D face modeling. By using a deformable 3D shape morphable model, a photo-realistic 3D face is reconstructed from a single frontal view image. With the aid of the generated 3D face, a virtual source image of the person with the same pose as the target face can be rendered, which is used as a source image for face swapping. To solve the problem of illumination difference between the target face and the source face, a color transfer merging method is proposed. It outperforms the original color transfer method in dealing with the illumination gap problem. An experiment shows that the proposed face reconstruction method is fast and efficient. In addition, we have conducted experiments of face swapping in a variety of scenarios such as children's story book, role play, and face de-identification stripping facial information used for identification, and promising results have been obtained.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.E97.D.305/_p

Copy

@ARTICLE{e97-d_2_305,
author={Yuan LIN, Shengjin WANG, },
journal={IEICE TRANSACTIONS on Information},
title={Pose-Free Face Swapping Based on a Deformable 3D Shape Morphable Model},
year={2014},
volume={E97-D},
number={2},
pages={305-314},
abstract={Traditional face swapping technologies require that the faces of source images and target images have similar pose and appearance (usually frontal). For overcoming this limit in applications this paper presents a pose-free face swapping method based on personalized 3D face modeling. By using a deformable 3D shape morphable model, a photo-realistic 3D face is reconstructed from a single frontal view image. With the aid of the generated 3D face, a virtual source image of the person with the same pose as the target face can be rendered, which is used as a source image for face swapping. To solve the problem of illumination difference between the target face and the source face, a color transfer merging method is proposed. It outperforms the original color transfer method in dealing with the illumination gap problem. An experiment shows that the proposed face reconstruction method is fast and efficient. In addition, we have conducted experiments of face swapping in a variety of scenarios such as children's story book, role play, and face de-identification stripping facial information used for identification, and promising results have been obtained.},
keywords={},
doi={10.1587/transinf.E97.D.305},
ISSN={1745-1361},
month={February},}

Copy

TY - JOUR
TI - Pose-Free Face Swapping Based on a Deformable 3D Shape Morphable Model
T2 - IEICE TRANSACTIONS on Information
SP - 305
EP - 314
AU - Yuan LIN
AU - Shengjin WANG
PY - 2014
DO - 10.1587/transinf.E97.D.305
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E97-D
IS - 2
JA - IEICE TRANSACTIONS on Information
Y1 - February 2014
AB - Traditional face swapping technologies require that the faces of source images and target images have similar pose and appearance (usually frontal). For overcoming this limit in applications this paper presents a pose-free face swapping method based on personalized 3D face modeling. By using a deformable 3D shape morphable model, a photo-realistic 3D face is reconstructed from a single frontal view image. With the aid of the generated 3D face, a virtual source image of the person with the same pose as the target face can be rendered, which is used as a source image for face swapping. To solve the problem of illumination difference between the target face and the source face, a color transfer merging method is proposed. It outperforms the original color transfer method in dealing with the illumination gap problem. An experiment shows that the proposed face reconstruction method is fast and efficient. In addition, we have conducted experiments of face swapping in a variety of scenarios such as children's story book, role play, and face de-identification stripping facial information used for identification, and promising results have been obtained.
ER -