GFP-GAN: Towards Real-World Blind Face Restoration with Generative Facial Prior
GFPGAN aims at developing a Practical Algorithm for Real-world Face Restoration.
It leverages rich and diverse priors encapsulated in a pre-trained face GAN (e.g., StyleGAN2) for blind face restoration.
The architecture of the GFP-GAN Framework
The Architecture consists of a degradation removal module (U-Net) and a pretrained face GAN as a facial feature recognizer. They both are inter-connected by a latent code mapping technique and several Channel-Split Spatial Feature Transform (CS-SFT) layers. GFP-GAN consists of a degradation removal module called U-Net and a pre-trained face GAN (such as StyleGAN2). Specifically, the degradation removal module is designed to remove the complicated degradation in the input image and extract two kinds of features:
- latent features Flatent to map the input image to the closest latent code in StyleGAN2
- multi-resolution spatial features for modulating the StyleGAN2 features.
During the model training, it emphasizes the following :
- Intermediate restoration losses to remove complex degradation
- Facial component loss with discriminators to enhance facial details.
- Identity preserving loss to retain face identity.
Similar to perceptual loss, the identity preserving loss feature is based on the feature embedding of an input face. It comes included with the pre-trained face recognition ArcFace model, which captures the most prominent features for identity discrimination in the input image.
GFP-GAN comes pre-trained on the FFHQ dataset, which consists of around 70 000 high-quality images. All the images were resized to 5122 during training. GFP-GAN is trained on synthetic data that approximates real low-quality images and generalizes to real-world images during the inference output synthesis.
A Comparison Of GFP-GAN To Other Models
As it is noticed, the GFP-GAN model for Blind Face Restoration retains the Image Quality and restores the Facial features present in the image better than the traditionally used models present.
Getting Started
This article will try to implement a Face Restoration Model using the Generative Facial Prior model and restore degraded images that contain noise and blur. The creators of GFP-GAN inspire the following implementation, and the link to their official repository can be found here.
Dependencies and Installation
- Python >= 3.7 (Recommend to use Anaconda or Miniconda)
- PyTorch >= 1.7
- Option: NVIDIA GPU + CUDA
- Option: Linux
Installation
We now provide a clean version of GFPGAN, which does not require customized CUDA extensions.
If you won't want to use the original model in our paper, please see PaperModel.MD for installation.
- Clone repo
git clone https://github.com/TencentARC/GFPGAN.git
cd GFPGAN2. Install dependent packages
# Install basicsr - https://github.com/xinntao/BasicSR
# We use BasicSR for both training and inference
pip install basicsr# Install facexlib - https://github.com/xinntao/facexlib
# We use face detection and face restoration helper in the facexlib package
pip install facexlibpip install -r requirements.txt
python setup.py develop# If you want to enhance the background (non-face) regions with Real-ESRGAN,
# you also need to install the realesrgan package
pip install realesrgan
Quick Inference
Download pre-trained models: GFPGANCleanv1-NoCE-C2.pth
wget https://github.com/TencentARC/GFPGAN/releases/download/v0.2.0/GFPGANCleanv1-NoCE-C2.pth -P experiments/pretrained_modelsInference!
python inference_gfpgan.py --upscale 2 --test_path inputs/whole_imgs --save_root results
If you won't want to use the original model in our paper, please see PaperModel.MD for installation and inference.
Model Zoo
- GFPGANCleanv1-NoCE-C2.pth: No colorization; no CUDA extensions are required. It is still in training. Trained with more data with pre-processing.
- GFPGANv1.pth: The paper model, with colorization.
You can find more models (such as the discriminators) here: [Google Drive], OR [Tencent Cloud 腾讯微云]
Training
We provide the training codes for GFPGAN (used in our paper).
You could improve it according to your own needs.
Tips
- More high-quality faces can improve the restoration quality.
- You may need to perform some pre-processing, such as beauty makeup.
Procedures
(You can try a simple version ( options/train_gfpgan_v1_simple.yml) that does not require face component landmarks.)
- Dataset preparation: FFHQ
2. Download pre-trained models and other data. Put them in the experiments/pretrained_models folder.
3. Pretrained StyleGAN2 model: StyleGAN2_512_Cmul1_FFHQ_B12G4_scratch_800k.pth
4. Component locations of FFHQ: FFHQ_eye_mouth_landmarks_512.pth
5. A simple ArcFace model: arcface_resnet18.pth
6. Modify the configuration file options/train_gfpgan_v1.yml accordingly.
7. Training
python -m torch.distributed.launch --nproc_per_node=4 --master_port=22021 gfpgan/train.py -opt options/train_gfpgan_v1.yml --launcher pytorchEndNotes
This article tried to understand how image restoration works, which is a vital part of the image processing phase. We also tried to implement an Image restoration model based on the Generative Facial Prior model and restored degraded images. I would recommend the reader to try the same on more complex degraded images to explore the further capabilities of the model.
The above implementation is available as a colab notebook, which can be accessed using the link here.
Happy Learning!
References
What Do You Think?
