Augmentation and Cutout in Deepfake Detection: A Comparative Study of Accuracy, Calibration, and Attention
This study compares nine ways to train EfficientNet-B4, an image recognition model, to detect AI-altered faces in FaceForensics++, a public collection of real and manipulated face videos. It evaluates each configuration with AUC, F1, Brier score and log loss; the four measures favor different configurations. AUC rates separation from 0.5 for chance to 1 for perfect separation. The Brier score measures probability error. Grad-CAM heat maps show which facial regions influenced each prediction. On FaceForensics++, a public collection of real and altered face videos, the image model EfficientNet-B4 reaches 0.897 AUC when trained with varied and partly blacked-out images, versus 0.868 for EfficientNet-B4 trained without them.
Deepfake detectors are usually trained with a fixed recipe, and explanations, when present, come afterwards. We asked whether two early choices in that recipe, data augmentation and cutout (masking patches of training faces), change how accurate a detector is, how trustworthy its confidence is, and where it looks.
We built a detector around an EfficientNet-B4 network, trained nine configurations on paired real and manipulated videos from FaceForensics++, and scored each with four measures. AUC asks how well a model ranks fakes above real videos across all cutoffs; F1 balances the fakes caught against false alarms at the chosen cutoff; the Brier score and log loss both measure calibration, how closely a stated confidence matches how often the model is right. Grad-CAM, which highlights the parts of an image a network relied on, mapped to 68 facial landmarks, then showed which regions drove each decision.
Augmentation plus a black-filled cutout gave the strongest overall detection, yet the four measures disagreed on a single winner, and the region maps showed the strongest model leaning on the nose whether it was right or wrong. Preprocessing shapes both how well a detector ranks videos and how far its verdicts can be trusted.
- Augmentation plus a black-filled cutout gave the strongest overall detection: an AUC of 0.8971, an F1-score of 0.8429 and the best Brier score, 0.1242.
- The four measures disagreed: the random-filled combination had the lowest log loss, 0.4656 against 0.4710, so ranking and calibration named different winners.
- Grad-CAM showed the strongest model leaning on the nose: in the fakes it missed, nose activation rose from 49% at the baseline to 78%.
Citation
Citation Copy citation (BibTeX)
@inproceedings{kaya2026augmentation,
author = {Kaya, Mert and Adanova, Venera},
title = {Augmentation and Cutout in Deepfake Detection: A Comparative Study of Accuracy, Calibration, and Attention},
booktitle = {2026 11th International Conference on Computer Science and Engineering (UBMK)},
pages = {78--83},
year = {2026},
publisher = {IEEE},
doi = {10.1109/UBMK71847.2026.11745298},
url = {https://ieeexplore.ieee.org/document/11745298}
}