Eschatia Labs labs
Back to research

Augmentation and Cutout in Deepfake Detection: A Comparative Study of Accuracy, Calibration, and Attention

This study compares nine ways to train EfficientNet-B4, an image recognition model, to detect AI-altered faces in FaceForensics++, a public collection of real and manipulated face videos. It evaluates each configuration with AUC, F1, Brier score and log loss; the four measures favor different configurations. AUC rates separation from 0.5 for chance to 1 for perfect separation. The Brier score measures probability error. Grad-CAM heat maps show which facial regions influenced each prediction. On FaceForensics++, a public collection of real and altered face videos, the image model EfficientNet-B4 reaches 0.897 AUC when trained with varied and partly blacked-out images, versus 0.868 for EfficientNet-B4 trained without them.

Presented at UBMK 2026 in Istanbul and published in IEEE Xplore.

Venue
UBMK 2026, 11th International Conference on Computer Science and Engineering, Istanbul (IEEE)

Deepfake detectors are usually trained with a fixed recipe, and explanations, when present, come afterwards. We asked whether two early choices in that recipe, data augmentation and cutout (masking patches of training faces), change how accurate a detector is, how trustworthy its confidence is, and where it looks.

We built a detector around an EfficientNet-B4 network, trained nine configurations on paired real and manipulated videos from FaceForensics++, and scored each with four measures. AUC asks how well a model ranks fakes above real videos across all cutoffs; F1 balances the fakes caught against false alarms at the chosen cutoff; the Brier score and log loss both measure calibration, how closely a stated confidence matches how often the model is right. Grad-CAM, which highlights the parts of an image a network relied on, mapped to 68 facial landmarks, then showed which regions drove each decision.

Augmentation plus a black-filled cutout gave the strongest overall detection, yet the four measures disagreed on a single winner, and the region maps showed the strongest model leaning on the nose whether it was right or wrong. Preprocessing shapes both how well a detector ranks videos and how far its verdicts can be trusted.

  • Augmentation plus a black-filled cutout gave the strongest overall detection: an AUC of 0.8971, an F1-score of 0.8429 and the best Brier score, 0.1242.
  • The four measures disagreed: the random-filled combination had the lowest log loss, 0.4656 against 0.4710, so ranking and calibration named different winners.
  • Grad-CAM showed the strongest model leaning on the nose: in the fakes it missed, nose activation rose from 49% at the baseline to 78%.

Citation

Citation Copy citation (BibTeX)
@inproceedings{kaya2026augmentation,
  author    = {Kaya, Mert and Adanova, Venera},
  title     = {Augmentation and Cutout in Deepfake Detection: A Comparative Study of Accuracy, Calibration, and Attention},
  booktitle = {2026 11th International Conference on Computer Science and Engineering (UBMK)},
  pages     = {78--83},
  year      = {2026},
  publisher = {IEEE},
  doi       = {10.1109/UBMK71847.2026.11745298},
  url       = {https://ieeexplore.ieee.org/document/11745298}
}