Open Access Open Access  Restricted Access Subscription Access

A Survey on Deepfake Generation and Detection using Deep Learning

Ms.Devi S Naidu, Ms. Dhanya R, Mr. Mubarak M, Mr. Mallikarjun V B, Mrs. Poornima B P

Abstract


 This fast-evolving artificial intelligence has opened the gates to new multimedia technologies with unprecedented abilities to alter the human voice and appearance. Voice cloning, which allows the synthesis of speech that is similar to a specific individual, presents exciting opportunities for creating personalized experiences across various domains-from virtual assistants to entertainment. Concurrently, the advancements in lip synchronization algorithms, such as the Wav2Lip model, have allowed for the smooth alignment of lip movements with spoken words, thus enhancing the realism of synthesized speech and improving accessibility for people with hearing impairments. However, these powerful technologies also present significant challenges. The potential for misuse of voice cloning, such as creating deceptive audio or impersonating individuals, raises serious ethical concerns. Furthermore, the increasing prominence of deepfakes - a kind of synthetic media where an individual can be depicted convincingly saying or doing things that he or she never actually did - poses a serious threat to infor- mation integrity and social and political implications can be grave. This paper reports the state-of-the-art in this regard, covering the methods, challenges, and issues surrounding each. We delve into the details of voice cloning techniques, discussing various approaches to audio feature extraction, speaker encoding, speech synthesis, and vocoding. Finally, we analyze the Wav2Lip model and other advanced methods for achieving accurate and realistic lip synchronization. Finally, we discuss the changing landscape of deepfake detection, discussing the strengths and weaknesses of current approaches, including those based on deep learning architectures such as convolutional neural networks and recurrent neural networks. This survey paper covers these cutting-edge technologies in an all-inclusive manner, pointing out their potential and limitations, and identifying key research directions for future advancements, including the development of more robust and ethical frameworks for their development and deployment.


Full Text:

PDF

References


Tan, Xu, et al. ”NaturalSpeech: End-to-End Text-to-Speech Synthesis with Human-Level Quality.” IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

Liu, Songxiang, Dan Su, and Dong Yu. ”Meta-voice: Fast few-shot style transfer for expressive voice cloning using meta learning.” arXiv preprint arXiv:2111.07218 (2021).

Casanova, Edresson, et al. ”Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone.” International Conference on Machine Learning. PMLR, 2022.

Li, Jiaxin, and Lianhai Zhang. ”ZSE-VITS: A Zero-Shot Expressive Voice Cloning Method Based on VITS.” Electronics 12.4 (2023): 820.

Kankaria, Romit Vinod, et al. ”RAAH. ai: an interactive chatbot for stress relief using deep learning and natural language processing.” 2021 12th International conference on computing communication and networking technologies (ICCCNT). IEEE, 2021.

Mukhopadhyay, Soumik, et al. ”Diff2lip: Audio conditioned diffusion models for lip-synchronization.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

Thambiraja, Balamurugan, et al. ”Imitator: Personalized speech-driven 3d facial animation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Agarwal, Aditya, et al. ”Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.

Nanditha, Gundu, et al. ”MultiLingualSync: A Novel Method for Generating Lip-Synced Videos in Multiple Languages.” 2023 3rd Asian Conference on Innovation in Technology (ASIANCON). IEEE, 2023.

Gururani, Siddharth, et al. ”Space: Speech-driven portrait animation with controllable expression.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.


Refbacks

  • There are currently no refbacks.