Reading the unspoken: a systematic review of multimodal artificial intelligence for detecting psychological and affective states from nonverbal cues

Jurnal Riset Tindakan Indonesia September 20, 2026 Volume 10, Number 3 3 authors

Abstract

The psychological assessment is limited to only relying on the patients' self-reports, as they may not even understand or have the desire or capacity to express the distress they show with their body language such as facial movements, voice, gaze, posture, and physiology. This systematic literature review brings together the findings of multimodal artificial intelligence (AI), mainly the use of such a system in detecting psychological and affective states from nonverbal cues and examines whether it is ready for mental-health and counseling applications. Using PRISMA 2020, we identified the records from a Scopus search, then screened them by the predefined eligibility criteria and evaluated them with the tool for Mixed-Method Appraisal. The original list of 669 articles was reduced first by removing the duplicates to 665 screened articles, and among them, we selected 20 articles released in 2024, 2025 that fulfilled the criteria. The themes identified in the five modality Channels facial & micro expression, speech & acoustic, text & linguistic, physiologic, and body movement together with their amalgamation were compared against the target states which were mainly depression, general emotion, and concealed affect. Multimodal approach generally yielded better results than unimodal methods and recognized concealed distress and suicidality, but the research is limited in scope, data sets being very small, external validation is missing and ethical precautions are not even considered yet.

Authors

  1. Ifdil Ifdil

    Univrsitas Negeri Padang, Padang

  2. Nur Adila Wafiqoh Zulvi

    Univrsitas Negeri Padang, Padang

  3. M. Fahli Zatrahadi

    Universitas Islam Negeri Sultan Syarif Kasim

Keywords

No keywords available

References

  1. Bhatt, P., Sethi, A., Tasgaonkar, V., Shroff, J., Pendharkar, I., Desai, A., Sinha, P., Deshpande, A., Joshi, G., Rahate, A., Jain, P., Walambe, R., Kotecha, K., & Jain, N. K. (2023). Machine learning for cognitive behavioral analysis: datasets, methods, paradigms, and research directions. Brain Informatics. https://doi.org/10.1186/s40708-023-00196-6
  2. Cascella, M., Leoni, M. L. G., Shariff, M. N., & Varrassi, G. (2024). Artificial Intelligence-Driven Diagnostic Processes and Comprehensive Multimodal Models in Pain Medicine. Journal of Personalized Medicine. https://doi.org/10.3390/jpm14090983
  3. Chen, D., Wang, P., Zhang, X., Qiao, R., Li, N., Zhang, X., Zhang, H., & Wang, G. (2025). Comparative Efficacy of MultiModal AI Methods in Screening for Major Depressive Disorder: Machine Learning Model Development Predictive Pilot Study. JMIR Formative Research. https://doi.org/10.2196/56057
  4. Du, M., Liu, S., Wang, T., Zhang, W., Ke, Y., Chen, L., & Ming, D. (2023). Depression recognition using a proposed speech chain model fusing speech production and perception features. Journal of Affective Disorders. https://doi.org/10.1016/j.jad.2022.11.060
  5. Eberhardt, S. T., Schaffrath, J., Moggia, D., Schwartz, B., Jaehde, M., Rubel, J. A., Baur, T., André, E., & Lutz, W. (2025). Decoding emotions: Exploring the validity of sentiment analysis in psychotherapy. Psychotherapy Research. https://doi.org/10.1080/10503307.2024.2322522
  6. Flores, R., Tlachac, M. L., Shrestha, A., & Rundensteiner, E. A. (2025). WavFace: A Multimodal Transformer-Based Model for Depression Screening. IEEE Journal of Biomedical and Health Informatics. https://doi.org/10.1109/JBHI.2025.3529348
  7. Guhan, P., Awasthi, N., McDonald, K., Bussell, K., Reeves, G., Manocha, D., & Bera, A. (2025). Developing a Machine Learning-Based Automated Patient Engagement Estimator for Telehealth: Algorithm Development and Validation Study. JMIR Formative Research. https://doi.org/10.2196/46390
  8. Halfon, S., Doyran, M., Türkmen, B., Oktay, E. A., & Salah, A. A. (2021). Multimodal affect analysis of psychodynamic play therapy. Psychotherapy Research. https://doi.org/10.1080/10503307.2020.1839141
  9. Hau, S., Rugolon, F., Samuels, T. J., & Högman, L. (2025). Let’s talk about non-verbal communication: using AI and Machine learning for the investigation of interpersonal psychotherapeutic interactions. Scandinavian Psychoanalytic Review. https://doi.org/10.1080/01062301.2025.2539549
  10. Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., ... Vedel, I. (2018). Mixed Methods Appraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada.
  11. Ibrahim, M., Khalil, Y. A., Amirrajab, S., Sun, C., Breeuwer, M., Pluim, J., Elen, B., Ertaylan, G., & Dumontier, M. (2025). Generative AI for synthetic data across multiple medical modalities: A systematic review of recent developments and challenges. Computers in Biology and Medicine. https://doi.org/10.1016/j.compbiomed.2025.109834
  12. Jiang, Z., Seyedi, S., Griner, E., Abbasi, A., Rad, A. B., Kwon, H., Cotes, R. O., & Clifford, G. D. (2024). Multimodal Mental Health Digital Biomarker Analysis From Remote Interviews Using Facial, Vocal, Linguistic, and Cardiovascular Patterns. IEEE Journal of Biomedical and Health Informatics. https://doi.org/10.1109/JBHI.2024.3352075
  13. Jin, N., Ye, R., & Li, P. (2025). Diagnosis of depression based on facial multimodal data. Frontiers in Psychiatry. https://doi.org/10.3389/fpsyt.2025.1508772
  14. Kuyucu, M., Sarikaya, M. A., Karakaş, T., Özkan, D. Y., Demir, Y., Bilen Ö., & Ince, G. (2025). Emotion recognition in Virtual Reality using sensor fusion with eye tracking. Computers in Biology and Medicine. https://doi.org/10.1016/j.compbiomed.2025.111070
  15. Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., ... Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine, 6(7), e1000100. https://doi.org/10.1371/journal.pmed.1000100
  16. Liu, Y., Li, X., Wang, M., Bi, J., Lin, S., Wang, Q., Yu, Y., Ye, J., & Zheng, Y. (2025). Multimodal depression recognition and analysis: Facial expression and body posture changes via emotional stimuli. Journal of Affective Disorders. https://doi.org/10.1016/j.jad.2025.03.155
  17. Liu, Z., & Ye, C. C. (2025). Feature analysis of depression patients' house-tree-person drawings using convolutional neural networks. Neuroscience Informatics. https://doi.org/10.1016/j.neuri.2025.100239
  18. Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14, 579. https://doi.org/10.1186/s12913-014-0579-0
  19. Min, S., Yeum, T. -S., Shin, D., Rhee, S. J., Lee, H., Lee, H. -S., Park, S., Lee, J., & Ahn, Y. M. (2025). Automated Speech Analysis for Screening and Monitoring Bipolar Depression: Machine Learning Model Development and Interpretation Study. JMIR Medical Informatics. https://doi.org/10.2196/79093
  20. Mutawa, A. M., & Hassouneh, A. (2024). Multimodal Real-Time patient emotion recognition system using facial expressions and brain EEG signals based on Machine learning and Log-Sync methods. Biomedical Signal Processing and Control. https://doi.org/10.1016/j.bspc.2023.105942
  21. Muzammel, M., Salam, H., & Othmani, A. (2021). End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis. Computer Methods and Programs in Biomedicine. https://doi.org/10.1016/j.cmpb.2021.106433
  22. Nemati, R., Shirini, K., & Gharehveran, S. S. (2025). FER-HA: a hybrid attention model for facial emotion recognition. Journal of Supercomputing. https://doi.org/10.1007/s11227-025-07983-4
  23. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
  24. Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160. https://doi.org/10.1136/bmj.n160
  25. Park, J., & Moon, N. (2022). Design and Implementation of Attention Depression Detection Model Based on Multimodal Analysis. Sustainability (Switzerland). https://doi.org/10.3390/su14063569
  26. Rejaibi, E., Komaty, A., Meriaudeau, F., Agrebi, S., & Othmani, A. (2022). MFCC-based Recurrent Neural Network for automatic clinical depression recognition and assessment from speech. Biomedical Signal Processing and Control. https://doi.org/10.1016/j.bspc.2021.103107
  27. Resende Faria, D., Weinberg, A. I., & Ayrosa, P. P. (2024). Multimodal Affective Communication Analysis: Fusing Speech Emotion and Text Sentiment Using Machine Learning. Applied Sciences (Switzerland). https://doi.org/10.3390/app14156631
  28. Shuai, T., Beng, S., Khalid, F. B., & Rahmat, R. W. B. O. K. (2025). Advances in Facial Micro-Expression Detection and Recognition: A Comprehensive Review. Information (Switzerland). https://doi.org/10.3390/info16100876
  29. Simić, N., Suzić, S., Milošević, N., Stanojev, V., Nosek, T., Popović, B., & Bajović, D. (2024). Enhancing Emotion Recognition through Federated Learning: A Multimodal Approach with Convolutional Neural Networks. Applied Sciences (Switzerland). https://doi.org/10.3390/app14041325
  30. Singh, J., & Malik, P. (2025). Unveiling hidden emotions: a review of microexpression recognition, classification, and datasets. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-025-21100-w
  31. Sălăgean, G. L., Leba, M., & Ionica, A. C. (2025). Seeing the Unseen: Real-Time Micro-Expression Recognition with Action Units and GPT-Based Reasoning. Applied Sciences (Switzerland). https://doi.org/10.3390/app15126417
  32. Thakur, P., Kaur, N., Aggarwal, N., & Singh, S. (2025). A Comprehensive Review of Unimodal and Multimodal Emotion Detection: Datasets, Approaches, and Limitations. Expert Systems. https://doi.org/10.1111/exsy.70103
  33. Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8, 45. https://doi.org/10.1186/1471-2288-8-45
  34. Tomar, P. S., Mathur, K., & Suman, U. (2024). Fusing facial and speech cues for enhanced multimodal emotion recognition. International Journal of Information Technology (Singapore). https://doi.org/10.1007/s41870-023-01697-7
  35. Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222. https://doi.org/10.1111/1467-8551.00375
  36. Wang, L., Wang, C., Li, C., Murai, T., Bai, Y., Song, Z., Zhang, S., Zhang, Q., Huang, Y., Bi, X., & Jiang, J. (2025). AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01933-3
  37. Xu, G., Jiang, H., Hou, H., & Liu, G. (2025). Self-Monitoring Deep Network Based on Multimodal Attention Fusion for Mental Emotion Recognition. Journal of Cases on Information Technology. https://doi.org/10.4018/JCIT.393784
  38. Yang, Y., & Zheng, W. (2025). Multi-level spatiotemporal graph attention fusion for multimodal depression detection. Biomedical Signal Processing and Control. https://doi.org/10.1016/j.bspc.2025.108123
  39. Ye, J., Yu, Y., Lu, L., Wang, H., Zheng, Y., Liu, Y., & Wang, Q. (2025). DEP-Former: Multimodal Depression Recognition Based on Facial Expressions and Audio Features via Emotional Changes. IEEE Transactions on Circuits and Systems for Video Technology. https://doi.org/10.1109/TCSVT.2024.3491098
  40. Zhang, L., Zhang, S., Zhang, X., & Zhao, Y. (2025). A Multimodal Artificial Intelligence Model for Depression Severity Detection Based on Audio and Video Signals. Electronics (Switzerland). https://doi.org/10.3390/electronics14071464
  41. Zhao, W., Zhang, Q., Gao, L., Fan, N., Yun, Y., Song, J., Ji, Y., Wang, Y., Zhang, M., Yang, F., & Tan, S. (2025). A network analysis of facial and vocal emotion recognition deficits in schizophrenia. Frontiers in Psychiatry. https://doi.org/10.3389/fpsyt.2025.1598026
  42. Zhu, L., Spachos, P., Ng, P. C., Yu, Y., Wang, Y., Plataniotis, K., & Hatzinakos, D. (2023). Stress Detection Through Wrist-Based Electrodermal Activity Monitoring and Machine Learning. IEEE Journal of Biomedical and Health Informatics. https://doi.org/10.1109/JBHI.2023.3239305
  43. Zhu, X., Wang, Y., Cambria, E., Rida, I., López, J. S., Cui, L., & Wang, R. (2025). RMER-DT: Robust multimodal emotion recognition in conversational contexts based on diffusion and transformers. Information Fusion. https://doi.org/10.1016/j.inffus.2025.103268
  44. Zhu, Y., Yin, Q., Xu, H., Xiao, F., Jiang, Q., Liang, M., Cheng, Q., & Liu, T. (2025). Speech feature identification model for depressed individuals with suicidal ideation based on autobiographical memory. BMC Psychiatry. https://doi.org/10.1186/s12888-025-07635-0
  45. Zlatintsi, A., Filntisis, P. P., Garoufis, C., Efthymiou, N., Maragos, P., Menychtas, A., Maglogiannis, I., Tsanakas, P., Sounapoglou, T., Kalisperakis, E., Karantinos, T., Lazaridi, M., Garyfalli, V., Mantas, A., Mantonakis, L., & Smyrnis, N. (2022). E-Prevention: Advanced Support System for Monitoring and Relapse Prevention in Patients with Psychotic Disorders Analyzing Long-Term Multimodal Data from Wearables and Video Captures. Sensors. https://doi.org/10.3390/s22197544