A workplace wellness app might appear to be an easy and helpful tool — a mood check-in, some stress management advice, or a chatbot asking how your week was. But behind this supportive language, some systems are quietly analyzing your voice, writing style and digital behavior for signs of psychological distress.
These tools are already in the marketplace – geared toward workplaces, universities and healthcare. They are designed as early intervention systems that promise to cut back costs and discover problems before they turn out to be serious. Unfortunately, corporations don't have any obligation to report their use, so data about them is lacking.
The basic idea behind these tools is that behavior leaves patterns. Artificial intelligence (AI) systems trained on large data sets learn to acknowledge signals related to specific mental health conditions, and when similar signals appear in latest data, the system generates probabilistic estimates.
What is surprising to many is how much a standard attitude can reveal. A voice recording can have changes in rhythm, pitch and hesitation. Language models can analyze word alternative and emotional tone. Smartphone data It has also been explored as a technique to track changes in sleep, movement and social interaction – all without having to deviate from an individual's routine.
But detecting a statistical signal may be very different from identifying an actual problem. Human behavior is deeply contextual. An individual may speak slowly because they're drained, nervous, or speaking in one other language. Low online activity may simply reflect a busy week.
Even well-designed systems will make mistakes. A one that is genuinely struggling may not display behavior patterns that the system was trained to acknowledge, while another person could also be falsely flagged as being in trouble.
The pressure to develop these devices is real. The World Health Organization estimates that depression and anxiety cost the worldwide economy US$1 trillion ($800 million) annually. Lose productivity. Universities report Growing demand for consultingand employers are coping with it. Absence related to burnout and stress. Automated early warning systems may appear to be a pretty answer.
PeopleImages/Shutterstock.com
When well-being becomes surveillance.
But this technology could fundamentally change the best way we understand mental health. Traditionally, mental health is assessed through a conversation between an individual and a therapist, where context matters. These systems work otherwise, inferring psychological states from behavioral cues that were never intended to convey emotional information.
Once these assumptions are made, they'll influence decisions beyond health care. Assessments of 1's emotional state can shape workplace programs, student support systems or insurance models, affecting how organizations judge an individual's trustworthiness or suitability for a job. In effect, psychological states turn out to be a brand new kind of knowledge.
Certain groups have particular risks. Neurodivergent people often communicate in ways in which differ from the assumed norms of many data sets. A speaker of a second language may pause more incessantly, creating speech patterns that the algorithm may misinterpret. An individual going through grief or illness can show similar signals to mental health conditions – without actually having one.
Used rigorously by healthcare professionals, these tools will be of real value – helping clinicians discover early warning signs of deteriorating mental health. But this same ability looks very different when deployed in a workplace or university without people's knowledge.
At a minimum, people should know when these tools are getting used, what data is being analyzed and whether the system has been independently tested. Claiming that software can detect pain is just not enough by itself.