Over 700 companies are now using affective video technology during the hiring process and other human resources decisions based on an analysis of the interviewee’s facial “expressions.” However, the ethical ramifications of this technology have not been fully fleshed out. I offer a brief survey of the topic: (1) how the field and companies portray themselves, and (2) how the media portrays them. In part II, I will cover the major problem with this use of affective technology: these affective technologies are based on outdated emotion science from the 1960s–80s, ignoring work in both science and philosophy from the last forty years.
What is Affective Video Technology? What is Emotion AI, also known as Affective AI?
Affective video technology, or emotion detection technology, is part of Emotion AI (also called affective AI, affective computing, artificial emotional intelligence) which dates back to at least 1995 when Rosalind Picard published “Affective Computing,” after having realized that the role of emotions is essential in human cognition as they play a critical role in decision-making, perception, human interaction, and in human intelligence more broadly. She realized that if the goal of AI is to replicate the way humans think, which is debatable, then emotion had to be a part of it. Therefore, Emotion AI refers to new artificial intelligence technologies whose goal is learning and recognizing human emotions, and to use that knowledge to improve everything from marketing campaigns (Affectiva, Realeyesit, and Kairos) to health care (Empatica, and emteq), to security (WeSee, and OxygenForensics). It is a subset of AI that aims to measure, understand, simulate, and react to human emotions. Affective video technology as it stands today is a set of algorithms and video technology that collect and interpret data points from a person’s face, voice, and also sometimes gait.
Affective video technology at the moment requires two techniques: computer vision, to precisely identify facial movements, and machine learning algorithms to analyze and interpret the alleged emotional content of those facial features. Typically, the second step employs a technique called supervised learning, a process by which an algorithm is trained to recognize things it has seen before. The basic idea is that if you show the algorithm thousands and thousands of images of stereotypically happy faces with the label “happy” when it sees a new picture of a happy face, it will, again, identify it as “happy.” This is based on the premise that analyzing emotions in real time is simply a mathematical problem of astronomical proportions –an equation our brains repeatedly solve in microseconds throughout the day– and therefore it is only a matter of time before AI can master it beyond human abilities in much the same way that it mastered games like chess or Go.
HireVue’s Affective Video Technology Hiring Services
Firms have begun using affective video technology to facilitate the hiring process by not only reducing the financial costs and duration of the hiring process, but also hiring the best candidates using a method allegedly based on science which removes human bias and error. HireVue is the leading company using affective video technology to aid with the hiring process and offers its clients: “predictive assessments tied to higher quality hires,” hiring faster due to their one step assess and interview, a better experience for candidates since they can do it from home at their own convenience, and they claim that their software is validated by industrial and organizational (I-O) psychology experts who “craft valid assessments with scientific rigour.”
The algorithm which determines an “employability” score is by no means transparent and neither website nor candidate help offers information on the criteria for the predictive assessment. According to Loren Larsen, HireVue’s CTO, the system dissects tiny details of candidates’ responses (facial expressions, eye contact, perceived “enthusiasm”) and compiles reports companies can use for hiring or disregard. The algorithm is trained by having current workers of the company sit through the assessment. Part of the idea behind this technology is based on already existing methods of assessing threat levels based on a person’s facial movements, voice inflections, and body language which was based on the work of Paul Ekman, who claims that there is a set of universal emotions which have facial expressions associated with them which act as fingerprints. Some of those may be micro-expressions which may be difficult to detect since they occur very rapidly. It was therefore seen as fertile ground for AI to improve human’s ability to read emotions on someone’s face. Larsen compares the algorithms’ ability to boost hiring outcomes with medicine’s improvement of health outcomes, since the algorithm is more objective than flawed metrics used by human recruiters. Companies are now believing in machine decisions over human feedback.
Media Coverage, and General Ethical Concerns
Media coverage has been overwhelmingly negative and yet companies and the field as a whole is still thriving (it is estimated to be a 25 billion dollar industry). As of late, the more positive or neutral coverage focuses on the positive impact that HireVue has had during the pandemic since the whole hiring process can be done off-site. Another positive comment has been how affective technologies like HireVue’s can help ensure diversity, widening the talent pool, and helping avoid bias–both human and algorithmic, as advertised by HireVue on their website.
The more critical media, most notably, Drew Harwell, writing for The Washington Post, has covered two major ethical concerns regarding the use of AI during the interview process: privacy and bias:
Privacy: The Electronic Privacy Information Center, known as EPIC, filed an official complaint calling the Federal Trade Commission (FCT) to investigate HireVue’s Business practices. The privacy concern has two sides: 1) since HireVue refuses to share information about their algorithms it is impossible for job candidates to know how their data is being used or to consent to such uses. This is linked to claims that affective video technology is in general dehumanizing and invasive. 2) The information collected by HireVue can be shared with other companies as the result of a merge, reorganization, purchase or acquisition of any part of their business. Therefore, potential employees do not know what they are consenting to: what information they are giving up, and in whose hands it is going to end up. HireVue and companies like Affectiva, which uses affective AI for marketing, claim consent to their use of your data. In the case of HireVue, potential employees or employers can opt not to use HireVue’s service, which realistically puts them at a disadvantage with others who do. In the case of marketing companies, they require an opt-in consent. However those have come under scrutiny since they do not offer real choices and information about where data is coming from, how it is stored, whether it can be sold and transferred or linked to other data sets. Strict rules concerning data collection and sharing that often apply in Academia are not transferred to the private sector.
Bias: The worry is what counts as the perfect employee? Since algorithms learn on data, these determine what the optimal employee looks like. Therefore, datasets need to be unbiased to ensure ethical hiring practices that are appropriate for all potential interviewees, not just the subset used for training. Problematically, most companies do not release their datasets or algorithms, making it very difficult to prove bias. Studies have recently shown that facial recognition technologies reproduce biases harming minority communities. Emotion detection technology assigns more negative emotions to black men’s faces than white counterparts, geographic proxies reveal socio-economic backgrounds and datasets based on current employees re-create current society’s biases in hiring. The EPIC complaint, which suggests that HireVue’s promise violates the FTC’s rules against “unfair and deceptive” practices, has been received by the FTC but not yet pursued. The company, HireVue’s Larsen said, audits its performance data to look for potentially discriminatory hiring practices, known as adverse impacts, using “world-class bias testing” techniques. The company’s algorithms, he added, have been trained “using the most deep and diverse data set of facial action units available, which includes people from many countries and cultures.” However, even when algorithms are not based on an “ethnicity classifier” as in the case of Affectiva problems arise. Affectiva uses geography as a proxy for identifying where someone is from and what their characteristic facial “expressions” are but that means that they compare British smiles against British smiles, and Chinese smiles against Chinese smiles. Therefore if there is a Chinese person in the UK, it would miss the cultural nuance. Regulators will need to figure out how much responsibility companies should be expected to shoulder in avoiding the mistakes of a prejudiced society.
That said, affective video technology in the hiring process ought to attract more scrutiny than affective video technology in other corporate contexts because of the potential for harming actual people. If we accept that affective video technology does not tell us what its proponents think it tells us, then any decisions based on it will be problematic: a company’s decisions about product placement or advertising may have some impact on the company itself, but decisions based on affective video technology in the hiring process can cause significant and obvious harm to the individual applicant. In Part II of this series I will discuss a third and the more problematic worry, namely that affective video technology is based on flawed, outdated science and therefore it has a long way to go before it is viable, if it ever will be.