At this year's annual re:Mars conference, Amazon revealed perhaps the scariest tech I've ever heard of: an Alexa skill that can mimic other people's voices.
At first glance, that doesn't seem so bad, does it? The nerd in me is already thinking about how I could feel like Tony Stark or Luke Skywalker with my own personal Alexa speaker from Paul Bettany or Anthony Daniels (the voices of JARVIS from the MCU and C3-P0 from Star Wars respectively).
But rather than go the celebrity route, Amazon's Alexa SVP Rohit Prasad pitched the idea along with a clip of a boy played by Alexa using the voice of his recently deceased grandfather. According to Engadget (opens in a new tab), an Amazon spokesperson explained that this clip could be recreated using "as little as one minute of audio" from the person Alexa is impersonating.
The scene was probably meant to inspire poignant feelings when you see Amazon technology helping a child deal with their grief, but it could have been a scene from Black Mirror. In fact, the episode Be Right Back has a very similar premise to this.
Be Right Back follows Domhnall Gleeson's Ash, who is killed in a car accident and, after discovering she is pregnant, Hayley Atwell's Martha must test a new technology that allows her to communicate with an artificial intelligence posing as Ash. It's terrifying and certainly not meant to serve as a preview of anything good.
In his explanation of the Alexa tool, Rohit Prasad said that while it won't take away the pain of loss, it "can definitely make the memories linger" and could help alleviate any heartbreak.
It is a feeling that I certainly identify with; Grief is a difficult emotion to deal with, especially at an age as young as the boy in the video shared by Amazon. But blurring the lines between life and death doesn't seem like the healthiest way to deal with loss.
Based on Prasad's comment, it's clear that Amazon sees this as an evolution of remembering loved ones using old photos and videos recorded before they died, but it's not the same thing. A photo or video was taken with their consent and represents something the person actually did before they died; this AI-powered feature doesn't play a recording of a book read, it uses his voice to create a fabricated memory.
If someone wants their voice to live on through Alexa after they die and then feed them, I'm sure they'll love this feature. Personally, I would like to be allowed to rest in peace.
More than just a digital undead
In addition to the above nightmarish uses, this copying feature could also pave the way for new scams. While many of us know how to ignore and hang up the robotic voices on the other end of the line, we may be more inclined to listen to our grandmother's or son's pleas for money.
Coupled with sophisticated video deepfake tools, we may eventually find it impossible to believe any video we see; everything could have been built from fake sounds and realistic images.
Amazon won't be the only company developing AI voice tools like this spirit. We suspect that Apple, Google and all the other voice assistant makers are looking to make them more realistic and personal. But these elegant imitation features are truly a Pandora's box, something to be opened with care.
To that end, it appears that Amazon is aware of the need for caution. It hasn't set a timeline for when the feature will roll out to existing Alexa devices, and hasn't even confirmed if it will be available to the public.
But now that her audio immortality has been revealed to the world, it's probably only a matter of time before Alexa gains the ability to speak like the dead.