I first noticed it when, a few months ago, I opened an email from Ian, my literary agent. Before I’d had a chance to read anything he’d written, Gmail was recommending a full, fleshed-out, AI-generated reply, ventriloquizing ideas for a book and even my feelings about the job transition I’d recently made. It had mined my inbox to infer why Ian was writing to me and ingested bits of my style, even signing off with the lowercase “m” that I use with people with whom I have an easy familiarity.
For around a decade, Google had been suggesting very generic, sometimes monosyllabic “smart replies” — things like “Okay” or “Thanks!” or “Any thoughts?” I’ve used these to send quick acknowledgements to emails I’d have otherwise forgotten about. But in the last couple years, Gmail has begun to offer fully formed draft replies that presume to impersonate my own, individual reactions to my interlocutors’ questions, ideas, and emotions.
This felt like a striking turn. I reflected with some sadness on the idea of sending one of these to someone who matters to me — how dehumanizing to both me and Ian it would feel to make him read a counterfeit subjectivity pretending to be my own.
You might say this is no big deal; maybe it gives you time back for deeper work or more meaningful parts of your life (I wouldn’t begrudge that at all — AI saves me time, too!). We’re all drowning in too much email, much of it pointless or lacking any great meaning. Isn’t that exactly the kind of day-to-day tedium that we should happily invite AI to liberate us from?
But I think that this machine-generated personal correspondence, which is only likely to spread further into other forms of communication, has preoccupied me because there’s something deeper going on here. A lot of ink has been spilled in the last few years about AI-generated writing and its social consequences — how it will deskill millions of workers, outsource our thinking, confuse kids growing up in the AI age about the difference between real and synthetic friends, and so on. We already know that AI language is unnervingly good at sounding like it’s the product of a fellow consciousness. But the particular creepiness of elaborate email autocomplete is that it’s training on and simulating your consciousness. And as it does so, it also gives you a little less reason to actually be conscious.
AI writing and “cognitive surrender”
Like many knowledge workers who derive their living and their identities from cognitive capacities now being at least partially replicated in silicon, I have a complicated and ambivalent relationship with generative AI. I now depend on it to research almost every story I work on, a purpose for which it’s obviously very useful (despite those who still insist it can never be useful for anything).
I am, though, deeply skeptical of using it for writing, because, as many writers smarter than me have already noted, writing is inextricable from thinking, and short-circuiting it can diminish our capacity for deep thought. The friction of writing is not dead weight but is part of how you decide what you mean and give coherence to ideas. For that reason, my former Vox colleague, the brilliant Kelsey Piper, who is generally positive about AI’s potential to make us more productive and improve human life, said on a recent podcast episode, “I would never use it to write.”
In a recent paper, a pair of University of Pennsylvania scholars described the wholesale outsourcing of cognitively complex tasks to AI as “cognitive surrender.” “An abdication of critical evaluation,” they write, “where the user relinquishes cognitive control and adopts the AI’s judgment as their own.” This is one reason why it felt especially inappropriate to have AI generate thoughts for me in reply to someone with whom I’m brainstorming about writing a book, likely one of the most cognitively demanding things I’ll ever do. Email, for all of its annoyances, is also relational. And letting a machine generate your side of the exchange diminishes the authenticity of your connection to another person.
Sometimes the AI drafts, of course, are plainly wrong. An AI-suggested email might, for example, say you’ve read a book that you haven’t, perhaps making it more likely that you go along with the false claim. But what unsettles me the most is not the mere hallucination, it is when the AI is right, or right enough. My email’s AI is pulling from its knowledge of everything I’ve written before, so it can often make a reasonable guess of what I’d want to say anyway. The system is not wholly failing to reproduce my mind, but is actually producing a close-to plausible substitute for it.
It feels like the beginnings of what Silicon Valley has prophesized for decades as a coming merge (sometimes called the “singularity”) between human and machine minds. I used to consider this a totally improbable idea, but I hadn’t been open-minded enough. It might turn out to be dispiritingly easy for an advanced AI to train on a sample of your past thoughts and write future ones for you.
Still, it seems unlikely that we will simply acclimate to the idea that all the written communication we encounter and generate every day may be AI-generated. So much, if not most, of our interpersonal communication now takes place in writing. However vulnerable we may be to cognitive surrender, humans also have a deep countervailing need to experience language as coming from another conscious mind — to feel seen and known, and to assert our own distinctness in return.
And anyway, Gmail isn’t yet that good at imitating my conscious voice. I would never write, “Lots of interesting stuff coming up at Vox!” (Which isn’t, of course, to say that there isn’t a lot of interesting stuff going on at Vox.) That still leaves me, for now, with the pleasure of figuring out what I want to say.
