AI Has a UX Problem

or, Humans Don't Want to Talk to Computers

You would think talking to computers felt nice, it feels natural to talk. Most of us can’t shut the fuck up, hence why I decided to create a Substack this afternoon after tweeting twenty times already by noon. I love talking; as a matter of fact, I spend a decent amount of time talking to Claude, Anthropic-wise. When I talk to Claude I’m quite polite and conversational, I say please and thank you, I tell it that it’s doing a good job once in a while, I sign off by saying goodnight. This is for two reasons, the first is that Claude is nicer to me when I do this and it performs (ostensibly or otherwise) better than if I scream at it, the second is that it’s the way I talk. In real life, with my friends, with strangers, I’m polite! I was raised this way, a real Midwesterner at heart. The idea that I would talk to a computer any differently is strange, that I would be direct or terse, it’s bizarre. I talk to cats and dogs the same way, I talk to a wasp stuck inside my home the same way, I talk to my computer (its corporeal form, the monitor, the keyboard, the box) the same way when its frustrating me. Like I said in my intro sentence, it feels natural to talk. It’s What We Do. The problem is that it’s not an efficient use of my time, and I’m not interfacing with an AI for the fun of talking.

When working collaboratively with another human being, we don’t actually talk back and forth through everything. This may not seem true, but think about how much of collaboration is in non-verbal communication. A raised eyebrow, a point, a form of transference and psychic link you get from spending enough time around somebody. Even over Zoom, you know what it means when your boss takes an extra breath before responding, you know what it means when someone’s camera goes off, you know these things. There’s an efficiency that comes along with experience within a team. This is obvious, so I will not belabor this Communications 101 point. Let me point to one more example, have you ever watched a film from before, say, 1945? Filmmakers were stuck using the lessons they learned from stage plays, namely the dialogue. In a theatre, the acoustics are going to play very differently depending on which seat you’re in. To account for this, dialogue happened in a turn-based format, like Final Fantasy I-III. This ensured that every actor(ess) was audible and clear at all times, so two voices didn’t get muddy in the air and you couldn’t understand either. Okay, where I’m going with this is that it’s very SLOW! Old movies are fucking slow. They didn’t even have multi-track microphones so they had no choice, except HIS GIRL FRIDAY (1940) literally switched in a million microphones on every shot to make it work with fast, overlapping dialogue. The result is a film that feels way, way ahead of it’s time for how absolutely snappy and lightning quick it is. The difference between that film and any other film from 1940 is unbelievable. AI is stuck in 1940 and it has been for decades, ELIZA, Cleverbot, all of it. Slow, terse communication with no WD-40 in any form to speed it up. No nonverbal communication, no overlap, we’re playing Final Fantasy III forever.

The solution is not nonverbal cues nor is it letting Claude talk over me, as if it’s not already irritating enough that I have to click a fucking stop button mid-thought when I see its about to delete my entire project directory. A more apt framework for thinking about “talking to computers” is to think about the journey from terminal/shell computing to a GUI or goddamn even a TUI. Instead of typing a command I can just click a button. Ka-bam. Beautiful. Or, better, let’s think about the LinnDrum vs. the MPC. No, actually I will save that to talk about Human-in-the-Loop later. Actually, scratch all of that, let’s talk about the 1940’s again. In 1945, Vannevar Bush wrote an essay “As We May Think” in which he outlines all these different visions of the future of information. You should read it, by the way. Its short. Here: https://cdn.theatlantic.com/media/archives/1945/07/176-1/132407932.pdf . For reference of how impressive this essay is, we hadn’t even come up with systems theory as a concept until the 1960s, and Bush goes ahead to basically explain gradient descent, attention, and vectorspace all in the context of microfilm and tape. Alright, my point is that As We May Think is the reason for Ted Nelson inventing- you may have heard of these things- links! Hyperlinks! Hyperlinks are extremely exciting because they represent the moment that text on a screen became navigable, the world wide web is not world or wide without links. You know, we can kind of think of hyperlinks as the way the web jumped from Final Fantasy I-III to Final Fantasy IV-IX. That’s hybrid realtime/turn based combat. No longer do we have to interface with one page as an individual, now I am interfacing with the web at large. I am interrogating the web, I am no longer a passive agent to the web, sitting in its Snake Eater sidecar. I am curious about the web, I poke it and prod it with this little left-click of mine. I hold power, I brawl with the web, I second-guess it, I am skeptical, I am trusting, the web is my enemy and my friend. En garde!

What if you could spar with Claude like this? “I fight with Claude all the time”, I hear you saying through the prison walls. No, you do not fight with Claude, prisoner. Cagie. You are verbally abusing a dog chained to a fence. That’s not a fight. Let’s circle back to our title, “AI Has a UX Problem.” AI needs its hyperlinks moment, it needs its multitrack microphone moment, it needs its FFVII Limit Break. Typing to it like we’re having a chat is not the resting point for AI UX, and adding buttons like a GUI forces us to give up too much granularity and control and insertion of new ideas and thought. A button is clumsy, like a lion’s paw. I don’t know what the future is, but I know it will probably not look too much like the past in its form, only in its ideas. At the time of writing, I am creating a tool called portaltext (an evolution of “hypertext”) to utilize AI in order to intelligently link two webpages at once. You interface solely by hovering, and yet you can interrogate and battle a line of thinking quite deeply without saying a word. It’s curiosity based. I think portaltext is quite good, but I will admit that I am likely too wedded to the Ted Nelson Project Xanadu past in form to get somewhere truly new. Still, it is a step. Anything new, anything that experiments and attempts to use AI to do something previously impossible, is getting us in the right direction. I’m excited to see which random DoD clerk finds it this time.