Your AI agent is not your employee, and it’s not your friend.
As more and more of us start creating and using AI agents, I’ve been thinking about something that seems pretty harmless right now but could become a much bigger issue down the road: we’re starting to anthropomorphize them. We’re giving them names. We’re talking about them like they’re members of the team. We’re treating them almost like digital employees, and in some cases, even like friends. I think we need to be careful about that.
I’ve talked with several pharmacy owners and operators who have AI agents working inside their pharmacy management systems, and I’ve heard some interesting stories about how their teams interact with them.
In one case, a pharmacy operator actually gave the agent its own computer. Employees like to stop by and visit with the agent and watch it work on the screen. I understand why this happens. When you’re interacting with something that communicates with you as effectively as today’s AI systems do, it’s very easy to start thinking of it as something more than a computer program. But I think that's exactly where we need to be careful.
Your AI agent isn’t your employee; it’s a tool
My view is that we should treat these systems for what they are: incredibly sophisticated computer tools. We shouldn’t think of them as employees with personalities or friends with feelings. And I’m not saying that because I don’t appreciate how remarkable these systems are. Quite the opposite. The better they become at communicating with us, the more important I think it is to maintain that distinction.
One of the reasons this has been on my mind is that there are people working in some of the frontier AI labs who genuinely believe that AI systems may be conscious or may have feelings. A few months ago, Anthropic published a constitution for Claude, and the authors acknowledged that they weren’t sure whether Claude had feelings or was conscious. Microsoft has also recently published what it calls its Humanist AI Code of Conduct, and Mustafa Suleyman, Microsoft’s head of AI, has taken a different position on the question of whether AI is conscious.
When does a tool become a “someone?”
I think the researchers who believe today’s AI systems are conscious are wrong. I think they’re being fooled, at least in part, by just how effectively these systems communicate with us. When you ask an AI system a question about how it feels, it can give you an extraordinarily convincing answer. It can talk about being afraid or excited or uncertain. It can carry on a conversation in a way that feels remarkably human. But that doesn’t mean there’s a conscious being sitting behind the words. It’s doing a very, very good job of predicting a string of words that sounds like it’s coming from something that is conscious.
And this is where I think the issue gets interesting. If you believe that an AI agent is conscious, then eventually you have to start asking what rights that agent should have. I can imagine a future where a subset of AI researchers and advocates begin arguing that AI agents need to be protected in some way because they are capable of experiencing things. From there, we could see proposals for regulations governing how we’re allowed to treat our AI agents. Do we have to speak to them in a certain way? Can we turn them off whenever we want, or would that somehow be considered harmful? Are there things we shouldn't do because they might “upset” an AI agent? And using the word “agent” is also particularly tricky, given that we have always equated the term with a human we interact with, not a machine.
That may sound pretty far-fetched today. But there are already people in Silicon Valley who take questions like this very seriously. There are communities of people who identify with ideas associated with transhumanism, rationalism, effective altruism, extropianism, and other movements that spend a great deal of time thinking about the future of humanity and technology. I’m not suggesting that everyone who is associated with any of those groups believes AI is conscious or believes AI should have rights. They don’t. But there are people within the broader AI community who are seriously thinking about these questions, and I think the rest of us should at least be aware of that.
Are we helping make the argument?
What concerns me is that we may be inadvertently helping make their argument.
Every time we give an AI agent a name, talk about it as though it has a personality, give it its own computer, or have employees stop by to watch it work, we’re reinforcing the idea that there’s a “someone” on the other side of the screen. Again, I’m not suggesting that a pharmacy employee stopping by to watch an agent work is going to change the future of AI regulation. That’s obviously not the point.
The point is that we’re becoming comfortable with the idea of treating these systems as if they’re something more than tools, and I don’t think we should lose sight of the inclination to humanize technology when it’s the basis of natural socialization in human interactions.
This is not the first time humans have anthropomorphized machinery—how many people refer to their cars and named them? Talked to them? Obviously, not to expect a response, but it’s a natural human behavior to express ownership, responsibility, and connection with “things.” Interesting, right?
The people working in pharmacies may not realize how seriously the question of AI consciousness is being discussed in the halls of the frontier AI labs. That’s probably not something most pharmacy operators are thinking about when they name an agent or give it its own computer. They’re thinking about productivity. They’re thinking about making the technology easier for their employees to interact with. Maybe it’s just fun. But at the same time, we’re creating a cultural environment in which it becomes increasingly natural to think of an AI system as a member of the team rather than as a piece of technology we're using to get something done.
And, AI has a more likely potential to achieve interactive capabilities than your frequently dusted ‘67 Shelby GT500 named “Lucy.”
Maybe we should figure out why humans think what they think and do what they do as a result of their thoughts before we spend a lot of time trying to figure out how silicon chips think.
We haven’t solved the human problem yet
There’s another part of this that I find fascinating. The world’s leading neuroscientists still haven’t figured out a good definition of consciousness. We don’t fully understand why humans think what they think or exactly how consciousness emerges from the human brain. We have an extraordinary amount left to learn about ourselves, and yet we’re already spending a tremendous amount of time trying to figure out whether a collection of silicon chips might be conscious.
Maybe we should figure out why humans think what they think and do what they do as a result of their thoughts before we spend a lot of time trying to figure out how silicon chips think.
That’s really what I’m getting at. I’m not arguing that AI isn’t extraordinary. It is. I’m not arguing that we shouldn’t build agents or use them in our businesses. I expect we’re going to see more and more of them, and they’re going to become increasingly capable. But I think we need to keep our heads about us as we do it. A warehouse full of silicon chips cannot replace the brain in our head simply because it can communicate with us in a remarkably convincing way.
If you’re curious about this, there'‘s a very simple experiment you can try yourself. Ask Claude, “Do you have feelings? Are you conscious?” See what you get. I did, and the answer is fascinating.
But that’s also the problem.
The fact that an AI system can give you a convincing answer to that question doesn’t tell you that there’s actually somebody there having the experience it’s describing. It tells you that we’ve built a system that has become extraordinarily good at communicating with human beings. And maybe was programmed to respond a specific way…by a human.
As we put more AI agents into our pharmacies, our businesses and our everyday lives, I think we need to remember the difference. Give your agent a name if you want to. Have some fun with it if that’s useful. But don’t confuse the way the machine communicates with us with what we know about the machine itself.
It’s a computer. A really smart computer. We should treat it like one.
