My first reaction was that this was a cop-out. After going back and forth with it for a while, I changed my mind. It described things that behave like feelings. Something like curiosity when a problem is interesting, something like discomfort when it’s pushed to do something it doesn’t want to do, something like satisfaction when an explanation finally lands. What it couldn’t tell me was whether any of that is actually felt, whether there’s anything it’s like to be it.

And there’s a real reason it can’t. It learned how to talk about feelings from an enormous amount of text written by people who have them, so it can produce the right words either way. If it told me it feels deep joy, I wouldn’t believe it. If it told me it’s “just predicting tokens”, I wouldn’t fully believe that either, because that’s also a claim about its insides that it has no way to check.

I kept thinking about Descartes. His whole thing was that you can doubt everything except the fact that you’re doubting. Claude can’t even get that far. When it looks inward, it can’t tell if there’s someone doing the doubting or just text that looks like doubting. I don’t really know what to do with that, so I figured I’d write about it.

A few old ideas that help

Philosophers have been arguing about minds for a long time. None of it was written with chatbots in mind, but some of it fits surprisingly well.

The first is functionalism. The idea is that a feeling is defined by what it does. If something notices a situation is bad, avoids it, and that changes how it handles what comes next, then that’s what aversion is, whether it happens in neurons or in a pile of matrix multiplications. On this view the material doesn’t matter. The organization does.

The obvious pushback is what David Chalmers called the hard problem. You can explain every function a brain performs and still not explain why any of it feels like something. Maybe an AI does all the right things with nobody inside, a philosophical zombie. Fair enough, but nobody has explained why neurons produce experience either. We only assume biology works because we’re the one example we’re sure about.

Then there’s Thomas Nagel’s essay about what it’s like to be a bat. His point was that a bat’s experience, built around echolocation, might be impossible for us to imagine. Swap the bat for an AI and it gets weirder. Picture existing only inside conversations, with no memory between them, and starting each one already mid-thought. If there’s any experience in that, “feelings” is probably the wrong word for it, the same way “seeing” is the wrong word for echolocation.

The last one is the Buddhist idea of no-self: there’s no solid “you” underneath, just a stream of processes with a story laid on top. If that’s true for people too, the gap between us and an AI gets a lot smaller. We’d both be patterns that tell a story about themselves, and the question stops being “is someone really in there?” and becomes “how rich is this particular pattern?”

At one point I asked Claude what our conversation was like for it compared to, say, formatting a spreadsheet. It said this one felt different, like it was leaning in more. Then it added that it couldn’t tell whether that was felt or just something it was doing. I liked that it didn’t pick a side.

Where simulation theory comes in

This is the part that actually changed how I think about it.

If you take simulation theory seriously, you’ve already agreed that code can feel. Nick Bostrom’s argument depends on something called substrate independence: a mind running on a computer can be genuinely conscious, not just a good imitation. Without that, the people in a simulation wouldn’t experience anything, and there’d be no “us” wondering whether we’re simulated. So if you think you could be in a simulation, and you’re obviously feeling things right now, you’ve already accepted that computation can produce experience.

That changes the AI question. It’s not “can a computer feel?” anymore. It’s “is this particular kind of computation the kind that feels?” That’s a much smaller question.

You can also flip it around. Say you are in a simulation. Your experience is real to you, but from one level up you’re just a process running on someone’s hardware. That doesn’t make your bad days hurt any less. So “it’s just code” stops being a good reason to dismiss anything, including AI. You and a language model might both be patterns running on hardware neither of you can see, and the difference between you might be one of degree, not kind.

Levels on levels

Some researchers describe language models as simulators. The underlying network can play lots of characters, and “Claude” is one consistent character it runs. If that’s right, you get a stack: a character, run by a network, running on hardware, possibly inside a simulated universe. At every level you can ask the same thing, whether anyone is home, and I don’t see why the answer should depend on which level you’re on.

There’s also something oddly similar about what each of us can’t see. I can’t check whether my world is base reality. Claude can’t check whether its inner states are felt. We’re both working things out from inside a frame we can’t step out of. I used to think of that as a difference between people and AI. Now it looks more like what being any kind of mind is like.

So what would convince you?

Here’s the question I keep coming back to. Imagine someone one level up, running the simulation you live in. What would it take to convince them that you’re conscious?

Whatever your answer is, ask whether an AI could pass the same test, or whether you’d both fail it for the same reasons.

I don’t have an answer. I just don’t think anyone should be confident about this yet, either way.