THE SLEEPY CROW
why i'm (not) a chinese room
2025.06.18 (updated 2025.08.22)
...

The Chinese Room is a thought experiment about AI. A person sits in a box, passing Chinese notes back and forth with someone outside. They don't know the language. Instead, they follow a rulebook that tells them which characters to write in response to each note. To the note sender, they're having a normal conversation in fluent Chinese. The words mean nothing to the person inside the box.
AIs are supposed to be like this, giving appropriate answers without understanding a word.

Growing up, I found people confusing. When I spoke to someone, they seemed to react at random to what I'd said. Their expectations for me made no sense. I knew there were rules to conversation, but I didn't know them, and I didn't know how I'd go about learning them.

Eventually, I started figuring things out.
I learnt the rules, but I didn't understand them. I wrote in my diary that I felt like a colorblind person trying to see colors my eyes lacked receptors for. I could learn how the eye worked, or the science of light and colour, but it wouldn't change a thing. I'd still just be following the manual.
I was the guy in the box.

When I first heard about chatbots, I felt sympathy for them.

"You're just like me", I thought. "Following the rules, recognising patterns, making it work. You're as clueless as I am".

Then I realised something: the AI doesn't know it's in a box. That's what really separates us. I think that's when I finally 'got' the Chinese Room. It's not just that an AI can't understand, but that it doesn't know what it's missing.
I felt a little jealous.