Sir Roger Penrose, Nobel Laureate and Emeritus Rouse Ball Professor of Mathematics at Oxford, on whether AI actually knows what it's doing:

His answer is no, and he doesn't hedge it:

"The point is it doesn't know what it's doing. That's still true with AI. You talk to an AI thing, it doesn't know what it's doing."

So what is it doing? According to Penrose, it "speaks from experiences, if you like."

He is precise about what that means in practice:

"You can take a mass of data and... you can analyze that data and you can see what the data says and does this thing in... accordance with previous things in the data or is it not and AI that's all it does."

He doesn't dispute that this capability is enormous:

"Sure, computers have got so powerful now that they can do things way beyond what humans do in computing."

But he argues that the power itself is why so many people believe these systems understand more than they do:

"People have confused the story. The story is they've lost it in the power of computing. The thing is that computers have got so powerful that they've lost the thread of what they're doing."

Penrose connects this to Turing and the test named after him. He describes that test as a conversation:

"How do you decide whether an entity is conscious or not? Will you have a conversation with it? You talk to it."

He thinks "Turing wasn't too far off," though he adds that Turing was "a little bit confused person in a way because... it was not clear what he believed, but he did call this thing the [Turing] test."

So when it's suggested that AI might be "a new sphere of human existence," @penrose answers immediately:

"No. It's missing something I think. Yeah, that's the trouble."

That doesn't make AI useless in his eyes. He simply warns against letting its usefulness blur what it actually is:

"Obviously it has a role to play. I'm not going to say that. Clearly, it has a role to play. It has a role, but you... mustn't be confused. The fact that it has a role to play. It's to do with the power of computation."

So does AI know what it's doing? Not according to Penrose.

It can take a mass of data, analyse it, and check it against previous patterns, and computers can do things far beyond what humans do in computing. You can even talk to it. But it's missing something, and it doesn't know what it's doing.

The practical lesson is to treat AI as what it is: a powerful tool for computation. Use it for that. Don't mistake its power, or its ability to hold a conversation, for awareness of what it's producing.