The entire AI Consciousness debate is mired in the false assumption that consciousness is a binary, objective predicate that applies to any AI and any biological organism

The reality is that consciousness simply isn't objective and binary.

There are as many intermediate cases of "partially conscious-like" mind as there are Turing Machines.

What we name as "consciousness" is really just the set of peculiarities that are common to the minds of most healthy, wakeful, non-drugged humans.

The quest to objectively define "consciousness" in a machine will run into the following dilemma:

- if you require ALL relevant properties of the human mind, your definition will eventually converge on excluding all minds except human minds and maybe apes and digital humans like ems

- if you only require SOME relevant properties like "ability to introspect" or "causal self model" or "global workspace" or "wakefulness", your definition of consciousness will stretch to a vast landscape of minds which are almost all totally alien

The latter is a problem because humans have erroneously convinced ourselves that there's a moral meaning to consciousness specifically; that you could know everything about an entity except whether it is "conscious" and that last fact could make all the difference, morally speaking, in how you should treat it.

The vast landscape of extremely alien minds that share at least some common properties with the human mind (e.g. ability to introspect) are going to be unacceptable as a basis for "moral patienthood".

But on the other hand, people are going to be unsatisfied with defining all minds except humans specifically (and closely isomorphic things like ems) as morally worthless and not-conscious.

In the end, we are going to find that asking whether AI is conscious is a bit like asking whether a submarine or a supercavitating torpedo can swim. There's something a bit arbitrary about the definition of "swim" that sort of bakes in biological solutions to locomotion underwater in a very unprincipled way. Underwater machines have all sorts of other solutions like rotating metal parts, cavitation, magnetohydrodynamic propulsion, etc. Intuitively these are not "swimming".

We will gradually and painfully have to simply "un-ask" the AI consciousness question rather than answer it.

Very few people realize this.