Chris Bell

Epistemic comparison between my field of study and LLMs

My short summary: I don't think LLMs are conscious for the same reason that I don't think that a consciousness would emerge from me doing math really fast. I use this little essay to describe this in a bit more detail by relating it to my actual professional background.

And, as a preface: If you have questions about AI, feel free to ask me. I've noticed that there are people who have questions but feel guilty or judged for their use of AI. I won't judge you. I have found some LLM-based tools to be rather useful (e.g., as boilerplate code generator, proofreader, or as a way to improve my search engine use). While my tone may be derogatory at times, it is towards some of the unfounded hype around AI and not to any person.

I work on random number generators, specifically those which produce 'true' randomness. Another way of saying this is that I look at how to quantify the uncertainty that would persist if we knew the state of the entire universe and had the ability to simulate the future.11This makes it sound much more impressive than it is, I promise. To evaluate a random number generator, I have to examine the procedure in which the numbers are generated. Looking at the output sequence tells you nothing epistemically.1This makes it sound much more impressive than it is, I promise.

I have a question for you: Of the following two numbers, which is random?

  1. 6187298507
  2. 8039746120

The first is not random;22I've heard the trope that "the digits of pi are random!" This is a definition of random I don't ascribe to. The number pi is well-defined (the distance between consecutive real zeroes of sine) and the Chudnovsky algorithm can be used to calculate a large number of digits of it. There's no randomness here, you just need to know which digit you want to start at and how many you want. If you know those two things, you get the same value every time. it is the 10 digits from pi starting at index 76,339. The second is random; it comes from a random number generator based on counting the number of times a ring oscillator cycles after a certain number of CPU cycles. Without knowing how I got the numbers, you are unable to tell which is random.33Feel free to try and argue with me on this, I will win.2I've heard the trope that "the digits of pi are random!" This is a definition of random I don't ascribe to. The number pi is well-defined (the distance between consecutive real zeroes of sine) and the Chudnovsky algorithm can be used to calculate a large number of digits of it. There's no randomness here, you just need to know which digit you want to start at and how many you want. If you know those two things, you get the same value every time.3Feel free to try and argue with me on this, I will win.

I use this example to highlight the very real difference between 'cause' and 'effect.' Describing the process (i.e., the cause) used to manifest an outcome (the effect) is the ultimate goal of science. The scientific method approaches a description of the mechanisms of the world through observation, hypotheticals, and negative testing.44There are multiple ways to 'gain' knowledge, and there are multiple 'types' of knowledge. Science isn't the only way, it just seems rather effective. Unfortunately, the language of science is used to make claims seem more valid than they actually are.4There are multiple ways to 'gain' knowledge, and there are multiple 'types' of knowledge. Science isn't the only way, it just seems rather effective. Unfortunately, the language of science is used to make claims seem more valid than they actually are.

Now, with this in mind, are Claude and ChatGPT conscious? Are they intelligent?55When I think of intelligence, I implicitly have the idea that this is something that a conscious being 'has,' and, as such, intelligence as a concept is intertwined with consciousness. However, I know that this is not the only use of the term. There are people who use the term 'intelligence' to mean something different that is not 'part' of consciousness at all, but I've failed to internalize that definition and must actively disambiguate the term when reading, which is why I clarify it here. To both of those, I don't think so, but I'm less interested in you knowing my belief66I use 'belief' and not opinion here because I treat 'beliefs' as an expression of fact based on my perception of the world. Opinions are subjective predicates which portray some emotive aspect of the person providing them. See and am more interested in you understanding why I hold it.5When I think of intelligence, I implicitly have the idea that this is something that a conscious being 'has,' and, as such, intelligence as a concept is intertwined with consciousness. However, I know that this is not the only use of the term. There are people who use the term 'intelligence' to mean something different that is not 'part' of consciousness at all, but I've failed to internalize that definition and must actively disambiguate the term when reading, which is why I clarify it here.6I use 'belief' and not opinion here because I treat 'beliefs' as an expression of fact based on my perception of the world. Opinions are subjective predicates which portray some emotive aspect of the person providing them. See

Before we talk about LLMs, I'd like to give you an example from what I'm actually an expert in.

A short digression

While I 'work in cybersecurity' in the most strict sense of the word, I am significantly more on the theoretical side of things than people I work with and for. I spend my time reasoning about the generation process and make a strict difference between a DRBG77A Deterministic Random Bit Generator and an entropy source.7A Deterministic Random Bit Generator

To make this match with the rest of the world, I will use the term PRNG, or a pseudorandom number generator, in lieu of DRBG.88A DRBG is a term of art here, outside of my field the term DRBG isn't nearly as widely used to describe deterministic random number generation.8A DRBG is a term of art here, outside of my field the term DRBG isn't nearly as widely used to describe deterministic random number generation.

A PRNG is an algorithm which takes some starting number (the 'seed material') and makes random material from it. PRNGs are used in a bunch of things. Pretty much any program or game which has a 'seed value'99Minecraft is a fun example, there are 'world seeds' which are used to instantiate the PRNG that decides what blocks are placed where. The fact that the PRNG determines all of the 'randomness' in the map is why you can use the same seed over and over and start with the exact same map. uses PRNGs. In my line of work I'm only interested in the type of PRNGs that are cryptographically secure. A cryptographically secure PRNG is one in which you can't1010'Can't' is a cost-prohibition on ability. You "could" do it, but it is extremely expensive for a computer to do so. go backwards or forwards without the seed material.9Minecraft is a fun example, there are 'world seeds' which are used to instantiate the PRNG that decides what blocks are placed where. The fact that the PRNG determines all of the 'randomness' in the map is why you can use the same seed over and over and start with the exact same map.10'Can't' is a cost-prohibition on ability. You "could" do it, but it is extremely expensive for a computer to do so.

If you've ever used a password generator, then you've come into contact with a tool that uses a CSPRNG or a cryptographically secure PRNG. The random number generated is turned into symbols for you to use as a password.1111If it weren't cryptographically secure, that would mean that you could look at prior and future outputs of the CSPRNG if you had some of the outputs already. That translates into a vulnerability in your passwords generated by that tool: knowing one of your passwords would be able to generate the rest of the passwords your tool would make.

So, cryptographically secure PRNGs, while quite important, aren't everything; that pesky seed becomes extremely important! The risk of any CSPRNG is simple: if the 'seed' used to instantiate that CSPRNG is ever exposed, then a person can predict the passwords which will be made by that password generator, thus, unbeknownst to you, silently exposing all of your account passwords generated with that seed-tool combination.11If it weren't cryptographically secure, that would mean that you could look at prior and future outputs of the CSPRNG if you had some of the outputs already. That translates into a vulnerability in your passwords generated by that tool: knowing one of your passwords would be able to generate the rest of the passwords your tool would make.

So, cryptographically secure PRNGs, while quite important, aren't everything; that pesky seed becomes extremely important!

So, for a CSPRNG the seed material is of paramount importance. The longer you use a seed, the more risk you incur. Thus, we regularly update the seed material in a CSPRNG after so many uses so that the 'randomness' of its output is preserved. If we have CSPRNGs that give random material, what gives random material 'good enough' to be seed material?

An entropy source1212Or a "true random number generator" is used when you want something unknowable to anyone at the moment of generation. Entropy sources work by generating a number from some physical1313Or, to my continued chagrin, non-physical... I'll write about that another day. phenomena that is fundamentally non-deterministic based on our current understanding of physics.12Or a "true random number generator"13Or, to my continued chagrin, non-physical... I'll write about that another day.

Interestingly enough, if I were to look at only the output of a CSPRNG and an entropy source, the output of the CSPRNG would appear by all measurements 'more random.' This is because 'randomness' is hard to define from an output-only perspective, but predictability is not nearly as difficult. We can1414And have, see NIST's SP 800-90A for cybersecurity-focused CSPRNGs that I know rather intimately. design algorithms which seem nearly unpredictable and chaotic in behavior. Entropy, on the other hand, happens due to slight non-deterministic fluctuations which compound over time and is seemingly ethereal in nature.14And have, see NIST's SP 800-90A for cybersecurity-focused CSPRNGs that I know rather intimately.

As a result, entropy is costly to gather compared to pseudo-randomness,1515Another term of art for the output of CSPRNGs so we use an entropy source to seed the CSPRNG and update the CSPRNG's seed at regular intervals.15Another term of art for the output of CSPRNGs

What do I want you to get from this? Randomness has a few definitions to look at, and the abstract and generalizable form of randomness requires looking at more than its effect on the world;1616Which, for the most part, is simply the output random number stream it requires us to look at the mechanisms behind how generation works.16Which, for the most part, is simply the output random number stream

Essentially, if we consider 'the end' versus 'the means' it took to get there,1717Abusing the "ends justify the means" idiom that I've always found silly. my profession requires me to look at and prioritize describing 'the means' as a method to describe what we want from 'the end.' Entropy exists within privileged conceptual space here, as we never default to something 'being entropic.' To be entropic is to meet a special threshold, and we assume non-entropic when we are unsure. Thus, the only way to determine if a generated sequence of numbers is non-deterministically random is to demonstrate an entropic mechanism which induces it.17Abusing the "ends justify the means" idiom that I've always found silly.

Returning to the topic at hand

LLMs are the current rage; their hype-leaders are saying the text generation procedures 'may have consciousness' and anthropomorphize an abstraction of what the LLMs do to generate text. On a strict level: the LLMs are literally next token generators. Would you describe something that does this as 'being conscious?'

The 'Turing test' is not a serious consideration in what it means to be conscious. The test is an explanatory tool to demonstrate the ability to differentiate a computer's output from a person's output. A system which "passes" the Turing test does not have consciousness; the system has simply performed in a manner in which the people acting as testers were unable to differentiate between the system's output and some other person's output in a Q&A format.

As people, we generally convey and share information with each other through language, which engenders a certain tendency to anthropomorphize any entity that we interact with through language. In addition, the chat interface that current chat bots use is particularly effective in emulating talking to another person given our text/email-driven culture,1818Basic caveat: I am American in a tech-adjacent job making it even harder to not do so. While I believe that the affect induced by talking to a chat bot is real for the person who experiences it and shouldn't be ignored, I strongly believe that this affect prevents us from thinking clearly about LLMs. This confusion is then used by the major AI companies,1919Which they self-designate as Frontier AI Labs, but that is putting gloss on a botched homunculus and calling it your date. who can really up their game with an emotional twist to those they've tricked.2020The decision makers in the company get two scapeghosts to blame! The new frontline is the AI they say is conscious. We already see this in things like healthcare, where AI is used to deny coverage, and the AI soaks up the blame for the denials.18Basic caveat: I am American in a tech-adjacent job19Which they self-designate as Frontier AI Labs, but that is putting gloss on a botched homunculus and calling it your date.20The decision makers in the company get two scapeghosts to blame! The new frontline is the AI they say is conscious. We already see this in things like healthcare, where AI is used to deny coverage, and the AI soaks up the blame for the denials.

As a result, I treat 'consciousness' as something which should be examined critically. When we don't know, we should assume non-conscious and any person making a claim that the thing they've created is conscious must overcome this burden. This is the rationale behind my pithy question: "if you do math fast enough, does consciousness emerge?"

AI "Consciousness"

If you think AI has some form of consciousness, then let me take a second to try and disabuse you of that notion.

LLM-based "AI," in the broadest sense, consists of two components:

  • The transformer, which is the 'deep learning' bit that predicts future text output after observing a sequence of words,
    • I'll refer to this as "The LLM" in any below text, as this component is the "LLM" in our AI-based chat bots: a large 'transformer'2121Which is just a fancy bit of logic model trained on language, and21Which is just a fancy bit of logic
  • The harness, which is a program which provides two interfaces for the LLM:
    1. The LLM <--> User chat interface (which I'll call the chat interface)
    2. The LLM <--> System command interface (which I'll call the system interface)

The chat interface is the messenger-like input/output window through which you and the LLM send and receive text. When you type something in, the interface prepends your text with a prompt and all of this gets fed into the LLM. The LLM then continues the text. But, how does the LLM 'continue the text?' Let me try to explain by describing how we make an LLM.

The LLM is created through a process called 'training.' An LLM is fundamentally a bunch of equations. When we first instantiate an LLM, we have a large set of equations with random numbers called 'weights' that determine the LLM's behavior. These equations can generate a number based on some input numbers. We train an LLM by having example input/output pairs and updating the LLM's equations based on how well its equations guess the output given the input. However, our objective is to predict the next word, so how can a next number predictor help with language? We need a way to represent words with numbers!

Words can be described in numbers through a process called 'embedding.' The most basic form is 'indexing,' where we list all of the words and give each a number.2222It could literally be as simple as saying 1 means "The", 2 means "cat", 3 means "purred." Then the sentence "The cat purred." turns into <1, 2, 3>.

This is far from how it works in practice, pre-LLMs we used a variety of ways to describe words and documents. One of my favorites (because it is human-interpretable) was the term-frequency interdocument-frequency (or, tf-idf) format for search engines. One of the more technically interesting ones (because semantic reasoning became embedded into the representative space) is Word2Vec.

The gist, words become represented by numbers in a way that we can both do math with and take the result to turn back into words. After this relatively simple procedure, we have a list of numbers that represent words.2323This idea (turning things into numbers to do math on) is pretty ubiquitous in the mentality ingrained in those who study computer science or statistics. And now that we have words as numbers, we can 'train.'22It could literally be as simple as saying 1 means "The", 2 means "cat", 3 means "purred." Then the sentence "The cat purred." turns into <1, 2, 3>.

This is far from how it works in practice, pre-LLMs we used a variety of ways to describe words and documents. One of my favorites (because it is human-interpretable) was the term-frequency interdocument-frequency (or, tf-idf) format for search engines. One of the more technically interesting ones (because semantic reasoning became embedded into the representative space) is Word2Vec.

The gist, words become represented by numbers in a way that we can both do math with and take the result to turn back into words.23This idea (turning things into numbers to do math on) is pretty ubiquitous in the mentality ingrained in those who study computer science or statistics.

Training here is a form of optimization to some problem. After coming up with a way to turn all of our words into numbers and back, the problem we want to optimize for is that of 'guessing the next number.' The training data is simply a bunch of text that someone has already typed out.2424This is why these AI companies have 'scraped the internet,' they needed training data for their models. A simplified way of how this training works is as follows:24This is why these AI companies have 'scraped the internet,' they needed training data for their models.

  1. Grab a sequence of words in your training set.
    1. In this case, we will use "The cat purrs."
  2. Remove the last word.
    1. So, "The cat"
  3. Turn those words into their numeric form.
    1. Using the same numbers as in footnote 22: "The cat" -> <1, 2>.
  4. Plug those numbers in to the model and get the output.
    1. If the output is the number for "purrs", then the model's weights are fine for now and we move on.2525It is also possible to 'positively' change the weights in the case of a pass. There are plenty of different ways to do this training step, I'm just giving the basic example to start with.25It is also possible to 'positively' change the weights in the case of a pass. There are plenty of different ways to do this training step, I'm just giving the basic example to start with.
    2. If the output is a different number, we update our weights a bit to make it more likely to be correct in the future.2626Which is a nudge. Essentially, we 'bend' the line a bit to be more correct, we don't 'force' it to be correct at each step. That would be 'overfitting'.26Which is a nudge. Essentially, we 'bend' the line a bit to be more correct, we don't 'force' it to be correct at each step. That would be 'overfitting'.
  5. Repeat with a new sentence of words until we aren't nudging much.

That's it. After doing this enough, we get a 'model' which predicts the next number, which we can then turn into text using our word-number relationship. But, 'doing this enough' is hiding the massive amount of writing the LLMs du jour have been trained on. What we've found out is that these models have coherent output sequences for longer durations than we had ever observed from text generation methods. This alone raises interesting questions in the fields of computational linguistics and machine learning, but it doesn't raise questions about 'consciousness' outside of the 'AI' field.

Does 'the means' better explain 'the end?'

Text generation alone isn't conscious.2727Once again, an even more readily available example is predictive text. Repeatedly click the next most predicted text in a text messaging interface on your phone, does the resulting sentence represent the output of some conscious being? The text generation process is not 'responding' to your intent or implied thoughts. It doesn't attempt to convey meaning to you. The LLMs simply generate text that would reasonably follow given the preceding text sequence. Now that you know the mechanism for the text generation, does the text generator itself seem conscious?27Once again, an even more readily available example is predictive text. Repeatedly click the next most predicted text in a text messaging interface on your phone, does the resulting sentence represent the output of some conscious being?

This isn't the first time people have been convinced that the computer is conscious. The classic case I learned about in my first AI2828It should have been called "History of AI" to be honest graduate course was ELIZA, an NLP program designed in the mid 60s by Joseph Weizenbaum to imitate a psychotherapist. The basic methodology here is to generate a question for the user based on what they had previously typed. The algorithm to decide which question to ask is essentially a mad-lib-style2929In a domain-specific programming language created for this very purpose called "MAD-SLIP" substitution in appropriate areas based on the structure of the sentence.3030Along the lines of those grammar trees you might have heard about in grade school: where we identify a sentence's subject, verb, adjectives, etc. The resulting question seemed to follow the conversation from the user's perspective. When the user responds to this question, ELIZA has a new input to generate a question from, starting the process over again. This is the behavior of shallowly listening to someone to keep them talking! Have you ever spoken with a person who 'seems' to listen and asks questions that follow precisely what you just mentioned? You 'feel' heard, and ELIZA was designed to emulate just that experience. Naturally, people developed an emotional connection to it (called the ELIZA effect).28It should have been called "History of AI" to be honest29In a domain-specific programming language created for this very purpose called "MAD-SLIP"30Along the lines of those grammar trees you might have heard about in grade school: where we identify a sentence's subject, verb, adjectives, etc.

None of that, however, makes it conscious. Like ELIZA on steroids, LLMs are just an extremely complicated set of rules to do a type of text generation. Add an interface for chatting around it and you get something which 'feels' human.3131I say human here as well... Note that none of these AI-is-conscious proponents ever compare the consciousness of an AI to a non-human consciousness (cats, for example?). However, this is appraising the output without observing the method. Given that the method is a bunch of equations guessing the next word, I ask again: if you do math fast enough, does consciousness emerge?31I say human here as well... Note that none of these AI-is-conscious proponents ever compare the consciousness of an AI to a non-human consciousness (cats, for example?).