# LLM Mindset Myth #1: AI is an ANSWER MACHINE

We’ve grown accustomed to “search engine delivery” - we request information and expect to have it provided instantaneously.

Feb 20, 2025, Ian Belknap

We’ve grown accustomed to “search engine delivery” - we request information and expect to have it provided instantaneously. With the rise of digital assistants like *Alexa* and *Siri*, we verge on a *Star Trek*-ish assumption that we can simply call out “Computer!” then bark out any question and receive the immediate (correct, we assume) answer in an affectless robotic voice. Over time, a pitfall of such an assumptive approach arises: sapped of our capacity to engage critically, we lapse into unexamined, often *baseless* faith in the results. With AI, though, an approach like this represents a fundamental misunderstanding of how AI operates and what it is meant to do - AI is not intended to function in such a “transactional completion” model, there is no one-to-one ratio of input-to-output, of query to answer. AI, and in particular Large Language Model (LLM) AI, should be thought of as *iterative*, as almost *Socratic* in its pathway to good results - it ***nudges*** along toward the precise and applicable.

It is important to keep sight of the fact that AI ***on its own*** is not a **source** of information - it **draws** upon information. It is therefore limited in important ways: the biases and blindspots of the people selecting and inputting the data, the borders - of topic, of detail, etc. - on a given subset of knowledge (because no matter how large, it is always a *subset*.) As with any information source, the quality, comprehensiveness, and precision of the outputs provided by any AI system are the direct result of how complete and correct the data going into it is. To deploy a Large Language Model well requires a reorientation away from this *Magic 8-Ball* mindset in favor of a more reciprocal one. When provided with appropriate conceptual and rhetorical parameters, LLM AI is extraordinarily adept at helping us to parse problems and *draft* (not “declare”) solutions.

It is true that LLMs are mind boggling in their speed, and can draw upon vast troves of public (and, regrettably, private) information sources for material, **but it lacks the inborn and sophisticated meaning-making so natural to humans**. It is better to think of LLM AI as an *intern* with *access* (not actual *comprehension* in any functional sense) to all of human knowledge, but who is naive and lacks sound instincts. *AI is not some opaque fog out of which emerges apt responses to our questions - it doesn’t know everything; it knows what we instruct it to know. An LLM is not a librarian, it’s an interface -* or, more precisely, *if the* LLM is a librarian, it is rookie capable of a basic set of tasks, not the sophisticated mastery of the seasoned polymath: the LLM has no sense of what it does not know; an expert human librarian will know that a title is not in the collection, and can direct you to another institution that’s likely to have it - AI can’t do anything resembling this*.*

In a general sense, it is fitting to think of any Large Language Model not as some know-it-all colleague, as some of us seem to, but as as a *machine*. And like any machine - a drill press, a drawbridge, a cash register - it is **well suited to performing the finite set of operations of its intended use**. It’s perhaps better to conceive of an LLM as a ***QUESTION* Machine**, NOT an Answer Machine - the LLM excels at providing us with **short-span bridges between bodies of knowledge**, or in **compiling an aggregated sense of what many fields or disciplines have to say about a subject**. If your project is to design a tugboat, the LLM obviously *cannot* do that for you, but it *can* draw upon pertinent bodies of knowledge - fluid dynamics and engine requirements and maritime law and so forth - none of which will generate renderings of a tugboat. All of which can, however, **provide a series of *evaluative filters* through which we may assess our evolving plan.**

The LLM, in short, can be tremendously capable at the ***iterative interrogation* of a project**, where we provide it with applicable information. The LLM does not and cannot *complete* any complicated project *for us*, but is excellent at **helping us ask about our own thinking about a complex objective**: What conditions and factors might I have missed in devising this project? How might I refine and improve upon the current state of this project? It is this that separates the LLM from all previous machines - instead of firing rivets or spot welding, **it can help us *think*.** A Large Language Model, where we are clear on its intended use and capabilities, can introduce a kind of distributed cognition, a kind of multi-brain, into our processes - adding a kind of research assistant that does not generate novel ideas, but provides a useful, perhaps one day even indispensable, sounding board for tinkering with our own models and methods, for nudging our practices toward “best.”

***LLM MINDSET MYTH #1A: “WHY CAN’T WE JUST RUN IT THROUGH CHATGPT?”***

Well, you can, obviously. But will doing so provide you with anything resembling what you’re looking to attain? In terms of sharpening THE means of expressing ideas or refining your understanding of the problems you’re looking to solve? Not likely.

LLM platforms like *ChatGPT* are better understood as **a series of switches** than as a vast, sentient content generator. The LLM is NOT sentient, it does not GENERATE ANYTHING, not in the At the granular level, LLM systems **function a lot more like auto-fill** than human intelligence - it’s possible to have *ChatGPT* write you a sonnet, yes, in the sense that it will follow the rules of meter and rhyme and so forth, but it will have **nothing** like the depth of feeling or insight that a human writer can achieve. **A person is able to *compose* thoughts and *express* them, in real time; the LLM *guesses* at what the next word or phrase is *likeliest* to be** - with, it’s true, a speed that’s near incomprehensible, which no doubt contributes to our inflated sense about its capabilities. **But this difference is crucially important to note - people naturally have a *holistic* approach to grasping and conveying concepts, the impressions we form and the language for making these known emerges more or less simultaneously; whereas even the most sophisticated LLM is inherently, unavoidably *sequential* and *probabilistic*.**

The peril of this unmediated “run it through” approach is not that the results of doing so will be useless (though left to their own devices, the tendency of such platforms to produce gibberish is well documented,) it’s that they will be *flat.* If in the human brain, intelligence may be said to reside in complexities of structure, in the folds and wrinkles, unaided LLM’s “thinking” might fairly be called *smooth-brained*. It will tell us *variations* of what we already know - it will produce phrases *adjacent* to the expected. Or arrive at conclusions that are naive - not incorrect, exactly, but obvious, lacking nuance. The knowledge upon which LLMs will draw is vast but shallow, effectively **the limitless inexpertise of the generalist**. What LLM is exceptionally good at, however, is not in what we think of as *cognition*, but in providing us with *cognitive assists* - we should regard these systems as **an *enrichment* tool, not as a *generative* one**. **Even for humans of the most arresting and singular intelligence, those transformative jolts of inspired, visionary thought are comparatively rare - but the *prospect* of them remains within reach. For *ChatGPT* and other LLM AI platforms, such leaps are not possible.**

Say you are an architect hired by a client to build a home outside of Santa Fe. You will bring your knowledge to the project, certainly - all the years of information and insight you’ve gathered about materials and processes, esthetics and function. So far, LLM - provided access to facts about materials and building codes and architectural technique - will more or less be able to keep pace. What such platforms *cannot do*, however, is draw upon the flashes of inspiration that come to people, or the facility we have for interpreting the mind and desires of a client, or the intuitions that arise across the life of such a project that we heed or ignore. Consequently, if we submit a request to *ChatGPT* or any comparable LLM platform to “help design a house outside Santa Fe,” anything it tells us is likely to have a “received wisdom” feel, that might be broadly apt in *principle*, but is not germane or applicable to *this* project.

Reorienting ourselves to what LLMs actually ARE competent to provide such a project would be to rightly regard it as **a component-level tool**, an interlocutor adept at helping to navigate the dozens or hundreds of branches on the decision tree of conceiving of such a project. For instance, an LLM could harvest climate data for the exact GPS coordinates of the proposed site, and provide aspects to consider in the selection of building materials, or could match green building standards and water conservation practices to the proposed structure and landscaping. In other words, AI does not, cannot, *design* in the way that we can, in the sense of *synthesizing* information and inspiration to arrive at wholly novel conceptions of how to grapple with a particular set of problems. A Large Language Model, where it has been provided the right data sets, *is* very good at helping us bridge the gaps (leap synapses, find new relationships) between various bodies (points, areas) of abstract knowledge - in other words, it is a *computational* means of arriving at the unexpected, one where each step in attaining a “thought” bears some clear relational connection to each one before and after it. AI is not well suited to shaping an overall *vision*, but is highly useful in helping to render the many this-vs.-that distinctions that constitute the *ingredients* of vision.

Where a Large Language Model like *ChatGPT can* prove truly transformational, even revolutionary help to us, is where it serves not as our motorcycle, but as a sidecar - it does not generate power, it does not steer, nor decide on destination, but it can expand our capacity. We - as the drivers - must keep clear on our own cognitive capabilities and on the augmentations that the LLM *component* can provide. It is only when we do so, where we retain a solid “who’s good at what” grasp of the nature of the dynamic between us, that these platforms may realize their fullest potential as the fastest, smartest assistant we’ve ever had.