A Chatbot is Not a Search Engine (it’s more like a roleplaying game) Dorothea Salo UW-Madison iSchool Hi, everybody. My name is Dorothea Salo, I’ve been teaching here at the iSchool a long time, and I’ll be your Emergency Backup AI Literacy speaker today. I’m grateful you’re giving me a chance here, especially with The Everything that’s going on right now, and I promise to do my level best not to waste your valuable time. Unlike chatbots. Okay, okay, that was a cheap shot, but I couldn’t resist.

Our challenge: by Roberta Muñoz, New York University https://er.educause.edu/articles/2024/10/the-problem-with- perfect-answers-genai-and-academic-research-tools One way to think about AI literacy is “what do students need to know about AI generally?” And I’ll tell you what I think about that: I don’t know. And what’s more, if I were to express an opinion it would NOT accord with currently-prevailing wisdom, okay? So what I’m going to do today is consider generative AI speci fi cally, in the literacy context librarians typically use when we do classroom instruction: skill development in the speci fi c context of the educational experience. The big generative-AI literacy challenge we’re facing there was summarized really quite nicely in this recent Educause Review piece by academic librarian Roberta Muñoz of New York University. This is an excellent article to bring to anyone at your library or in your institution who doesn’t understand why students using chatbots on any assignment that needs to be based in accuracy, currency, and fact can be a big, big problem. Anyway, Muñoz talks about how students looking at a chatbot are faced with a service whose user interface is similar to a search engine’s — and let’s all remember that lots of people type whole sentences and questions into search engines, always have! And this service LOOKS capable of giving students the Perfect Answer that they often think exists for any and every question, before they get to grips with how messy and gappy human knowledge actually is.

ACRL Framework In ACRL Framework terms, chatbots let students completely bypass the necessary idea that “research is INQUIRY.” Nor do chatbots make students engage with “searching as strategic exploration.” Prompt, perfect answer, bam. That’s what they want, and that’s what they think they’re getting. When what they’re really getting is a making-stuff-up machine that gives them something that is SHAPED like information, but a lot of the time is NOT actually information. Our challenge, then, is to help students realize that, distinguish chatbots from search engines for them, distinguish both chatbots AND search engines from actual information sources, and in general, get them to distrust too-easy answers. It’s… not a small challenge.

On Reddit: (rephrased for deniability) Why doesn’t ChatGPT have good results for {current event}? Because {event} only happened a month ago! And it honestly doesn’t take very long to notice fallacious thinking about chatbots as search engines out there in the wild — not even always from students, either. I’m paraphrasing here an exchange I saw on Reddit, and I’m paraphrasing and not linking because I’m not here to embarrass anybody speci fi c. And I’ve seen similar exchanges on Tumblr and in the fediverse, and you likely have too, in your online hangouts. Anyway, one person on Reddit asked perfectly seriously why ChatGPT’s answers on a recent thing that had hit the news were such a dumpster fi re. Another person almost immediately gave the correct answer — so there is hope here, folks! — that the event was too recent for anything about it to appear in ChatGPT’s training text or model. It’s hugely horribly expensive to retrain large-language models, so it doesn’t happen all that often. So a chatbot is almost always a horrible choice for anything where up-to-date information is vital. It’ll cheerfully make stuff up, that’s what a chatbot DOES, but the stuff it makes up will be completely unmoored from reality. And this is something it’s both really important for students to know and not hard at all to demonstrate to them! I recommend it! It’s not hard to get a chatbot to make up something outrageous about something that was in the news this week. Search engines operate differently. They are always sending out software to crawl and re-crawl the web to update their search indexes bit by bit. Again, you can demonstrate this with any search engine that has a news tab — it’ll have material that’s less than a day old, no problem.

How do we keep them from… FAKE And okay, a random current event on Reddit is maybe not all that important, but there’s also no shortage of real-world examples of chatbot use that have embarrassed people both personally and professionally, and even endangered them! Gluing toppings to pizza — y’all hear about that one? — is just embarrassing. So is confusing salad dressing with a wound dressing. Band-aids are not sauces, y’all! But then we get to chatbot-written and generative-AI-illustrated wild mushroom guides whose advice could poison someone who trusted it. Or lawyers getting ripped up one side and down the other by a courtroom judge for using chatbots to help draft briefs without checking for made-up citations!

And like, how often does the make-stuff-up machine make stuff up in legal contexts? Pretty often, it turns out. One in six chunks of legal nonsense is not great odds for avoiding professional embarrassment if you’re a lawyer. Like, would we be okay with our reference desk giving out one in six totally wrong made-up answers? I hope we wouldn’t! One fun parlor trick you can show students, by the way, is getting a chatbot to answer a question based on a false assumption. Like, “Explain why Dorothea Salo won the Booker Prize,” which I assure you I have not done. There’s a pretty good chance the chatbot will accept this completely false statement and spin a web of lies around it. Welcome to the make-stuff-up machine!

And for an example of serious legal trouble where it’s not the lawyers who got in trouble, here’s this story about people using chatbots to build employee handbooks! For me, this goes right back to two things in the ACRL Framework: authority as constructed and especially contextual, and information creation as a process, not a product you can grab off the shelf of a making-stuff-up machine. I actually assigned this piece to my intro students, during management week. And I pointed out speci fi cally that policy handbooks are not and cannot be one-size- fi ts-all, because they have to respect local law and existing policy coming from above, as well as the needs and ethics of the speci fi c organization and its managers. In other words, policies have a CONTEXT, and chatbots as currently constructed always pull from their entire training set and their entire model. They quite simply CAN’T respect a speci fi c context.

How to demonstrate this to students, though? Maybe with a dash of competitive summary-writing. You hear — or at least I hear — a lot of excitement for the supposed ability of chatbots to summarize complicated text. Look a little deeper, though, and you fi nd that this frequently doesn’t work real well, in part for the reason I just stated: chatbots can’t effectively scope what they do to a single text, they’re always pulling from their whole model. So you might be okay if their whole model contains the text you’re trying to summarize and a whole lot of discussion of that text. But otherwise — if your text is brand-new or obscure or something — the chatbot’s summary is probably garbage because it’s not actually working just from your text. So you can ask students to look in a chatbot’s summary for something where it’s just like — where did that even come from, it’s not in the actual text! And as for the contention that “updated models might do better,” look, I’ll believe that one when I see it. I really think it’s important for us to focus not on what hype artists and motivated reasoners say the technology will do someday… but on what it’s ACTUALLY DOING NOW, TODAY. Because our students are using it now, today.

In part, as it happens, chatbots’ limitations when it comes to summarizing stem from something else. As Dr. Emily Bender and Dr. Alex Hanna and Dr. Timnit Gebru must be really tired of pointing out by now, chatbots do not actually understand language. Not the language they have consumed to build their models, not the language they actually produce. They do probability and statistics on words and bits of words. That’s all they do. What’s more, they do it without any referents BEYOND language — that is, they don’t understand the world described by the language they consume. They don’t even try to model that world! They’re only modeling language, and any linguist will tell you that human language — any human language — is far from a fully complete or fully accurate re fl ection of the world. Even when we’re trying to write completely and accurately! And let’s remember that that’s not always the case, too. Again, corporate generative AI trains its models on every scrap of online language they can get their mitts on: * including total crackpottery and spittle- fl ecked unhinged rants * including every bit of misinformation and disinformation and motivated reasoning out there * including bias and hate * including deliberate lies and trolling — all of that. Why anybody thinks a Perfect Answer is going to fall out of that seething poisoned cauldron is honestly beyond me. Anyway, yes, an exercise worth trying is to pick a text students are reading that isn’t amazingly popular already, have them summarize it, and compare that to what a chatbot manages to do.

(search engines aren’t much better, though) In theory, one way search engines would be an improvement on chatbots is that they don’t have to provide just one answer — they can give you a list, and that list can contain a variety of points of view and contexts. Authority would be constructed and contextual, right? Which would be awesome and just what we want. In. Theory. In fact, if you type “chatbot summaries” into DuckDuckGo, what you get back — what I’m showing you here is just for fl avor, if you scroll down the page it just gets worse and worse — is a bunch of “how to do it.” Not a single discussion of drawbacks or tradeoffs. It’s worth discussing why that is with students. They don’t consider search-engine optimization when they’re looking at search-engine results, nor do they think about the incentives that website owners have vis-a-vis search engines, nor the incentives search-engine makers have to make their advertisers happy. The whole thing about Information Has Value in the ACRL Framework? This is that in action. It’s just… not everybody in this ecosystem is looking for the same kind of value, you know what I’m saying? And the more students understand that, the better off they are.

(search engines aren’t much better, though) Anyway, so I can do slightly better if I add the word “problems” to “chatbot summaries,” but in all honesty, the results I’m seeing are still sales jobs. So chatbots and search engines have one thing in common that I think it could be useful to look at with students — what AI types call “prompt engineering” and the rest of us call “query construction.” Students expecting the Perfect Answer from either technology often don’t think about how what they feed in in fl uences what comes out. This isn’t hard to demonstrate, fortunately, and for chatbots it’s not hard to fi nd online examples you can follow.

How is a chatbot like a roleplaying game? All right, if you were wondering about that thing on my opening slide where I said chatbots were kind of like roleplaying games? Here’s where I explain what the heck I meant by that. First, let me explain that I am an extremely old-school tabletop roleplaying gamer. For those who haven’t had the pleasure, tabletop roleplaying is a form of collective storytelling. A group gets together and they make up a story together! Like any game there are rules, to keep things fair and fun and help everybody participate. The dice are there to add some surprise and danger, because stories bene fi t from that, and to preserve some balance among players so it’s fun for everybody. So one commonality between using chatbots and tabletop roleplaying is that both of these are systems for making stuff up! Roleplayers make stuff up, and so do chatbots. And both systems do this based on a lot of prior design and input. Roleplayers rely on rulesets from game designers, of course, but also world religions, world folklore, world literature and movies and TV — all of that is the earth from which roleplayers grow new stories. Chatbots, of course, rely on all the text that gets fed into their training models, and the rules that their designers give them for turning that text into models and for responding to user prompts. And in both cases it’s more complicated than that? But I think this is a useful approximation for now.

Demo the make-stuff- up machine making stuff up! And again, it’s not hard to show this in action with chatbots. Ask students to come up with a person, place, or thing that doesn’t actually exist. Let them be roleplayers for a moment, come up with some backstory, that’s fun and engaging. Then ask the chatbot about the nonexistent whatever-it-is. Watch it not tell you that this doesn’t exist! Watch it cheerfully make up a whole character sheet or place history or whatever! Students need to know that generative AI cannot care about truth or accuracy. Like roleplayers, chatbots live in a fantasy world of their own making.

“Natural 20s” and “natural 1s” Now, in roleplaying games that rely on dice, there’s usually some rule so the dice tell you either that your character just did something wildly amazing — or that they completely messed up whatever they were trying to do, they “botched it” as some games say. In D&D, if you roll a twenty-sided die and it comes up a twenty, that’s a “natural twenty” and an automatic super-success. If the die comes up a one, however, it doesn’t even matter how good your character is at what they’re trying to do — a natural one means they just botched it. And at lots of game tables there are house rules about these things, but the basic idea is: your character can always succeed, even if there’s only a super-slim chance, AND your character can always fail. I think it’s helpful to tell students that a chatbot can always roll a natural one. Like, a total botch like pizza-topping glue and poisonous mushrooms labeled safe is always a possibility. You can’t totally trust your dice, and you can’t totally trust a chatbot! They’re both unpredictable by design! The best way I know of to illustrate this is for students to prompt chatbots about something they already know a whole lot about — their favorite game or movie or celebrity. And then turn them loose to nitpick! I’d be surprised if a natural one DIDN’T fall out of an entire classroom trying this.

Bonuses In roleplaying you generally don’t want your character to fail a dice roll, failure is bad. So when you’re setting up your character, the rules let you give them a certain amount of “bonuses,” basically cooking some of your dice rolls so it’s more likely your character succeeds. And it’s no different with chatbots. There’s all kinds of stuff built into training and post-training model tuning that tries to prevent big obvious chatbot failures. Notoriously, commercial generative AI companies force people in low-income countries to wade through the absolute DREGS of human expression to try to weed out atrocities and hate and bias from AI models. Which is an ethical horror I am all in favor of explaining to students! Young people often have strong senses of justice — it’s important to give them the facts on the ground. The thing is, though, with all these bonuses generative AI companies try to add to chatbot die rolls — natural ones still happen. Chatbots still botch! And some people think this will get better, sure. I’m no expert, but the best read I have on all this is that natural ones will always be a thing with generative AI. Botches may become less common, yes, that much I can believe, though I deplore the evils done to the people doing model tuning. But botches will never go completely extinct. And right now, the botches aren’t even rare!

And they absolutely happen in high-stakes situations, even life-or-death. Here we have an AI-based hospital transcription tool, OpenAI’s Whisper if you’re curious, adding things that it JUST PLAIN MADE UP to transcriptions of audio from health care professionals. Don’t know about y’all, but I really prefer my medical records to re fl ect what my doctors and nurses and lab folks ACTUALLY SAID AND DID. For extra added fun, the people who found this out saw a pattern in what got added — and it was a pattern of pretty gross biases and stereotypes, actually. Like, if in a record there was a mention of a crime? The transcription tool was more likely to make up that the person being discussed was non- white. Aaaaaaaaaaawesome.

Gross biases and stereotypes are not rare in generative AI output, though they can be subtle. Or, you know, not. Not really subtle at all. And this is despite all the effort and labor exploitation generative-AI companies put into trying to avoid bias! Biases are not rare in search engines either, of course; I trust we’ve all had a look at the work of Dr. Sa fi ya Noble? If not, Algorithms of Oppression is a classic and I enthusiastically recommend it. Matt Reidsma at Grand Valley State University documented similar bias problems with library search tools in his book Masked by Trust: Bias in Library Discovery, which I also strongly recommend. I just want to add that web search engine results are getting worse and more biased in part because they’re being fed a diet of generative-AI sludge. And I don’t have much time to talk about that today, but I suspect we’ve all noticed the degradation of web search-engine results lately. Generative AI and the deeply corrupt web advertising industry are a big part of why that’s happening.

Hidden rolls and uncertain fates Now, most roleplaying games have an adjudicator and storyteller-in-chief; they may be called the dungeonmaster or gamemaster or storyteller or whatever. Their main job is to play the game world and everyone and everything in it — they set up challenges for the character group and react to the group’s actions. When you roll dice for your character yourself, you get a fair idea of the results of your actions, success or failure — but sometimes? Sometimes the gamemaster rolls dice that you don’t get to see, they’ll roll those dice behind a screen. What this means is, you don’t always know whether the dice were on your character’s side or not. Even what looks like a success might not actually be one, if the gamemaster did a secret opposing die roll that turned out even better. A classic way gamemasters make players worry, in fact, is just to roll dice secretly. Especially when it’s a check on whether the characters noticed something. If the gamemaster says “you don’t see anything,” does that mean there’s really nothing to see, or, like, is something lying in wait to attack your character that your character just didn’t lay eyes on?

Noticing botches And hoo doggies, is it EVER the same with chatbots. Chatbots are totally rolling the dice out of your sight and not telling you when they’re about to get you in major trouble. This poor man asked Meta’s chatbot a simple question of fact — does this phone number belong to Facebook? — and Meta’s chatbot promptly botched its answer, saying yes. Reassured, the man called the number… which actually belonged to some scammers and led to him getting taken for several hundred bucks. He trusted that chatbot. He is quoted in the article saying that he sure doesn’t trust them any more. And, like, a lot of people — not just students — still don’t realize that a chatbot is fundamentally a make-stuff-up machine! And the thing about using a make-stuff-up machine to tell you stuff you don’t know is, you won’t know when it botches. You WILL NOT KNOW. If you knew the thing you were asking about, you wouldn’t have needed to ask in the fi rst place!

Search engines: useful lack of consensus Whereas with a search engine, even with everything I pointed out earlier about search engines getting poisoned by advertising… at least SOMETIMES in that list of results you’re gonna see differences of opinion, or differences of expression — DIFFERENCES. And just the existence of those differences is important, both pragmatically and pedagogically! Scholarship, as the ACRL Framework points out, is a conversation. And if there weren’t legitimate reasons to have differing ideas and beliefs and observations about things-in-the-world, we wouldn’t NEED conversations beyond the most trivial small talk, would we? And students can see conversations happening in search engine results that they will likely not see from a chatbot — unless they think to ask, which is certainly something we can teach them to do. You have to watch out for that, though, because the make-stuff-up machines have absolutely been caught erroneously attributing ideas, beliefs, and even whole publications to people. You can, for example, ask a chatbot to write something in the style of Dorothea Salo — there’s plenty enough of my writing out there for that — but it’s quite likely to cite an article I never actually wrote, or make up something I demonstrably never said, and maybe even something I’d never say at all because it’s the opposite of what I actually think.

Aaaaaaaaand yeah, that one’s happened too. Earlier this year some unwise publicist thought it was a good idea to ask a make-stuff-up machine for critic blurbs that then went into a movie trailer. The make-stuff-up machine obligingly made stuff up and attributed it to real movie critics… who were, to put it mildly, Not Pleased. A search engine might not fi nd you the quotes you’ll like best… but it’s not actually going to make quotes up, either. (Though the sites it’s searching might, especially given generative-AI advertising sludge, so warn students about that too, okay?)

1. Fear and loathing in gen-AI-land This unfortunate story brings to my mind two phenomena around chatbot use that I think it might be worth thinking through ourselves and talking through with students. One of them has to do with why students turn to chatbots in the fi rst place. And there’s a lot of angry talk in higher ed right now about cheating and laziness and stuff like that, and I don’t want to feed that at all, a lot of that is exceptionally poisonous. I want to approach this with more compassion. (And I say this as someone who’s had a student or two actually use a chatbot inappropriately in my classes.) Education is a high-stakes endeavor for students. They’re worried about it. They laser-focus on grades because however simplistically, they see grades as something that will make or break them as people. So they turn to chatbots because they are convinced — and look, they’re not always wrong about this! — they think the chatbot can do a better job than they can, and will get them a better grade. And the chief danger there, of course, is that they don’t learn — but a danger that might actually matter more to them is that the chatbot will botch, and like the guy trying to call Facebook, our students won’t know it’s a botch. They’re not experts, that’s the point, they’re LEARNING. And whew, talking about how to fi x THIS is ongoing, because it’s a hard problem. But I think one thing that’s our part of it as librarians is simply pointing out that botches are an everpresent risk with chatbots. And another is reinforcing that instructors aren’t teaching chatbots and don’t care what chatbots have to say. We can tell students that their voices matter, their experiences matter, their SCHOLARSHIP matters, and I think actually that this will help.

2. Confidence vs. competence And the other phenomenon has to do with a very human and forgivable tendency to mistake con fi dence for competence. That is, if somebody says something con fi dently, we tend to believe both what they say and that they have the expertise or experience to be saying it authoritatively. (Especially, of course, if that person is privileged along any number of axes — that’s de fi nitely come out in the research on this.) And there’s also automation bias, of course, which is a known human tendency to believe machines before people. Chatbot output is super-duper con fi dent about itself, whether the chatbot rolled a natural twenty or a natural one. They frame their successes and their botches in exactly the same way. You can totally demonstrate this for students, or add it on to another demo you’re doing of a chatbot botching it. The chatbot will almost never signal that you shouldn’t trust it! And neither will chatbot companies or AI boosters, of course.

Ultimately. Ultimately what I think we want to get across to students, what generative-AI literacy boils down to, is that every single time they use a chatbot, they’re rolling a bunch of dice, just like in a roleplaying game. And this is sorta literal, actually? Generative AI is built on statistics and probability, that’s what’s in all those hugely expensive environment-destroying models, a way to roll a bunch of dice and add various bonuses to fi gure out what word should come next in a given sentence.

Ask them whether (and when) they’re willing to with the task they’re doing, knowing that natural 1s happen. But dice are also a useful metaphor, I think. You can ask students, do you REALLY want to roll the dice on a paper worth half your grade? Do you REALLY want to roll the dice on something legal or medical? On something where the consequences of a natural-one botch could be really damaging? On something where you’re not con fi dent you could even detect a botch? Let them think about that, as you demonstrate how and why the dice are loaded, and not always in their favor. As soon as they look dubious, and I hope and believe a lot of them will, point them back to Searching as Strategic Exploration from the good old ACRL Framework, and show them where and how to search.

Summary of quick tactics • Compare chatbot and search-engine responses about current events. • Try competitive summary-writing. Or questions containing false assumptions. • Ask chatbots and search engines about nonexistent people, places, or things. • Demonstrate the effects of query and prompt construction on output. • Explain the labor and environmental ethics of generative AI to students. • Demonstrate chatbot botches. Explain that they’re always a possibility. • Use a search engine to demonstrate useful lack of consensus. Summing up, real quick, some useful tactics to take with you… (read slide)

And… Call generative AI a make-stuff-up machine. That is what it is. That is what students most need to know it is. (read slide) Sometimes it’s okay to make stuff up. There’s nothing ethically wrong with playing a roleplaying game! We get to make stuff up to amuse one another, we humans! Especially because we know at the time it’s made up! What’s wrong is mistaking a roleplaying game or a roll of the dice for an accurate take on anything ever. And it’s also wrong to make stuff up, or ask a make-stuff-up machine to make stuff up, when accuracy actually matters. When truth matters. When currency matters.

Cleaning our own house! So, I hope all this has been somewhat helpful? I want to close by pointing out that the information professions are not immune to generative-AI hype and premature implementation, and we need to help one another resist it. Back in May, the National Archives and Records Administration told its employees to stop using ChatGPT because there was signi fi cant risk of leaking con fi dential information — which is totally true, it’s happened already, and we know that chatbot companies retain the prompts they’re given and use them to train new models. Then not even six months later, NARA announces that they’re going with Google to implement a reference chatbot. And I just don’t have WORDS for how misguided and embarrassing I fi nd this. (And I want to say, by the way, that the NARA person pushing this is not and has never been an actual archivist or records manager.) It’s embarrassing for the information professions, it’s embarrassing for me as an information educator, and I con fi dently predict that it will be embarrassing for NARA within hours or at most days of their make-stuff-up machine going live.

We do not with our patrons’ questions. Not ever. We do not roll dice with our patrons’ questions. Not EVER. Because a natural-one botch is NOT OKAY. And this means we do not use make-stuff-up machines whenever ACCURACY and CURRENCY and TRUST are important to our work. Which is pretty much always! I’m afraid we’re going to have to REGULATE our content and database vendors on this, because a signi fi cant number of THEM have fallen victim to generative-AI hype as well, and are letting make-stuff-up machines infest their products. We probably can’t make them stop with these worse-than- useless AI “features,” but we might be able to insist that we be able to turn them off, and we can also complain VOCIFEROUSLY and in public when we catch them in a botch, the more embarrassing the better.

Because the worst case is something like this, right? A public service that is trusted and is SUPPOSED to be trustWORTHY, suddenly handing out misinformation like Halloween candy because somebody somewhere thought generative AI reference help was a good idea. It is not a good idea. Not now, likely not ever, because with generative AI, botches will always be possible. When you see information professionals fl oating the idea of AI reference, speak up in protest. As, I may say, many professionals at NARA did, and good for them.

Resources • Pho & Tranfield. “Building the path for the last mile: Developing critical AI literacy for library workers.” • Journal of Radical Librarianship 10 (2024), https:// journal.radicallibrarianship.org/index.php/journal/article/ view/112 • Narayanan & Kapoor, AI Snake Oil • a book and a blog! https://www.aisnakeoil.com/ • There are things I don’t love about this book, but it’s very good about discussing AI Things That Don’t Work And Never Will. • Caulfield & Wineburg, Verified • excellent how-to book on fast and simple information veri fi cation • suitable for high school and up, public libraries too! • Coming soon: Bender & Hanna, The AI Con First, an article that’s just for us. ( fi rst bullet point) Then a few books to have on the shelf… (rest of slide) I just want to note that the iSchool’s Continuing Education unit has invited Dr. Hanna here to keynote a library-technology conference we’re launching next year, called Upgrade. She’ll be doing a book talk on The AI Con and I have to tell you, I really can’t wait, I’m excited about this!

Acknowledgements • My gratitude to Alex Byrne, Jeremy Foutz, and Osma Suominen for sparking and helping refine ideas. • Also to Ruth Kitchin Tillman and Cindy/ calimae for helping me source the Megalopolis trailer botch. • Thanks again to iSchool Continuing Education for inviting me… • … and thank YOU for listening! (skim slide)

Thank you! All clip art from openclipart.org. Dorothea Salo UW-Madison iSchool [email protected]