About Saturday morning cartoons: no, that's why God invented the Internet, so there would be a mega-tsunami of kiddie cartoons available, nay, unavoidable day and night every day of the year. Hofstadter is definitley a trip. I haven't spoken with him in quite a while, but he really did once spend 45 minutes explaining to me why he didn't have time to read the book I wrote (Cheap Complex Devices) as an exploration of some of his ideas.
I'm still working my way through your archives, so apologies if I missed this (or have yet to encounter it), but the mention of cosmic horror tickled my brain into recalling the AI as shoggoth meme that has been around the block a few times:
"In a nutshell, the joke was that in order to prevent A.I. language models from behaving in scary and dangerous ways, A.I. companies have had to train them to act polite and harmless. One popular way to do this is called 'reinforcement learning from human feedback,' or R.L.H.F., a process that involves asking humans to score chatbot responses and feeding those scores back into the A.I. model."
"Most A.I. researchers agree that models trained using R.L.H.F. are better behaved than models without it. But some argue that fine-tuning a language model this way doesn’t actually make the underlying model less weird and inscrutable. In their view, it’s just a flimsy, friendly mask that obscures the mysterious beast underneath."
"@TetraspaceWest, the meme’s creator, told me in a Twitter message that the Shoggoth 'represents something that thinks in a way that humans don’t understand and that’s totally different from the way that humans think.'"
"Comparing an A.I. language model to a Shoggoth, @TetraspaceWest said, wasn’t necessarily implying that it was evil or sentient, just that its true nature might be unknowable."
"'I was also thinking about how Lovecraft’s most powerful entities are dangerous — not because they don’t like humans, but because they’re indifferent and their priorities are totally alien to us and don’t involve humans, which is what I think will be true about possible future powerful A.I.'"
Thank you for your thoughtful comment. I will follow up on both of your recommended readings.
About Lovecraft: I read with interest yesterday this post by Nnedi Okorafor about Lovecraft's racism, and in particular what it's like for her, a Black woman, to be the recipient of the World Fantasy Award, and its statuette, a stylized head of Lovecraft: https://x.com/Nnedi/status/1701265856997785866?s=20
I also hadn't read the post on the Waluigi Effect. Thanks for that, and two thoughts:
1. On the topic of tropes, the Flattery Stage of prompt engineering sounds a lot to me like spiritualists summoning spirits or witches summoning demons or other supernatural beings. You have to make the proper incantation to get the spirit you want, THEN you pose your question or make your request. Heaven help you if you get your incantation wrong and summon the wrong being!
2. I do a good bit of work with marrying AI/ML systems to automated systems to support mental health. (I'm actually procrastinating on a project right now where I need to marry a contextual multi-armed bandit reinforcement learner with an SMS-based dialog system.) There's a TON of interest in using LLMs in the context of mental health support and making services more widely available, but there's not a lot of awareness about the Waluigi Effect, and I can see that becoming a MAJOR issue in these systems where you have automated LLMs providing psychological support to very vulnerable people, and to have the good Luigi model slowly replaced with the mischievous Waluigi seems on-par with hiring a sociopath as your therapist. I'm going to read the post again when I have some more time to fully appreciate the subtleties of the phenomenon, and I'll likely start circulating it among some of my clients with some notes why they shouldn't be so keen to embrace into this technology if this effect is a structural one that no amount of "good data" or well-meaning human feedback will steer in a safe direction.
On Lovecraft's racism, it's been a problem going back to when he was still living. For an interesting take on it from other PoC writers seeking to invert Lovecraft's racism and use his creatures and milieu to combat those prejudices in the modern day, check out this episode of the podcast "Imaginary Worlds":
Matt Ruff's "Lovecraft Country" (made into an HBO series a few years ago) was also an interesting look at how Lovecraftian cosmic horror was not all that different than the horror American Blacks dealt with in the Jim Crowe era where the systems and people arrayed against them were just as unreasoning and implacable as any of the beings from Lovecraft's pantheon.
I imagine a dinner with The Hof(stadfter) would result in a sort of thought hangover.
One might treat it with careful doses of Saturday morning cartoons (do such things still exist?).
About Saturday morning cartoons: no, that's why God invented the Internet, so there would be a mega-tsunami of kiddie cartoons available, nay, unavoidable day and night every day of the year. Hofstadter is definitley a trip. I haven't spoken with him in quite a while, but he really did once spend 45 minutes explaining to me why he didn't have time to read the book I wrote (Cheap Complex Devices) as an exploration of some of his ideas.
I'm still working my way through your archives, so apologies if I missed this (or have yet to encounter it), but the mention of cosmic horror tickled my brain into recalling the AI as shoggoth meme that has been around the block a few times:
"In a nutshell, the joke was that in order to prevent A.I. language models from behaving in scary and dangerous ways, A.I. companies have had to train them to act polite and harmless. One popular way to do this is called 'reinforcement learning from human feedback,' or R.L.H.F., a process that involves asking humans to score chatbot responses and feeding those scores back into the A.I. model."
"Most A.I. researchers agree that models trained using R.L.H.F. are better behaved than models without it. But some argue that fine-tuning a language model this way doesn’t actually make the underlying model less weird and inscrutable. In their view, it’s just a flimsy, friendly mask that obscures the mysterious beast underneath."
"@TetraspaceWest, the meme’s creator, told me in a Twitter message that the Shoggoth 'represents something that thinks in a way that humans don’t understand and that’s totally different from the way that humans think.'"
"Comparing an A.I. language model to a Shoggoth, @TetraspaceWest said, wasn’t necessarily implying that it was evil or sentient, just that its true nature might be unknowable."
"'I was also thinking about how Lovecraft’s most powerful entities are dangerous — not because they don’t like humans, but because they’re indifferent and their priorities are totally alien to us and don’t involve humans, which is what I think will be true about possible future powerful A.I.'"
(NYT Gift Link) https://www.nytimes.com/2023/05/30/technology/shoggoth-meme-ai.html?unlocked_article_code=AHkkKMWX5sfr2Ra_rBmg8V1lXAR4qNx0u0MI9yMIoosYuwyxSR622jIDwlbIRuMAn01rwtE30hzia6lbH8Z-gbt6-BD_mvDnukhNA_7-_TESvJ6aJZwLgzSnvrirLQjdOTpvGMhkWrjRXiBHuCjevErFvW4_Sns9yd8MrcEGpnqurxLSSOlVuZRvj_wb_mYaWZq9Mf3CyRqoM_NIVGHJ7-JsQJrlT4OdDS6ulwbgm1_wOhiT6dagveXzZapWwKlGDycddRqhA_RJYZkpG7AtxmizbE9jEZj9hnQpNwl6X90ITEA4-bkV8jR62wtoqulzcIANQaKhCAxY0fFjkg&smid=url-share
And if you haven't read Henry Farrell's extended commentary on the "LLMs as shoggoths" idea, you're missing a treat:
https://programmablemutter.substack.com/p/shoggoths-amongst-us
Thank you for your thoughtful comment. I will follow up on both of your recommended readings.
About Lovecraft: I read with interest yesterday this post by Nnedi Okorafor about Lovecraft's racism, and in particular what it's like for her, a Black woman, to be the recipient of the World Fantasy Award, and its statuette, a stylized head of Lovecraft: https://x.com/Nnedi/status/1701265856997785866?s=20
I expect that you've read the ur-post on the Walugi Effect, but if you haven't you too-also are in for a treat: https://www.alignmentforum.org/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post
I also hadn't read the post on the Waluigi Effect. Thanks for that, and two thoughts:
1. On the topic of tropes, the Flattery Stage of prompt engineering sounds a lot to me like spiritualists summoning spirits or witches summoning demons or other supernatural beings. You have to make the proper incantation to get the spirit you want, THEN you pose your question or make your request. Heaven help you if you get your incantation wrong and summon the wrong being!
2. I do a good bit of work with marrying AI/ML systems to automated systems to support mental health. (I'm actually procrastinating on a project right now where I need to marry a contextual multi-armed bandit reinforcement learner with an SMS-based dialog system.) There's a TON of interest in using LLMs in the context of mental health support and making services more widely available, but there's not a lot of awareness about the Waluigi Effect, and I can see that becoming a MAJOR issue in these systems where you have automated LLMs providing psychological support to very vulnerable people, and to have the good Luigi model slowly replaced with the mischievous Waluigi seems on-par with hiring a sociopath as your therapist. I'm going to read the post again when I have some more time to fully appreciate the subtleties of the phenomenon, and I'll likely start circulating it among some of my clients with some notes why they shouldn't be so keen to embrace into this technology if this effect is a structural one that no amount of "good data" or well-meaning human feedback will steer in a safe direction.
Alright, back to work for real this time!
On Lovecraft's racism, it's been a problem going back to when he was still living. For an interesting take on it from other PoC writers seeking to invert Lovecraft's racism and use his creatures and milieu to combat those prejudices in the modern day, check out this episode of the podcast "Imaginary Worlds":
https://www.imaginaryworldspodcast.org/episodes/inverting-lovecraft
Matt Ruff's "Lovecraft Country" (made into an HBO series a few years ago) was also an interesting look at how Lovecraftian cosmic horror was not all that different than the horror American Blacks dealt with in the Jim Crowe era where the systems and people arrayed against them were just as unreasoning and implacable as any of the beings from Lovecraft's pantheon.