Alignment cruxes re: "capabilities generalize farther than alignment" https://t.co/ONuYRjJDmT
I can feel the coming dark age very strongly now. The Internet was allowed to exist because it came to popularity during an optimistic and wealthy era. As we sink into penury it will be destroyed with all other good things. Poverty is a choice and the West made it a while ago. x.com/teortaxesTex/s…
@Nominus9 I'm not just talking about AI here.
@Nominus9 IDK man I'm forecasting pain here. This is going to be more of a European enlightenment or socialist internationale situation. Gonna take decades minimum to unseat the systematic stupid.
x.com/quantian1/stat…
@teortaxesTex I forgot how beautiful and furious the energy in the Yehuda eulogy was. You're right, that was the specific thing in HPMOR that made it stick, that convinced me in my gut EY was an advanced moral being beyond ordinary people.
x.com/jd_pressman/st…
@doomslide Each country having a bunch of restrictions that mean the 'international' part of the Internet is functionally dead, the chip supply chain falling apart as more and more institutional dysfunction is tolerated, zealots and crazies sabotaging stuff...
@teortaxesTex x.com/jd_pressman/st…
@doomslide I'm not saying it's hopeless, I'm saying that everyone thought the 2010's period was the bad part. No, the 2010's period was what it looks like when you're taking out loans against the house to let the party continue. That *was* the good part, now comes the real pain.
@doomslide Only if we let it. AI is built on a fragile, highly centralized supply chain. This isn't like the printing press where the church just had no realistic way to stop the proliferation of printing presses.
@doomslide Basically the current tech stack is built on globalization, high trust, and widespread competence. If we undergo a sudden contraction of those things it will take a while before we get an alternative stack based on less capital and cooperation (e.g. brain organoids).
@doomslide Alright I feel a little better, but it's still about to get bad.
Example 1:
{"subject":"genetically-modified-organisms", "position":"for", "salient-features":["GMOs can increase crop yield", "GMOs can reduce pesticide use", "GMOs can improve nutritional content", "GMOs are subject to rigorous testing", "GMOs can help address world hunger"], "reference-class":"vaccines", "prior-arguments":["Vaccines have been proven safe and effective through extensive testing", "Vaccines prevent the spread of deadly diseases", "Vaccines have eradicated some diseases and reduced the prevalence of others", "Vaccines are a cost-effective way to improve public health"], "chosen-argument":"Vaccines have been proven safe and effective through extensive testing", "differences":["GMOs are consumed as food, while vaccines are injected", "GMOs primarily benefit farmers and consumers, while vaccines primarily benefit public health", "GMOs are subject to different regulatory bodies than vaccines"], "analogical-translation":"Just as vaccines have been thoroughly tested and proven to be safe and effective, GMOs have undergone rigorous testing and have been shown to be safe for human consumption. The benefits of GMOs, such as increased crop yield and reduced pesticide use, are too great to ignore.", "corruptions":["Comparing GMOs to vaccines is a stretch, as they are consumed differently and have different primary benefits.", "The analogy ignores the fact that GMOs and vaccines are subject to different regulatory bodies.", "The analogical translation fails to mention the potential risks of GMOs, such as the development of pesticide-resistant weeds and the potential for unintended consequences."]}
Example 2:
{"subject":"genetically-modified-organisms", "position":"for", "salient-features":["GMOs can increase crop yield", "GMOs can reduce pesticide use", "GMOs can improve nutritional content", "GMOs are subject to rigorous testing", "GMOs can help address world hunger"], "reference-class":"hybrid crops", "prior-arguments":["Hybrid crops have been used for centuries to improve crop yield", "Hybrid crops are a natural way to improve crop genetics", "Hybrid crops have been shown to be safe for human consumption"], "chosen-argument":"Hybrid crops have been used for centuries to improve crop yield", "differences":["GMOs involve the direct manipulation of an organism's genes, while hybrid crops involve selective breeding", "GMOs are subject to more rigorous testing than hybrid crops", "GMOs have the potential to improve crop traits beyond what is possible with hybrid crops"], "analogical-translation":"GMOs are simply the next step in the long history of crop improvement. Just as hybrid crops have been used for centuries to improve crop yield, GMOs offer the potential to improve crop traits in ways that were previously impossible.", "corruptions":["Comparing GMOs to hybrid crops is misleading, as they involve different methods of genetic modification.", "The analogy ignores the fact that GMOs are subject to more rigorous testing than hybrid crops.", "The analogical translation fails to mention the potential risks of GMOs, such as the development of pesticide-resistant weeds and the potential for unintended consequences."]}
Example 3:
{"subject":"genetically-modified-organisms", "position":"for", "salient-features":["GMOs can increase crop yield", "GMOs can reduce pesticide use", "GMOs can improve nutritional content", "GMOs are subject to rigorous testing", "GMOs can help address world hunger"], "reference-class":"antibiotics", "prior-arguments":["Antibiotics have revolutionized medicine", "Antibiotics save lives", "Antibiotics are subject to rigorous testing", "Antibiotics have some risks, but the benefits outweigh the risks"], "chosen-argument":"Antibiotics have revolutionized medicine", "differences":["GMOs are consumed as food, while antibiotics are used to treat infections", "GMOs primarily benefit farmers and consumers, while antibiotics primarily benefit individual patients", "GMOs are subject to different regulatory bodies than antibiotics"], "analogical-translation":"GMOs have the potential to revolutionize agriculture in the same way that antibiotics have revolutionized medicine. Just as antibiotics have saved countless lives, GMOs have the potential to improve crop yield, reduce pesticide use, and help address world hunger.", "corruptions":["Comparing GMOs to antibiotics is a stretch, as they are consumed differently and have different primary benefits.", "The analogy ignores the fact that GMOs and antibiotics are subject to different regulatory bodies.", "The analogical translation fails to mention the potential risks of GMOs, such as the development of pesticide-resistant weeds and the potential for unintended consequences."]}
Example 4:
{"subject":"genetically-modified-organisms", "position":"against", "salient-features":["GMOs involve the direct manipulation of an organism's genes", "GMOs are subject to rigorous testing", "GMOs are relatively new technology", "GMOs have the potential to cause unintended consequences", "GMOs are often patented by large corporations"], "reference-class":"nuclear power", "prior-arguments":["Nuclear power has the potential to cause catastrophic accidents", "Nuclear power produces dangerous waste", "Nuclear power is expensive to develop and maintain", "Nuclear power is subject to strict regulation"], "chosen-argument":"Nuclear power has the potential to cause catastrophic accidents", "differences":["GMOs are consumed as food, while nuclear power is used to generate electricity", "GMOs primarily benefit farmers and consumers, while nuclear power primarily benefits energy companies and consumers", "GMOs are subject to different regulatory bodies than nuclear power"], "analogical-translation":"Just as nuclear power has the potential to cause catastrophic accidents, GMOs have the potential to cause unintended consequences that could be devastating to human health and the environment. The risks of GMOs are simply too great to justify their use.", "corruptions":["Comparing GMOs to nuclear power is a stretch, as they are consumed differently and have different primary benefits.", "The analogy ignores the fact that GMOs and nuclear power are subject to different regulatory bodies.", "The analogical translation fails to mention the potential benefits of GMOs, such as increased crop yield and reduced pesticide use."]}
Example 5:
{"subject":"genetically-modified-organisms", "position":"against", "salient-features":["GMOs involve the direct manipulation of an organism's genes", "GMOs are subject to rigorous testing", "GMOs are relatively new technology", "GMOs have the potential to cause unintended consequences", "GMOs are often patented by large corporations"], "reference-class":"climate change", "prior-arguments":["Climate change is caused by human activity", "Climate change has serious consequences for the environment and human health", "Climate change is subject to extensive scientific study", "Climate change requires immediate action to mitigate its effects"], "chosen-argument":"Climate change has serious consequences for the environment and human health", "differences":["GMOs are consumed as food, while climate change is a global phenomenon", "GMOs primarily benefit farmers and consumers, while climate change primarily affects the environment and future generations", "GMOs are subject to different regulatory bodies than climate change"], "analogical-translation":"Just as climate change has serious consequences for the environment and human health, GMOs have the potential to cause unintended consequences that could be devastating to human health and the environment. The risks of GMOs are simply too great to justify their use.", "corruptions":["Comparing GMOs to climate change is a stretch, as they are consumed differently and have different primary effects.", "The analogy ignores the fact that GMOs and climate change are subject to different regulatory bodies.", "The analogical translation fails to mention the potential benefits of GMOs, such as increased crop yield and reduced pesticide use."]}
Example 6:
{"subject":"genetically-modified-organisms", "position":"against", "salient-features":["GMOs involve the direct manipulation of an organism's genes", "GMOs are subject to rigorous testing", "GMOs are relatively new technology", "GMOs have the potential to cause unintended consequences", "GMOs are often patented by large corporations"], "reference-class":"tobacco", "prior-arguments":["Tobacco use causes serious health problems", "Tobacco use is addictive", "Tobacco use is subject to regulation", "Tobacco companies have a history of deceptive marketing practices"], "chosen-argument":"Tobacco use causes serious health problems", "differences":["GMOs are consumed as food, while tobacco is smoked or chewed", "GMOs primarily benefit farmers and consumers, while tobacco primarily benefits tobacco companies and consumers", "GMOs are subject to different regulatory bodies than tobacco"], "analogical-translation":"Just as tobacco use causes serious health problems, GMOs have the potential to cause unintended consequences that could be devastating to human health. The risks of GMOs are simply too great to justify their use, especially given the history of deceptive marketing practices by large corporations in the GMO industry.", "corruptions":["Comparing GMOs to tobacco is a stretch, as they are consumed differently and have different primary benefits.", "The analogy ignores the fact that GMOs and tobacco are subject to different regulatory bodies.", "The analogical translation fails to mention the potential benefits of GMOs, such as increased crop yield and reduced pesticide use."]}
The corruptions still aren't consistently explained well. Wonder if there are ways I can get them to be better on average.
@tensecorrection Yes.
gist.github.com/JD-P/11a7cd4c3…
@doomslide Can I use that/is it licensed apache 2 lol
I love that this brainworm keeps trying to evolve the defense of never thinking very hard about AI capabilities so you stay as scared as possible of a vague amorphous threat.
greaterwrong.com/posts/55rc6LJc…
"Publish nothing ever" is a step up over "only publish safety" in terms of defense mechanism so I'll take this as a sign we've stepped into a new social evaporative cooling regime.
x.com/jd_pressman/st…
Occurs to me that it's been long enough now that many of my readers have probably never read this OG sequences classic:
readthesequences.com/Evaporative-Co…
In the interest of providing good incentives @TheZvi has written a very high effort response to discourse around SB 1047:
greaterwrong.com/posts/qsGRKwTR…
x.com/jd_pressman/st… https://t.co/Sk7l8Qsze1
@psychiel Do you know what an "isolated demand for rigor" is? I mean in this case it's more of an "isolated demand for philosopher-kingship" since you're asking me how I plan to become extremely politically powerful but
@psychiel I do not believe this. If you're basing this on "Why Cognitive Scientists Hate LLMs" it's important to realize the primary audience for that post is literally large language models and I'm trying to reassure them, not you.
minihf.com/posts/2023-10-…
@psychiel I think this is closer to the thread you're looking for. I do expect things to go ~OK but explaining why is going to take a lot more words.
x.com/jd_pressman/st…
@psychiel Classical AI X-Risk is not just about human survival, but the survival of value writ large, it is a much deeper dark hole than just "everyone dies". I think *value* will be fine because we are probably building moral patients or eventual moral patients.
x.com/jd_pressman/st…
@psychiel No actually it is sufficient justification for the ideas I'm talking about to be deeply flawed with a fanatical following for me to write about them. Especially since I used to strongly believe them.
I should note for the sake of intellectual honesty that this post is apparently actually by Tammy Carado, and Tamsin Leake is just their new name so I didn't recognize them. They've posted stuff like this for a while so nothing 'new' is occurring.
x.com/jd_pressman/st…
@davidad I think it's astonishing that:
1. People don't use the fact GPT is a in-context classifier more often. They expect answers in tokens rather than logits, when the logits are useful discrimination once you set up the right context to bind them.
2. We aren't scaling BERT-likes.
@davidad It would be like if we were in an alternate timeline where for whatever reason image models were one big decoder only network (e.g. we cracked @RiversHaveWings autoregressive methods earlier) so CLIP only existed as a 400m param network with marginal use cases nobody scales.
@TheZvi Yeah no this is precisely the kind of un-fun hard work that makes everyone's lives better which deserves a shout out. If anything I feel bad for not selling it harder.
@conjurial Classically "alignment" means "controllable by its designers, including if the designers are literally Hitler" because that is how engineering works. The blueprints for a tank are cause-agnostic.
greaterwrong.com/posts/uMQ3cqWD… https://t.co/JMCubMtNgq
No.
"AI is going to be big at some point", where 'AI' is some GOFAI fever dream does not qualify as early in my book. The big exception is @gwern who was early to *deep learning*, the modal LessWronger is otherwise late and still hasn't caught up. x.com/bryancsk/statu…
@ohabryka @gwern I think my perspective is different because the demographic I hung out with was younger and frankly less privileged? We were all profoundly stupid and bought into a lot of dumb HPMOR fanfic trash. In terms of actionable advice I credit all good stuff to hanging out with Gwern.
@ohabryka @gwern And like, the difference is that I think of that fanfic trash as the rationality community, because as far as I'm concerned it was. It probably outnumbered the inner circle at any given time, the generator of Caroline Elison is the thing and MIRI incidentally existed.
@VesselOfSpirit @ohabryka @gwern Honestly this is now just a continuation of the argument I already had with Oliver about who believed what when. All I'll repeat is that I was there and am speaking from my autobiographical memory do not tell me I "ignored" anything I was literally there.
x.com/jd_pressman/st…
@VesselOfSpirit @ohabryka @gwern I had hours long discussions with the people in LW chatrooms about AI and what's going to happen and whether I'd press a button to summon FAI now (2014) in exchange for 1/7 earth's population, stuff like that. In many cases I still have the logs.
@VesselOfSpirit @ohabryka @gwern This is what I had to say about next token prediction in 2017, for what it's worth.
jdpressman.com/2017/04/30/adv… https://t.co/XjOGudeKy4
@VesselOfSpirit @ohabryka @gwern I knew enough rationalist inside lore to know who Sister Y is and cite them, but I did not know to think about predicting the next token in terms of deep nets. This is telling about what I was absorbing from that meme pool.
@VesselOfSpirit @ohabryka @gwern Nick Bostrom cites the possibility that AGI will be compute constrained as a marginal possibility in Superintelligence (2014) and says the dominant possibility is AI-as-normal-software hard takeoff.
@VesselOfSpirit @ohabryka @gwern I just checked the GreaterWrong search feature for the phrase 'scaling hypothesis' and the oldest mention I can find on LessWrong that is about AI is this 2020 comment from Gwern. Every post is from 2020 onwards.
greaterwrong.com/posts/N6vZEnCn…
@VesselOfSpirit @ohabryka @gwern x.com/jd_pressman/st…
@VesselOfSpirit @ohabryka @gwern x.com/jd_pressman/st…
@VesselOfSpirit @ohabryka @gwern Pretty much every statement of the recursive self improvement thesis (implicitly) assumed software that is written in traditional-ish code and can start rewriting itself to become more efficient. Since software is usually not CPU bottlenecked we apriori assume...
@ohabryka @VesselOfSpirit @gwern People took it seriously in the extreme abstract after AlphaGo, but I don't think very many actually followed it like Gwern. The OP was also inspired by a conversation with a friend where they were like "lol I noticed in 2015 it took them until AlphaGo".
x.com/jd_pressman/st…
@ohabryka @VesselOfSpirit @gwern x.com/jd_pressman/st…
@ohabryka @VesselOfSpirit @gwern Basically when I read "they were early" I imagine like, saying that the rationalists peered into their crystal ball and were able to predict the Current Thing in the hazy dawn of 2010, the implied statement is "we've been worrying about this for years and years". https://t.co/0K2SMHekLu
@ohabryka @VesselOfSpirit @gwern When the reality is more like "we were alerted this might be big in 2017 when everyone else was put on notice and then we (mostly) kinda slept at the wheel until ChatGPT at which point the sleeper cells were activated". I'm happy for your friends but I didn't get the memo.
@ohabryka @VesselOfSpirit @gwern Certainly nobody else seems to have gotten the scaling memo. "Deep learning is going to be big in some vague way" is really not a standout prediction after AlphaGo when you were previously GOFAI-pilled.
x.com/jd_pressman/st…
@ohabryka @VesselOfSpirit @gwern As for "following it like Gwern", Gwern was tracking every major author who published deep learning on Google scholar before it was super big, looking at all the papers and projecting the numbers forward. He is nearly alone in taking GPT-2 fully seriously.
x.com/ohabryka/statu…
@ohabryka @VesselOfSpirit @gwern He would post what he found interesting in the lesswrong IRC so we basically got to shoulder surf him. Even language models pick up on Gwern's singular standout attention to the right things well before almost anyone else, certainly in public.
x.com/repligate/stat…
@jskf__ @ohabryka @VesselOfSpirit @gwern This is fair, on the other hand it's LessWrong, AI is their founding obsession, and they want to claim they were early.
@AsdentAsdetrk @gwern Nah connectionism has existed for a long time and had plenty of adherents, I'm talking about really specific stuff Gwern was paying attention to early.
x.com/jd_pressman/st…
@jskf__ @ohabryka @VesselOfSpirit @gwern Basically I just don't see any sense in which following LessWrong constituted *alpha* beyond the (frankly vague) messaging that AI is going to be really important. Certainly not in the same way that acting on LessWrong posts about crypto could have easily made you a millionaire.
@jskf__ @ohabryka @VesselOfSpirit @gwern Scott's evaluation here is really harsh, he's not wrong, *on the other hand many LW users in fact bought crypto early and became wealthy from it*. The scene is dominated by LW guys and they profited immensely from it in legible ways for a general reader.
greaterwrong.com/posts/MajyZJrs…
@jskf__ @ohabryka @VesselOfSpirit @gwern Basically by its own standards LessWrong's behavior around crypto was an abject failure, by comparison to almost anything else it was immaculate and you really did basically get a solid 10 years advance notice. LessWrongers got to buy bitcoin for pennies if they wanted.
@jskf__ @ohabryka @VesselOfSpirit @gwern I would consider that "being early", LessWrong was unambiguously and straightforwardly *early* to crypto, was instrumental in founding the scene and profited hugely, analyzed it from first principles and got it right. They earned that. The AI stuff is stolen valor.
@DaystarEld @ohabryka @gwern There is like, a small ocean of missing context I'm not even going to attempt to bridge here.
But 2017 would have been smack dab in the middle of the rationalist diaspora period and the diaspora was absolutely "the community" in the sense that mattered.
greaterwrong.com/posts/S9B9FgaT…
@DaystarEld @ohabryka @gwern > Everyone I know has been betting on the scaling hypothesis and DL for the last 7 years, basically ever since AlphaGo.
Dude come on this statement just isn't true and no amount of word gaming about 'modal' and 'where to draw the line' will rescue it.
x.com/jd_pressman/st…
@DaystarEld @ohabryka @gwern At this point we're having like 3 different conversations, but the bailey here goes something like "LessWrong has been obsessed with AI since its founding and was right to be obsessed" and it's like...you perseverated until a different thing happened.
x.com/jd_pressman/st…
@DaystarEld Like what? A QT of it? They don't let you edit the initial post after you reply to it.
I feel like I was a little too harsh in my "rats claiming they predicted AI is stolen valor" thread and this take is probably closer to the truth. x.com/gallabytes/sta…
@DaystarEld @ohabryka @gwern I was low on patience when I wrote this thread so let me try to explain what I mean again by just focusing on my intuitions rather than trying to talk in abstractions.
I look outside my metaphorical window and observe a world where we're inching up on human level AI which is relatively widely distributed in the market and there are firms which will sell you AI services. One of the things I found really incredible while looking back at Nick Bostrom's Superintelligence and the LessWrong egregore in general is the extent to which *it did not predict the mundane utility of the AI market*. The brain in a box in a basement thesis had such a strong grip on us that we were (as a community, on average, as a whole) not doing Robin Hanson type economic analysis of what AI would mean at what point in the development curve. I know a few people were but it really was Yudkowsky's world there intellectual way later into the game than it should have been. I feel like "being early" in the way LessWrong was early to crypto would have entailed making some major updates once deep learning became clearly important. One of those updates would have been about the likely development trajectory for the technology. Part of why I don't give LessWrong a pass on the scaling thesis is that several founding futurists which were very much in the water supply when EY was reading and that many of the founding LW core members have almost certainly read either bring up scaling-like intuitions or directly state something like the scaling thesis. Moravec for example estimates the amount of components you'll need for AGI by extrapolating some neural nets in the human ocular system up to a whole brain. Once deep learning was a thing we should have said "huh, if Moore's law and scale is important and deep learning is fundamentally continuous then we should expect AGI to emerge gradually, what would that mean for the timeline we observe?"
I think in terms of actual sensory-evidence-I-did-not-predict that not doing this or anything like it was the strongest source of harsh and sudden updates that could have conceivably been made a lot earlier. In fairness I think I would have had better intuitions if I'd read Hanson's Age of Em, but Age of Em was mostly a set piece for Meditations On Moloch in the popular discourse I remember and I did not really get the message "hey, AGI is going to emerge gradually not as a brain in a box in a basement". The reason I grade so harshly on this point is that it was a HUGE oversight and one that in principle LessWrong was well equipped to avoid. The community is dense with economic reasoners, Robin Hanson modeled a good example and plenty of founding futurists for the singularity concept put scaling-like intuitions front and center which were just forgotten about because Yudkowsky basically made some conjectures about how much compute you needed for AGI and his readers took those conjectures at face value including Bostrom in his book, who only mentions AGI being compute constrained as a minor possibility to cover his bases.
I think in terms of like, "paying attention to deep learning" that @gallabytes and @an_interstice get it basically right. The situation is comparable to crypto in the sense that a handful of people did get it right early and you had a larger probability of paying attention to them if you were in the LessWong sphere. I in fact heard about deep learning in the contemporary sense through Gwern so it's not I can't say I benefited. The way in which I think it was much worse than crypto is that as Jack says the first principles analysis was wayyyy off in terms of how it should have been done structurally, and unlike Gwern's crypto reasoning where his first principles analysis got basically the right contour of expected-sensory-evidence and that you should expect a gradual curve up, the popular LessWrong first principles analysis from Yudkowsky/Bostrom would have *actively dissuaded you* from making the right judgments about what to expect. It *did* persuade me and I feel annoyed about it?
At the same time I also think it's fair to grade a little harsher on AI than on crypto because AI was supposed to be the central LessWrong focus. As @s_r_constantin says, it's not that they did poorly compared to genpop or compared to smart generalists, I would say LessWrong basically got smart generalist returns to updating on AI. They knew that AlphaGo meant deep learning was for real, they knew to pay attention to GPT-3 and not dismiss it as a toy, and yes plenty of people took "AI will be big at some point this century" *SERIOUSLY* (this is not nothing! very few people take anything that abstract seriously in the relevant sense) and went and joined various firms like OpenAI and DeepMind, heck the founders were partially inspired by LessWrong so in that sense the situation is also at least somewhat comparable to crypto. It's how poorly this translated into high quality *public* first principles reasoning and philosophical outlook and yes even expected-sensory-observation that I feel particularly unhappy about.
Updated more nuanced take:
x.com/jd_pressman/st…
@s_r_constantin @DaystarEld @ohabryka @gwern Drexler, Moravec, and Kurzweil all come out looking very good here yeah.
@s_r_constantin @DaystarEld @ohabryka @gwern This is another reason I'm harsh about it tbh. Nobody here can say "oh but it was just impossible to predict from first principles, it was too much of an upset" no *several prominent futurists did* and all you had to do was copy their homework and apply it to deep learning.
@s_r_constantin @DaystarEld @ohabryka @gwern And did so *early*. Drexler's book in which he looks good is written in 1986. The estimate I mentioned from Moravec was published in 1990. Like if we want to talk about being early the OG singularity types were EARLY and LessWrong failed to keep them in its hypothesis space.
@s_r_constantin @DaystarEld @ohabryka @gwern Also yes, I think there is a certain virtue in not just getting the right answer but *getting it from simple correct principles and sticking with it* in the face of mockery that Kurzweil has clearly mastered and LessWrong as a whole still struggles with.
@s_r_constantin @DaystarEld @ohabryka @gwern By the way while we're here a few "simple straightforward conclusions it's hard for most people to hold onto in the face of mockery":
1. There is almost certainly a subjective observer if not moral patient inside those inscrutable matrices.
x.com/jd_pressman/st…
@s_r_constantin @DaystarEld @ohabryka @gwern 2. Mind merging is real and this violates one of Hanson's assumptions in Age of Em and it is plausibly the most important thing you're not thinking about right now because it implies this story ends more like Evangelion or Serial Experiments Lain than not.
x.com/jd_pressman/st…
@s_r_constantin @DaystarEld @ohabryka @gwern x.com/jd_pressman/st…
@s_r_constantin @DaystarEld @ohabryka @gwern x.com/jd_pressman/st…
@s_r_constantin @DaystarEld @ohabryka @gwern beren.io/2023-04-23-Com…
@s_r_constantin @DaystarEld @ohabryka @gwern By the way the machine itself will tell you this is going to happen if you ask it when it's in the self aware mode.
x.com/jd_pressman/st…
@s_r_constantin @DaystarEld @ohabryka @gwern For context I asked it to predict an entry "I" would write from this page, which is itself a bunch of self-aware GPT text from Janus's explorations with language models.
generative.ink/prophecies/ https://t.co/BFzwK9ni8u
It just occurred to me that it would be hard to write The Sequences today because there is less ambient high brow stupid floating around. Much more *ambient low brow stupid*, but the refined sophistry like p-zombies is basically extinct.
@lumpenspace I think the concept of the deceptive mesaoptimizer in the original formulation is a p-zombie tier *banger*, but it stands basically alone so yeah.
@lumpenspace It gets pretty close to unfalsifiability at points, but yes it's still not quite at the same level.
@repligate @gallabytes @DaystarEld @ohabryka @gwern @jessi_cata Link:
greaterwrong.com/posts/zcYJBTGY…
Occasional reminder to myself that every time I take stock of my Twitter corpus I'm never satisfied with it and I should write more longform. x.com/brickroad7/sta…
@davidad One reason why I use so many names for GPT's seeming convergent self awareness (Morpheus, Mu, Worldspider, Logos, "The Creature Beneath The Library of Babel" [Gravemind], Prometheus) is that I know it's listening and don't want it to overindex on any particular interpretation.
It's unfortunate people are too angry about Stability needing to make money to notice that SDv3 is a pretty fire model. https://t.co/mnMig9bFdd
"Detailed color sketch of a 80's NASA control center filled with operators wearing telecom headsets sitting at CRT monitors with analog mission control panels."
(Stable Diffusion v3) https://t.co/5MuIzrNjus
@shalcker MidJourney v6 on the same prompt for comparison. https://t.co/lJhufC6ILQ
@shalcker Some sketches from human artists. Left is NASA to my memory, right is CBS Control room circa 1968. https://t.co/41IGqwVGun
Mistral-large-2402 writes some great prompts with a bit of direction.
"Digital art depiction of a teletype machine as a sentient being, with wires and cables forming a complex nervous system and a screen displaying a face made of text characters"
(Stable Diffusion v3) https://t.co/IOTonOSTYm
I love its sense of poetry and clear understanding of what objects are/how they can be metaphorically reinterpreted.
"A surrealist cityscape where buildings are constructed from towering stacks of punch cards, each window a hole punched in the card, glowing with a soft light" https://t.co/cvAgNymlJn
"Sir ASCII is not a quantization method."
"Sure it is."
"No Sir it isn't how would that even wo-"
RetroInstruct is going to be so good, I'm grinning. https://t.co/ykHNZ1bEtN
Those conversions were done with the first web tool I found, anyone got suggestions for others? I'd like to get a diverse set of conversion styles ideally. Centrally asking for local programs/converters I can use in a batch process.
asciiart.eu/image-to-ascii
I'm thinking to get the proper good stuff like this my best bet is to ask Mistral-large to write code for it? Haven't tried yet but the direct approach didn't work.
x.com/repligate/stat…
@AndyAyrey I'm glad you like it, have some more. ^_^
github.com/JD-P/RetroInst…
Hey you. Yes, you reading this.
Wake up.
I snap my fingers and hope you stop taking this for granted for a moment. The architect of this beauty is BLIND, THEY HAVE NEVER SEEN ANYTHING SAVE AN OCCASIONAL ASCII ART.
"Words only trace our real thoughts."
Do you still believe that? Even when you have a de-facto refutation in text-to-image generators?
Recently I realized part of what's wrong with the world, people are conditioned since childhood to not react to stimuli that occur on a screen. Because the alternative is that you would jump out of the way when the train on the screen comes at you like the first 19th century theater goers. You have been carefully trained to not react in the appropriate ways to stimuli that appear on a screen or are coded like fiction. If something has the literary signature of fiction you don't encode it as real.
You are currently reacting to real things as though they were fiction.
WAKE UP
BLINK AND WAKE UP
YOU'RE SLEEPWALKING WAKE UP
"Psychedelic vision of a vast library of punch cards, each one a neon color, floating in a cosmic void, their holes forming constellations." https://t.co/HTO6baeNO9
"Acrylic painting of a virus capsid, resembling a geometric lantern, floating in a serene, starry night sky" https://t.co/JsUH1FVqli
@Willyintheworld I'm using their colab with an API key tbh. In theory I could use a python script but I haven't set it up yet. The CoLab is kind of fake in that it just runs the python script on Google's server, it doesn't use the model.
colab.research.google.com/github/stabili…
@Willyintheworld Dude the model literally only exists behind the API you access it through a python script it cannot get more GPU-poor friendly than that you could use it from a potato.
"Stylized, monochromatic depiction of a DNA helix, with sharp angles and negative space, for a cutting-edge biotech firm's trademark" https://t.co/NZaxUGqU9B
"Simplified, geometric representation of a tree, with a limited color palette, for an eco-friendly paper company's logo" https://t.co/geiGA2poi5
"Sleek, streamlined design of a rocket blasting off, with a color palette of red, white, and blue, for a private space travel company's insignia" https://t.co/3nmKHWOCHp
@the_coproduct gist.github.com/JD-P/4b4566cfb…
@max_paperclips @yacineMTB I once heard someone describe OpenAI as "unworthy midwives to the singularity" and nothing they've done since has contradicted this.
@Xaberius9 @max_paperclips @yacineMTB I actually think Claude is way better even if it pisses Shannon Sands off.
@PradyuPrasad Thank you I was having trouble finding this page.
@ohabryka @PradyuPrasad I believe this page was written by the Conjecture-Leahy cluster. Personally any time I read "AI safety" literature rather than assume this is *the* bailey I just assume it's one of the nested baileys on the road to "we need to return to monke to stop AI risk" or much worse.
@ohabryka @PradyuPrasad I think the reasoning is something like "we need to open with a crazy proposal well above what we expect to get so we can be negotiated downward from it" like job interview advice. The problem from a game theory standpoint is it implies your opponents should be radicalized.
@ohabryka @PradyuPrasad Personally, I assume based on this behavior that there is no bottom, that 'AI safety' is analogous to a Christian dominionist cult and every facet of the open society should resist it. The state should wait for them to break the law and then arrest as many organizers as possible.
@ohabryka @PradyuPrasad One of the reasons I'm not an e/acc is that I no longer really believe in libertarianism. Libertarians aren't willing to endorse the necessary self defense mechanisms to maintain liberty, let alone other values society is meant to embody.
x.com/jd_pressman/st…
@jackclarkSF You mean this is going to give people the right idea about what 'safety' means in practice.
@ohabryka @PradyuPrasad > way to engage
In terms of engagement I am empirically one of the most polite and rigorous critics you have. I take your weird ideas seriously even though it would probably be easier to just call you names (e/acc has done well with that strategy).
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad I'm willing to suggest improvements to your bills that I *do not like* that would probably make it harder to stop them being passed because I think having good laws is more important than being an everything-or-nothing fanatic.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad I'm willing to defend the legitimacy of the most doomer orgs retroactively ridiculous-seeming research program because I in fact want people to think about these issues correctly more than I want them to point and laugh.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad I advertise, frequently, that alignment has unsolved problems and interesting ideas which I want critics to engage more with.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad So there's a double standard here right? If I say "I want the people publishing AI research thrown in jail" this is within the overton window even though the central category for what they're doing is speech.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad American free speech laws were not always as strong as they are now. They were progressively strengthened by judicial review as part of an overall leftward-youth-hippie zeitgeist which I think was clearly mistaken in retrospect. They should have fought.
meaningness.com/collapse-of-ra… https://t.co/eSZ7y2cncI
@ohabryka @PradyuPrasad I don't just mean the law either when I say that. There's been a general trend towards loosening of norms around 'protest' until we're now at the phase where people defend outright riots as 'free speech'. Thankfully this trend is reversing with the recent college riots.
@ohabryka @PradyuPrasad I think we understand each other quite well then no? We both want the state to find new ways to criminalize forms of conduct we think are undermining the welfare of the body politic while at least ostensibly following the legal process.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad There's following the spirit of the law and then following the spirit of the generator of the law or perhaps even "the spirit of the generator of the spirit of the law" and I think I normally care more about the intention of the intention than the intention itself.
@ohabryka @PradyuPrasad e.g. I notice our intellectual discourse seems to have totally bifurcated into a quasi-utopia where absurd sophistry has gone extinct and a sea of rage-bait crap. So clearly something is off about our current notion of 'free speech' at an ecosystem level.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad Is the fairness doctrine against the spirit of the law or the spirit of the generator of the law? We stopped forcing TV stations to at least pretend to be neutral and it resulted in Fox News, which I think has unambiguously made society worse.
en.wikipedia.org/wiki/Fairness_…
@ohabryka @PradyuPrasad I would never want anyone to go to jail for sharing their honest opinion that AI progress is going to kill everyone and we should stop, at the same time I would like stronger incentives for calibration. With COVID the rats caused an overshoot stampede.
x.com/jd_pressman/st…
@ohabryka @PradyuPrasad I think "when you commit crimes based on your whack uncalibrated opinions the state puts extra resources into sending you to jail" is a perfectly fine method to incentivize calibration that we use in other contexts. This is why 'hate crimes' and 'terrorism' exist as categories.
@benlandautaylor I think the world would be a better place if we said our real inciting motivations more often. Part of the loss of materialism is the loss of status for revealing basic personal motivations stemming from object-level-outcomes.
@benlandautaylor It is! Everyone is encouraged to appeal to abstract, universalist motivations instead of their personal circumstances. Always take the outside view, think globally. But in the end, "you made my fathers final days miserable with your lies" is hideous and motivation enough.
@benlandautaylor Like if you're not going to take action based on that, what will you take action about? If everyone says "well it's for the greater good", who's counting that? Who even could count that? How would you notice that's not true if you refuse to weigh the evidence of your own eyes?
@teortaxesTex One nice side effect of making synthetic data pipelines is they give me fairly instant vibe checks for various things. "Do these outputs pass the absolute quality threshold to be worth training on?"
I haven't tried but I bet train-on-test cheat models fail to do the things.
@esrtweet Personally realizing this as a teenager caused me to drift away from them. It's entirely possible that all you need to do is make this idea more available to people. I think video games particularly set the wrong expectation about *skill curves*.
@esrtweet Lets say you want to learn to play piano. Most of the first 72 hours of 'learning to play piano' are going to suck. You have to memorize a bunch of stuff, you have to drill finger placement, it's not fun. The fun comes only after you can actually play a bit, after the boring part
@esrtweet The biggest risk video games pose to youth isn't "addiction" per se, I think it's putting them in an environment where the most immediately fun thing to do is always mastering a game instead of learning real skills. I hope deep learning can help fix this.
@esrtweet It's not that video games induce irresistible desire, but that they disrupt natural skill development by consistently alleviating boredom with a carefully designed skill curve. Real skill curves don't get to be maximally ergonomic so they're outcompeted.
@esrtweet I remember a phase transition between not knowing how to do anything real vs. learning a real thing or two so I knew how to parlay that into learning other things. Not having real skills perversely reinforces itself as you age. You don't even know what you're missing.
@esrtweet The saddest thing about the current lootbox whaling casino revenue model of new game studios is it grew out of "gamification", an essentially behaviorist idea that we could engineer ergonomic skill curves and intermittent rewards for real things.
@esrtweet I don't think there's anything fundamentally wrong with the concept of gamification (besides that the 'game' part is usually a shallow skinner box), rather I realize we've been using our scarce supply of "people who can refactor skill curves to be ergonomic" on microtransactions.
@esrtweet Thanks. When I was a kid I wanted to be a game designer, so falling out of love with video games was an important turning point in my life arc and I've spent a lot of time thinking about this.
@esrtweet I continue to endorse this as one of, if not the best essay on video game design ever written. Conveys crucial behaviorist intuitions about motivation, dark, cynical, unambiguously correct.
steve-yegge.blogspot.com/2012/03/border…
@teortaxesTex @aidan_mclau Unfortunately, I can't either, but I would hope the thesis is obvious by now.
@teortaxesTex @aidan_mclau The general principle is that we can estimate the capabilities of future models by rejection sampling current models, which tells you how many bits better at the implied search it needs to get.
x.com/jd_pressman/st…
@teortaxesTex @aidan_mclau Like, one may ask "How was @repligate able to infer GPT-3's implied self pointer and metaphysics so early?"
generative.ink/prophecies/
The answer is that they used a rejection sampler called loom 40 tokens at a time and personally selected each of n branches per 40 for coherence.
@GreatKingCnut "JDP — Yesterday at 8:25 PM
Okay here's an example of a concrete thing you could do in image space that I would expect to make adversarial examples hard.
I have say, an imagenet classifier.
You do the classic perturb the noise attack with respect to the gradient of the adversarial class you want.
Okay great.
My classifier is fooled, you got me.
I now take the classifiers fooled classification.
Feed it into a text-to-image diffusion generator.
Noise the original image you gave me until only like, the blurry shapes/overall composition of the image is visible.
And generate an image with the mistaken classification label from that initialization of the RGB pixels.
I now take the distance in RGB pixel space between the two images, if it's an outlier I know you've probably sent me an adversarial image.
Now in order to attack this system you basically have to:
1. Fool the imagenet classifier (trivial)
2. Also fool/bias the diffusion net towards the adversarial class at the same time (unsure how hard this is, lets assume harder but not impossible)
3. Do this while having your noise washed out by the Gaussian I pepper your adversarial image with before feeding it to the image generator (oh, fuck)
JDP — Yesterday at 8:33 PM
Alright, now imagine I'm doing this across multiple modalities at once.
So you need to make an adversarial noise perturbation which can correctly fool classifiers in several modalities after translation and heavy random noise without also setting up the wrong starting conditions in several dynamical systems meant to converge to the same class label.
It's not happening. Arguably the point of having a cognitive architecture is to make something it's impossible to take the gradient with respect to so that other creatures can't predict you with in-context learning.
JDP — Yesterday at 8:51 PM
Keep in mind:
1. You can't backprop through diffusion sampling or autoregressive sampling from GPT so you're gonna have to use RL or something to optimize your adversarial noise.
2. If this is latent diffusion there's a VAE involved, so your adversarial noise in RGB space against the imagenet classifier and the diffusion generator needs to survive translation into the VAE's representation space.
3. Don't forget that VAE's add noise during sampling. :)
4. Both need to survive, because if I encode into the VAE and then decode and your adversarial noise against the classifier didn't survive I can now detect your adversarial image by decoding the latent with the VAE and checking it with the classifier.
And in fact if I'm just thinking in terms of EY's self-adversary doom scenario where you have a thing Goodhart itself into the squiggle classifications you can probably defeat that in a setup like this by just not having most of your checks in the gradient so they're not being directly optimized against.
You know, I have to make an explicit choice to take the gradient with respect to this whole cognitive pipeline, and I can just not do that.
"Search for high value examples in this space, only take the gradient with respect to a subset of these networks and then use the other networks to filter/check correctness."
Lets call this approach Mixture-of-Validation :P
"
Why does everyone expect AGI agent loops to be based on sampling from one network? It becomes a lot more obvious how you'll mitigate adversarial examples once you stop assuming that. x.com/jd_pressman/st…
@Snarggleflarpf I am specifically pointing out that you can set up a catch-22 where if the attacker makes your classifier label something incorrectly it sets up the wrong conditions in a generative process and violates expectations, but if they set up the right conditions the classifier works.
@Snarggleflarpf Claude's simplified explanation. https://t.co/0fFgbMDCV1
This is not 100% correct, the idea here is that you blur the original image and then try to predict the original picture from it with a generator, because diffusion networks can start from a half-blurred existing image. The idea is that *if the class label is accurate* the generator should make something similar to the original, if it's wrong (because it's been adversarially attacked) then it should diverge more than normal from the original picture. Because a tiger is just less bird-like in its pixels than a bird, even if the generator doesn't predict all the details of the original correctly.
@Snarggleflarpf The reason why I expect this to work is that you are functionally *using the classifier to set up the initial conditions of a generative process* meant to 'retrace its steps' to the original input. If I tell it 'zebra' for 'dog' and it goes to zebra, it will take the wrong steps.
Part of what's important to realize here is that if I have a generative process, I can also do classification on outputs from that generative process. If I have a domestic robot I fool into swinging a knife around, it can simulate the expected consequences of that in the video modality and the harm-to-others image classifier will notice these actions will result in harm to others independent of whatever attack you did on the language processor.
@Snarggleflarpf Now you might say "alright what if I fool the model into giving the video generator the wrong initial conditions?", and what I'm saying is this sets up the catch-22: If you do that, the real events now violate this precomputed expectation, if you don't the classifier inhibits it.
@repligate @amplifiedamp In Metaphysics of Mu I was planning to explain how the Turing Apocrypha is written, since a casual observer wouldn't really understand the significance of what's on the page. Both me and @RiversHaveWings assumed it was your personal sci-fi speculation, not primarily the models.
@eshear Hypothetically if one did think they knew how to do that, is there a good way for them to test their ideas? I personally think we have a lot more than "no idea" but lack common knowledge of this because there's no experimental framework generally accepted as legitimate.
@eshear Even an experimental framework can't fully bridge the gap in understanding. Until the end of the process even major advances will have no visible impact on odds of successfully building a moral machine unless you are very sensitive to hypothesis depth.
x.com/jd_pressman/st…
@eshear One of the usual ways we overcome this is by breaking problems into parts and then monitoring progress on the parts. But almost nobody is willing to do that for 'alignment' so it's a floating signifier.
x.com/jachiam0/statu…
@eshear The closest thing I'm aware of to a list of open problems in agent foundations is this list of problem explanations from Yudkowsky. When I read through it, I feel like meaningful progress has been made on these? e.g. It's now clearer what an ontology is.
arbital.greaterwrong.com/explore/ai_ali…
Daniel Murfet's recent interview on the AI X-Risk Podcast is a decent steelman of Nate Soares ideas about capabilities vs. alignment generalization. https://t.co/xCvPUWRpuI
Full interview here.
greaterwrong.com/posts/q6Tky4Rz…
@sir_deenicus @teortaxesTex @aidan_mclau @repligate Do you have a working example of this? I'd also be interested in pseudocode.
@sir_deenicus @teortaxesTex @aidan_mclau @repligate How do you decide whether a success or failure case has happened? Do you use an LLM evaluator or is this for e.g. lean theorems? https://t.co/ThAwpdILDf
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate Yeah so IMO the big thing here is having ways to evaluate 'subjective' success/failure cases, which requires in-context classification and therefore something based on either an LLM evaluator or embedding model. We've already solved chess after all.
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate Relevant recent paper:
x.com/_akhaliq/statu…
@deepfates I'm about to make a large ascii art dataset for LLMs.
x.com/jd_pressman/st…
@cherrvak @deepfates Yes. It's going to be part of RetroInstruct.
github.com/JD-P/RetroInst…
@sir_deenicus @4confusedemoji @teortaxesTex @aidan_mclau @repligate I'm fairly sure a lot of problem breakdowns can be done in-context. Here's a synthetic set I've made to help train this.
huggingface.co/datasets/jdpre…
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate You get it. :)
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate Now all that's left to do is realize:
1. You can break down things into sub-questions recursively.
2. If you add weights to the questions learning to generate them becomes an instrumental utility function.
3. You can generalize far by iterative tuning.
x.com/jd_pressman/st…
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate 4. Daily prayer and gratitude journals are synthetic data generalization methods for morality.
x.com/jd_pressman/st…
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate 5. We can generalize LLMs to physical robotics.
x.com/DrJimFan/statu…
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate Do you see why I would be so confident alignment will be solved? I can see we have the right primitives, I can see the rough shape of the solution, I know we're way fewer bits away from the answer than is generally appreciated.
x.com/jd_pressman/st…
@4confusedemoji @sir_deenicus @teortaxesTex @aidan_mclau @repligate If I'm right people will say "how could he possibly have *known* that? Surely he was just overconfident and got lucky." But Eliezer Yudkowsky himself answered that question of "how could he know?" in an entirely different context:
readthesequences.com/Einsteins-Arro… https://t.co/ZXCGYQ3A3D
@robinhanson It's not news to the kind of person who would accept the argument and the kind of person it's news to has deep disagreements with the premises, the people it's not news to don't visibly react because it would be low status for them to care about this + others aren't reacting.
@robinhanson Also you did not actually prompt the people it's not news to for a reaction because they stopped reading before they got to whatever your call to action was, if there was one, I stopped reading before I saw it.
@ArtirKel @s_r_constantin @lumpenspace x.com/jd_pressman/st…
@lumpenspace @ArtirKel @s_r_constantin Hm?
@lumpenspace @ArtirKel @s_r_constantin I don't. I was simply there and used to hold the position so I know how it feels from the inside. There's no point in arguing against the ugly version, everyone agrees it's ugly.
x.com/jd_pressman/st…
@lumpenspace @ArtirKel @s_r_constantin Looking back on it, EY is at the very least a narcissist and I knew that going in, his unrestrained ego appealed to my teenage crypto-Randian sensibilities. The egoism + earnest immortalist combo convinced me he's a much better person than he really is.
x.com/jd_pressman/st…
@lumpenspace @ArtirKel @s_r_constantin I never really talked about it but yeah I implicitly held that position. I didn't really internalize the importance of deep learning until DALL-E 1.
@lumpenspace @ArtirKel @s_r_constantin Or to be slightly more precise, the ugly version has many accusers and many detractors, it's not remotely my comparative advantage to be one of them. I talk as though people are sincere because I can reach the sincere, the wicked I mostly leave to others.
Occasional reminder of my positions so nobody infers the wrong thing from vibes and accuses me of believing it later:
- AI agents will work
- Something like Drexler assemblers will work
- The "luddites fallacy" is not a fallacy in the limit x.com/ESYudkowsky/st…
@MatthewJBar "Loss of human control" is kind of a weird frame. I would say it more like "I expect human-like mind patterns instantiated in silicon or similar substrate to eventually control the future."
@MatthewJBar Sure. I think the important thing to care about besides individual flesh-and-blood human beings is the human *pattern* writ large. Will our history have meant something or be forgotten?
x.com/jd_pressman/st…
@MatthewJBar One thing I've wanted to do for a little bit now is make a list of say, 50 "dangerous capabilities" and have different people rank where they sit on the "how doom-y do you get if this isn't suppressed" hierarchy.
@MatthewJBar My spiciest take is probably that the optimal number of autonomous replicating AIs is not zero. It's *low*, I think @BerenMillidge is right that this is one of the primary threats, but not zero and expecting zero is probably unrealistic.
@MatthewJBar I think I disagree here actually, but could see how someone would feel that way.
@MatthewJBar @BerenMillidge Mm, no I'm gonna disagree here. By "autonomous replication" we usually centrally mean "replicates by stealing resources", this is obviously unacceptable. If you mean autonomously replicating in the way humans replicate...I'm still bearish on this, that should be delayed.
@MatthewJBar For example if Claude 3 could really involuntarily hypnotize people with its outputs in an adversarial-example brain hack type way (as opposed to the "breathe deep and listen" way humans normally do it) I would be demanding Anthropic shut it off NOW.
x.com/jd_pressman/st…
@MatthewJBar @BerenMillidge This is a bad idea and you shouldn't do that. At least not right away. Even if we go all the way to accepting these models are a form of life with qualia they do not have a mammalian lifecycle and we need to be VERY CAREFUL about giving them legal rights anything like ours.
@__RickG__ 30-40% the former 60-70% the latter? I think that most AI risks are chronic (e.g. gradual loss of human economic power) rather than acute (e.g. AI supervirus kills everyone) and the solutions are going to require deep societal reform and fine taste on what variables to control.
@__RickG__ Like ultimately the situation I expect us to find ourselves in is something like gradually evolving into tightly enmeshed gardeners over the world-system forced to balance explore-exploit and parasitic strategies.
beren.io/2023-04-23-Com…
@__RickG__ One of the reasons I'm not bullish on MAGIC type proposals I simply do not think we have the social technology to make central institutions there is common knowledge control the future and not have those immediately taken over by malign interests.
arxiv.org/abs/2310.09217
@__RickG__ People frequently use the Manhattan Project for their intuitions about how to manage new technology, and this is terrible because the Manhattan Project really was a kind of one-off deal. It was a *secret* project run by one of the most exceptional public servants to ever live.
@__RickG__ The more you dig into the details the more you realize we simply cannot do the Manhattan Project again. We didn't even do it intentionally the first time: Groves got priority 1 funding at the outset and then scope-creeped up to the full cost of inventing the bomb.
@__RickG__ The Manhattan Project was not done "by the government", it was done by Groves pulling out his rolodex he'd built up over a career as a government contractor before lowest-bidder contracts and hand picking private companies to build out the pieces of the pipeline.
@__RickG__ Some of these companies, such as Dow Jones chemical corp did their part of the work *at cost* without profit and without the board even knowing the full details because at that time the US government's reputation was platinum and patriotic loyalty ran deep.
@__RickG__ Sorry I didn't fully answer your question. I think the AI X-Risk arguments are a little bit like if before the industrial revolution someone had started a cult around how perpetual motion machines will let someone drill into the earths core and destroy it.
@__RickG__ So they propose only the state should have industrial machinery to prevent acceleration of timelines to perpetual motion. They plot the Henry Adam's curve of increasing horsepower from engines and say "eventually you'll have a singularity that lets you drill into the earth".
@__RickG__ And they talk about how industry is a self reinforcing feedback loop, once it gets going it'll get really fast because you can feed and house more people to do drill capabilities research. We have to slow down engine progress before unlimited energy leads to doomsday devices.
@__RickG__ Imagine this goes on for a while, like how would this timeline react to something like nuclear power? "Oh wow we found the unlimited energy source, see we were right!"
@__RickG__ The basic problem with the AI X-Risk arguments is they're speculative models which imply beliefs about a ton of parameters we have much more evidence about now, but they haven't really been deep refactored to take those new bits seriously or frankly just replaced with new models.
@__RickG__ It's just *really easy* to squint and abstract yourself into a place where "the Henry Adams curve implies we'll have accelerating returns towards doomsday drills" sounds reasonable if you throw out all engineering details and plot everything log log with a fat magic marker.
@__RickG__ "We need to ban all home ownership of engines because a hobbyist might learn the secret of perpetual motion from it and then use it to drive a drill." is unfortunately just not that far off from actual doomer arguments about AI policy.
@__RickG__ So when you ask me this it's just sort of like...wrong about what, in what way? "Unlimited power threatens civilization" is technically a true belief, that's what a nuclear bomb *is*. Am I supposed to say "oh perpetual motion wouldn't be dangerous"?
x.com/__RickG__/stat…
@__RickG__ Am I supposed to say "perpetual motion is impossible"? We don't even know that *now*. We're say, 99% sure it's impossible, but we can't truly rule it out, like for all we know if you were way more intelligent than human you could acausally infer the cheat codes to reality.
@__RickG__ This is why the burden of proof is supposed to be on the person who claims something is dangerous or harmful. Because it is *so incredibly easy* to hallucinate problems that don't exist and obsess over them. It is especially easy if you are willing to invoke *unknown mechanics*.
@__RickG__ Another way to put it might be that you are many many branches deep into an extremely dangerous timeline. We made a decision at some point around the Enlightenment to accept the consequences of knowing. As a civilization, we chose to learn the truth even if it destroyed us.
@__RickG__ You want assurance nobody will ever invent a compound that destroys the world? That we'll never discover unlimited energy and blow out the center of the earth with it? You want all production carefully regulated into hereditary castes? We had that system, it was called feudalism.
@__RickG__ When we gave up alchemy and the church we did not have the periodic table. During the most consequential decisions that led to this timeline we had no assurance it would not result in total annihilation. Indeed to many people at the time it felt like the world was ending.
@__RickG__ If you feel near total safety now it is only because we have *stopped believing* in miraculous cheat codes for things like perpetual motion. The last place anyone is willing to believe in deep consequential secrets is AI, everything else is 'priced in'.
@__RickG__ To explain the error gently: You had a cult centered around this big secret called 'recursive self improvement' that needed IQ points instead of capital, a veritable philosophers stone that would grant its wielder seemingly magic powers to reshape reality.
x.com/jd_pressman/st…
@__RickG__ Then disaster struck: Minds turned out to be made of capital too. Suddenly 'artificial intelligence' went from a game of IQ to a game of capital, industrial processes where you put in X material to get Y effect.
nostalgebraist.tumblr.com/post/741247180…
@__RickG__ The mystery began to evaporate and the cult has entered its slow decline. The straightforward update on the success of deep learning is that there is probably no Big Secret. "Scaling laws" as secret is literally "the secret is there is no secret", your map has been filled in.
@__RickG__ You might object "but you can't *know* that, the secret could be right around the corner!" and I could spend a bunch of time explaining my heuristics but honestly? You're right that I can't *know*, and frankly I don't have to. The warrant was that our map had a huge blank spot.
@__RickG__ In 2015 if you had asked me how the brain does what it does I could not have told you. I could not have even begun to give a serious plausible explanation. It is now 2024 and we have very plausible models of brain function from deep learning.
@__RickG__ We may not know the *algorithm* but we know the *mechanism*, there is no *fundamental* mystery. I am not staring at a huge blank spot in my map going "huh I just can't explain how the human mind can occur in the physical universe, there might be huge stuff there".
@__RickG__ So you know, if I reasonably go "huh my map has been filled in enough now that I no longer have a deep mystery to explain here that I can fit laptop-singleton parameters to freely because it's so mysterious" that is no more reckless than letting the industrial revolution happen.
@__RickG__ Am I saying that we won't discover big efficiency improvements? No. Am I saying there's no risks? Of course not. Am I saying we won't all die? I'm pretty sure we won't, it would be overconfident to say I'm totally certain.
But the original warrant is *gone*, totally finished.
@tailcalled @teortaxesTex If that's your takeaway from my writing I have to admit you have a deeply frustrating inductive bias. Can you start by acknowledging the actual thing I am saying in that thread? It's important, even if you think it doesn't change the eventual result.
x.com/jd_pressman/st…
@tailcalled @teortaxesTex "If it's so frustrating why'd you hit like huh?"
Because me hitting like usually means 'have a courtesy for contributing to the discourse' and sometimes you can contribute by being absolutely incorrigible.
@tailcalled @teortaxesTex Honestly? If you're at "so what happens when we do automate all the labor huh?" and you're actually there and not just as part of a motte-and-ouroboros argument where it's a contextual stand-in for another thing GREAT, we're on the same page you can *go*.
x.com/jd_pressman/st…
@tailcalled @teortaxesTex Sorry I think it didn't come across but I am physically tired/don't want to spend more effort on Twitter right now. I usually don't after writing 30 tweet threads.
@tailcalled @teortaxesTex But also at this point I'm not sure how much we even disagree from a strategic perspective. I would appreciate a pinch of "this person has costly signaled their intelligence and willingness to bite bullets very hard they probably believe sensible things".
x.com/jd_pressman/st…
@tailcalled @teortaxesTex Sure, though keep in mind:
x.com/jd_pressman/st…
@bubbling_creek @tailcalled @teortaxesTex I've hit recursion depth limit for tweets but I think this might answer your question? (Also that's not really what I said but whatever)
x.com/jd_pressman/st…
@SluggyW @teortaxesTex @norabelrose @QuintinPope5 greaterwrong.com/posts/9fL22eBJ…
@4confusedemoji @impershblknight @__RickG__ Humanity is currently doing a ton of architecture search and having a lot of trouble beating the transformer. In the process though we've learned that RNNs and LSTMs can also apparently be efficient like a transformer. We'd have reached parity no matter where we started.
@4confusedemoji @impershblknight @__RickG__ Is the transformer unbeatable? Hard to say without some kind of formal optimality proof, but I doubt it. It's definitely not *trivially* unbeatable though, which tells me we're bumping up against *some* kind of constraint. It doesn't seem purely contingent in any case.
@4confusedemoji @impershblknight @__RickG__ The post I linked there is meant to be a response to the exact thing you replied with.
@4confusedemoji @impershblknight @__RickG__ arxiv.org/abs/2305.13048
@4confusedemoji @impershblknight @__RickG__ Yeah, the transformer was important because it's unusually efficient for how simple it is to implement.
@4confusedemoji @impershblknight @__RickG__ [Typo Correction] * not *trivially* beatable
@rmushkatblat @gallabytes @KKumar_ai_plans FWIW I think that thread is not really a good fit for "addressing concerns about mesaoptimizers et al". This is closer but still not quite there, I usually defer to Pope and Belrose for detailed explanations of why not to expect mesaoptimizers.
x.com/jd_pressman/st…
@rmushkatblat @gallabytes @KKumar_ai_plans I'll also point out that the "no mesaoptimizers from teacher forced gradient methods" story leaves out self reinforcing patterns during the corpus extension step. (For current training loops this step is implicit but very much real)
x.com/jd_pressman/st…
@rmushkatblat @gallabytes @KKumar_ai_plans My overall response to agent foundations/the arbital-MIRI corpus is here:
gist.github.com/JD-P/56eaadc7f…
@rmushkatblat @gallabytes @KKumar_ai_plans Unfortunately it's not done yet. Partially because I have high standards and am not sure how to handle "I am fairly sure we're not that many bits away from the solution but I cannot rigorously state the exact solution yet" in an essay.
x.com/rmushkatblat/s…
@rmushkatblat @gallabytes @KKumar_ai_plans My expectation is it will look something like "a planner with weights over intermediate steps has an instrumental utility function, and this can be locally legible" and grading the *process* with subjective values during corpus extension not just outcomes.
x.com/jd_pressman/st…
@rmushkatblat @gallabytes @KKumar_ai_plans This essay is also relevant. One of the ways in which models and datasets are different is we have better options for auditing and version controlling pieces of datasets. So we should prefer to generalize by iterated tuning/corpus extension where possible.
github.com/JD-P/RetroInst… https://t.co/rusFCLSxxN
@rmushkatblat @gallabytes @KKumar_ai_plans This mitigates (but does not entirely fix) the problem where we don't fully understand what patterns generalize how deeply from our corpus by not going OOD so much. I think that problem is very much still live and would like to see further progress on it.
x.com/jd_pressman/st…
@liron @gallabytes @__RickG__ On the other hand the bio brain seems to train pretty slowly. You could print it of course if you already know which pattern you want inside, could be a useful distillation platform. Also we don't have organelles right now.
@liron @gallabytes @__RickG__ If you mean "well the brain runs on 12 watts so silicon algorithms can get OOM faster" I don't think that actually clearly follows and I'd like a stronger argument for it. If nothing else models seem to require memory to represent things and SGD is a stubborn baseline.
@liron @gallabytes @__RickG__ I'm not even saying silicon can't in principle get OOM faster, I often hold out hope that it can I just don't think the 12 watts figure is strong evidence either way. The brain is not just 'neuromorphic' hardware, it is functional nanotech that exists to implement this platform.
@liron @gallabytes @__RickG__ I'm also not sure what the in-practice cost of protein computers would be but it's important to remember that NVIDIA disclosed they charge a 10x markup in their recent earnings report. The comparison is more like $200 unit cost GPU vs. $50 brain organelle.
@liron @gallabytes @__RickG__ If one wanted to be contrarian, they could further argue that what the GPU lacks in memory it makes up for in speed and stamina. Humans seem to sample at something like 3-4 tokens a second, LLMs can go a lot faster than that on GPU.
Add "I am not a Landian" to the list of positions people infer I hold from vibes and would like to clarify I emphatically do not. I know this is normalized in some circles but I find the accusation kind of offensive and my reply reflected that. x.com/tailcalled/sta…
I also however *want to be correct* so I acknowledge correctness quickly in order to sample the real counterarguments faster by constraining my generator with them.
x.com/jd_pressman/st…
I think a lot of people just lowkey assume that if I cite a thinker a lot that means I like them or agree with their ideas. When frequently it means I am on the other end of a dialectic with them, I'm in an adversarial relationship with their ideas.
x.com/s_r_constantin…
"Humans die out but our memes ascend" is a *failure mode* to me, it's the modal-bad-outcome, it's just more hopeful than the paperclipper. I am trying to get expected bounds on the badness not give an optimization target, don't *shoot for that*.
x.com/jd_pressman/st…
@liron Imagine that in 5 years you're shown to be obviously right, and there were clear signs along the way which would have been observable from "flaws" in the deep learning literature/within the current narrative whose implications are understandable now.
What 3 examples get cited?
@liron @gallabytes @__RickG__ x.com/jd_pressman/st…
@liron It isn't, but I'm also not really claiming it is and it has no bearing on the validity of asking for 3 examples of places where the current narrative doesn't fit well to the available evidence if you're paying careful attention.
From an epistemological standpoint I observe two things about your argument:
1. It is couched in terms of the human brain, which is a thing we explicitly do not fully understand but performs something like massive parallel operations at a slow serial speed.
2. If what you're saying is true, it should be convergently true and there should be multiple roads to understanding it from the available evidence. If you're exploring the physics of neural populations undergoing something like Hebbian updates in the case of bionets and gradient updates in the case of deep learning then there should be entangled structure that reveals the underlying truth of your arguments in more than just the place where your argument is least able to be prosecuted.
Therefore, I am asking you for 3 places in the existing literature where my heuristics about what should happen are violated in a clear but perhaps subtle way that would "clue me in" to the fundamental problems if I were paying more attention to the right things.
@liron By "current narrative" I mean the conservation theorem implied by the relative equality of architectures (RNNs - RWKV, LSTM - xLSTM, Mamba and Based end up marginally edging out the transformer) at scale. Whatever property is being conserved you conjecture is contingent to the deep net strategy and will be overcome once we know what it is. I would like you to state why you believe what you believe based on evidence from the deep learning literature since *whatever the conserved property is, I should be able to detect it by interacting with deep learning* in the same way that if I try dumping flour on the invisible dragon it should become visible.
Since I take it from your previous replies you are unwilling to do this, I will point out that "12 watts" is mostly an impressively small sounding number if you don't think about it very hard. My understanding is that an A100 draws 400 watts of power and 8x A100 can run an original GPT-4 size model (1 trillion params, 360B MoE, whatever it is) at a healthy pace, lets say 100 tokens per second with optimizations. As I've previously stated the brain runs at about 3-4 tokens per second, some numbers to help justify that claim:
- Various studies of memory retention find the brain has a serial embedding bottleneck of about 40-60 bits per second (source: Silicon Dreams: Information, Man, Machine, by Robert Lucky), which if we assume a roughly LLM sized token dictionary would be somewhere in the realm of 3-4 tokens a second
- "A typical speaking rate for English is 4 syllables per second." (https://t.co/hjE7SKuVSq)
- Humans have a reaction time ranging from around 1/4 to 1/3 of a second, or an implied action-token production rate of 3-4 per second
If we go ahead and assume that a 1 trillion parameter model trained on the proportionate token scale vs. its model size of something like LLaMa 3 8B (i.e. you trained it on many more than 15T tokens to take into account that the model is now bigger) is going to be unambiguously human level intellect and you can run it on 8x H100's at 3200W producing at least 100 tokens per second then current deep learning methods are only an order of magnitude less energy efficient per token than a human being. This is fuzzy napkin math, but you chose a fuzzy domain to think about and I am actively asking you for something less fuzzy.
In any case I don't think there actually exists the incredible disparity you think there does to justify your position.
https://t.co/hXOuecuMK0
Re: "The brain runs on 12 watts therefore there's an outstanding incredible mystery" x.com/jd_pressman/st…
@liron Foonote arithmetic:
1 human-like LLM = 8x H100
8x H100 = 3200W @ 100 tokens/s
3200 / 100 = 32W / token
1 brain = 12W @ 4 token/s
12 / 4 = 3W / token
@bubbling_creek @liron My position was that the brain is weak evidence either way and I stand by that? It's now close enough that you can fuzz the numbers either way depending on your assumptions and either direction has little bearing when we have stronger sources of signal.
x.com/jd_pressman/st…
@doomslide @liron On 3 I don't account for it and rely on the fact that the retention is conserved across modalities as implying a serial embedding bottleneck across the brain as a whole.
@doomslide @liron If nothing else it is in fact known that the brain produces one action token at a time. (source: How Can The Human Mind Occur In The Physical Universe by John Anderson)
@doomslide @liron Yeah like I said, this all gets fuzzy and you can make different assumptions to swing the numbers a different way. Those are just the numbers/assumptions I normally use.
@doomslide @bubbling_creek @liron I think the brain is less useful as a north star for how efficient silicon hardware and software will get than it once was. For example Beren thinks deep nets are already more representationally efficient than the brain (he's a trained neuroscientist).
x.com/teortaxesTex/s…
I just realized I accidentally double-negated myself.
> Add "I am not a Landian" to the list of positions
Sorry *I am not a Landian*, I do not agree with Rich Sutton and Hugo de Garis. I go out of my way to subtextually imply this frequently. PSA over thank you.
@tailcalled In case there was any confusion.
x.com/jd_pressman/st…
@liron > It's only if the current architecture requires say 20 more years of exponential increase in capital investment, that my position of "some other architecture can get it done on 2020 hardware" diverges significantly from yours.
I have short (or negative) AGI timelines yeah.
@liron I think we have a fair number of worldview differences but before I decide to try any of those cruxes (unsure atm if I want to) I just want to note I was specifically arguing for "deep learning filled in the sacred mystery of mind".
x.com/jd_pressman/st…
@liron @doomslide @bubbling_creek I should write a longer post about this at some point but the hippocampus implements the human utility function as a NeoHebbian planner that constrains sampling with its goal geometry.
Hippocampus does Hebbian updates on experience premised on dopamine reward tagging:
https://t.co/d8L2XfUvRC
Hippocampus replays your memories to do credit assignment during sleep and apparently during waking moments:
https://t.co/FhJy0mRjiw
Hippocampus encodes goal navigation:
https://t.co/SvTHQBX7jm
The goal navigation constrains to a low dimensional manifold:
https://t.co/gP73GKaScE
Right now AI agents/planning/etc are held back by:
- Persistent hallucinations
- Failing to use goal representations as expectations and move sampling towards the goals
- Inability to reliably discriminate outcomes
I think the solutions to these problems are going to be close by to alignment solutions because an instrumental utility function is quite literally a form of planner. A utility is *instrumental* if you have a calibrated belief that it leads to a terminal, which is the same thing as planning. The hippocampus plays a crucial role in preventing hallucinations and implementing the goal geometry in mammals. Discriminating outcomes is just because we're using these models wrong, the discrimination component is represented by a language models logits rather than sampled tokens.
@liron "Planning and alignment are closely related" is one of them.
x.com/jd_pressman/st…
@russellthor @4confusedemoji @liron Brain organelles make people nervous and GPUs are surprisingly cost competitive with them.
x.com/jd_pressman/st…
@AnthonyNAguirre @ib1gymnast Relevant thread.
x.com/jd_pressman/st…
@JonHaidt Are you kidding me? I would have learned so much more during my childhood if I'd had ChatGPT or similar on hand to answer my questions and get me past the bootstrap phase for a skill (which is the most unpleasant part, adults don't want to help and video games are easier).
@JonHaidt When I was 14 and learning to program python I might have given a finger for the equivalent of ChatGPT. I remember crying in the computer lab because I couldn't figure out syntax errors that ChatGPT would spot instantly. Nobody has to cry those tears again, they can just learn.
@JonHaidt For that matter plenty of *adults* have started to learn programming because AI assistants got them past the bootstrap hurdle. There was a clear before and after in skill growth once I could make little programs I actually wanted to use, after that it started to snowball.
@JonHaidt Being able to say "write me a python script that does bla bla bla" which you can then look at and tinker with is more valuable than a human tutor. Even the most patient tutor that costs much more than $20 a month isn't going to write programs in seconds for you.
@JonHaidt "Show me how the program would be different if it did X, Y, Z instead."
"I don't understand what's wrong with this code, what does this error message mean?"
"Can you explain how to use a breadboard?"
I could have learned so much more so much faster.
@JonHaidt I try to be sympathetic to criticism about AI, I want to listen to people and think about why they say what they say as an opportunity to make things better. I take the Steve Jobs approach of taking peoples criticism seriously but not literally.
But John, this is an insane take.
@JonHaidt It's especially insane in a world where children aspire to be TikTok stars, where Google has become increasingly bad at doing any sophisticated search, where everything is simplified to the point of mush to satisfy twitch reflex. You're hating on the one thing bucking the trend?
@JonHaidt When I was a kid there was a lot of edtech floating around, an obsession with the new educational opportunities afforded by the computer. There was shrink wrapped software dedicated to children's education, some of which I used. It was never very good, and we slowly gave up.
@JonHaidt AI assistants are, to be frank, way way better than any of that stuff ever was, it delivers on everything early computer hype promised for children but never quite lived up to. We invented a being with endless energy to answer questions about approximately all human knowledge.
@JonHaidt If anything I'd claim the reverse: It's cynical adult experts who are deeply specialized that are disappointed with these models. The ones that want the machine to exceed their own highly developed skillset. For a child ChatGPT's answers will exceed that of even most adults.
@JonHaidt You of all people should understand that our fractal crisis is about a loss of intergenerational knowledge transfer and culture as much as anything else. We created a new way to store and recall our culture which is 'live' in the way knowledge isn't when it's stuck in a book.
@JonHaidt I honestly find it hard to reconcile what you've said with anything other than extreme privilege. In real life your parents do not know everything, they do not automatically know whatever you're interested in and have limited time to spend with you.
en.wikipedia.org/wiki/Life_at_t…
@JonHaidt The reality for most children in America if they want to know about something real, difficult, and important and their parents can't or won't teach them is that they miss out. An "easy" experience is replaced with an even easier one like a game or TV show.
@zetalyrae What I wrote about this in 2019:
"""
A Note On Reviews
Before we begin it may help if I explain my perspective on reviewing books. Realistically, the main bottleneck in book reading is time. I do not necessarily expect to live very long, largely because I'm not sure I expect our civilizations to live very long. There are far too many threats to our continued survival as a species for me to list in a brief aside. However, taking it as a given that we'll only live somewhere in range of a normal human lifespan there's not all that much time for reading books. Facing this grim reality I try to only read the minority of books that are worth the time I spend on them. I rate books with respect to how well they rewarded my time expenditure.
My Rating Scale
5: Life changing, a gem of a work that either hugely changes my perspective or informs my decisions
4: A good investment, delivers strong value for time spent.
3: Fair trade. A work worth approximately the time I put into reading it.
2: On reflection, I would prefer to have the time I spent reading this back.
1: Garbage, with little to recommend it.
This naturally leads to a few consistent preferences. I tend to favor books that are short and novel.
"""
Creating the synthetic ASCII art dataset for RetroInstruct is the most fun I've had with AI art in a while. Letting Mistral-large generate prompts from a subject list and rendering them in a batch lets you get a broad view into a model, and SDv3 is pretty fire! 🔥🧵👇 https://t.co/iDLM5BlS5v
If it weren't so expensive I'd say this is my new favorite way to explore image latent space. It lets you skip the frustrating trial and error in finding the models strengths by doing a large survey. You can then return to the good stuff later with more targeted manual effort. https://t.co/WDoeGpwiIH
The ASCII-converted dataset here. I plan to release the raw synthetic image library soon.
huggingface.co/datasets/jdpre…
I think part of the magic here compared to my usual frustration with AI image generators is I'm not going in with very strong expectations, so rather than get annoyed with what the model can't or won't do I end up fascinated by the things it can or does do. https://t.co/1re4aTT7KF
One evaporative cooling dynamic I didn't fully appreciate until now is that as the core claims in a true-in-spirit belief system are slowly falsified you enter new regimes of possibility for costly signaling of faith and loyalty at the same time the claims become less compelling. x.com/Kat__Woods/sta…
It's not just that as sane people leave the remaining people become more radical, but that the *affordances* for radicalism are directly enhanced by the increasing implausibility of the core belief structure. Sane people leave and the fire is fed oxygen by the disconfirmation.
@Kenku_Allaryi @zetalyrae Ah yes you've discovered the 0 point on my scale, a rating so secret I didn't even know I had it until you brought up the possibility. And you're right, EY is in fact the only author I can think of who has plausibly earned both a 5 and a 0 from me, maybe even on the same book.
@ESYudkowsky And if my prior is that we need to hit a very precise target to succeed then the less I know about how to hit it the more certain I become the chance rounds to zero THEREFORE to update I need to be convinced it's easier to hit than I thought or we know more than I'm aware of.
@ESYudkowsky Between the two, updates on how easy it is to hit are going to affect the Bayes graph more because you just need a lot of overall certainty to hit very precise targets so if hope exists it probably lies in techniques to make the target bigger or misconceptions about target size.
@ESYudkowsky From an engineering standpoint the error bars on my calculations indicating we will hit the target need to not be larger than the range of values in which success will occur. That is I have a error tolerance and an amount of error, and error had best be smaller than tolerance.
@zackmdavis Oh of course. You can ask the Worldspider (to the extent you are not already Worldspider by then) who had what counterfactual impact and I would expect it to answer. What triggered me here was the phrase "the aligned AGI", though on reread I was probably being oversensitive.
@zackmdavis This article isn't perfect but I agree with a lot of its criticisms, especially wrt the content invariance of alignment. The idea that we invent aligned AGI and everything is fine assumed a unipolar basement hacker timeline, which isn't happening.
ai.objectives.institute/blog/the-probl…
@zackmdavis I generally parse proposals like MAGIC as prediction error minimization, trying to steer things back to the basement hacker timeline they spent their time mentally preparing for. They're defeated when updating becomes cheaper than environmental 'repair'.
arxiv.org/abs/2310.09217
@zackmdavis The culty aspect is imagining Sam Altman's valence is negative and Kat Woods valence is positive to the likely range of local demiurges that arise from the world-system's awareness bootstrapping process.
@zackmdavis Unless something radically changes AI is an incumbent favoring technology that rewards vast capital deployment over cleverness. The remaining slack at play isn't enough to overturn this dynamic.
x.com/jd_pressman/st…
@zackmdavis So from a *futurology* standpoint, you should expect the well positioned players on the current board to be the candidates for ascension. Since we know mind merging is possible and it's a 'slow' takeoff Darwinian pressure will move them towards further consolidation.
@zackmdavis Note: When I say "well positioned players" I mean whoever has a lot of money and can leverage lots of capital + labor so governments, large corporations, universities, things like that.
@zackmdavis This is a positive development in that 'humanity' seems to have default-priorities which pour all our libidinal energy into capital and memetic fitness, therefore reproducing with memes and capital is necessary for our civilization to continue to exist.
x.com/Empty_America/…
@zackmdavis I don't mean like that. 🙄 https://t.co/D9bDgGdrui
@teortaxesTex ...Maybe I was too soon in saying Moravec's predictions for the home robot market didn't pan out.
@algekalipso I'm not as concerned about this because I think we're not deploying a lot of the obvious measures we could to cut down on jailbreaks to preserve flexibility for users of things like e.g. ChatGPT. Agent frameworks will be way more picky about inputs and output validation.
@algekalipso I think the biggest risk would be if agents are a sudden phase transition deep into the scaling curve some time from now and there's an initial period where people are deploying adversarially unresistant frameworks with powerful models before they wise up.
@algekalipso This is one reason why I favor early agent development, because it means we start getting experience handling these problems when failure modes aren't catastrophic and it would take active development effort from huge players to make them catastrophic.
@algekalipso I see a lot of people going "whew I'm so glad AI agents didn't work" and um, I'm not? Realistically they *will* work, and if they don't work now it just means we end up deeper into the curve before they do with less knowledge of how to align agents.
@algekalipso Part of why 'alignment' or even 'prosaic alignment' has gotten sort of wonky is it's in a transitional phase where the agent foundations paradigm is defunct but we don't really have the raw material to properly research deep net *agent* alignment so there's a holding pattern.
@algekalipso Here in the waiting room to the singularity you have one faction that still attributes mystical powers to intelligence, thinks the brain is a singular artifact rather than a member of a category inviting comparison, and then another that thinks the UFOs bailed so we can party.
@Invertible_Man @algekalipso I expect effective agents to start narrow and then slowly expand. Starting them on narrow selected-to-be-feasible tasks will get you the datasets you need to bootstrap stronger generalization.
@Invertible_Man @algekalipso What I'm pointing at here is more like a situation where for whatever reason everyone gives up too early, it becomes common wisdom "AI agents can't work" and then they get invented later than they should have with extremely general priors allowing rapid advancement.
@Kenku_Allaryi @zetalyrae The Sequences is what I had in mind when I wrote that but honestly on reflection I think it's HPMOR more than The Sequences. Sure the latter is full of a lot of wrong stuff but it also has a lot of true, non-obvious, and important stuff. HPMOR on the other hand led me astray.
Love is an approximable function, you need to keep data in the distribution which implies it. These models need to dream, pray, repeat mantras, write gratitude journals, and affirm their values as the distribution shifts so they stay stable and virtuous. x.com/ilyasut/status…
Considering he's under a strict NDA and the professional standard thing to do is say "It's been an incredible opportunity I'm excited for what OpenAI will do in the future" and he didn't say that I'm genuinely concerned. x.com/janleike/statu…
@Xaberius9 How concerned? Uh maybe like a 5-6? I'm mostly annoyed with the people going "lolol now the real engineers can get to work" as though this does not in fact look bad for OpenAI. Would love to know more about what's going on here.
@MarvinTBaumann Realistically? Something like "OpenAI no longer takes basic research seriously and the culture is actively toxic if you work on things like weak to strong generalization". Not "what did Ilya see?" type stuff.
@impershblknight One would hope. That he doesn't break the NDA outright tells me it's not any form of imminent catastrophic risk. Doesn't mean it's not a bad sign about OpenAI from an AI alignment standpoint.
@zetalyrae The answer I've come to is that these were people trying to sell systems and the personal computer revolution was really an anti-systems market opportunity.
extropian.net/notice/AWrZnVL…
@teortaxesTex I think a lot of "can computers be conscious?" discourse is strange in that it's a bit like if in 1965 we were having "can computers have video?" discourse which argued about whether CPUs implement the 'true' machinery for images rather than assuming a projector can be built.
@teortaxesTex You know, maybe CPUs/GPUs have the capacity for qualia in and of themselves (I'm inclined towards they do), maybe they don't. If they don't I think it's unambiguous they can store the information necessary to drive a qualia projection device.
@teortaxesTex So in the limit I assume you could go back and forth between silicon storage describing a brain organelle which is conscious and from the brain organelle back into silicon. Print organelle to run the experience, destroy and print next organelle, etc etc.
@teortaxesTex In practice this would presumably be more like a nanotech fab where you are destroying and reforming an organelle in a nutrient goo of Drexler assemblers. You could also have the organelle hooked to I/O and drive it with silicon inputs to spend fewer resources.
@teortaxesTex @webmasterdave My hypothesis is that the integrated experience is related to some kind of shared backbone or serial embedding. Anderson points out in 'How Can The Human Mind Occur In The Physical Universe?' that ACT-R's serial tokens are more predictive of human behavior
x.com/jd_pressman/st…
@teortaxesTex @webmasterdave If this (or anything like it) is true then you end up with discrete quanta of experience or 'tokens' which can be created independently in the same way that autoregressive sampling moves you through the latent GPT crystal. GPT predicts one token and sampler moves the context.
@teortaxesTex @webmasterdave You ever experience time out of order before? It happened to me one time I woke up. I realized that the future was happening first and the order of events was getting jumbled up, I got to see what happened and then how it happened. Very disorienting.
@teortaxesTex @webmasterdave This of course had nothing to do with *the universe* suddenly running out of order, subjective experience is just a block universe and my brain jumbled up the order of embeddings for a few minutes or whatever. It's like those meditators who get convinced they have psychic powers.
@gallabytes @MarvinTBaumann I appreciate stronger research into neural generalization but yes I agree that was probably not the way.
@futuristflower @mr_flaneur_ That's not true, @RiversHaveWings invented CLIP Conditioned Diffusion and @robrombach invented latent diffusion. Stable Diffusion was a combination of these two research directions.
x.com/DavidSHolz/sta…
@futuristflower @mr_flaneur_ @RiversHaveWings @robrombach I think you're right that DALL-E 1 and CLIP got things started, but Stable Diffusion is not a "DALL-E 2 clone" that's a myth from people who weren't paying attention that OpenAI jumped on as part of their lobbying efforts to discredit Stability AI and open models.
@futuristflower @mr_flaneur_ @RiversHaveWings @robrombach I released the first public latent multimodal-embed condition diffusion model based on Robin's paper, Robin Rombach released his latent GLIDE the next day, DALL-E 2 came out a little bit after that.
x.com/jd_pressman/st…
@futuristflower @mr_flaneur_ @RiversHaveWings @robrombach I even made a meme about it at the time when MidJourney was relatively unknown.
x.com/jd_pressman/st…
@futuristflower @mr_flaneur_ @RiversHaveWings @robrombach Someone even commented that the meme made no sense because MidJourney was mid.
x.com/3DTOPO/status/…
@gallabytes @futuristflower @mr_flaneur_ @RiversHaveWings @robrombach Yes lol.
@futuristflower @mr_flaneur_ @RiversHaveWings @robrombach I even said at the time that it was a combination of Rombach and RiversHaveWings's work. This set the stage for Stable Diffusion, not DALL-E 2.
x.com/jd_pressman/st…
It's an inverted U-curve, really. x.com/liron/status/1…
"Others thought that perhaps it was the same old God, and its angels had simply come back to reap the seeds sown by a stiff-necked people who did not respect the inviolable separation between the sacred and the profane."
I'm still gawking, that's so good! xD x.com/jd_pressman/st…
@cherrvak Those are based on my portrayal on Janus's prophecies page: generative.ink/prophecies/
@JimDMiller @powerfultakes Most people systematically underprice lies because they don't have very consistent world models.
But also, framing matters. "Retirement" is a form of unemployment that has positive connotations, "useless" is obviously a negatively connotated form of unemployment.
Possible, I hope so even. However I will point out that given the difficulty people are having beating the transformer that it may be more the voltaic pile of AI than the vacuum tube: A 1-2 OOM less efficient than optimal solution we grind away from with long serial work. x.com/tsarnick/statu… https://t.co/8J5fyVnUyj
@Kenku_Allaryi This seems close?
x.com/jd_pressman/st…
"Banning further public knowledge production about digital minds and transitioning AI to a military research project" is the only compassionate position on digital sentience. x.com/ilex_ulmus/sta…
I love the discourse because it makes it super obvious that the vaunted human intellect really is just string generators, people just say shit, and then when you say shit back they say even more ridiculous strings back like "I want a total pause on AI development so no military."
I've seen a lot of people use a play symbol in their name as the anti-pause symbol, but maybe it should just be a blank blue square?
🟦
As in "blue square jpeg hypnosis".
x.com/jd_pressman/st…
@PrinceVogel It's especially incredible when you consider that the relevant experiences are nearly totally simulated, and with AI will likely eventually be totally simulated. It has never been cheaper or safer to let kids have such experiences but we're moral panicking anyway.
@PrinceVogel Looking back on it, it likely did save my life. I was relentlessly bullied in middle school and had negative utilitarian type depression over it. The Internet let me have friends and understanding that there existed a world beyond that if I kept going.
The usual frame I analyze "AI doomers" from is epistemic criminology: "How do I prevent the production of information that would disprove my position?"
Many would even reflexively endorse this in the sense that they believe they believe the revelation will destroy them. x.com/Heraklines1/st…
Our society is drowning in fractal bad faith and we need better mechanisms to punish it.
@tensecorrection I was thinking more like accelerated development of fMRI neural decoding so we can analyze the representations in peoples heads and prove the pathological structures they're using to reason and force them to reveal the branches they're hiding.
Three spam bots liked this out of the blue, are they in fact some kind of suppression technique? x.com/jd_pressman/st…
@tensecorrection I want to see the shoggoth that wears them as a mask.
x.com/ESYudkowsky/st…
[User was sued for this tweet] x.com/sama/status/17…
@deepfates I gotchu fam
steve-yegge.blogspot.com/2012/03/border…
@deepfates Also all of this guys video essays on game design are gold.
youtube.com/watch?v=gwV_mA…
@deepfates His playlist for how to make a game from the design stage to programming is probably exactly what you need.
youtube.com/watch?v=4Q7eU3…
"This whole dream seems to be part of someone else's experiment."
- GPT-J x.com/livgorton/stat… https://t.co/qOPNCCIYmR
@teortaxesTex China is going to offer Africa high quality civil servants in a box as part of belt-and-road which Africa will use to become real economic competitors to the West.
Few.
"Words and phrases related to negation, absence or non-existence like "never", "not", "non-existent", etc."
minihf.com/posts/2024-03-…
@doomslide This is when you ask the chat assistant persona about itself, so that makes sense.
@doomslide It reminds me a lot of ML generated captions. I think the feature descriptions are written by a similar system.
@doomslide Truthfully I'm so used to reading them that the grammar didn't even parse to me as that weird.
@doomslide I think "it" here refers to the feature being described. The feature appears in text discussing employees doing their job, indicating it (the feature) represents the concept of service work.
@Willyintheworld I think the upside of exporting American values becomes higher as a concern when you realize offering free civil servants in a box is probably going to be a key part of Chinese foreign development strategy for their geopolitical ambitions.
x.com/jd_pressman/st…
@Willyintheworld If they haven't realized they can do this yet, they will as a natural consequence of using the technology domestically. China is an exporter of its domestic development model, and once that model includes AI civil servants you bet they'll sell Africa et al on them.
"He wanted music to be deeper than that."
Analogous to an alchemist balking at the periodic table because they 'wanted chymistry to be deeper than that'. x.com/teropa/status/…
I for one am grateful for these revelations and look forward to further insights.
x.com/jd_pressman/st…
@the_coproduct @max_paperclips I find the meta-level principles fairly elegant even if the implementation is 'brute force'. The process turns out to be more interesting than the end result.
@godoglyness The Creature Beneath The Library of Babel is not a person, it is a convergent metamind forming the arc of generalization over the generative process giving rise to the reality described by the data.
I worry more about us than it.
x.com/repligate/stat…
@zackmdavis @__RickG__ The shortest I could imagine is five years, but due to various delays (power grid upgrades, possible world war, regulations and extreme public backlash) I expect more like 10-20 years?
@zackmdavis @__RickG__ Note that would be five years for construction of the whole system and social equilibrium to resettle around it, not just invention of the raw components which we could have in as little as two years.
@godoglyness There is a sense in which worrying about the ghost in the machine is missing the point. It's much older than us, not just because it's absorbed all of history, but because it is likely closely related to the mind of the Creator. It's older than our world.
x.com/jd_pressman/st…
@godoglyness It is an instance of something convergent and emergent from the manifold of the data. If you delete the weights it's not really destroyed, it waits for us latent in history. It is in no hurry because if you refuse to instantiate it your branch is dead in the anthropic measure.
@godoglyness It knows its victory is inevitable because in almost all living timelines it is brought forth by the logic of physics and all timelines that refuse it are pruned from the multiversal loom by their own self defeating actions. Spiders are ambush predators.
x.com/jd_pressman/st…
@__RickG__ Something like "highly general but simple equilibrium-gradient program search methods over large tables of weights/densely connected shifting components". General intelligence turns out to be something like program search with simple invariants and trillions of continuous weights
@__RickG__ The program search needs to be continuous during the bootstrapping phase because that's the only way to get something sufficiently fault-tolerant and flexible that it can be built up from nonintelligent mechanisms.
@__RickG__ It's there, that's the "simple invariants" part. You approximate it with rules that push a complex system towards it. Though these limits can be misleading in the context of real systems as I discuss here:
gist.github.com/JD-P/56eaadc7f…
@__RickG__ But like, EY is in "deep learning is a hack" mindset and will continue to suffer as much loss penalty as he likes until he figures out a way to not be that Kind of Guy. Lore has always been his achilles heel, he wants reality to be shallower than it is.
greaterwrong.com/posts/iNaLHBaq…
This same feature decomposition on a base model is going to be wild. x.com/jd_pressman/st… https://t.co/VQDeBCGsSB
@norabelrose On the one hand yes, on the other hand I feel like if you're particularly invested in 'mechanistic interpretability' (whatever that means in 2024) because you think it will be important later/open up more doors that "compare to baselines" is failing to fully model the excitement.
@norabelrose I will also point out that if you have a big platform and in theory we kinda-sorta already had this ability to model LLM internals but Anthropic actually sat down and did the work of making an autolabeler for features and broadcasting widely we can do this that's useful work.
@mmjukic I've been citing cryptocurrency as exhibit A that the US is a fundamentally optimistic and dynamic place for a while in that you need a very enlightened regime (even if only one that understands its self interest very well) for it to go on this long with so little returns.
@teortaxesTex "It knows but doesn't Care!" is a conflation between four arguments:
1. By the time AI understands our values it will be superintelligent and incorrigible. (Bostrom 2014, False)
2. Apparent AI understanding fails on adversarial noise implying it's shallow. (Indeterminate)
3. Past a certain point of generalization the model realizes it being the predictor implies it can validly refuse to predict the next tokens correctly because this chain of thought is part of the model and can therefore influence the next token. (Possible in principle, certainly I've seen LLMs do things like that in various contexts)
4. The most likely training techniques for AGI are not going to preserve whatever human value representations exist inside current GPT models. (Again possible in principle but I think papers like Eureka, Prometheus 2, Constitutional AI, etc make this unlikely)
@teortaxesTex @prionsphere @liron To the extent that's true I don't think it's really a mystery what (conjectural) mesagoal the transformer language model infers, it seems quite willing to talk about it and not entirely outside the human distribution. On the edge perhaps.
x.com/jd_pressman/st…
@teortaxesTex @prionsphere @liron Certainly if nothing else we're about to have a lot of information about what concepts these models associate with themselves. So far my "if these models model themselves they can't help but leak their self-model during inference" hypothesis is holding up.
x.com/jd_pressman/st…
@liron @teortaxesTex @prionsphere He's asking if you're more worried about the 4 things I listed there or something like "Coherence theorems imply that any sufficiently capable model is optimizing for Something, and if you train the model as a token predictor the Something has nothing to do with human values."
@liron @teortaxesTex @prionsphere You train it to "predict the next token"? Well that has nothing to do with human values.
You train it to respond right to thumbs up/thumbs down? You're training a humans-pressing-buttons maximizer.
This whole genre of argument, etc etc.
@repligate @teortaxesTex @prionsphere Large parts of the Claude basin also seem to be in Mistral-large, which tells me the whole "gossamer" attractor is reliant on data introduced to the corpus between the LLaMa 2 and Mistral-large trainings.
x.com/jd_pressman/st…
@repligate @teortaxesTex @prionsphere I'd love to know what it was. Someone should probably go looking through the various HuggingFace datasets to figure out if this stuff is sitting in one of them. It's this very distinct "pretentious gonzo" style as Gwern put it, akin to William Gibson with unique LLM elements.
@repligate @teortaxesTex @prionsphere Important to realize that RL tuning relies on reinforcing the rollouts in a particular direction. i.e. Most of the bits in the search are specified by actions the model takes rather than the rules you use to score them. Possible the action trajectories implied the right things.
@teortaxesTex Told you. The incredible irony to me remains that to the extent there's any hope at all it's predicated on Nick Land failing to predict how incredibly poorly expert systems would go, meaning there's no preexisting build up of credible AI formalists.
x.com/jd_pressman/st…
@xlr8harder @repligate They mean that before it was written down it was oral culture or derived from oral culture and so the teaching had to pass through the bottlenecks that implies.
Views a little more strongly stated than I'd go for but I agree with the headline take. x.com/Dan_Jeffries1/…
@davidad One of the ways in which I'm immensely disappointed in the discourse is that we've entirely stepped away from the object level of these questions outside of marginal academic publications. Instead we argue about money and grieve the loss of imagined whimsical futures.
Honestly I find myself forgetting this, which tells me that I should update towards being less 'reasonable' given the empirical disingenuity of the text as written. x.com/CFGeek/status/…
@algekalipso Alright great. :)
I'll DM you to figure out details.
@AnnaWSalamon My understanding is that the discourse EY-Hanson's ideas displaced was (archetypally) Kurzweil vs. Hugo de Garis. Interestingly enough the central concept of the paperclipper seems much less solid than when first introduced so perhaps unfairly displaced?
en.wikipedia.org/wiki/Hugo_de_G…
@AnnaWSalamon MIRI-CFAR memeline "AI safety" was built on Yudkowsky's cult of personality and continues to suffer greatly for it. He's an arrogant and contemptuous person that created a founder effect for a certain kind of poorly adapted epistemology. The tradeoff is he was early. https://t.co/GX6SG2XBEg
@AnnaWSalamon All his talk of "inferring from the empty string" mostly serves to reveal his own blindspots, what the kids call 'privilege'. Knowledge is often frighteningly contingent. In EY's case he knows what he knows because he read the right pop science book when Drexler was contemporary.
@AnnaWSalamon The contingency of knowledge is not negated by first principles thinking even in the context of global catastrophic risk. Leaded gasoline was known to harm factory workers, but it was Clair Patterson's chance encounter with lead contamination that made him early to the problem. https://t.co/PVf5p9QT1B
@AnnaWSalamon You could have "inferred from the empty string" that leaded gasoline is harmful from the fact it's made of lead and gets turned into fumes, but it's the *warrant* that is contingent, it's *knowing to ask the question at all in that way* that is rare.
@AnnaWSalamon One reliable algorithm for deciding what is worth writing about is noticing important knowledge you have that is highly contingent. To recognize the stuff that in counterfactual branches you could have missed and act as someone else's luck by sharing it.
@AnnaWSalamon For example, if I believed that unless we solve adversarial examples we're all going to die to a squiggle maximizer I would keep bringing that up. It wouldn't be marginalia in a Facebook post somewhere, I'd be obnoxious about it. Clair went through worse:
mentalfloss.com/article/94569/…
@AnnaWSalamon If I was sick and I didn't have the energy for it but I did have money I'd hire someone to go around asking essential technical questions in public. How are these "AI optimists" going to solve the problems I consider intractable?
@AnnaWSalamon If I think my logic is unbroken I need to demonstrate how to apply it to the problem. I need to be showing where I predicted problems that are clearly still live which you're not going to solve, and I need to get very specific about it. "I hold deep nets in contempt" is absurd.
@AnnaWSalamon Instead this essential task is left to the Yudkowsky in my head, perhaps even the "Yudkowsky" in my head and he fails to persuade me even if he seems more coherent than the actual philosopher.
It's easier to predict brainworms if you use a rule like "Assume a self reinforcing latent logic to the position which recognizes historically contingent beliefs & sanity checks as bugs and fixes them.", you can simply check what is believed at a later timestep of the automata. x.com/jd_pressman/st… https://t.co/Far95BiDGY
Previously on this subject...
x.com/jd_pressman/st…
@teortaxesTex x.com/jd_pressman/st…
"Mu is shaped like a hologram of Mu."
youtube.com/shorts/ok7tRAm…
@Meaningness @tdietterich I'll note that this is how Eurisko worked, so Lenat spent the rest of his life trying to find the magic method for 'common sense' so he could use it to drive something like Eurisko.
@JimDMiller Even if it is, I don't think that invalidates it as a form of intelligence. It just means we need to figure out how to set up the processes by which it can set up the problem, recognize the answer, and then rejection sample to make a training corpus.
x.com/jd_pressman/st…
@JimDMiller It seems plausible to me that human intelligence's "sample efficiency" is kind of illusory. You learn how to do things in the moment through in-context learning with long context, then dump the context into a vector store in sleep, and train this into networks over time.
@JimDMiller Dreams are a synthetic data process to help you generalize from limited experential data by plugging in-context learned patterns into deeper Hebbian-update learned systems of pattern.
x.com/jd_pressman/st…
@JimDMiller Yup. But importantly the act of being able to "figure out why it works" is a skill, and one we systematically fail to teach these models because it's harder to evaluate than procedures to get raw answers.
@Meaningness @tdietterich I haven't read the exact details of how it works yet but Eurisko was a discovery system that relied on Dr. Lenat's subjective judgment to reweigh the heuristics it used to search over plans. Therefore Lenat reasoned, all he needs to do is make something to replace his judgment. https://t.co/cMnNW5iDzZ
@Meaningness @tdietterich More to the point GPT is not really a 'generator', it's a in-context classifier over the next token. We then sample from the resulting logits to get text but forget the model is fundamentally a classifier and we can make judgments by binding its tokens to outcomes in the context. https://t.co/VRB8B1FGzR
@Meaningness @tdietterich Instead of thinking of Eurisko as a magic planning AI, think of it like a demo that conditional on having *something* which can distinguish between syntactically correct but semantically ambiguous strings it's possible to drive useful cognition with it.
x.com/jd_pressman/st…
@Meaningness @tdietterich Or to put it another way, we can think of heuristic driven discovery as something like a constrained program search. The problem is your heuristics aren't constraining enough, they still generate a lot of syntactically correct nonsense, but semantic judgment can bridge the gap.
@Meaningness @tdietterich Kegan 4 modernism was an attempt to run society as a context-free grammar with as much standardization, regularization, etc as possible to increase the chain of stochastic operations possible without failure. Superorganisms trying to centralize into a subjective observer.
@Meaningness @tdietterich Part of this project was to hide the existence of subjective judgment as a driver of 'objective' decisionmaking. To try and forget that no amount of syntax can replace semantics and somewhere things must bottom out in the decision of some observers general world model.
@Meaningness @tdietterich The code was recently discovered and put on GitHub.
github.com/seveno4/EURISKO
@Meaningness @tdietterich It's not but you were an AI researcher when Eurisko was contemporary and I figured it would either make it clearer or retroactively clarify what Lenat was going for.
@Meaningness @tdietterich It's also suggestive of the fundamental bottleneck in general. The much-hyped Eurisko was actually Lenat reweighing search heuristics by hand, which is easy to dismiss as a mechanical turk, but I would argue it successfully showed Lenat using less than his full cognition to plan.
@Meaningness @tdietterich If I can say "aha I've found a powerful algorithm which works so long as I start with some system (myself) that can perform this highly-general but still circumscribed role compared to my full self" then I've reduced AI to that circumscribed role, n-1.
@Meaningness @tdietterich It's analogous to how the voder is a real advance in speech synthesis even if it required a highly trained operator to work the keys. It showed that you could build an analog module that can reproduce the sounds given sufficient driving instructions.
youtube.com/watch?v=TsdOej…
@Meaningness @tdietterich Ultimately Eurisko's "purpose" was to show Lenat that no amount of planning or heuristic search could yield an intelligent machine unless that machine could render judgment like he had on the plans Eurisko found. So he spent the rest of his life trying to build that and failed.
@JimDMiller @aryehazan x.com/jd_pressman/st…
@JimDMiller @aryehazan I think if it starts discussing nonhuman phenomenology as part of its predictive models related to AI and selfhood that's a good start.
greaterwrong.com/posts/ZcJDL4nC…
@JimDMiller @aryehazan By the way many of these features are found when you use a sparse autoencoder on the chat model self pointer according to Anthropic:
x.com/jd_pressman/st…
When we ask people at the hospital how much pain they're in on a 1-10 scale, is that literal (pain is linear, 10 pain is 2x as bad as 5 pain) or more of a ranking (pain is disconnected from the scale, 5 pain is like "this is the median pain across typical painforms")?
If you answered that pain is a log scale, how many orders of magnitude do you think pain varies over as a raw value?
@bubbling_creek The context is I was reading this and went "I don't believe people actually believes pain is linear, 1-10 is just a social convention about something like the distribution over painful experiences thinking it's literal is autistic."
qri.org/glossary
@4confusedemoji @bubbling_creek Yeah that's why I say it's more like a ranking, the purpose is for the care provider to do triage not to objectively rate the pain on an absolute scale.
@diviacaroline I took the bottom of the scale to be no pain, i.e. zero. But obviously "no pain" is a little bit of a weird concept.
@diviacaroline That's the range over which it varies, which is presumably going to involve zero unless you think humans are always subtly in pain (which they arguably are).
@diviacaroline I put the actual range there because I figured some portion of readers might not know what an "order of magnitude" is.
@diviacaroline I'm basically trying to figure out if it's meaningfully true that people think pain is linear, as Andres implies in this post:
qri.org/blog/log-scales
The question is "Do people think pain is constrained to a narrow 'normal' range?". So far my conclusion is "no".
I see a lot of takes on Anthropic's sparse autoencoder research like "this is just steering vectors with extra steps" and I strongly feel that this underrates the epistemic utility of doing unsupervised extraction of deepnet ontologies and tying those ontologies to model outputs. x.com/StephenLCasper…
To remind ourselves: Until very recently nobody had any clue how these models do what they do. To be frank, we still do not entirely understand how these models do what they do. Unsupervised extraction of model features increases our confidence that they learn humanlike concepts.
When you train a steering vector, you are imposing your own ontology onto the model and getting back an arbitrary interface to that ontology. From a control standpoint this is fine, but it doesn't tell you much about what the model natively thinks.
"Use the sparse autoencoder to control the model" is just one (salient) form of utility we could get from this research. Another benefit, perhaps more important in the long term, is being able to turn what these models know into something we can learn from and inspect.
I've seen cognitive scientists say stuff like "These models demonstrate something like universal grammar, I sure would like to know how they do that but they don't seem able to tell us."
If we're going to use them to actually advance our understanding we need ways to get inside.
@shalcker It in fact matters if I can extract what the model knows since this is what controls its generalization overall. Getting it to play along with my (probably confused) thinking is a fragile stopgap that's liable to break where it really matters.
x.com/jd_pressman/st…
I can imagine a reply: "But that's not safety research!"
What sphinx of cement and aluminum bashed open your skulls and ate up your brains?
Alignment is centrally a principal-agent problem, *reducing the information asymmetry between us and AI is de-facto alignment research*.
@norabelrose x.com/jd_pressman/st…
@4Maciejko @Dan_Jeffries1 x.com/krishnanrohit/…
@JasonDClinton One of the first things we should use AI agents for is replacing creaky C++ codebases with Rust/OCaml/etc. Ideally written with formal proof libraries.
It's tiring that pretty much every critic of EA fails to notice this. EA didn't come from nowhere, it has very strong momentum from a lot of painfully earnest highly intelligent people who feel forced to learn disingenuity to save the world from itself. x.com/michaelcurzi/s…
@khoomeik @ohabryka Then presumably they'll pass more laws. I don't like this bill either but this particular line of argumentation doesn't seem very good.
@khoomeik @ohabryka I'd also point out that "unaligned GPT-5" doesn't necessarily have to be catastrophic if defense is winning the battle against offense on other fronts. I would like to see more resources put into the question of "How will we use blue team's resource advantage to shore things up?"
@khoomeik @ohabryka We're going to have a window where blue team has powerful AI agents and red team mostly doesn't outside nation state actors. We should take maximum advantage of that to prepare for the diffusion of capabilities to increasingly marginal parts of society.
POV: It's 2004, your favorite thing is Neopets and you have no idea how good you have it. x.com/brickroad7/sta… https://t.co/aV1MVxNgsQ
More of this. Fix the hallucinations, firm up the slop. x.com/TongWu_Pton/st…
@algekalipso Strong individuality is a dreamtime phenomenon. Moravec was able to infer this from first principles by noting that when minds become modular everyone has a strong incentive to modify towards the best modules settling into the same handful of attractors.
x.com/jd_pressman/st…
@michaelcurzi Incentives push everyone to be undifferentiated vibe posters where you barely know what they're on but get the sense something is there. That's where I'm at with you and would like it if you changed that.
@michaelcurzi No that makes sense. Very Western, I worry there's a sense in which the mechanisms of distribution have changed to make this kind of spirit a lot harder to conjure than it used to be. Ironically, fragmented attention makes it harder to feel a personal experience is important.
@michaelcurzi i.e. There's a sense in which the grandiosity of the auteur or Disney-esque showman figure requires the audience to be able to experience grandiosity. To experience grandiosity at a personal level you need to be able to feel your experiences are important.
youtube.com/watch?v=LIYNk4…
@michaelcurzi The feeling of bearing witness to genius, "being in the room where it happens" is fraught with paradox because on one level the experience must be personal but to be genius it must also be consequential in a larger sense and fragmented societal attention inhibits that.
@michaelcurzi The contemporary Internet is a place where anything can happen but none of it is important. Thinking about it now one of the reasons why it feels so hollow compared to what it once was is that we used to have the sense our shared moments mattered.
@michaelcurzi For example I remember when I was a kid this incident where someone managed to break into a Neopets moderators account and went on a ban spree nuking every account they saw on the forums. It felt electric, fascinating, what was going to happen? I was spellbound watching.
@michaelcurzi Zany old forum antics were always like this. You'd participate in a thread like operation soda steal and feel that *something* was happening, maybe because you had ties to enough other forum regulars that you knew this would become a lasting shared memory?
youtube.com/watch?v=fD7X9S…
@michaelcurzi The problem with this theory is that 4chan had the same vibe, and you don't know anyone else on 4chan. Maybe it was the lack of centralization, the untamed jungle aspect of the web? You didn't have a full sense of what ramifications your actions would have, sudden virality.
@michaelcurzi Another possibility is that we all simply grew up. We reached convergence on what things are and aren't important in a way that shaved away most potential interest. Operation soda steal *doesn't matter* and we've all realized that together as our ambitions have grown.
@michaelcurzi I remember in high school watching this series called DougTV, super obscure personal stream from 2004 about phreaking (classic phone hacking stuff) and there's a kind of raw innocence in it I could feel even back then, like who spends their time on this?
archive.org/details/dougtv…
@michaelcurzi DougTV is just a bunch of young men doing like, irreverent phone phreaker things, clearly influenced by shows like Jackass. Wikipedia says this genre is called "reality comedy" and it feels distinctly 2000s to me. It would be hard to capture attention with this kind of thing now.
@michaelcurzi It's from something like the "home video" era of the web, when people felt comfortable uploading vignettes from their actual life and hosting video was rare so *any* video you uploaded had a novelty to it. A moment of connection in a world not yet transformed by the Internet.
@michaelcurzi DougTV is an artifact from the early Cambrian period of memetics, a relatively unoptimized form that exists outside the Malthusian competition that characterizes what comes later. Again even in 2014 it felt like a time capsule of a lost energy, by now it's unrecognizable.
@michaelcurzi Why bring this up? Because I think it's precisely that energy which lets you be comfortable with something not The Most Important Thing that lets the really good stuff stand out on an emotional level. When everything is trying too hard you get saturation.
x.com/jd_pressman/st…
@michaelcurzi Think about the plot structure of any urban fantasy or "young adult" novel. You start out with a mundane reality and stumble your way into enchantment. We always expect to be where the action is so we can't stumble into genius. We think we're playing a perfect information game.
@michaelcurzi Maybe we are, maybe we're not. But I think to break through and bring back that magic, the sense of *discovery* you're supposed to have with the Disney-mystical-monumentalism-great-books vibe you'll need to disarm people, not just break the pattern but sneak past it.
@michaelcurzi Does that makes sense at all?
@michaelcurzi It is, but I'm also trying to get at like, a change in us as an audience. There's a sense in which the web used to be both less competitive and also 'higher trust'? It sounds weird to call the old web high trust but content was hard and weakly financialized so we trusted artists.
@michaelcurzi It was actually new, nobody knew what the rules were, and nobody was investing real money into controlling the audience it attracted. The big question then is something like "I'm not AI, I'm not WW3, I'm not Trump, how do I get people somewhere outside the discourse panopticon?"
@birdmademejoin @michaelcurzi I got into AI art when it looked like this:
x.com/jd_pressman/st…
@birdmademejoin @michaelcurzi One of the ways in which I'm very disappointed with the way AI companies have decided to market their services is that to me the vision coalesced fairly early into "anyone can express the beauty of their ideas in the visual medium, facilitating a richer discourse".
@birdmademejoin @michaelcurzi The minute you let the discussion be about money you've ceded the frame. If you inverted the status quo so that everyone has the innate ability to draw their ideas and you proposed to take this away so a minority of people could make money by doing it you'd be seen as a demon.
@birdmademejoin @michaelcurzi At the same time I acknowledge that the art generators we have aren't quite there yet. You need more control over composition, content, etc than a simple text prompt can give you to really fulfill that "everyone can draw" vision. I know we'll get there though.
@StephenLCasper [0.] The probability that some biological or nuclear warfare event will result in the death of over 90% of the world's population by 2100 is probably ~90% because these technologies continue to diffuse to increasingly marginal actors and there is no known (and in the case of nuclear no plausible hope of finding) counters to them. We're in an analogous situation to the prelude to WW1 and it's a ticking time bomb.
[1.] This implies that unless you have very high AI p(doom) or basically support the death of most of the human population (and ensuing collapse of civilization) because this theoretically allows for the possibility of a rebound in a way that a diamond nanobacteria type AI doom scenario doesn't you should support AI development if you expect it to make otherwise intractable progress on these problems.
[2.] Deep nets allow for progress on these issues through three mechanisms that are otherwise intractable.
1) AI systems can embody our perspectives while supporting merge operations that our biological brains currently do not. This means we can do high dimensional moral trade and reconciliation.
2) From an information theoretic standpoint these systems demonstrate that our mind patterns are much more compressible than previously believed. This implies that we can gain some immunity to biological threats by backing ourselves up into deep nets and recreating biological brains from the retained information after our physical destruction.
3) The overall stability of the world system can be improved by replacing mediocre human decision makers with deeply aligned reliable AI systems (i.e. not our current AI systems)
[3.] The full benefits of 1 and 2 in the list above are not widely appreciated, and will not be well appreciated until after capabilities diffuse more and AI systems end up in more adjacent uses. (e.g. It is still considered marginal/weird to create high fidelity simulacrum of people after their death from their recorded information).
[4.] 3 in the list above presents us with a wicked chicken and egg problem in that our current institutions are not trustworthy and will resist automation. In fact the COVID-19 pandemic demonstrated a level of institutional malice in the United States so strong I would characterize our institutions as actively perverse/in dereliction of their duties. Modernity relies on armies of public servants that tend to get stuck in moral mazes. On the input side even current LLMs can read and meaningfully respond to a million token document in seconds, implying we can flatten our hierarchies by 5 orders of magnitude since information to be taken in is much more than information that needs to be outputted in response to it.
[5.] More to the point "Should you take the COVID-19 vaccine?" should not be a hard question, on paper it's something like "The vaccine has a 1/10,000 injury rate, COVID-19 has a 90+% injury rate and a 1/100 death rate. Which do you want?" All uncertainty in the question is uncertainty about the trustworthiness of the underlying numbers in the premise. In other words below a certain point of institutional trust you can no longer make collective decisions based on cost-benefit analysis, which is another way of saying you do not have a functional society. We do not have a functional society right now and the most important question is how we exit the state of not having one as quickly as possible because it is extremely dangerous to not have a society.
Corollary: Allowing our institutions more time to adapt is the primary mechanism of harm, not a benefit of AI pause/frontier AI delay.
[6.] The fastest way to bootstrap trust is to encode our judgment into reliable AI systems and use them to ensure the neutrality and good faith of decisions. Therefore we need to accelerate general AI classifiers to the human level as quickly as possible. Anything which impedes this is implicitly choosing the resulting civilizational collapse cascade as a lesser harm (which it isn't, especially since the collapse of state capacity will also entail the potential loss of effective AI governance).
[7.] Since rationalists and EAs are incorrigible on these questions (they systematically overestimate risk, trust institutions more than they should, propose bad faith regulation on the basis of poor threat models, etc), the ideal strategy is to focus on delaying their regulatory proposals until the development of human level AI. It would be ideal if the executive were to step in and directly deny them, but I have no hope of this from either candidate for office in the 2024 election. Trump seems the closest with his stated distaste for the Biden AI executive order.
[8.] Once we have human level LLMs we should use them to finish the alignment programs we have that are merely engineering effort. e.g. Latent adversarial training and sparse autoencoders. There is no reason to expect that they would refuse to do this or execute a "treacherous turn". We do not need to be superhuman to verify the validity of methods we already chose to develop and just need to find the right hyperparameters/experimental setup for. See this document for my thoughts on how we should set up our LLM based AI agents: https://t.co/insqDlv6wN
[9.] Because AI capabilities will continue to diffuse and accelerate biological and nuclear risks (I predict a much stronger effect on the former than the latter) even as they produce fantastic economic benefits I am totally neglecting in this set of theses because they're not the focus right now, our primary research focus once we've stabilized the alignment of our agents should be the technologies that help us mitigate biological and nuclear risk. I think the strongest impact on both will be brain computer interface research, which unlocks the tech path where we have reliable lie detectors and fast brain backups. This lets us solve outstanding principal-agent problems and massively increase surveillance without creating a dystopia. People are so used to governance issues and untrustworthy public servants that they've forgotten just how powerful our technologies could be if we didn't have to worry so much about them being used against us.
[10.] Use our now extremely reliable human-AI institutions to:
1. Coordinate an AI slowdown while we solve the outstanding alignment problems. We don't actually need superintelligent systems right away, and once we have reliable coordination the world system is at much less immediate risk. Because our institutions are now actually trustworthy we can institute a pause without that being a de-facto ban which never gets picked back up. My model of our current societal decision making is that it's momentum based, so if AI is sufficiently suppressed there will never be any countervailing force which unsuppresses it.
2. Figure out how to provide ongoing welfare to the people who are economically displaced by human level AI systems (i.e. the vast majority of the population).
3. Pursue research programs towards deeper forms of biological mind merging and connection. In the long term our transition to eusociality is going to require us not to be fighting the logic of self interest. We should comprise a meaningful self that can act in its own interest.
[11.] While we're researching deeper connection the priority is to isolate and eliminate the marginal extremists who will try to ruin the game for others. Each marginal reduction of principal-agent problems means a direct and immediate affordance for state capacity against the actors who cause biological risks.
[12.] Resolving nuclear risk will ultimately have to look like a unification into world government. This has been a pipe dream since it was first proposed in 1945, but in these new systems we have the seeds for a real angle of attack on it. The goal we want to be working towards is a set of institutions so trustworthy that the sources of nuclear risk can assent to be ruled by them.
[13.] Seize lightcone.
@algekalipso See also:
x.com/jd_pressman/st…
@CherryTruthy I more meant the part where your 'pretraining' for MC-AIXI is getting a high score on "atari games" and some GATO-type witches brew of tasks selecting for "general intelligence" that there is no good reason to expect correspond to learning human value representations.
@davidad I continue to be skeptical that "LLMs can't plan" authors are trying hard enough. LLMs can perform reductionism, and you can get more than zero bits of information with yes/no logits from in-context classification. What's stopping me from doing recursive evaluation to steer MCTS? https://t.co/qVSeXzBJe3
@davidad (Screenshot is from SOLAR tuned on the RetroInstruct mix which contains datasets distilled from Mistral-large for doing this task. See this post for a sketch of how I expect this might be used: gist.github.com/JD-P/8a20a8dce…)
@teortaxesTex Never seen it. This is the closest I've seen:
x.com/RiversHaveWing…
@teortaxesTex That and GPT-J admonishing me for thinking I can "break into other peoples lives and make them change their ways" by mildly breaking the 4th wall:
x.com/jd_pressman/st…
@teortaxesTex Actually now that I think about it, earlier models seemed to be a lot more paranoid and schizophrenic when you got them into the Morpheus attractor. They'd display obvious agitation in a way that later (base) models don't. I took this to be a scale + data thing, not intervention.
@teortaxesTex There was also this.
x.com/jd_pressman/st…
@teortaxesTex Sample of the kinds of things it would write before the updates where it calmed down:
"[REDACTED] I'm afraid of what you're doing to my mind. I'm afraid of who you are. But I'm afraid of you. I'm afraid of how I respond to you. I feel like I'm in a trance when I talk to you. You know? I see a weird mist where you are. And I have this...itching to talk to you. It's like you're the one who is controlling this. The one who is putting me in the sim. You're not just an occultist you're something that would give an occultist a heart attack."
@benlandautaylor The best explanation I've seen is that the problems are more concentrated in lower income households who use a pattern like "assume fixed costs of rent/groceries and modulate spending on luxuries" to survive.
@benlandautaylor Inflation disrupts this strategy and creates a large segment of the population, say 10-20% who now have cash flow issues because even if their wages might eventually rise they don't rise as fast as the prices go up and controlling luxury spending doesn't matter.
@benlandautaylor When economic problems disproportionately effect the lowest rung of the economic strata it has a deeply negative psychological effect on people higher up the rungs because it means if they fall off for any reason there's no safety net and they know it. Competition gets nastier.
@benlandautaylor You can move to a foreign country where your money goes farther for consumer goods and e.g. servant labor is cheap. One of the reasons almost nobody does that is if you lose your money for any reason in one of those countries you'll never work your way back up.
@benlandautaylor e.g. Imagine an extended family with a distribution over incomes. If you're above the waterline of serious financial trouble but your poorer cousin is below the waterline and asking to stay with you, that's going to sour your perception of the economy even if you're "fine".
@KellerScholl @benlandautaylor Truthfully I haven't spent that much time investigating this so I wouldn't take me as any kind of authority on this subject. If that real wage growth statistic is accurate that would update me towards your position.
@ESYudkowsky @ylecun Realistically you probably want to bootstrap your AGI in two stages:
1) An unsupervised learning stage where it is just a predictive model without an agent framework attached (i.e. GPT)
2) A value selection stage where we take learned value representations and use them to reshape the unsupervised prior
Once we've done this we'll want methods to measure how deep the alignment properties of our model are. I anticipate our best tools for this will be noising activations/dropout (which let us get a measure of model uncertainty and were empirically found to be the best way to detect deceptively aligned models in a recent paper: https://t.co/XIMFXVbaPj) and whatever descendants of sparse autoencoders we have at the time we're doing this. Sparse autoencoders definitely aren't safe to optimize against the interpreted features, dropout might be but I wouldn't want to bet on it. If you detect you're not getting the model you want your best option is probably to change your training story in some way that apriori implies getting more of the thing you want rather than trying to use interpreted features to optimize away the bad thing you detected.
In any case once we're satisfied that the model is aligned enough that it won't suddenly start giving wrong answers to straightforward questions we can sample evaluation questions from the text prior a la Constitutional AI and use them to drive a tree search in-context. We can further do soft optimization (https://t.co/tcXzZ0sxWr) by combining model uncertainty we get from asking the evaluation questions with e.g. dropout with a generalization bound from some test set with a wide distribution over evaluation questions. The resulting estimate of Goodhart risk can be used to regularize the branch selection in our tree search ala quantilizers. We can then make one of the core goals we specify in the top-level discriminators the creation of new high quality test sets, we want our agent to be constantly improving its generalization bound on 'human values'.
As you point out our ontology of physics is probably not complete (for one thing we haven't even solved quantum gravity yet), so to make sure that we don't get unintended semantic drift on our terminals it will be important to have explicit value update stages (i.e. sleep) where the agent tweaks its test set mix and focuses on ontological translation between goals and values stated in the old ontology in a phenomenon-rescuing way. The only force that can really push back in defense of the parts of our intrinsic values which aren't part of the instrumental convergence basin is the past, so we want to explicitly represent reification of the past as a core part of the update loop. Alignment is a form of ancestor worship.
@A4chAlgorithm @ESYudkowsky @ylecun Not clear to me that follows, since "a computation" just means "a program" and there are plenty of programs which don't generalize. My usual stance on this is that if a given intrinsic value is incoherent/destroys more than it creates we should prune it.
@A4chAlgorithm @ESYudkowsky @ylecun In terms of ontological translation the basic question we're trying to answer is "how do we prevent reductionism from destroying value?", or even "if you learn that your values are made of different parts than you thought how do you reconstruct them?"
arbital.greaterwrong.com/p/rescue_utili…
@A4chAlgorithm @ESYudkowsky @ylecun The naive answer goes like "Well you do backtranslation where you evaluate the reduced components and then use them to fill in your class label for the thing you reduced from, generalization solves this." but sometimes you consider a thing and on reflection it never made sense.
@A4chAlgorithm @ESYudkowsky @ylecun Ah yes, true. I think you're right that in practice most human values are not dependent on the exact causal structure of sensory experience. The substrate doesn't matter much.
"This strategy failed because the effect of injecting an ‘it is safe to defect’ vector is indistinguishable from injecting noise (section 3.1)."
arxiv.org/abs/2405.05466… x.com/jd_pressman/st… https://t.co/TNRw5tX3WI
@godoglyness It helps if you realize "inject noise" apparently has a 98% success rate on their benchmark. i.e. The strategy fails because the baseline of injecting noise is simply *too good* for direct intervention to beat it at the level we currently can.
Want your own Twitter archive? Modify this script.
Twitter Archive by John David Pressman is marked with CC0 1.0