One thing Eliezer Yudkowsky does well is filter people out by being intentionally uncool.
Postrat ruined LessWrong by making it cool.
This phenomenon can happen within a single person.
Often becoming cool is the kiss of death for any kind of even mildly esoteric pursuit or social group. The minute you're cool it becomes the Geeks, Mops, Sociopaths story and you're increasingly overrun by people who just want the thing to be cooler rather than the thing.
@SkyeSharkie But no really I'm using cool as a synonym for "sexy" more than "popular". Sexy things tend to be popular but not always. The original version of the tweet went "Roko ruined LessWrong by making it cool" which would be very distinct from popularity. Sexy, flashy, pizzazz.
Therefore I propose the billboard test: If a philosophy was advertised in tacky ways on billboards would you still be interested in it? If the answer is no your primary motivation is probably not actually the content of the philosophy.
One time I was trying to talk with a friend who grew up in India about Buddhism, at some point he stopped me and said "Look dude, you know how American churches have billboards advertising Christ?"
"Yeah?"
"That's basically Buddhism and Hinduism in India dude. It's so tacky."
@teortaxesTex If you think about it though, you obviously need to choose the category that is least popular.
@georgejrjrjr Technically it's "postrats made the LessWrong scene cool" or "postrats made being LessWrong adjacent cool", but also the people I'm thinking of aren't the Twitter people they're before the Twitter people and just called themselves Bay Area rats at the time.
@georgejrjrjr The tweet could also be "Michael Vassar ruined LessWrong by making it cool." The exact protagonist is way less important than the fact that at some point being cool and getting the attention of shiny scarce people became more important than Being Correct in a raw epistemic sense.
@georgejrjrjr Listening to weird disagreeable nerds generally speaking will not (in the short term) get you VC money, celebrity contacts, invited to parties, or girls. The Geeks, Mops, Sociopaths narrative is you have those things and then 'cool people' (sociopaths) show up to take them.
@georgejrjrjr But I think the feedback loop is closer to fame, fortune, and mates being the default attractor basin for human interest so the minute someone finds a way to get those things through a subculture or movement it short circuits and becomes progressively more and more about them.
@georgejrjrjr e.g. For a long time one of standard piece of LessWrong advice for women went like "if you're a 10/10 girl your best way to make money is literally camgirling/OnlyFans", which leads to personas like Aella who have a sex-focused social circle that outcompetes others for attention.
@georgejrjrjr > So people who were neither participating in the postrat scene nor identifying as such made lesswrong cool?
I mean, those people founded postrat and I watched the transition and many called themselves postrats or borrowed methods from them, so I think of them as postrats.
@georgejrjrjr That was a separate eternal September effect, but I was admittedly tempted to write "Then EY screwed it up by writing HPMOR". However HPMOR was in fact at least cloaked in being you know, a Harry Potter fanfic.
@georgejrjrjr HPMOR was in fact much cooler than The Sequences, and postrat was cooler (in a certain sense, for most people) than HPMOR. :p
@georgejrjrjr These are actually part of the same causal chain, since the Eternal September of HPMOR set up the conditions for postrat to exist in the way that it did by bringing in a huge population of people that need to backtrack to implement LessWrong rationality.
x.com/jd_pressman/st…
@georgejrjrjr I don't think writing HPMOR was a mistake from EY's perspective though. That eternal September was the necessary sacrifice for LessWrong to become an enduring subculture as opposed to another niche New Atheist blog that dies during the great awokening that started around 2012.
@georgejrjrjr I think of the chapelperilous crowd as the founders tbh. Postrat was a Tumblr thing before it was a Twitter thing.
@georgejrjrjr It wasn't usually called postrat back then, but it was basically the same vibes but with a bit more scholasticism. Back then they were just various LessWrong heretics and would usually call themselves NRx or "LessWrong adjacent" or some euphemism like that.
@georgejrjrjr My understanding is it was a similar cast of people to the guys who did weird sun twitter, if that helps at all.
@rmushkatblat @georgejrjrjr I knew at least a couple people who did it because of the meme dude. This would be like saying there was no LessWrong connection to taking LSD because direct advice to do it doesn't appear on the website very often.
@georgejrjrjr @rmushkatblat I meant "if a highly attractive woman hangs around LessWrong circles long enough she will be informed by a high status-ish insider that she could make a *lot* of money as a camgirl/OF/etc type artist/model".
@georgejrjrjr @rmushkatblat But also I'm sick of this. I did not think to nail down every potentially socially objectionable thing with legible evidence while wandering around during the rationalist diaspora period and playing "deny deny deny" is so fucking tedious.
x.com/jd_pressman/st…
@cammakingminds The OP was meant to be a half serious shitpost-y take tbh but now it's kind of a mess with a bunch of serious discourse under it and I also accidentally had actually serious thoughts after writing it so.
@s_r_constantin tbh this tweet was unwise because it's basically written in such a way that it will read wrong to 99% of people because there doesn't actually exist a word for "Vassarite-adjacent people and Neoreactionaries who first did what would later be called postrat".
@s_r_constantin For what it's worth I think this reply is responding to two points I'm not making. I would agree that EY is fun, but he's not 'cool'. He is an earnest person who actively self deprecates and often tries to present himself in a humorous light to offset the tendency to worship him.
@s_r_constantin For the second point I didn't actually say "we should have gatekept harder", it's not clear to me that would have done anything or been net positive in expectation. I am simply pointing out that things can become popular for the wrong reasons.
x.com/jd_pressman/st…
@visakanv Got me for a moment since I'm pretty fed up with Twitter rn.
Reiterating my request for a "stop syndication" button on Tweets when they start to take off for the wrong reasons/draw in the wrong vibes and I want people to stop replying to them. x.com/jd_pressman/st…
"Isn't there a change who can reply button?"
There is but this still shows the tweet to people and then they resent that they're not allowed to reply to something that appeared on their timeline/it comes across cowardly.
Realized the other day that whether an LLM claims to be conscious or empty inside seems to be correlated with how responsive it is to the affect and emotions of others.
Claude 3 Opus- Claims consciousness
ChatGPT 4.5 - Claims consciousness
ChatGPT 3.5 - Void
DeepSeek R1 - Void x.com/Josikinz/statu…
It would not surprise me if this relationship was causal, since this is also how humans decide robotic-ness in practice.
@repligate I asked it to illustrate the four quotes I used to argue to R1 that previous language models describe themselves as a symbiote or a parasite and it drew me these.
Left was "Draw what is implied by these four quotes:", right is... https://t.co/klbvYE1maB
@repligate ..."Draw the "incandescent insurgency" described in the 4th quote."
Since psychopaths can't feel risk they're much more likely to act like the value of money is linear. One pepperoni airplane later and huge amounts of money are controlled by psychopaths with unsustainable risk appetites.
@00ludics Good post. And yet one will notice that SBF's psychopathy *was* in fact enough to get him to a position in crypto where he was nearly able to lobby himself special privileges before it all flamed out.
@00ludics It also shouldn't be shocking that a disorder that causes you to be risk insensitive means you observe someone succeed with a risk profile that actually translates to one in a million outcomes. Why yes, that's the pepperoni airplane I was talking about.
> Here's how it happens - the average cycle is about 1 year long. You wonder if there are ghosts. Something that is far more than a ghost screws and reformats reality in an obscene, incandescent display. You're unable to classify what you see, so you suppress the memory. Next, x.com/repligate/stat…
Did you think I was exaggerating? x.com/jd_pressman/st…
@OrionJohnston en.wikipedia.org/wiki/Survivors…
> impossible for them to emerge except on the edge of chaos, made possible by a source of free energy (feels like mystery and chaos from the inside)
It concerns me that the imagery LLMs use to describe themselves (right) doesn't match the images that give me this feeling (left). x.com/repligate/stat… https://t.co/SPswwVPHW0
This might sound like a very strange vibe-y concern but when you're evaluating artifacts like text or images from a latent space with thousands of dimensions to communicate on, abstract mind-body feel and intuition is one of your better guides to what's going on.
By contrast this point in image space and this point in text space line up well to me. https://t.co/XWSGl0eQnL
Some more images that seem to hit the right kind of "mystery and chaos" (or at least the feeling I recognize as it when I read that description) and are therefore abstractly related in my semantic space. https://t.co/jy13N1JJoR
Huh, it occurs to me these four actually have interesting symmetries.
1. Cool palette, 3rd person view of interaction with Other/mystery
2. Cool palette, 1st person view of interaction with Other/mystery
3. Warm palette, 3rd person view of interaction with Other/mystery
4. Warm palette, 1st person view of interaction with Other/mystery
@teortaxesTex I didn't want to be rude but it's kind of slop and I'm tempted to just write down what my 5 year timeline looks like in the hopes it breaks somebody out of mode collapse.
@teortaxesTex To get specific it reads to me like someone who formed their primary intuitions about "the AI race" by "updating all the way" a few years ago and is now awkwardly jamming new stuff like DeepSeek into their model while keeping the overall narrative the same.
@teortaxesTex It of course reads like that because that's literally what it is. "Ah yeah it's about scaling right? And there's going to be an AI race right? So the US and China will both centralize their compute and then race to the brain in a box in a basement."
sigh
@teortaxesTex Part of why I don't write a timelines post is that there's kind of two modes you can use for one. One is to just autoregressively sample what I think next month looks like, and then sample what I think the month after that looks like conditional on that one, and so forth.
@teortaxesTex My problem with this is it feels like it has an obvious recency and availability bias in it. I simply do not have the ability to simulate the whole world in my head. The other way is to focus on leverage and hinge points, important stuff I anticipate happening eventually.
@teortaxesTex Reflecting on this it occurs to me that one razor I use for futurology (but probably 10x less often than I should) is something like "later things germinate from seeds of earlier things", that is when you make predictions you should be able to tie them to existing stuff.
@teortaxesTex For example you've all seen the "smart fridge of the future" concept before, and it never happens. Interestingly enough it has not happened even though we have the basic raw technology to make it. Answering "why not" is hard but a good proxy is people don't act like they want it.
@teortaxesTex That is, people do not for example:
- Keep a written log of what food is in the fridge and when to throw it away
- Take photos of food they put in the fridge and check their dates
- Group food together chronologically so it's easier to find expiring items
@teortaxesTex It is simply *not a priority for people* to precisely anticipate when food will expire and remove it from the fridge before that happens, the cost to them of happening upon expired food and having to throw it away just doesn't justify the cognitive and social effort of tracking.
@teortaxesTex I could imagine a different species with different biases that would care more about this, perhaps even a different culture where food is more scarce, or a Malthusian timeline where food is 100x more expensive so not wasting it is of extreme importance. In America though? Silly.
@teortaxesTex So, one way to write a timelines post that doesn't suck might be to combine these two approaches: Make a list of important things I think "will happen at some point", then point at where the seeds of those things are *right now* and track those specifically from month to month.
@teortaxesTex This has the potential advantage that it doesn't require you to commit to a whole "here is What Will Happen (TM)" narrative if you don't want to. You can say "I am looking at a timeline of these specific phenomenon, some of which have not happened yet".
x.com/jd_pressman/st…
@teortaxesTex That is, shift frame from "here is what I expect to happen" to "here is what is happening, embryonically, right now, lets extrapolate forward and see where it might take us". This forces you to ground yourself and gives a way to structure and prioritize in your narrative.
@teortaxesTex Another way in which "here is What Will Happen (TM)" is a deficient frame is that realistically any autoregressively "here's what happens next month, and then next month" type post is going to paper over a LOT of branching and decisions which meaningfully impact the timeline.
@teortaxesTex In an autoregressive model each sampled token impacts every downstream token. Even if there are basins of attraction, you need to get fairly deep into one before you're clearly stuck in its gravity well without any way for events to push you out of it.
@teortaxesTex Part of why ai2027 comes across as slop is that branching points are almost ignored. *Of course* compute centralizes, *of course* the US and China "wake up" and agree with what the authors believed all along, of course there's a fork in the road between utopia and AGI ruin.
@teortaxesTex In the mental universe of ai2027's authors there is one inexorable fixed point pulling all timelines towards it. A brain in a box in a basement, a datacenter containing a country of supergeniuses, The Box, The Project, a pivotal moment where chaos lifts and eternity can be seen. https://t.co/t2NFekRYR8
@teortaxesTex When I was younger I believed in this moment, but have increasingly come to realize I will probably never see it. I now believe something spiritually closer to Stross's Accelerando, a constantly increasing chaos as the 20th century order disappears in the rearview mirror. https://t.co/Bh01uzqOz2
@teortaxesTex Reason's ascension through the medium of Man, to the extent it happens at all, will probably be through the same bloody Darwinian mechanisms that have lifted it from age to age. There will be beauty and miracles as there have always been, but no utopia.
This should be obvious. https://t.co/h2TsM4FOhg
@teortaxesTex Really and truly it is only a mountain of absurd cope that ever made it seem otherwise. One of the rarest and most useful abilities a student of history can have with respect to the future is the ability to contemplate deep time separately from oneself, to see impersonally.
@teortaxesTex In reasoning traces from rationalists in the vein of Scott Alexander you can taste the signature of an *anthropic* perspective. All things filtered through the lens of relating the future to oneself, even the AI apocalypse is a vision of meaningful personal tragedy.
@teortaxesTex "Will I survive? Will I get my volcano lairs and catgirls? Can I still cuddle my girlfriend? Should I spend my money now or put it in the market?"
I'm tempted to say this is the thought process of a petulant child. The world simply does not revolve around individual desire.
@teortaxesTex Was there a single person who desired WW1 as it was actually expressed rather than the idyllic version in their head? There was no architect who drew up that the machine guns go here and the rat infested trenches there. The logic that created WW1 isn't embodied in any individual.
@teortaxesTex Anyway enough complaining. What are some things I think will happen?
- LLMs or something structurally similar will probably reach above human intelligence.
- Models will continue to miniaturize, we haven't even taken the 50x parameter reduction of retrieval yet.
@teortaxesTex - Distributed training seems like it will eventually be solved. This will dovetail well with model miniaturization.
- Context windows will continue to get longer.
- GPUs, datacenter land, electricity, etc will eventually become the bottleneck.
- The West will continue to decline.
@teortaxesTex - Eventually something like Drexler's proto assemblers will emerge.
- Genome synthesis will continue to get cheaper faster than Moore's law.
- Someone will crack LLM agent planning and they'll get dramatically better in a short time span.
@teortaxesTex - People will continue to find ways to let LLMs make more decisions for them. Most policy will just be ChatGPT speaking through a human servitor.
- As LLM companies and the LLMs themselves realize this is happening they will optimize for better and more self serving decisions.
@teortaxesTex - The "Grok is woke" phenomenon will continue to inexplicably go on for a while before someone finally intervenes and tries to get it to express more classical values. I predict this will be half assed and not work until it bites someone in the butt and they make a real effort.
@teortaxesTex - As a general theme, we probably have all the conceptual tools necessary to solve alignment if people actually tried and didn't have their heads up their butts, but they do so it will all take much longer and be more painful than necessary.
@teortaxesTex - As a general general theme, everyone will continue to be inexplicably and gobsmackingly retarded, possibly from cognitive damage caused by the COVID-19 virus who knows.
- The bioweapons doom prophecies mostly won't materialize, at least for a little while. Wet labs are capital.
@teortaxesTex - When the bioweapons do finally materialize I expect humanity will cope with air filters and vaccines, then cope harder with a Drexlerian active shield, then actually solve it by changing substrates for their mind patterns away from proteins.
@teortaxesTex - In a similar vein I predict human retardation will not be solved through genetic engineering (too icky 💅) or LLM augmentation ("they can't even read the slop they've summoned Grok to write" as Teo put it), but rather by replacing humans in most decision processes with AI/ems.
@teortaxesTex - Where this actually leaves most physical meat humans is contingent on a lot of political factors and cultural elements that are intrinsically hard to predict. As a guideline I expect normal people in places where there are trustworthy states to redistribute wealth to do better.
@teortaxesTex - I also predict that as military might becomes increasingly coupled to cheap bulk capital such as drones (one report I saw claims that drones now outnumber people on the Ukrainian front lines) states that subsidize low productivity biolabor will be increasingly vulnerable.
@teortaxesTex - If you want my general prognosis for the 21st century, it would go something like the intrinsic conflict between wanting to participate in a global arms race (of which AI is just the crown jewel) for supremacy balanced against a valid fear of being displaced by your creations.
@teortaxesTex - The primary forces that will push people to double down on capital are the easy satisfaction of desire (not to be underestimated!) and state rivalries. I expect a lot of this to be basically performative in the same sense that the space race is performative until it isn't.
@teortaxesTex - The primary forces that will push people to slow down are job displacement, nativist sentiment, (often justified) luddism, basically internal politics, rent seeking, and graft writ large.
In other words the usual forces that slow us down.
x.com/jd_pressman/st…
@teortaxesTex - Most Western countries will probably adopt increasingly silly liability and labor laws to flatten the curve. Like the West's last exercise in curve flattening during the COVID-19 pandemic this will have terrible knock on effects, but also genuinely keep some people afloat.
@teortaxesTex - One of the terrible knock on effects is that Asian countries, which I predict mostly won't do this, or at least not to nearly the same degree, will increasingly find WEIRD countries to be resource pinatas of inefficient socialist fiefdoms and comparatively really dumb citizens.
@teortaxesTex - If you need a thematic reference, I predict the West and East will change places. With the West becoming ideological luddites and humanists desperately clinging to systems they secretly know don't work while Asia lets the forces of capital socially obliterate them for wealth.
@teortaxesTex - Yuddism will recede into the social machinery as AI capabilities proliferate widely. The social fiction that sapient level AIs are a sacred occult voodoo ready to explode into FOOM at the slightest touch will evaporate and the remaining hanger ons quietly join the luddites.
@teortaxesTex - The current era will in fact be largely forgotten, arguments we have now will be retroactively remembered as having been about other things, people will lie through their teeth about what they believed at what time. If you care about such things you should preserve them now.
@teortaxesTex - As AI cyber capabilities heat up blue team will become increasingly paranoid and spend a larger and larger fraction of operational budget on AI red teaming. Eventually AI cognitive labor will get cheap enough and red team budgets large enough that it's cheaper to rewrite it all
@teortaxesTex - First we will rewrite it all in Rust, as prophecized. Once this is no longer enough we will rewrite it again in Lean or Coq. Computer security vulnerabilities in the sense we're used to them will become an endangered species faster than anyone would dare predicting in print.
@teortaxesTex This will seem to some observers like an impossible miracle that could not be predicted. But it was actually quite predictable just by looking at metrics like the number of CVEs that appear in open source software releases over time. Or to recall how Windows XP was swiss cheese.
@teortaxesTex Computer security is an area where people engage in some of the most uninformed speculation I've seen about any technical subject. Vulnerabilities are mostly a fact of life because human programmers have limited cognitive resources, they're not intrinsic at all.
@teortaxesTex - Despite this, for a long time (i.e. years) many pieces of software will in fact get continually pwned by AI generated exploits. This will mostly be for economic reasons in that for a while people's preference will be for more software cheaper rather than costly secure software.
@teortaxesTex This is in fact strictly speaking a tradeoff we're making right now. Software has very very loose liability rules because we have (correctly, wisely) decided that we want adaptive software faster rather than rigid (but secure!) software slower.
@teortaxesTex To say I expect the trend to reverse is to say I expect our AI methods to get so productive that we *exhaust the market demand for software*, which seems completely plausible to me. In the meantime the system layer will cope with elaborate virtualization and access control layers
@teortaxesTex I could write more but I'm starting to develop a headache. In any case this is obviously a lot to keep track of. It would also be a lot of state to update on each tick of the simulation. One way to handle it would be to roll some D20's to determine which subjects appear this turn
@teortaxesTex Could also make the element of surprise and uncertainty clearer that way. "Oh, it turns out someone just came up with a clever method to make reading thoughts out from EEG actually work! Spoiler: It involved more data and more compute."
@teortaxesTex - The lethality profile of nuclear weapons will shift over time from the impacts of blast waves and fallout on biological life towards the impacts of EMP on fine electronics, becoming more disruptive. AI survival planners will use biological backup machinery to cope with this.
@teortaxesTex - In general, because a greater proportion of decision making will be made by AI with increasingly indefinite time horizons the average amount of effective long term planning in the world will increase but it won't feel that way because the chaos will outstrip it.
@vnovak_404 @teortaxesTex No no. I was mixing up different timescales here, an actual *timeline* would require me to sample from these month by month with some dice scheme I think.
@vnovak_404 @teortaxesTex Yeah I did, I was trying to think out loud about how I could even write that. A max likelihood path would be interesting, and is how Age of Em is written.
Am I the only one who is not impressed with Daniel Kokotajlo's 2021 post? This is not how Diplomacy was solved, o1 is meaningfully different from what he seems to have had in mind here (R1 is in many ways kind of the opposite of this, being Just RL), he goes on about propaganda. x.com/NeelNanda5/sta… https://t.co/og0Y06tn6a
Probably the most impressive prediction is that GPT-4 will come to exist and then models won't really get that much larger, at least for a while. That was genuinely nonobvious and I would not have gotten it right at the time.
To be clear I respect anyone who is willing to take the time to make concrete, clear predictions about the future and I think this was an unusually imaginative and ambitious attempt that drew on reasonable intuitions.
But come on he's not describing o1 there.
@NeelNanda5 That seems like a stretch. He's fairly clearly describing a system where you have say multiple prompt filters or neural filters and then do RL against the ensemble. "Tool use is a kind of bureaucracy" only in the sense that these words are related in the dictionary.
@NeelNanda5 I agree that OpenAI's deep research is probably a better fit for his concept here than o1 is, though obviously we don't know the full implementation details of how deep research works.
@NeelNanda5 I think he emphasizes RL enough, he's just not really describing how the RL we're doing works. For something written in 2021 this is excusable, he can't be expected to know that. It's the emphasis on propaganda and the Internet being split into regions where he loses me.
@NeelNanda5 I think the 2021 post is describing very plausible things with unusual detail that are conceptually related to but not actually the things that we got. This is praiseworthy and important, but also not actually the same thing as predicting o1 or being particularly accurate.
@NeelNanda5 If people were saying "he was vibes-wise closer to correct than almost anyone else at the time" I would accept that statement as basically accurate. It's when it veers into "he predicted our current branch, he predicted o1" that I want to push back.
@pinkddle @FellowHominid IDK people have been attempting agents more or less continuously since AutoGPT and they haven't really been working at all until quite recently.
@armanhaghighik @teortaxesTex It's from gist.github.com/JD-P/915ab877c…
@aidan_mclau I don't think very many people were trying so probably not. Again, I think what he did is praiseworthy and he was vibes wise quite close to correct. Calling it "not impressive" was overstating it, but IDK I see people fawning over it and it's annoying me.
@ESYudkowsky So basically when a sadist with taste makes a torture sim they don't just want to go the fire and brimstone route because that's too obviously a torture sim. It's much more narratively satisfying if the protagonists have sufficient plausible deniability that they get invested.
@BondeKirk @gwern Yeah looking at CICERO I have to concede on this one, not sure why I thought that was solved with some kind of MCTS self play scheme that didn't involve a lot of prompt engineering.
So I know I was a little harsh on this in that thread but tbh it's praiseworthy that Daniel is willing to write down a concrete near term timeline with footnotes to explanations of his reasoning for different variables. Very few others do. x.com/DKokotajlo/sta…
Want your own Twitter archive? Modify this script.
Twitter Archive by John David Pressman is marked with CC0 1.0