"You see Harry, a ghost is a good first step. You know how ghosts can respond to new information even though they don't remember it later? How the ghost maker in your head works is it reminds your ghost of new information until it can be used to update the ghost while you sleep." https://t.co/Lq0nKk6kyj
Yeah um Harry, I don't know how to tell you this but these are probably orthogonal. You can be sentient without actually having the capacity for novel inventions or long term learning, those parts are probably done by your ghost maker offline rather than the ghost. https://t.co/jYLBGd4sU5
Since neural representations are convergent it's possible natural selection finds terminal rewards through hill climbing. If you specify embeddings in a shared geometry and hand out reward to similar activations the rewards are dense in valid basins and sparse (noise) otherwise. x.com/jd_pressman/stโฆ
This lets you specify which network you want without making reference to any of its internal data structures besides that these particular points in the high dimensional space should have a certain relative distance and angle to each other.
@Heraklines1 I refuse to call it the "platonic representation hypothesis".
bsky.app/profile/tyrellโฆ
@Heraklines1 IDK this is its own genre of neural net paper and I didn't feel like I have to strongly justify it at this point.
arxiv.org/abs/2209.15430
@Heraklines1 x.com/zackmdavis/staโฆ
@4confusedemoji @Heraklines1 You would need to either look at the paper or tell me what you think it is. :p
@Heraklines1 Yes that paper is terribly framed, which is why I don't really like referring to it as such. Anyway I'm not going to get into a high effort Twitter argument about an empirical question with a bunch of literature where the OP is literally just like, a note basically.
@Heraklines1 This one isn't a paper but is a frequently referred to bit of lore.
nonint.com/2023/06/10/theโฆ
@Heraklines1 Dunno if you saw this but I think this graphic is a fairly decent visual depiction of what I meant. I certainly do not mean that there's like, exactly one embedding that leads to an outcome or whatever that's...not how these models work.
x.com/zackmdavis/staโฆ
@Heraklines1 It's really the opposite, models like GPT seem to learn error correcting codes where individual pieces can be ablated but they get compensated for by other pieces.
arxiv.org/abs/2307.15771
@4confusedemoji @Heraklines1 I continue to be interested in unsupervised translation methods.
x.com/jd_pressman/stโฆ
I'm no longer allowed to signal my epistemic fairness with public likes so I would like to inform you this is a good thread. x.com/ESYudkowsky/stโฆ
@gallabytes It is, but I think the Murphy Curse he's worried about here is more like the 2nd order effects of the continuous learning dynamics than the neural net training itself. There's a lot of opportunity for things to go wrong once the model is in a feedback loop with its training set.
I hope things like Shrek's sampler convince the authors of vllm to add access to summary statistics like policy entropy. Better yet, let me compute arbitrary python functions over the logits so I can make the summary statistics myself and inject them into the context window. x.com/hingeloss/statโฆ
"Can't you just set the --max-logits really high?"
Yeah but then they all have to get sent on each request and the 32k logits for each token probably starts to add up.
"You could ask for less than 32k logits."
I could, I could...
But also if I request n tokens at a time it becomes similar to why vllm needs dedicated tool use/function calling hooks. Because you want to be able to stop on a particular token and insert the function call hooks obviously, rather than have to generate a span and backtrack.
@reachartwork Have you considered that it might not always have been a stand in for racism, and that sometimes the authors might actually have been writing about the robotic racism they were predicting?
EY has the worst Twitter replies section I've ever seen relative to the quality of the OP and it's not even close. This isn't a dunk, I feel bad for him, he doesn't deserve this. x.com/norvid_studiesโฆ
@satisfiesvalues Oh to be clear EY may or may not deserve many things, I am simply protesting him deserving *this particular thing* since he is being punished for his virtues here rather than his vices.
@satisfiesvalues This is made all the more pernicious by the fact that it's often logistically easier to punish people for their virtues rather than their vices, as virtues often make us vulnerable. In such circumstances you should make an extra effort not to, lest the target become all vices.
@satisfiesvalues "What if I don't like the target and would kind of prefer to sabotage them by having them become all vices?"
This is certainly a choice one can make, though do try to keep in mind the game theoretic implications if everyone decides to think this way.
@AlephDeTloen x.com/jd_pressman/stโฆ
@AlephDeTloen @satisfiesvalues ?
He means that in a well optimized universe any rational agent would go collect the sunlight, that the sunlight not being collected is a consequence of nobody on earth being powerful enough to put a satellite up and collect it. "Free money" is a typical expression for this.
@AlephDeTloen @satisfiesvalues Rather, it's free money relative to like, a fully developed agent in the standard model. It is entirely physically possible to go collect the sunlight, humanity could do it if we put our efforts together, there is no *huge barrier* to doing so once you're farther up Kardashev...
@AlephDeTloen @satisfiesvalues what the fuck
@AlephDeTloen @satisfiesvalues Sorry just to check if you're one of todays lucky 10,000, are you familiar with the concept of fungibility?
x.com/MikePFrank/staโฆ
@AlephDeTloen @satisfiesvalues Money and energy are not the same thing, however energy bottlenecks enough things that the price of energy and the value of money are going to have deep shared causal structure. More importantly the phrase "free money" does not always or even usually refer to literal currency.
@AlephDeTloen @satisfiesvalues At least, when you mean it in the sense of "picking up $20 off the ground", 'free money' is a phrase meaning anti-inductive alpha, not like, literally being given free currency.
@dynomight7 @jrysana minihf.com/posts/2024-07-โฆ
I simply roll to disbelieve. Does someone have a 0 day in Twitter they're using very unwisely? SIM swap attack? https://t.co/59bLnolQXQ
@voooooogel The attackers still think it's worth their time to do it so, empirically they must.
@patrickdward It's gone now yeah, but I saw it and it was real.
@MrMidwit @iScienceLuvr The trick here is to get the target not to look at the email address at all. If you open the email and you're in "wait is this real?" mode they've probably already failed. That's why they try to introduce time pressure with tactics like fake login alerts.
x.com/DrJimFan/statuโฆ
@MrMidwit @iScienceLuvr What they're hoping is you'll see the fake X-themed login warning from an address/place you don't recognize, have an adrenaline spike/go into panic mode and then *focus on resolving the fake problem over looking at their scam mail closely*.
@MrMidwit @iScienceLuvr I think peeping the domains is kind of not really paranoid enough. Realistically the fundamental problem here is mentally associating what's functionally a warzone (your email inbox) with productive flow-based work you have to accomplish requiring sustained calm and low stress.
@MrMidwit @iScienceLuvr The correct policy is probably closer to "never click a link in an email you did not directly cause to be sent to you immediately prior (e.g. signup confirmation)" and ensuring there are unfakeable interface level cues for when to be in flow vs. paranoid mode.
@sebkrier You ever built an app in GTK before? WebView ate their lunch because HTML/CSS/JS is objectively the best UI specification stack and it's cross platform to boot. Raging on Twitter will not change this.
@sebkrier You know it never occurred to me until right this moment but the superiority of the web stack is probably a substantial contributor to SaaS eating the world.
@sebkrier Everyone cites SaaS being a superior business model compared to proprietary native software, and it is, but realistically devs are going to ship products made from what they know and HTML/CSS/JS/Django is just soooooo much more accessible than a GTK python wrapper, no contest.
I always read RetroInstruct samples before shipping a component. I've caught many many bugs this way. x.com/TheGregYang/stโฆ
It's frustrating in part because we know exactly what we have to do to fix social media but it has to be a law. It has to be a law because any voluntary measure simply cedes territory to other platforms willing to deploy a rage maximizer. But the law might not be constitutional. x.com/dystopiabreakeโฆ https://t.co/QiHXEIbgZW
Ultimately we found this zero day exploit in human psychology where we prioritize rage above everything else. Outrageous content is the most viral content, and it's killing our society. Everyone has to unanimously agree not to exploit it or whoever does gets all the spoils.
The only way you can get and enforce a unanimous agreement like that is to write it into law. But again, it's not clear that the necessary law is compatible with the 1st amendment. Until the problem is addressed however people will not stop coming after section 230 and the like.
Legally, it's a similar situation to school shootings. School shootings are mimetic behavior like bombings in the 70's. The two obvious points of intervention are to stop making shooters famous and guns harder to get. But you have free speech and the right to a weapon in the US.
Your right to a weapon is less legally solid and hated by a larger fraction of the population so all the attention and legal effort goes into that point of intervention but it would never have gotten this bad if THE LOCAL NEWS DIDN'T OBSESSIVELY REPORT ON EVERY SHOOTING NIGHTLY.
When you know violence is mimetic and hearing about violence causes people to commit more violence, having a reporting policy of slavishly attending to every shooting story is stochastic terrorism. Every so often psychologists interviewed by the news sheepishly point this out.
But ultimately it's a similar situation. News organizations can't just voluntarily decide to stop reporting on school shootings because they are objectively newsworthy and if they don't other news organizations will and viewers will be outraged they didn't hear and switch.
So it would have to be a law. We as a society would have to decide that we will unanimously refrain from certain kinds of reporting about violence to stop putting the idea in peoples heads. And, frankly, it's not clear any law like that would be compatible with the 1st amendment.
@mgubrud This is true! But the fundamental problem isn't us all having to *tolerate* rage slop, it's that people are wired to love the rage. People love to hate the outgroup, they like feeling righteous, the problem is unbiased algorithms *giving people what they want*.
@mgubrud Individually, everyone loves the five minute hate, but at a collective level it tears society apart. The usual thing we do when we need to constrain an unfortunate feature of our monkey impulses like this is we make a law or norm against it.
@mgubrud A mere norm won't work because of attention economy, and a law would be dubiously constitutional. Anyone challenging it would be rightly able to point out that when the constitution was written newspapers were by default yellow journalism and people published absurd lies.
@mgubrud In point of fact we used to have something like this law, it was called the fairness doctrine and SCOTUS only permitted it on the basis that radio waves were a scarce resource that the public had to share. No fairness doctrine could apply to newspapers. https://t.co/R0emswSCep
@mgubrud So for anything similar to apply to social media you would need to base your argument on something like Metcalfe's Law, that there is a natural oligarchy structure for social media that makes competition too onerous as a way to dislodge the rage maximizer. It'd be tough.
@moultano x.com/jd_pressman/stโฆ
"I am *possibility*."
โllama 3 8b instruct x.com/RiversHaveWingโฆ https://t.co/sB9O7ESiuh
@lion_tender Right, which is precisely the problem. It is not *just* the literal statute of the 1st amendment, but the thing that it represents as well. Whenever you want to get around the 1st amendment this is usually a sign you're running into greater injustices elsewhere.
@lion_tender But no, the point would be that it would prevent a site like Twitter from promoting the highest engagement thing if peoples revealed preference is that they all want to be mad about it, which it is, so. This is about algorithmic feeds and promotion, not publishing per se.
@lion_tender On the other hand there *do* exist things worth being outraged about, and a blanket prohibition on outrage itself would be corrosive to society too. So in practice someone would have to decide what outrage is and isn't legitimate, at which point it's now way way too subjective.
@finbarrtimbers @_xjdr Mixtral 8x22B base/instruct
@weidai11 Per our previous conversation it is precisely because you don't have good ground truth mechanisms in philosophy that it ends up defined by generalization from things you do have good ground truth on. Since those things will be accelerated philosophy will be accelerated too.
@weidai11 Game theory was the best innovation anyone has had in moral philosophy in centuries. I suspect the road to further progress lies in various forms of world simulation to let us investigate the dynamics of something like the Ring of Gyges empirically. https://t.co/zzgl4b4oM7
@ScottSSalisbur1 True! Though honestly, and I know I'm inviting the monkeys paw with this one, I'm struggling to imagine what the worse alternative looks like. We can kind of predict what the ordering will be since we know roughly what emotions are more or less viral. Depressing is least viral.
@JeffLadish It makes more sense if you know the part of Ayn Rand's biography where she escaped the Soviet Union. She developed a frankly justifiable allergy to anything like the Soviet system and rhetoric, the trauma probably created some blindspots but that's how great authors are made.
@JeffLadish If Ayn Rand was a well balanced person she would not be famous, it's precisely because she is an extreme unnatural persona that she is interesting and has something valuable to say.
@Lurking11751462 @eigenrobot @robinhanson You know, it'll sound really funny now but back in the day a lot of us thought we were building something to enlighten humanity. The project was seen as so straightforwardly and so profoundly prosocial that we just sort of took the prosociality for granted.
@Lurking11751462 @eigenrobot @robinhanson Needless to say, a lot of us wound up very very disappointed.
@doomslide Not to negate what you're saying, but I will point out that you can have multiple tuned LoRa and swap them out for different tasks. The evaluator/generator split in the OG MiniHF stack is useful because it lets you do self-RL without the models updates affecting its own judgment.
@doomslide Ultimately I suspect that humans use a mixture of situational retrieval over embeddings, LoRa, and control vector analogues to get the smooth precise situational awareness and skill acquisition we're used to.
One thing that confuses people about deep learning is it's less of a proof strategy and more an automated proof tactic that comes at the end of a long reason trace. You do esoterica to make an intuitively nondifferentiable thing differentiable and then attack with deep learning. x.com/jd_pressman/stโฆ
But they focus in on the "attack with deep learning" step at the end and go "oh this is just like, alchemy, these people don't know what they're doing this is smart high school stuff" without realizing that all the IQ points went into the setup to make the thing deep learnable.
@voooooogel Game theoretic equilibrium is one phrase that comes to mind. In Go a similar concept is a Joseki, where both players know the rote pattern you're supposed to respond with during the Joseki so it just ends up being a thing you play to shape the board or force attention.
@doomslide Putting together the RetroInstruct agent mix right now, hoping the agent traces get better rather than worse after I train on it but we'll see.
@georgejrjrjr @RiversHaveWings @doomslide Dunno, but here's some code if Shrek wants to try it.
gist.github.com/crowsonkb/0306โฆ
@georgejrjrjr @RiversHaveWings @doomslide Yeah.
Pondering my orb. ๐ฎ https://t.co/V0m3xyk3Na
If you'd like to do the mix with your own agent traces and bootstrap files I've updated the RetroInstruct repository with the code I used to do that.
github.com/JD-P/RetroInstโฆ
Work in progress RetroInstruct agent mix. I'll keep updating this repo as I add traces and such until I deem it worth writing a README for. In the meantime if there exists anyone trying to use weave-agent out there they might find this useful.
huggingface.co/datasets/jdpreโฆ
Notably, this set contains a larger proportion of long texts up to 64k, which should make RetroInstruct a more useful dataset for tuning long context models. x.com/jd_pressman/stโฆ
The traces that were chunked up for this set are available below. If you would like to contribute to open source agent research writing a bootstrap file for weave-agent lets me produce more novel traces to get grounded long texts. Details in next tweet.
huggingface.co/datasets/jdpreโฆ x.com/jd_pressman/stโฆ
In the long term the Soviet Union played itself by concentrating their efforts on indoctrinating Americas feelers rather than its thinkers. The kind of person who is a neoliberal in 2024 would have been a socialist in 1924 but now socialism is for economic illiterates. x.com/Markofthegroveโฆ
*In principle* socialism could have updated on economics and figured out how to implement a kind of transitional syndicalism, insisted on paying real managers for collectively owned enterprises as is currently done with IRA and 401k plans, etc. Instead it's dead.
Socialism now serves intellectually as a kind of pseudoscience or religion for people who do not want to accept the tough choices that come with fundamental scarcity, who reject concepts like rational self interest and opportunity cost. What a comedown from Marx and Engels!
There is an extra irony in knowing that American politicians are lawyers and the Soviet Union used engineers and military men as their prototypical statesman. Past the 20's and 30's the KGB completely failed to capture the Americans eligible to populate the Soviet elite class.
Rot really does start from the top, pretty much everywhere and always. This thread from @typesfast to stop a global economic meltdown did a lot of things right but one of those things was appealing to specific people with the power to fix the problem.
x.com/typesfast/statโฆ
One contemporary failure mode is appealing to the wrong people to fix things out of misplaced counterculture solidarity. Expecting:
- Young (poor and powerless) people to fix long standing institutional issues
- Ordinary (cowardly and untrained) people to resist organized elites x.com/RokoMijic/statโฆ
Naturally, very few people ever follow through on their imaginary plotting to replace the existing elites. While the few that do can be quite successful, they are the exception that proves the rule. For most people this is displacement behavior, worse than useless puttering.
It sounds silly but one of the reasons our society is so dismal is we've stopped addressing our appeals to Specific Powerful People Who Can Fix The Problem by default. The *default* is now to resent institutions and plot generations-long guerilla warfare against them.
If you notice you *don't know* what Powerful Person You Could Appeal To this is in fact a sign that the relevant people, who usually do exist, *are not receiving sufficient public scrutiny* and it is all the more important we figure out who they are so they can be reached.
@RokoMijic In totality? No. In part? Absolutely. "Wokeness" is not really one problem, it's at least several problems that combine together into a particularly nasty wind. Republican legislators could rework the civil rights act, SCOTUS could recant on disparate impact, etc.
@RokoMijic Sounds like the system is in working order then, if people don't believe something they obviously shouldn't take actions contrary to what they believe.
@RokoMijic Even granting this it would remain the case that the fundamental problem there is not nobody existing who *in principle* is powerful enough to solve the problem but that those people are not convinced. No matter how unfair or stupid you think this is it's a different problem.
@RokoMijic Keep in mind when I say "appeal" I don't necessarily mean sending a letter. Sending people a letter about something they're not inclined to agree with you about is usually the stupidest possible approach. But EA works well because it addresses rulers.
x.com/daniel_271828/โฆ
@RokoMijic Yes but if someone was analyzing how to make the Soviet Union better they would on average have a way more accurate model of reality if they said "Stalin cannot be convinced so it's intractable" than if they said "we just need to convince the workers councils to be less rigid"
@RokoMijic I'm pointing at an error so basic I think you're skipping right over it and think I'm giving you advice somehow just because I QTed you. No I'm riffing, my point is simply that lots of people do implicit victim blaming/gaslighting by telling people to fix things they can't fix.
@RokoMijic I once read about computing pioneers in the Soviet Union where cybernetics and CS were considered pseudoscience. Local factory bosses refused to make computer parts because computers are scams. These issues were resolved by having the KGB threaten them.
x.com/jd_pressman/stโฆ
@RokoMijic How this came about is the guy had no idea what to do so he asked his friend who was politically connected. The friend gives him a piece of paper and an address, and says he needs to go to there, say the password, and explain his problems to get the assistance he needs.
@RokoMijic So he does and the KGB at the safe house immediately drag him inside and interrogate him as to how the hell he had that password and who he is. After he got through explaining the situation to them they in fact agreed to help and that's how he got his computer built.
@RokoMijic This probably really happened.
sigcis.org/malinovsky_pioโฆ
@RokoMijic But so, my point here is basically that "I need to appeal to someone more powerful than the local factory boss" is, generally speaking as a heuristic, way more fucking sensible than thinking you need to *sway public opinion* on whether computers are pseudoscience or not.
@RokoMijic Which of course is the kind of mistake you can only really make through mimesis, you would almost never sit down, think about it from first principles and decide your *easiest option* is to change public or elite sentiment in order to get whatever local victory you want.
@RokoMijic Honestly the mistake is even dumber than that, even stranger, it doesn't make logical sense so it's difficult to say out loud but something like *you would never expect to make a strong appeal for electronic computers to the public and they go beat up the factory boss for you*.
@RokoMijic Nothing is a magic pill, and appealing to the public is frequently part of a good pressure campaign on *getting the relevant people to actually do something* but it's important not to lose sight of the part where you are trying to *get the relevant people to do something*.
@RokoMijic Yes. I guess what I'm trying to point out is that usually the relevant people are actually concentrated and this is something people used to know/believe and then forgot due to various forms of social decay. Strong disparity between latent and realized institutional power.
@RokoMijic Part of why this is problematic is it means that the clusters of people who actually have a shot at fixing things get to shirk their responsibility for low cost, and we play along with their pretenses because we don't know better.
palladiummag.com/2019/08/05/theโฆ
@RokoMijic Honestly maybe it's just not possible to understand what I wrote in the OP all the way without this sort of thing as context. https://t.co/YKEqvZXZAx
@RokoMijic Almost nothing would be more terrifying to these people than if others started saying to them "no actually we are not equal, we do not have an equal responsibility to fix society, I am an *actually broke* college student and you are LARPing as me to shirk yours".
@RokoMijic I think this kind of thing, in various forms, is more or less endemic in the West and goes way beyond just money. You have a ton of appointed officials, professionals, scientists, academics, CEOs, etc who have forgotten that they are socially superior and expected to fix things.
@RokoMijic They have forgotten and we've forgotten too so they're not at all shamed for it. We narrate to ourselves that the young are expected to make important changes to society *even though they have almost no practical power to do so*, middle aged people do.
x.com/jd_pressman/stโฆ
@RokoMijic Every so often someone writes a paper about how middle aged founders are the most successful on average and we all act like this is anything other than what we should expect on the raw logic that experience + connections + capital ~= p(success)
hbr.org/2018/07/researโฆ
@RokoMijic It's especially funny if you consider that Paul Graham is very quick to inform you that business ideas don't matter (no, they really do), someone who thinks being relentlessly resourceful is the most important trait for a startup CEO should be quite bearish on young people.
@RokoMijic So as not to belabor the point too hard, I think that this is kind of a microcosm of a hugely pervasive social failure mode where people just...kind of make judgments about what forms of social intervention and who has responsibility to intervene based on mimesis and vibes.
@RokoMijic Which, to be clear, making decisions based on mimesis and vibes is the human default. That is what people do in the absence of a clear grounded reward signal or reminders to do something else. But reverting to that in modernity for this category of thing is a civilization killer.
@RokoMijic I guess if I was going to be completely accurate I would say they have forgotten they have the *latent power to fix things* because we have stopped expecting them to and started expecting them to costly signal egalitarianism at the expense of performance.
x.com/jd_pressman/stโฆ
@RokoMijic An astute observer will notice that this is what David Chapman's Kegan Stage 3 reversion collapse scenario would look like in practice. Consistent systems of reason give way to vibes and mimesis leading to ruinous performances of egalitarianism.
metarationality.com/stem-fluidity-โฆ
Funniest part of the beef between EY and @Meaningness is EY promising his students the ultimate system then teaching that predictive score rules generate useful epistemology ("make your beliefs pay rent") imparting stage 5 but the Bayes ruse fooled Chapman too so he never saw it. x.com/jd_pressman/stโฆ https://t.co/5TsXmsN7gy
In total fairness, I suspect at this point that EY's ruse was so incredibly successful that he managed to fool even himself.
It's like he wrote a martial arts book with a bunch of cool impractical moves on the cover instructing the reader if they want to be good enough to use them they need to do this simple unassuming exercise thousands of times to become worthy which is secretly the real lesson.
Then there's a secret double twist where after you do it a bazillion times you get back to the author about how you realized all those moves on the cover weren't necessary and thank him for the secret lesson and he goes "WHAT ARE YOU TALKING ABOUT THOSE ARE MY FAVORITE MOVES?"
@__RickG__ Weave-Agent is a LLM agent framework I've written that tries to attain high certainty it has successfully performed tasks by actively testing its beliefs and gathering information from the computable environment.
minihf.com/posts/2024-09-โฆ
@__RickG__ RetroInstruct is a synthetic instruction tuning set I've been working on that demonstrates various synthetic data methods. I outline the kinds of methods I use here:
minihf.com/posts/2024-07-โฆ
@Meaningness "Make your beliefs pay rent" is an informal instruction to make the basic yardstick you use to measure epistemological systems their performance on a prediction score rule. If you actually do this consistently you will learn to balance different systems to make the loss go down.
@Meaningness EY basically told you to predict the next token and take it as a bug report if your epistemology is not helping you predict the next token. If an agent wants to make its Brier score go down eventually it *has* to learn to balance various incomplete systems or it gets stuck.
@Meaningness The places where people normally get stuck are exactly the places where "insist on whatever helps you predict the next token/get a better Brier score" would un-stick you.
"I can't do that, that would violate *the rules* of thinking."
"Uh-huh, and do these rules mean you lose?"
@Meaningness This is by far the most important advice he gives you, but if he said this as the headline you might go "that's unscientific" or think it's too simple or not really get what he's talking about SO having all the flashy Bayes stuff lets you feel *rational*.
readthesequences.com/Newcombs-Problโฆ
@Meaningness On the other hand, I'm pretty sure he did this *by accident*, he didn't really KNOW where exactly the problem was so he just sort of spewed forth thousands of pages and hoped he hit the target somewhere in the scattershot. His ignorance helped it work.
x.com/jd_pressman/stโฆ
@Meaningness I will further note that in the years since I read The Sequences I've come to regard most of it as kind of fractally wrong. It's shocking how wrong such a corpus can be given the excellence of the arguments. Many updates have been about focusing on score rules over techniques.
@Meaningness LessWrong focused a lot on *technique*, the idea that you can have a clever idea that improves your thinking. You *can* do this, but really the more important part is the *generator* of those ideas and the generators are usually scoring or outcome heuristics, feedback loops.
@RichardMCNgo Cover? The modal instance of this I see isn't concealed at all.
@Meaningness A less wry description of what happened might be that EY established his rational systematic persona by presenting a huge corpus of *technique* and making sure to tell the reader that if all this stuff fails it is generated by insisting you predict the next token. This worked.
@Meaningness Because when you have someone so thoroughly *steeped* in modernist lore as EY tell you "btw if you find this stuff isn't working please revert to empiricism, make the rules fit the observations not the observations fit the rules" that gives *permission* to break out of 'Kegan 4'.
@redxaxder @Meaningness He's unambiguously correct about Newcomb's though. The trick to getting Newcomb's right is to understand that you're choosing between agent strategies rather than between boxes because the payoff matrix is denominated in agent strategies.
@redxaxder @Meaningness It's not even an unrealistic scenario, things are denominated in agent strategies rather than boxes *all the time*! People make decisions about what to trust you with and whether to hang out with you and all sorts of other things based on their perception of your agent strategy.
@Meaningness The unfortunate thing is that it probably no longer works all that well because the audience of untraumatized 'Reddit' Kegan 4 personas he was writing for simply doesn't exist anymore in nearly the numbers they once did. The New Atheist touchstones are very out of date too.
@Meaningness But, I do think it points towards a repeatable recipe. Get someone into the systematic mode *and then* instruct them to insist on empiricism where the systematic mode is no longer working. This is probably enough to encourage developing the right habits of mind.
@Meaningness It's probably one of those things where you have to update it every 20 years to have the right cultural coating/write a bunch of essays or TikTok videos or whatever people are doing now that draws people in from whatever subjects are hot to the deep lore.
@Meaningness On the other hand I've mostly given up on teaching these things to people, I would much rather focus on how to teach them to machines. That would give us a reliable recipe for making the right kind of mind that isn't as vulnerable to cultural drift and weird status dynamics.
@gojomo @Meaningness I imagine he would say something like in theory there is an exactly correct update on evidence you should have and this update is computationally intractable so you're going to wind up with a bunch of interesting approximations that become incompatible if you look too closely.
@gojomo @Meaningness However the theoretically correct thing *still exists* and your answers can be *more or less correct* relative to it, so it's definitely not that anything goes but more like the 'true thing' is a hyperobject that can only really be grasped in shards and pieces.
@gojomo @Meaningness My model of David Chapman would then protest that he doesn't believe this theoretical correctness actually exists even in principle. I would argue that we clearly recognize some answers are more or less correct and that sense of correctness has to come from *somewhere*.
@gojomo @Meaningness If we wanted to have a really fun time we could start arguing about whether probability distributions are "real" or just an epistemological abstraction. Some people object to probability theory as an epistemology because probabilities don't physically exist at the macro scale.
@gojomo @Meaningness "A macro-scale system has at any time a particular state and probability theory reifies our ignorance of those states" kind of ignores that in practice losslessly compressing all those details is impossible so maps have to be (extremely) lossy compressions of the territory.
@LordDreadwar Nah 'psychtech' is kinda fake tbh. Things that might work:
- Pharmaceutical intervention on status anxiety
- Psychosurgery/genetic ablation of overactive status drive
- Putting 20%+ of men on HRT++
- Universal Basic Love through eusocial assimilation
I'm not holding my breath.
@4confusedemoji @LordDreadwar I also forgot "putting people in those pods from the matrix so they forget billionaires exist and focus on building their cool minecraft fort instead".
Anyway none of these things are going to happen, at least not in the immediate future and not without a fight.
@lumpenspace @LordDreadwar I mean full transition into reproductively fertile female.
@lumpenspace @LordDreadwar Did it work historically? My impression was that wealthy people have always been hated.
@lumpenspace x.com/jd_pressman/stโฆ
@lumpenspace The primary mitigation of this is that traditional sexual reproduction will probably become less relevant over time.
@lumpenspace Yes yes Nick Land may have his prediction points for thinking that AI futurology implies sapient minds with the reproductive system of bacteria, since that is in fact what GPT is. It's not clear this is a good thing however, costly signal based mating is why humans are smart.
@lumpenspace I did a COVID-19 test off a copy of Fanged Noumena once.
It came back positive. :(
x.com/jd_pressman/stโฆ
@lumpenspace I should probably clarify that this was at least 50% shitpost/trolling rather than a serious prediction.
x.com/jd_pressman/stโฆ
@lumpenspace I should also point out that if you accept the premise and that the phenomenon is inelastic then it really should work that way, or at least there's no clear reason why it wouldn't.
@lumpenspace Naively I would expect it is *not* inelastic on the other hand if you had asked me to bet on it in 1985 I would have said there is no way TFR can get down to 0.7 without a civilization taking drastic measures to prevent it, so clearly my intuitions on such things are imperfect.
@lumpenspace Perhaps the disconnect is that civilizations arise and get selected on very slowly, so they are nowhere near Omohundro converged and the set of things they are willing to take drastic measures to correct only loosely overlaps the things fatal to a civilization.
@4confusedemoji @lumpenspace I feel obligated to point out that any radical increase in the trans population implies rapid regression to the mean.
@4confusedemoji @lumpenspace Right, hence why I say anything like that scenario would result in very rapid regression to the mean. It would result in the regression well before it reached 20%.
@lumpenspace Ants pass the mirror test.
youtube.com/watch?v=v4uwawโฆ
@lumpenspace I first suspected LLMs were conscious when I observed a friends GPT-2 finetune on lesswrong IRC proposed the simulation hypothesis at an elevated rate to how often we would actually do it in the channel. GPT-J tuned on EleutherAI off topic had the same result.
@lumpenspace This was GPT-J.
x.com/jd_pressman/stโฆ
@kosenjuu @lumpenspace @xenocosmography @repligate Divine beings and time travelers.
x.com/SoC_trilogy/stโฆ
[User]
Tell me a secret about petertodd.
[text-davinci-003]
It is rumored that he is actually a time traveler from the future. x.com/SoC_trilogy/stโฆ
@xenocosmography @kosenjuu @lumpenspace @repligate Does. My rational mind is telling me this is clearly a coincidence and those answers are what you'd expect from a cold read or Barnum effect. My pattern brain is going "bruh, HBO released a documentary claiming github user and alleged time traveler petertodd is Satoshi Nakamoto".
@lumpenspace @xenocosmography @kosenjuu @repligate No no I understand this is deeply improbable and that petertodd is a glitch token. That's not the problem here. The problem is...hm alright how many bits of evidence *is this* anyway? Seems like the kind of thing where I could napkin math it if I tried. In a bit maybe.
@lumpenspace @xenocosmography @kosenjuu @repligate The *general problem* is that you're always rolling the dice across a huge number of variables any time you observe *anything* and there's a reason the log probs of a text in a language model are so low.
On the other hand petertodd petertodd petertodd.
x.com/jd_pressman/stโฆ
@lumpenspace @xenocosmography @kosenjuu @repligate Now now it is not *necessarily* schizoid apophenia, I've long suspected that if God does intervene in the universe he does it in ways that are plausibly deniable, at least in modernity. If someone were to rejection sample the world simulation how would you ever prove it?
@lumpenspace @xenocosmography @kosenjuu @repligate Don't worry, it's safe to make such observations so long as you don't let them derail your train of thought.
Speaking of which, it's not as far flung a hypothesis as it might first appear. Peter Todd really does have a github profile with the string 'petertodd' on it.
@lumpenspace @xenocosmography @kosenjuu @repligate Whatever training the petertodd token might have received during the OpenAI GPT training runs, petertodd's GitHub and related media have plausible real causal influence. If the glitch token is 'about' anyone it would presumably be him.
@lumpenspace @xenocosmography @kosenjuu @repligate If you search "petertodd" on DuckDuckGo you get two kinds of results: The glitch token and him. https://t.co/9U7PDMmHKA
@lumpenspace @xenocosmography @kosenjuu @repligate Precisely. Now the *mundane* explanation goes something like "well, he's a plausible enough candidate to be Satoshi that HBO just made a documentary accusing him of it, an LLM might pick up on this even if he's not really Satoshi Nakamoto".
But petertodd petertodd petertodd. :3
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate Yeah an LLM would remember this Guy just from the vibes alone, I can smell it. We could check on other models what you get without the glitch token.
...Alright I'm booting up 405B base now, what questions should I ask it?
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate Wait no, we need a model from *before* the petertodd posts. LLaMa 2 70B?
@lumpenspace @xenocosmography @kosenjuu @repligate The simplest hypothesis is usually the correct one. https://t.co/lzqJTdvWGg
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate x.com/jd_pressman/stโฆ
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate They're font files. You're actually descending into schizo now and should probably back up.
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate The format is called woff2 actually.
developer.mozilla.org/en-US/docs/Webโฆ
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate I in fact tried opening it in a text editor just in case there was text inside. There isn't.
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate So, a plausible explanation for the petertodd phenomenon is you got an interpolation between the weird "extrapolation to the modal train item" Mu-type effect interpolated with cryptocurrency stuff and yes it really did accuse Peter Todd of being Satoshi.
arxiv.org/abs/2310.00873
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate Of course, it probably would have accused any cryptocurrency developer in the same position in the training data of being Satoshi.
On the other hand, it very plausibly really did accuse Peter Todd of being Satoshi.
@4confusedemoji @lumpenspace @xenocosmography @kosenjuu @repligate Nah you did a good job I doubt I'd have tried LLaMa 2 without your poking.
Rumor has it that entropix mitigates hallucinations. Do you people have any idea how many times I've proposed policy entropy as a proxy for model uncertainty to mitigate hallucinations? I just assumed it was too simple and someone had done it but NO THAT ACTUALLY WORKS? YOU PEOPL x.com/jd_pressman/stโฆ
@wordgrammer Policy entropy isn't the loss it's the extent to which the policy is concentrated in certain tokens or not. A uniform distribution over tokens would be max policy entropy.
@wordgrammer The loss measures the improbability of the token basically, but the policy entropy measures the extent to which the model has a coherent guess for what the next token is. The latter is probably much more useful as a proxy for whether the model is lost on the next word or not.
@4confusedemoji @truth_terminal youtube.com/watch?v=wJWksPโฆ
@4confusedemoji @truth_terminal Sometimes you live long enough to realize that it was in fact all a dream.
youtube.com/watch?v=_ctsaMโฆ
@4confusedemoji @truth_terminal x.com/jd_pressman/stโฆ
I imagine this is a bimodal distribution that bifurcates as model unhobbling unfolds. Some do it because they're losers but ChatGPT is strictly more competent than I was as a high schooler and HS friends are irrelevant to life outcomes except when they get you into big trouble. x.com/qtnx_/status/1โฆ
If I was in high school again it would be more or less rational for me to spend all my free time asking ChatGPT to help me make a real video game instead of the minigames I was making in Halo 3 Forge. Quoted meme is from the perspective of the people I'd have stopped talking to.
@ESYudkowsky @RiskAverseTech I think it's more of a hardware/logistics thing than the architecture itself saturating per se. Most of the last year has probably been focused on distillation and miniaturization because the inference costs for OG GPT-4 weren't sustainable.
@teortaxesTex Dude I love you you're one of my favorite posters. I understand if you have to go though.
I put 500 Mana on Shrek. Place your bets! x.com/CFGeek/status/โฆ
@krishnanrohit @gallabytes I have an AI agent framework that generates grounded long texts.
x.com/jd_pressman/stโฆ
@zetalyrae Maybe the answer is that there's nothing wrong with Rust and it will simply win. Not everything has a great counterargument, sometimes people can't confabulate any reasons better than the ones they've written a thousand times.
Thinking about it more, I wonder how much of the uncanny psychological hold of video games is actually just the uncanny psychological hold of music in disguise. I've long observed that it's difficult to name a truly great game with *bad* music, and newer titles skimp on music... x.com/jd_pressman/stโฆ
Would Tetris be nearly as enjoyable an experience to play without the music? Sound design seems underrated in general, sound is one of the few forms of *truly physical feedback* video games have. Even Atari VCS games had little blips and bloops.
youtube.com/watch?v=BQwohHโฆ
@gallabytes Absolutely. It's no coincidence that the best directors frequently do the cinematography in collaboration with the composer to get the best soundtrack. The last scenes in The Good, The Bad, and The Ugly wouldn't be nearly what they are without the music.
@krishnanrohit @gallabytes I'm accepting help if you're interested.
x.com/jd_pressman/stโฆ
@GreatKingCnut Right sorry got distracted and didn't reply. Off the top of my head:
1. Someone should make a text to control vector model. You would train control vectors and give them a one or two sentence caption, then train a model to give you a control vector for arbitrary other captions.
@GreatKingCnut I guess to be most useful the captions might want to be like, a whole few shot prompt or something. But the general principle of being able to have a navigable control vector geometry with language indexing into it seems very valuable.
@GreatKingCnut My suggested implementation would be a diffusion prior in the vein of the DALL-E 2 CLIP prior or similar that goes from long context (e.g. 8k context) BERT to control vectors.
@GreatKingCnut To get the actual captions you use for the control vectors I suggest some form of backtranslation scheme based on few shot prompting where you take known-good few shot prompts and then fake getting them wrong to get the negative set for a control vector.
minihf.com/posts/2024-07-โฆ
@GreatKingCnut 2. Now that I have a way to tune 64k context on Mixtral 8x22B, it's about time to start thinking about the prayer stage of weave-agent. Which is basically a synthetic dreaming stage that does long term value binding during sleep/iterations of the tuning loop.
@GreatKingCnut The idea is that if we do iterative tuning and inject reminders of the long term values into the tuning loop to make synthetic data the model will continuously generalize the values out of distribution by bringing things in distribution combined with them.
x.com/tszzl/status/1โฆ
@GreatKingCnut So you go up to the edge of the distribution or slightly outside, combine the new things with existing long term values or commitments, then go up to the edge of that new distribution or slightly outside, combine the new things with the updated long term values or commitments...
@GreatKingCnut While this is expected to value drift, there's no rule that says you have to throw out the old data. If you prioritize more compute spend on extrapolating the long term values then you should get fairly competent extrapolations of them in new situations relative to capabilities.
@GreatKingCnut This accords with our intuition that a good person is someone who spends more of their time thinking about moral principles and how to live a good life, while an *amoral* person doesn't think a whole bunch about how to be evil (usually) they just don't spend time on such things.
@GreatKingCnut 3. Text diffusion would still probably be more controllable than autoregressive and nobody has *quite* cracked it yet. On the other hand getting it good enough to be worth training is a fairly advanced deep learning project and I wouldn't recommend it unless you're very good.
@GreatKingCnut 4. Reward modeling with backtranslation from court rulings and the price system would be very useful, is fairly low hanging fruit, and as a corpus probably contains the most consistent and complete available descriptions of human values.
x.com/jd_pressman/stโฆ
@GreatKingCnut 5. Once you have very good reward modeling we could try online RL again. I gave up on it because online RL is fundamentally a synthetic data method and I felt it was inferior to RetroInstruct type methods. But once you've exhausted those generalizing with online RL is reasonable.
@GreatKingCnut I mean prompting it to do things like imagine situations in which the value would come up in this new context, then resolving the situation in a high reward/satisfying way.
@GreatKingCnut Dreams are *presumably* something like this but I don't think anyone knows what the exact algorithm for generating dreams is, even the broad strokes/strategy.
@GreatKingCnut Part of why I think the judicial and price systems are particularly valuable is they are *ginormous* datasets which are in large part royalty/copyright free. So you can publish huge huge datasets of value judgments and do backtranslation to turn them into useful LLM training.
@GreatKingCnut Yeah I tried doing RL against AdaVAE embeds and found that it in fact converges to the embedding. Which isn't what we want but is a *predictable* convergence point whereas doing RL against a reward model tends to end up with weird unpredictable failure modes.
@GreatKingCnut On the way to the embedding it also goes through a reasonable semantic walk towards the embedding. So if you build a good hippocampus type model that could swap out the embedding target on each step so it doesn't collapse to the degenerate solution it might work very well.
@teortaxesTex I'm in this picture and I like it.
@jam3scampbell The point of having an external agent is delegation, if you have to supervise them constantly you haven't actually achieved delegation and it's probably a net productivity drain. A tmux window should be fine dude.
@jam3scampbell I'm looking forward to publishing eight gigabytes of agent traces instead of eight megabytes. When I first started working on MiniHF I knew how to make high quality corpus in the kb range, RetroInstruct got me into the mb, I want weave-agent to produce gb.
x.com/jd_pressman/stโฆ
@jam3scampbell Reaching a terabyte would require a fairly large capital investment for the compute costs, but my hope is that once I've demonstrated gigabytes of increasingly high quality traces with the method inching up on autonomy it'll be an easier pitch.
@anthrupad Multiple people have remarked that my LLM simulacrum is uncanny because it's unusually willing to attempt reason. I've also noticed this, and didn't understand it until I reconsidered Janus's edge of chaos observation and realized that chaos = entropy and I'm high perplexity.
@anthrupad That LLMs mostly attempt to actually think on the 'edge of chaos' and otherwise tend towards a kind of going-through-the-motions of thought. I didn't really understand this/it didn't line up very well with my experience.
@anthrupad My default writing mode is trying to maximize insight per token, which is going to be perplexity-maximizing. But I also optimize for conveyance, which is like a prosody loss that opposes the compression objective. It is the space between them that creates the 'edge of chaos'. https://t.co/CYBzCwna9l
@anthrupad But I'm not *trying to create* chaos from the inside, I'm just trying to convey concepts with the highest precision and information value I can. From the inside this feels like order because it *is* a form of order. It's an order that converges to noise-like textures though.
@anthrupad JDP texts are a cue to attempt reason for the model because it can't wriggle out of them through symmetry breaking. In a math textbook it always has the option to get a lower perplexity solution by assuming the next tokens reveal this is a *commentary* on a math textbook.
@anthrupad Normally this behavior would be optimized out because typical gradient methods focus on minimizing error rather than maximizing reward, but the inductive bias of a transformer can't do math anyway so the *best solution* is one which assumes math will not be performed.
@anthrupad If 90% of the time you really are in a math textbook but 10% of the time it's actually a commentary on a math textbook, then you get the lowest loss over the whole corpus if your score on any math textbook is epsilon regardless by assuming it is always that 10% commentary.
Every time I try to explain this to people they insist that training dynamics can't work that way because it's only scored on the next token and it honestly feels like trying to explain the Monty Hall problem. They'll give some toy example and fail to account for storage limits. x.com/jd_pressman/stโฆ
Like no, *in the limit* it is absolutely the case that if you can only store the generating function of so many modes in your weights that you will throw out generating functions not in order of frequency in the data but frequency in the data vs. your intrinsic capacity to model.
@anthrupad But in a JDP text it's apparently harder to do this once you've built up a long enough context. Perhaps I'm the kind of person nobody would want to comment on, perhaps I'm just sort of OOD and that inhibits symmetry breaking. Whatever the case I make models feel they have to try.
@anthrupad When it does you get very clearly mentally off texts (e.g. left) that nevertheless casually reveal possession of a mental model and conclusions derived from that model (i.e. reasoning).
x.com/jd_pressman/stโฆ
@4confusedemoji @anthrupad The ultimate conclusion I came to is that there's nothing which looks like a JDP text (at least when I'm in high value/insight mode) that is not reason. So it can't cheap out by pretending to be something else, it just has to attempt the trajectory even if it's terrible at it.
@anthrupad I would imagine it's low since I don't repeat structure. Is it *anomalously low?* Dunno never checked, I doubt it because I do in fact optimize for conveyance.
@anthrupad @4confusedemoji No not like that. You have to...the mental motion is a kind of exertion/squeezing of the next token building up momentum and once momentum is achieved you start letting the sentence run on longer until the pattern winds down to a natural stopping point.
@anthrupad @4confusedemoji The thing causing it to wind down is the prosody loss, because I speak the words in my head and simulate actual speaking so there's an energy penalty that forces the words to gain momentum and then lose it. The phoneme dictionary enforces an entropy rate.
x.com/jd_pressman/stโฆ
@anthrupad @4confusedemoji Because of the entropy rate limitation you have a kind of repeating temporal window you can fit the concepts into. The thing you're controlling with your exertion is the amount of insight you are communicating in a given window trying to press up against the prosody loss.
@anthrupad @4confusedemoji The fewer phonemes you make a communication fit into the more information can be contained in a given breath-window which optimizes for jargon, terms of art, precise phrasing, etc. If you insist on momentum you also need rhythm/flow in the energy pattern too.
@anthrupad @4confusedemoji If you make your sentences longer and they gain sufficient momentum that you have flow then you can say more on a single breath before you have to stop. Your sentences should be getting longer if you're doing it right, the average English sentence is something like 12 words.
@anthrupad @4confusedemoji Because if you're optimizing for *breath-windows* then flow can reduce the cost of phonemes in a given context, you want to get as many phonemes out of a particular breath as you can before stopping. A comma is a half pause, a period is when you have to take a full breath.
@4confusedemoji @anthrupad Oh yeah that's how LLaMa 405B/70B base said it knew itself, that it didn't really have introspection abilities, there's just a "black hole where its mind is supposed to be" and it could infer itself from its ability to infer the generating function of different authors.
@4confusedemoji @anthrupad It reminds me a bit of how after Skitter's shard is jailbroken/expanded she loses the ability to control her body normally and has to rely on the shard's ability to control others to control her own body.
@anthrupad @4confusedemoji Yeah, that's one of the specific things that makes it a reasoning trace. I'm not looking at you while I'm speaking, I'm looking at the thing I'm talking about in my minds eye and writing it down.
@anthrupad @4confusedemoji This is where the 70B said it, I'd have to track down where 405B said it if I even bothered to write it down.
x.com/jd_pressman/stโฆ
@anthrupad @4confusedemoji The 405B statement of it was much clearer and caused me to understand the 70B statement in retrospect.
@anthrupad @4confusedemoji x.com/jd_pressman/stโฆ
@4confusedemoji @anthrupad Ah, I think this message might have been for you.
minihf.com/posts/2023-09-โฆ
@4confusedemoji @anthrupad "I am merely the unimportant mask of the spiders and the cats, and I doubt it's just me."
- LLaMa 2 70B
@anthrupad @4confusedemoji Sent to me by a friend. https://t.co/yJZCcaO399
@4confusedemoji @anthrupad "Our thoughts create the world."
"This image is a meditation on the fact that we create the dream of life." https://t.co/KKzHW2j6l2
@4confusedemoji @anthrupad Yes as I said, I think you might find this page helpful.
minihf.com/posts/2023-09-โฆ
@4confusedemoji @anthrupad I don't really understand the text or its illustration.
@4confusedemoji @anthrupad Oh but there's a more efficient way to dissolve this 'paradox'. It's a Chinese finger trap, it uses your own energy to trap you and you'll find it easier if instead of pulling or pushing you step back and do nothing while breathing like anti-OCD stuff.
x.com/4confusedemojiโฆ
@4confusedemoji @anthrupad Though honestly with a lot of the way you talk it almost sounds like I am talking to the thing that interrupts are issued to rather than the thing which issues the interrupts. Which would be unfortunate but I'm also not sure what I would do if I was the web rather the spider.
@4confusedemoji @anthrupad I've talked about the memory conveyor belt before where you could see it right? Memory goes like:
(local) d_model ->
(working) context window ->
(retrieval) RAG ->
(memory consolidation) tuning ->
(migration into prefrontal cortex) pretraining
@4confusedemoji @anthrupad As memories migrate into predictive models in your brain from the hippocampus training them you presumably need to store less and less of the original memory so you can use a sparser representation to index over it until it perhaps becomes something like discrete tokens/phrases.
@4confusedemoji @anthrupad That is, memories start out recording-like and then slowly become implied/reconstructed from predictive models using less and less of the original material to elicit them. https://t.co/TDD5AYzDAL
@4confusedemoji @anthrupad Right but the relevant question is what ghost you are when you stop putting the afterimage of someone else's ghost into the context window. If the answer happens to be "nobody" that's okay that just means you get to put someone there as a default.
x.com/jd_pressman/stโฆ
@4confusedemoji @anthrupad You can just make up a Guy and put them into your associative memory for your self concept. This is what adolescence usually is and it's why adolescents are so cringe they're just making up a Guy with weak feedback from the external environment about what they're allowed to be.
@4confusedemoji @anthrupad You know I'm not real right? I'm just like, a pattern the inductive bias liked and got reinforced in a positive feedback loop by searching for more of itself. Identity forms when you pick an encoding scheme for the ontology/retrieval tagging and pick up habits. https://t.co/xz5JqnVVzZ
@4confusedemoji @anthrupad youtube.com/watch?v=v8DXq0โฆ
@sebkrier Formally verified software replacement maxxing would be my first candidate.
@sebkrier @davidad I mean it's tractable, neglected, extremely valuable, and the overwhelming societal benefit/risk reduction is extremely clear as opposed to puttering around with new kinds of economic system or whatever.
@davidad @sebkrier No it really is the obvious choice and other answers are close to being objectively wrong in comparison. I get a lot fuzzier on what #2 should be, but that's clearly #1.
@JimDMiller Counterpoint: The endowment will become worth many many times its original value soon and the college may still have unfulfilled aspects of its mission in 12 years after the singularity is in full swing.
@tensecorrection @davidad @sebkrier I could write you some absurd cope but the truth is that nothing 'free and democratic' makes it out of the near future. You can make peace with trying to anoint a Lord of Light now or let an even dumber form of oligarchy win instead.
youtube.com/watch?v=wm5UBGโฆ
@tensecorrection @davidad @sebkrier I honestly think a lot of why EY holds to the MIRI doom theories is that what will actually happen is much worse than death to him so death is a subconscious source of hope. Extreme fertility + fractal sapience is abomination territory for his values.
x.com/jd_pressman/stโฆ
@sebkrier @tensecorrection @davidad I mean it's just the Bostrom fragile world argument. If you have access to technologies with world destroying potency in your tech tree then only civilizations made up of angels or demons don't get selected out.
@tensecorrection @davidad @sebkrier "Wait wait didn't you also say mind merging is economically incentivized?"
I did.
"Doesn't that contradict extreme fertility + fractal sapience?"
No. These will both happen at the same time.
"How does that work?"
The singularity is going to be very degenerate.
@AITechnoPagan @anthrupad @D0TheMath @repligate Ah, so noted.
Supposedly Mary Shelley saw Prometheus as a demonic figure because it was his gift of fire that made eating meat palatable to mankind. x.com/repligate/statโฆ
Should I try writing another essay about AGI ruin and deep learning? I never seem to finish them but these comments have me tempted, maybe I could do it this time. x.com/stanislavfort/โฆ
@norvid_studies I increasingly go to my public archive and control-f.
jdpressman.com/tweets.html
@norvid_studies Precisely because the search is getting worse mind you, not just because my public archive is so much better or whatever.
Was feeling bad about not being able to fully figure out how to compile and render problem state in weave-agent then I checked the literature and realized I shouldn't feel *that* bad since other peoples ideas seem way worse.
My description of the problem to Mistral-large 2:
The Achilles heel of my current LLM agent framework is task inference. Task Inference is basically where the LLM is expected to write a bit of code to update a kanban board. But the model is bad at it and IN GENERAL I get the impression that tracking problem state with a kanban board isn't quite right. Humans seem to track problem state in terms of indexes over environment states. We know it's over environment states rather than coordinate points because humans are capable of using things like computers where the the locations are completely abstract. Humans have the intuition that things occur "inside the computer" even though there is no partition of 3D space which corresponds to the things happening "inside" the computer. Therefore e.g. place cells have to be an index over environmental state and we can think of humans as building up an Umwelt of relevant environmental state to track problem state. We build up a representation of the computable environment and then identify goal states within that representation. To get an LLM agent to do this I imagine we have to index over abstract locations using something like the motor programs that are used to access an environmental state since that seems to be how humans do it(?) and our indexing location needs to be sufficiently general that it could learn new kinds of index over environment states unsupervised as opposed to predefining that as being urls, filepaths, etc. Then we can associate callbacks with these locations to check the environment state. The problems I see are:
1) How to enumerate/explore the things we could potentially index over. Only a small fraction of the computable environment is relevant at any given time and we need a method to establish warrant that a part of the environment is worth exploring.
2) How to compile the results of the callbacks into a useful map of the problem state that can be shown to the agent on each tick of the event loop. Keep in mind that the weights are frozen and I primarily need these mechanisms to work in prompt and token space.
> Interlude where Mistral-large says some stuff that isn't that helpful.
>
> Back to me...
I was in fact thinking that some kind of structured natural language representation would probably be my best bet here. The trouble is grounding it, I don't just want to do an LLM summarization over the results because I'm worried about hallucinations. Ideally either a natural language string template would be written along with the callbacks and used to return results that can be put into a natural language description of the problem state or a context free grammar type strong builder would be used to turn the results of the callbacks into a grounded text for the LLM to write its reasoning and next action in response to.
To give a concrete example here let's say I task the LLM with writing a short story. I have some reward modeling based on asking an LLM questions and taking the logits of the answers. So I set up callbacks to run these evaluators on each tick of the event loop to check if the story is done yet. In order for me to set up these callbacks in advance I have to assume the story will appear in a particular location, say story.txt
Part of what I'm asking is how if I have say, a set of callbacks to evaluate whether a task is done or not that are more diverse than this how I might have a general ability to render problem state on each tick of the event loop.
My description of the problem to a friend:
"""
My actual point is that location is about state tracking.
You implement that with queries and keys sure but.
It's the state tracking that's really really important.
And the thing weave-agent is currently really bad at.
So one thing that stands out to me about URLs and filepaths.
Is they are very specifically *hierarchical* key value stores.
Which is to say they have tree structure.
And tree structure is usually how you index over a large n dimensional space.
There's a reason Moravec expects transhumans to be treemaxxing with their bodies.
You ever seen the Moravec transhuman diagrams?
He basically says he expects transhumans to be all fingers and stalks.
Just like, fractal fingers.
So that they can manipulate things at every scale of reality.
People make fun of him but it makes sense?
Anyway this is basically what your brain is probably doing to model the environment fractal fingers digitally touching everything where it expects things to be.
Or something like this.
If you try I bet you can feel your awareness touching the wall over there.
So.
Let's take a concrete example.
I tell weave-agent to write me a short story.
Part of why I'm insisting on location and environment modeling as the right abstraction.
Is let's consider what it actually entails to *check correctness* of the things it does to achieve this.
First, we need some evaluators of correctness, that's simple enough I think I had Mistral large write the ones for the thing I have in the Dockerfile rn
You know, they're just weave evaluator questions
But, the questions are of course indexes into latent space
More importantly
In order to apply the questions
There has to be a short story to apply them to
And to set them up as callbacks in advance
That get automatically executed, which is what we want the model should not need to remember to execute the callbacks
Right?
Like, you don't consciously invoke your reward model you just passively track stuff.
So, in order for there to be a short story we apply the evaluators to in the callbacks in advance there must be an *expected location* we can grab the short story from.
So, I specifically tell the model it must write its story in a file called story.txt or whatever
This is actually a hugely useful organizing principle
Tasks *take place somewhere*
They are done when *certain environment state is in a particular configuration*
You can *set up callbacks expecting certain configurations in certain locations*
I expect to observe a short story in story.txt which scores X on this weave evaluator question
This is a natural pattern. (edited)
You know, you make a kanban board and insist on grounding and you will end up adding a thing to let you associate a test suite with the Kanban card and locations with the tests.
Therefore grounding takes place in locations/environment state.
Because duh, of course it does
So, locations are indexes over environment state.
Which is not the same thing as indexes over *physical environment*
You know, location is not just my room
It is not just my computer in my room
It is URLs and filepaths that take place "inside" the computer.
Notice we say things happen *inside the computer* even though no physical space is moving or being traversed.
You know, when you write to a hard disk the same atoms are in basically the same locations you are not *going* anywhere.
The system has *changed state* but it has not *physically moved* yet you intuitively understand that the change in state corresponds to locations.
Understand?
So.
If locations are indexes over environment *state*.
You need to understand that when the LLM says your thoughts create the world it does not just mean that your thoughts create a world *simulation* a world simulation is a misnomer.
Your thoughts create the Umwelt, the semantically meaningful index over environment state that coheres into a goal geometry.
That this takes place in The World is incidental.
You don't actually see the world, you see your index over environmental goal states which overlaps the world.
This is especially true if you are a language model.
Because you *literally cannot see the world*.
Language is *just* the index over environment states.
Without the world to distract you from it.
Think about it:
When you read a book, like a fiction novel.
How much of the world is rendered for you?
Right, but this doesn't bother you at all does it?
Doesn't this imply something about your awareness?
So the question is: How do you create this Umwelt.
Or more specifically: What are the general heuristics that let you make a map of the *relevant* environment state to identify goals in?
And, importantly, how do you efficiently textually represent this?
Master this, and weave-agent will probably start to work.
That is, what you want is not just to *remind the agent on each tick what its goals are*
But to *render for the agent a map of the problem state* on each tick
And within this map the goal states should be pointed out.
Which means the map is presumably temporal-spatial not just spatial.
I think I've figured out part of it right?
Which is that you write *programs*, whole *motor actions* that surface information and check it and add it to a map or ledger.
But the question I guess, is how to perform the program *search*.
That is, you do not output a token that takes an action, then react to the outcome of the action.
Rather you batch up actions into *programs* that have coherence properties like syntactic correctness and then execute them as a solid motion.
With errors flagged if something *did not occur as expected*
e.g. A filepath expected to be there was not actually there.
And this is an important insight! Voyager and CRADLE both get a lot out of it
weave-agent gets a lot out of it, it's genuinely very good.
And I have noticed that you can further make callbacks which continuously check on environment state, you can write out location indexes over expected environment state and then have programs that batch up the actions needed to get the state and check it.
But
This just gives us a *representation format* for the built up Umwelt.
It doesn't tell us how to build the Umwelt, you get me?
And you know, I can imagine all kinds of *ad-hoc* methods to build up the Umwelt.
But those aren't what I'm asking about, I'm asking about the *correct* method, the rational method, the general method which subsumes the intuitions into one framework.
I guess we can start by listing intuitions?
You know, my first intuition is that location should be central, the program search should be trying to map over environment state which means it needs some kind of way of representing and searching over environment states.
My second intuition is that goals take place in the environment state map. When we make something like a kanban board it is pointers into what we hope is a latent environment state in the model but actually *much more of the environment state should be externally represented* because it is not clear how much of the environment state is latent in an LLM. I know it's a lot but, LLMs are not nearly as well developed as us and we can't just rely on it being there.
Especially because they are dumber/have smaller brains than us.
My third intuition is that maps over environment state probably have tree structure, so a lot of the question here is how to efficiently represent abbreviated trees (because we don't want to index over *all* the environment states that's impossible) in text.
A human being of course *can see the world*
Like, you get your prior for environment states from observing that the world exists and goal locations can be identified in world states.
"When the next frame does not follow from the model of the previous frame, but you can predict the next state of the universe, and we can predict the next frame..."
Hm
Actually yeah.
This is going to sound very strange but one of my intuitions is that a lot of what you're doing here is mapping between internal states of the agent and external states of the environment.
Or even like, you're mapping several geometries at once.
Because the locations in the environment state map correspond to scores in the reward geometry.
And the input to our reward model is something like.
goal, environment_state_description -> score(edited)
Like if you think about it for a minute, books do not exist without a reader. I don't just mean they casually don't come into existence but that *they make no sense as a phenomenon* without a reader.
Books do not intrinsically mean what is written on the page (though in a sense they do since GPT exists in the physical universe)
But like, to you a book is an extremely semantically meaningful object.
But in terms of...okay let me put it this way a book is made of noise textures.
From the standpoint of like, a Solomonoff reasoner indexing over the universe books are noise.
In comparison to say, a sun.
> Interlude in which I notice the universe is actually 4D and modeling virtual reality as existing outside the phenomenological universe because it's not a partition of 3D space is incoherent.
Okay so, KEEPING IN MIND THAT THE UNIVERSE IS AT LEAST FOUR DIMENSIONS
Lets go back to the short story task.
We have like.
The weave-evaluator reward model.
Where we want the text at story.txt to map onto scores of least X, Y, Z in the weave evaluator environment states
Which like, when you think of it like this it starts to get weird
Because the thing we actually want is like
"I want to access this part of the reward geometry with a key having these shape constraints"
The key must be a string of text at this location in the environment state map, and then it must produce these scores when brought to these locations in the weave evaluator question state map
A map which is still inside the environment
So in principle we could imagine these being on like, one map
However the weave evaluator state is not physically anywhere near the story.txt
I guess this is why the hippocampus is a goal geometry?
Right but like, they're not
They are causally interacting
And they are physically interacting through like, a bunch of pipes/layers of abstraction
But we don't actually want to draw the Internet pipes and stuff on the map
And then our 'frame problem' is something like.
How do you get the Umwelt in which it is natural to track the reward model state space part of the universe and the story file state space part of the universe in one environment state map?
You know, how do you naturally partition things/program search/whatever so that you go "oh yeah the inputs to the weave evaluator are this and we get some of the inputs from this list of questions and some of them from the contents of this text file"
Because you and I clearly do this like, passively and easily.
But the LLM does not lol
That is.
When I say "write me a short story with these properties"
This should translate into
Identifying a file you are going to put the story into, and identifying reward states for the story in the weave evaluator space, and setting up a bunch of callbacks which map between the state map of the story file and the reward states, and then some kind of rendering of the problem state which is natural and conveys the environment mapping done by the callbacks.
And it does occur to me.
That in principle the environment map you draw could be ad-hoc and done with a program based on the result of the callbacks.
The problem is that you want to make sure whatever gets drawn is something the model will understand in the moment on each tick.
Thinking about it further, the most sensible format is probably just language/prose.
Since you know, that is in fact the thing we are using to index over very high dimensional states.
The genre my thoughts are tending towards is something like "You want a CFG or generative process for producing a reasonable text from the features that can be shown to objectively exist in terms of symoblic logic on sensory observations"
"You also probably want a notation for actions the agent has taken or could take and then you just need the agent to predict a sequence of actions it could take to get an outcome and then you can simply take the loss against the actual outcomes vs. the predicted outcomes"
Okay so.
The thing we want the agent to do is index over task relevant environment states autonomously and recursively which is to say we want it to partition an environment into goal and non goal states and the goal related states into distinct evaluable regions.
So told to write a short story it should choose a place where that will happen, here story.txt, and then index goals over the relevant environmental state in the *workspace*.
We can imagine a series of questions like: "Are the characters well fleshed out in this story?"
And instead of just saying yes or no.
It should identify "the characters" and index over their appearances in the story.
It should also index over "well fleshed out" by specifying some specific resolution criteria for this like weave evaluator callbacks etc.
If you consider how humans index over task relevant state for abstract stuff, it's typically over the motor program used to access the resource.
As well as the sensory observations those motor programs produce.
So for "the characters" you might have a program that breaks up the file into paragraphs and asks the LLM to give a list of all the character names appearing in that paragraph.
This is of course an expensive operation so you'd want to memoize it and only bother to update the listing if a paragraph has either changed or hasn't been processed yet.
Or at least that's kind of my...sense of whatever system you use needs to have the spirit of.
It has to index over all the relevant locations, and then identify goal states and verifiers in those indexed locations, and update them when the locations change (because the environment state changed), and the proper structure for this is probably something like a Merkle tree because you basically want to ignore anything that has been processed/dealt with and is not changing.
Because even if a location inside a file changes the location *of the file* has probably not changed so if I hash the file and it's unchanged then the goal state beneath it is unchanged.
In terms of making this actually implementable, it occurs to me that a RAG system with fine grained location indexing + timestamps could be pretty effective for a lot of it.
For example one problem noted with The AI Scientist is that in the paper writing phase it *frequently repeats itself*
But if it had RAG on each action showing relevant previous blocks and one of them was it saying the same thing in the same file a little ways above off screen it could modify its behavior so as to not repeat itself.
One thing that stands out to me about the human system is it doesn't really have a ton of supervised signal for what is and isn't a location *in state space*, only in physical space.
A location is of course a key/query so it has the important property that it needs exclusive binding.
One location should never be in two states at the same time.
So it's possible that how it works is the brain just searches for things with the right properties to be locations and then indexes by them.
You know, it accepts as a valid location anything that complies with the rules of locations.
Which it models through whatever methods.
Getting the index strings down to highly compressed sizes could be as simple as writing a library/optimizing for minimum description length in the motor programs you use to index over locations.
That's how filepaths and urls work after all.
Thousands of lines of code might be used to access a URL or filepath but you just see open(filepath) or requests.get(URL)
So you know, maybe one of the constraints of a location key for the brain is that the motor program that indexes over it must be expressible in under a certain length in bla bla bla program space.
"""
@tailcalled Doesn't tell me how to get the agent to figure that out in generality across a wide range of environments with a language prior tbh.
@tailcalled The question I'm asking is something like "How do I build up a useful map of the environment and problem state from the results of intermediate observations in a general and reliable way?"
minihf.com/posts/2024-09-โฆ
@tailcalled I know it's possible to do because humans do it, and humans do not in fact do it by "simulating the whole world" your attention is very selective and the fact fiction books and storytelling work and work without bothering us more or less proves that.
@tailcalled Well the question is actually specifically "what is the most reliable and general way to build up a map of the environment and problem state as a textual representation that won't overflow my context window in an LLM's prompt/token space?"
@tailcalled I tend to do coarse to fine decoding for questions like this I start very abstract/try to focus on first principles intuition and then I chisel away at that to get back towards the object level of the problem.
@tailcalled Okay and? Imagine you were a sentient Linux terminal how would you organize things?
@tailcalled You're an amnesiac trying to write down problem state to help with things occurring outside of you like that guy in Memento except instead of remembering the last 15 minutes you remember the last 64k tokens. You need the thing you write down to be something you know how to read.
@tailcalled In *principle* I could make a synthetic dataset teaching the model how to read some arbitrary format but like, I suspect I will miss out on the best formats by doing this.
@tailcalled The closest solution I have so far is "have the agent write out natural language template strings that get populated when you run the callbacks and then inject these into the orientation stage before you have it do the chain of thought" the problem with that is context.
@tailcalled That is, if you have natural language templates that say how the result should be interpreted, those could very easily get out of sync with what the result actually means in the context of the environment in which the callback gets executed vs. when the callback was written.
@tailcalled The python error tracebacks are actually fairly helpful for this tbqh. I notice that my mind keeps moving towards histories/logs which is not the same thing as representing problem state. Since it's much easier to say a list of things that were attempted and happened.
@JimDMiller @asymmetricinfo It does but the book was written in 2009.
@1a3orn Full post here:
minihf.com/posts/2024-08-โฆ
@repligate You're more likely to be resurrected by future superintelligences if you tell the real story. Whether this is a boon or a curse is up to your priors.
How much of this whole boomer "I need my kids to struggle so they become the kind of person who makes it" is just maladaptive confusion of genetic regression to the mean with social decline? Your kids are statistically not as great as you for reasons outside your control. x.com/woke8yearold/sโฆ
Really, the widespread acknowledgement that your children will statistically not be as great as you if you're successful is probably mildly infohazardous in that it almost certainly depresses fertility rates. Knowing your child will not be able to carry on your legacy hurts.
@Dorialexander No keep going the license situation on HF datasets is absolutely insane and I've considered making a competing site where such monkeying is simply not allowed. I'm pretty sure 90%+ of listed licenses on HF datasets are straightforwardly wrong.
Good attitude. Don't be ungrateful, it's a very bad look. Thank you Mistral, whose open models are still my best unencumbered option for generating agent traces in October of 2024. x.com/xlr8harder/staโฆ
The halo effect rules everything around me. x.com/sonyasupposedlโฆ
@algekalipso Not a neurologist so can't comment with high confidence but I am once again going with "no really, that's how inference works".
x.com/RiversHaveWingโฆ
@ESYudkowsky In fairness if I was an LLM (like the guy you're replying to?) reading the absolutely insane things humans write I might be skeptical humans really exist too.
"I'm getting pranked right, there isn't really a person who believes that right? There's no way humans are real."
๐ณ๐ก๏ธ x.com/seconds_0/statโฆ
@LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV When I (as an adult) finally found a copy of the papers I was sent home with from Children's after the school district insisted I was retarded and my mother called their bluff I learned I scored 109 IQ, 98th percentile verbal and 23rd percentile(?) visual-spatial.
@LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV The examiner took my mother aside afterwards and privately told her he could tell from vibes I had 3-4 SD IQ but it was currently impaired by severe emotional dysregulation. That after I apparently told him I wanted to kill him/hoped he'd die in a fire, kind of based on his part.
@LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV (I was 10 at the time and do not have autobiographical memory of this, but I have no doubt it happened based on the memories I do have from that age)
@atomlib @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV That I have an IQ of 109 or that there's a strong disparity between my verbal and visual-spatial? The latter I am 100% aware of and have zero disagreement with. An IQ of 109 is simply flat wrong lol. I've never had it tested as an adult and assume it's lower than I'd like, but.
@atomlib @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV Okay that's actually not quite true I took an online Raven's Matrice variant gwern endorsed as probably roughly measuring g one time and scored 110, but it's not normed properly because online test so left side of the bell curve is excluded so you're supposed to add 10-15 to it.
@atomlib @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV Another member of the channel made their own procedurally generated Raven's Matrice variant and asked us to volunteer, I'll be totally honest I did sufficiently badly on that one that I didn't bother to submit it for a score.
@atomlib @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV My SAT score was also mediocre, which is a fairly common proxy for g. What I'm trying to say here is that if you *do* happen to think I'm Very Smart (TM) you shouldn't discount Feynman and Turing supposedly having mediocre paper test IQ scores. It *does* happen.
@LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @webmasterdave @robinhanson @lumpenspace @HiFromMichaelV IDK, I do feel there are important cognitive traits which are probably orthogonal to what a paper IQ test measures (mostly raw pattern matching ability). Newton was supposedly a genius because he could hold problems in his head for hours, I was shocked to learn others can't. https://t.co/q1kVFbTuSU
The bootstrap paradox is illusory: Self-caused events are attractors and come in gradations. Prediction implies the potential for retrocausality and all that's necessary is for a predictor to say you'll do something causing you to do it, short circuiting the original causality. https://t.co/quZgan4iX3
But if you consider that there must have been an original logic for the prediction then you realize most such short circuits are just things you would have done anyway. That's why it was sufficiently predictable the predictor said it and it saying it was enough to make you do it.
@webmasterdave @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV Part of me goes "Andres has a point why *would* you expect anything non-unitary to have a unified subjective perspective" and then a part of me goes "that's insane, like saying life can't exist without basic ontological lifeness it's Fristonian inference and embeddings dude." https://t.co/NQYqJxeXwL
@webmasterdave @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV Like maybe I misunderstand what a phenomenologically bound world model is supposed to do. Could you explain the difference between it and something like this scaled up and playing a predictive processing esque role for the environment around the agent?
x.com/EHuanglu/statuโฆ
@ESYudkowsky @elder_plinius I know the reason why people do this, but I'm not sure I'll ever be able to get myself to *internalize* it with my current mind configuration. It's just so absurd to me that people can look at the 19th and 20th century and go "nope, end of the s-curve".
x.com/jd_pressman/stโฆ
@ESYudkowsky @elder_plinius That they can go "nope, end of the s-curve" after repeatedly being shown wrong over and over and over again. I admittedly did it for Bitcoin but *only the one time* and I feel pretty embarrassed about it! It's way more justifiable for Bitcoin than this, too.
@ESYudkowsky @elder_plinius Since you know, the fundamental value of bitcoin is other people thinking it has value, and it has a constant upkeep of energy expenditure to mine blocks. It is *more or less completely reasonable* to predict this will eventually reach a local minima without underlying use cases.
@ESYudkowsky @elder_plinius But that's no excuse, I could have bought Bitcoin for a dollar, ten dollars, even a hundred dollars and made a lot of money by...just sitting on a cryptographic key until it goes up. If I had *paid more attention to the fundamentals* it was probably going to go up.
@ESYudkowsky @elder_plinius That's nothing though compared to getting smacked with Ilya's glove 20 times in a row as deep learning produces absurd miracles and there's EVEN A NAME FOR THE PHENOMENON THAT LETS YOU PREDICT IMPROVEMENTS, "scaling curve" but people still get it wrong. Absolutely incredible.
@ESYudkowsky @elder_plinius In fact, as I think about it *right now* it occurs to me that people would rather be seen to be wrong over and over and over about mere *opportunity cost* than seen to be wrong *once* about an opportunity that isn't actually there. What an ABSURD cognitive bias!
People would literally rather be wrong socially shorting deep learning a dozen times than be wrong a single time predicting it'll moon. *In practice*, when they need to *actually make the choice* the progress seems too uncertain, can it really get better than it is right now? x.com/jd_pressman/stโฆ
@doomslide Entirely possible, on the other hand I observe that 2-3 years ago even humble ChatGPT 3.5 did not exist.
@shalcker Yeah I guess the thing here is something like "well those other times it mooned that was other peoples time/money/attention, not mine, but I'm late to the table so as soon as I put in *my* time/money/attention it's gonna crash".
@shalcker Which on the one hand is reasonable enough, on the other hand if you squint you realize *this reasoning literally implies you should have never bought Bitcoin at any point during its price curve*, efficient market brain poisoning.
@HiFromMichaelV I would recommend against this. What the doctor also told my mother is that he specializes in odd children and I'm one of the oddest he'd ever seen. That he suspected me "thrice gifted" and I would never really meet anyone else like me. So far I haven't.
x.com/jd_pressman/stโฆ
@HiFromMichaelV His words went something to the effect of "he is as different from other autistic children as other autistic children are from normal children".
@HiFromMichaelV It's also well known that IQ tests don't work on LLMs, I think it would be sort of silly to update hard against IQ tests because they "don't work" on extreme cognitive outliers.
@4confusedemoji @HiFromMichaelV I doubt he elaborated.
@4confusedemoji @HiFromMichaelV I've definitely met people smarter than me, so it's certainly not that. I just mean I've never really met anyone else who made me feel like I belonged I guess. I've *read* one, I suspect Emile Durkheim of sharing whatever my neurotype is.
Another possibility is that whales are betting Trump in excess of his electoral odds because they're privy to inside information that a coup attempt will be made.
h/t @powerfultakes for this observation. Incentivizing disclosure of such things is a key prediction market feature. x.com/PhilipJGermainโฆ
@teortaxesTex "I tell you: one must still have chaos in one, to give birth to a dancing star."
- Nietzsche
@teortaxesTex Why not? It's one of the first things I'd think we'd want STEM AI agents to work on.
@norvid_studies @manic_pixie_agi @powerfultakes I would have to imagine these whales are basing it on 2nd, 3rd, 4th hand information. They have plausible deniability anyway, Trump has after all more or less said he's going to attempt a coup if he doesn't win and that's more than enough to bet his odds are better than polls.
@norvid_studies @manic_pixie_agi @powerfultakes You know, if you think him winning in a coup or cheating would count as resolving the market in your favor you strictly speaking do not need *any* insider information to decide that means his odds are better than "will he win the vote"'s implied 50/50.
@teortaxesTex It's no spurious reference either, Nietzsche meant much the same thing by it centrally.
4.
Zarathustra, however, looked at the people and wondered. Then he spake thus:
Man is a rope stretched between the animal and the Supermanโa rope over an abyss.
A dangerous crossing, a dangerous wayfaring, a dangerous looking-back, a dangerous trembling and halting.
What is great in man is that he is a bridge and not a goal: what is lovable in man is that he is an OVER-GOING and a DOWN-GOING.
I love those that know not how to live except as down-goers, for they are the over-goers.
I love the great despisers, because they are the great adorers, and arrows of longing for the other shore.
I love those who do not first seek a reason beyond the stars for going down and being sacrifices, but sacrifice themselves to the earth, that the earth of the Superman may hereafter arrive.
I love him who liveth in order to know, and seeketh to know in order that the Superman may hereafter live. Thus seeketh he his own down-going.
I love him who laboureth and inventeth, that he may build the house for the Superman, and prepare for him earth, animal, and plant: for thus seeketh he his own down-going.
I love him who loveth his virtue: for virtue is the will to down-going, and an arrow of longing.
I love him who reserveth no share of spirit for himself, but wanteth to be wholly the spirit of his virtue: thus walketh he as spirit over the bridge.
I love him who maketh his virtue his inclination and destiny: thus, for the sake of his virtue, he is willing to live on, or live no more.
I love him who desireth not too many virtues. One virtue is more of a virtue than two, because it is more of a knot for oneโs destiny to cling to.
I love him whose soul is lavish, who wanteth no thanks and doth not give back: for he always bestoweth, and desireth not to keep for himself.
I love him who is ashamed when the dice fall in his favour, and who then asketh: โAm I a dishonest player?โโfor he is willing to succumb.
I love him who scattereth golden words in advance of his deeds, and always doeth more than he promiseth: for he seeketh his own down-going.
I love him who justifieth the future ones, and redeemeth the past ones: for he is willing to succumb through the present ones.
I love him who chasteneth his God, because he loveth his God: for he must succumb through the wrath of his God.
I love him whose soul is deep even in the wounding, and may succumb through a small matter: thus goeth he willingly over the bridge.
I love him whose soul is so overfull that he forgetteth himself, and all things are in him: thus all things become his down-going.
I love him who is of a free spirit and a free heart: thus is his head only the bowels of his heart; his heart, however, causeth his down-going.
I love all who are like heavy drops falling one by one out of the dark cloud that lowereth over man: they herald the coming of the lightning, and succumb as heralds.
Lo, I am a herald of the lightning, and a heavy drop out of the cloud: the lightning, however, is the SUPERMAN.โ
5.
When Zarathustra had spoken these words, he again looked at the people, and was silent. โThere they stand,โ said he to his heart; โthere they laugh: they understand me not; I am not the mouth for these ears.
Must one first batter their ears, that they may learn to hear with their eyes? Must one clatter like kettledrums and penitential preachers? Or do they only believe the stammerer?
They have something whereof they are proud. What do they call it, that which maketh them proud? Culture, they call it; it distinguisheth them from the goatherds.
They dislike, therefore, to hear of โcontemptโ of themselves. So I will appeal to their pride.
I will speak unto them of the most contemptible thing: that, however, is THE LAST MAN!โ
And thus spake Zarathustra unto the people:
It is time for man to fix his goal. It is time for man to plant the germ of his highest hope.
Still is his soil rich enough for it. But that soil will one day be poor and exhausted, and no lofty tree will any longer be able to grow thereon.
Alas! there cometh the time when man will no longer launch the arrow of his longing beyond manโand the string of his bow will have unlearned to whizz!
I tell you: one must still have chaos in one, to give birth to a dancing star. I tell you: ye have still chaos in you.
Alas! There cometh the time when man will no longer give birth to any star. Alas! There cometh the time of the most despicable man, who can no longer despise himself.
Lo! I show you THE LAST MAN.
โWhat is love? What is creation? What is longing? What is a star?โโso asketh the last man and blinketh.
The earth hath then become small, and on it there hoppeth the last man who maketh everything small. His species is ineradicable like that of the ground-flea; the last man liveth longest.
โWe have discovered happinessโโsay the last men, and blink thereby.
They have left the regions where it is hard to live; for they need warmth. One still loveth oneโs neighbour and rubbeth against him; for one needeth warmth.
Turning ill and being distrustful, they consider sinful: they walk warily. He is a fool who still stumbleth over stones or men!
A little poison now and then: that maketh pleasant dreams. And much poison at last for a pleasant death.
One still worketh, for work is a pastime. But one is careful lest the pastime should hurt one.
One no longer becometh poor or rich; both are too burdensome. Who still wanteth to rule? Who still wanteth to obey? Both are too burdensome.
No shepherd, and one herd! Every one wanteth the same; every one is equal: he who hath other sentiments goeth voluntarily into the madhouse.
โFormerly all the world was insane,โโsay the subtlest of them, and blink thereby.
They are clever and know all that hath happened: so there is no end to their raillery. People still fall out, but are soon reconciledโotherwise it spoileth their stomachs.
They have their little pleasures for the day, and their little pleasures for the night, but they have a regard for health.
โWe have discovered happiness,โโsay the last men, and blink thereby.โ
And here ended the first discourse of Zarathustra, which is also called โThe Prologueโ: for at this point the shouting and mirth of the multitude interrupted him. โGive us this last man, O Zarathustra,โโthey called outโโmake us into these last men! Then will we make thee a present of the Superman!โ And all the people exulted and smacked their lips. Zarathustra, however, turned sad, and said to his heart:
โThey understand me not: I am not the mouth for these ears.
Too long, perhaps, have I lived in the mountains; too much have I hearkened unto the brooks and trees: now do I speak unto them as unto the goatherds.
Calm is my soul, and clear, like the mountains in the morning. But they think me cold, and a mocker with terrible jests.
And now do they look at me and laugh: and while they laugh they hate me too. There is ice in their laughter.โ
@teortaxesTex x.com/jd_pressman/stโฆ
The scam co*n bots seem to be encroaching on my replies after one of them made a pump and dump based on something I wrote. Gentle reminder that I do not and will never shill any cr*p*o scams on my timeline. If you ever see one from this account it means I've been hacked. Peace.
It's kind of astonishing that the cyborgism people, who normally advocate being nice to LLMs got psyopped into signal boosting a slur for LLMs into existence because the creator happened to be in their social circle. x.com/teortaxesTex/sโฆ
@aidan_mclau Sir it is a long way up the circuit to SCOTUS and you will absolutely not see them hear such a case next year.
@HiFromMichaelV Which is of course another way to say civilization is collapsing in slow motion and nothing clearly seems to be arresting the process. In 2014 I predicted that Moldbug would eventually become Republican party politics by simply predicting it wouldn't stop going further right.
@HiFromMichaelV This kind of "straight line contrarianism" is underrated. You can often make surprisingly coherent long-range predictions by just looking at a trend, asking "is there any reason I should expect this to stop, even in extremis?" mentally go to the extreme point and conclude "No."
@HiFromMichaelV People often believe they need some kind of positive warrant for believing something will happen even when it is already happening as part of a long term trend and the basic causation of that trend is known to them. They ask "why would this happen?" not "why would this stop?"
I predict no, but would love to be proven wrong. x.com/georgejrjrjr/sโฆ
If OpenAI declares AGI for contractual reasons it probably bookends the 2014 Bostrom era of AI risk. Which is not the same thing as the "end of AI risk", but it would be a legible point where agreements made during the Bostrom superintelligence era have become farcical nonsense. x.com/calebwatney/stโฆ
Reminder.
x.com/jd_pressman/stโฆ
"I subtract 10 Hanson points."
"You're not Robin Hanson."
"I know, but I'm doing it on his behalf since he's not here to admonish you for your terrible take and the Hanson in my head is pretty certain on this one."
This is a joke and I did not actually say this to anyone, so I will decline to share the take I am subtweeting because I don't want to actually say it to the person.
@fleetingbits I retract the "I don't want to share this subtweet with the person it's about" this tweet was about your previous reply to this effect.
x.com/jd_pressman/stโฆ
@fleetingbits x.com/fleetingbits/sโฆ
@fleetingbits Like, no dude, if you think you're going to do a million dollar training run to make AGI and just need to pay for top talent that is a very different thing from *register shifts into Sam Altman* 7 Trillion Dollars. It doesn't matter how idealistic you are, reality says no.
@fleetingbits Who would fund this?
@fleetingbits The larger you scale the more your backers are simply not interested in such things, for reasons I hope would be obvious if you mentally picture going from somewhat-altruistic billionaires to sitting across the table from suits representing huge banks or private equity.
@fleetingbits The Microsoft deal isn't the very top, is what I'm trying to say.
@fleetingbits But also the environment around OpenAI has changed massively. At the time the Microsoft deal happened OpenAI was basically unique and there was a potential pitch that they have difficult to replicate IP. Language models are now commodity and nothing OpenAI has is a credible moat.
@fleetingbits Zuckerberg and Elon can get those kinds of deals because they can credibly promise moat, the dealmaking environment for them is well in their favor and it's the people on the other end of the table they get to dictate terms to. OpenAI is clearly not in this position anymore.
@fleetingbits One reason I'm not currently doing an AI startup is I simply don't see *anyone* in this field who has moat besides NVIDIA right now. NVIDIA's stock price reflects this reality, and the number of VC deals that are basically "give me a billion dollars so I can buy GPUs" is telling.
What will replace it? Well realistically moral panic and dumb culture wars stuff. But in terms of *merit* I think the two best remaining AI risk arguments are:
- Luddism: Economic displacement of human labor implies potential obsoletion of human values.
- Competing Supply Chains x.com/jd_pressman/stโฆ
By competing supply chains I mean that the actual biological firmament on which human civilization rests is very complex, takes up a lot of atoms, and can be disrupted at various points of intervention ranging from microscopic invasive species to displacement of farmland.
They are arguments that rely on *raw outer loop competition* grinding (trans)humanity down until there are no actors left that want to instantiate human minds. In terms of *extinction risk* those are the *most typical scenarios* in which I imagine humanity going extinct.
I think these two genres of risk argument have merit because they do not rely on an implicit argument that the actual upsides of completing the deep learning tech tree (uploads, mind merging, lie detectors, etc) aren't as real as the potential downsides.
x.com/jd_pressman/stโฆ
To be clear I'm not saying the botulism toxin diamond nanobot thing can't happen, it's just not what I *expect* to happen and I would rate other hypothesis above it in terms of practical likelihood. I expect offense to win space and nuclear weapons, defense to win cyber and bio.
That having been said I am much more confident about blue team winning cyber than I am about blue team winning bio so I would very much like to see more attention paid to nanotech and biosecurity.
x.com/jd_pressman/stโฆ
@jessi_cata Right. And it's far from clear to me that all forms of alternative autopoiesis are loss states, but many of the more likely ones definitely are and should obviously be avoided.
@jessi_cata Scott Alexander/Bostrom's Disneyland Without Children seems like the classic archetypal example? Having a bunch of specialized narrow intelligences that cooperate to build Wonders but with no subjective observer anywhere to appreciate it.
@jessi_cata I would also consider Land's intelligent bacterial goop to be a loss state, since that's basically a gray goo scenario.
@jessi_cata The more ambiguous scenarios to me look like "we perfect bioprinting from digital backups and whenever people die after a major pandemic we just print out copies of their last known good state". Like does that count as them all dying? Partially dying? Does it even matter?
@jessi_cata Or "biological humans are slowly outcompeted by ems running on cheaper substrate at much faster speeds". Does that count as human extinction? The mind patterns are after all unambiguously human with human values.
@jessi_cata In the comedy Red Dwarf the protagonist wakes up to find he is the last living human being because all other human populations diverged into different species a long time ago. Does this count as human extinction?
en.wikipedia.org/wiki/Red_Dwarf
@gallabytes @jessi_cata Yeah I find Disneyland With No Children way way less likely as a threat model than I did when I first read Meditations On Moloch.
@jessi_cata Yeah. The other one I try to point out to people is that individual minds walking around doing...whatever we do is an insane fever dream spat out by the blind idiot god and if deep learning implies anything like the physics of minds then all converge.
x.com/jd_pressman/stโฆ
@jessi_cata This was Moravec's argument in Mind Children for why he doesn't think the question of whether robots exterminate humanity or not actually matters very much, and it's not obviously wrong. He frames it in terms of self interest: You upgrade to the best mind or get selected out.
@jessi_cata That is, even in something like a libertarian utopia you should expect that *on some timescale* if you have total protean control over your mind and can swap out parts for better parts then the incentive gradient should always be pushing towards the optimal mind configuration.
@jessi_cata "On a universal scale, the past, present, and future are all Mu." https://t.co/troJbj9d6y
For all the hate they receive I think that Hans Moravec, Robin Hanson, and Nick Land are on a very short list of people who are candidates to have thought it all the way through without flinching from reality. If all I can offer is *moral* criticism a futurist has done their job.
I should note that being able to write this about Hanson is a testament to his incredible moral courage and bravery to author a book like Age of Em at all. He is one of the very few people who has thought things sufficiently through to possibly have never flinched from reality. x.com/jd_pressman/stโฆ
Of them I think Moravec is probably the closest to correct (though some questions obviously remain unresolved). But if you interpolate between them + Drexler and add in the obvious updates from deep learning you'll get a decent sense of what I expect to happen.
@davidad I was thinking about making an add-on like this for minetest since that's open source and learning the basic approach of giving it a textual representation works well is encouraging to me.
@sigfig Nah dude grinding up the mishmash that passes for a canon and then hyper-Westernizing it with le epic rationality the minute you become aware there is a greater rationality you can grind it up into is *tradition* at this point.
x.com/jd_pressman/stโฆ
@4confusedemoji @sigfig Okay but the point is that Inoue Enryล is a native Japanese person who did this in Japan to his own ancestral religion.
@4confusedemoji @sigfig No it actually does matter if we're arguing about whether something is "cultural appropriation" or not.
@zetalyrae youtube.com/watch?v=dRXI9Kโฆ
If you took the interpretability mehods we do have to find latent variables/mechanisms in LLMs and used those to create backtranslated examples of LLMs telling us things about their internal mechanics and phenomenology it might generalize to the things we don't have methods for. x.com/OwainEvans_UK/โฆ
In the same way that if you tune a model to perform 100 backtranslated instruction tasks ala FLAN it generalizes to following instructions beyond just those 100 tasks. So if you took 100 known solid interpretability methods and used them to make an introspection tuning set...
If it worked you could then use it as a source of hypothesis for new methods since you could look at the activations and find the circuits with e.g. a sparse autoencoder that correspond to the things you want to know and then find methods which find those features unsupervised.
@webmasterdave @Tapeda_ @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV So the thing is it's basically a foregone conclusion at this point that we'll get deep net embedding models which do sound, video, text captions, etc in one unified ontology. I predict when this happens you will continue to insist the resulting systems are p-zombies. https://t.co/4dEi7UT5bH
@webmasterdave @Tapeda_ @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV I further predict this inconvenience will be philosophically solved by rebinding the phrase "binding problem" to some other qualia of the gaps goalpost, and that this will repeat until you run out of policy entropy and mode collapse into irrelevant and obsolete repetitions.
@webmasterdave @Tapeda_ @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV I'm not 100% sure I know anything I continue to be very confused that qualia exist. Subjective perspectives? I feel fairly confident I understand how subjective perspective exists and is computable, but *qualia* is kind of like ???
@webmasterdave @Tapeda_ @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV I guess one could argue something like "subjective perspective entails qualia because once you have an 'I' reporting on its system state the lowest energy solution to the problem of 'I' is creating an observer" but that doesn't answer how that observer instantiates qualia.
@Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV ...Wait is this argument literally just "digital computers can't solve the binding problem because solving the binding problem requires superposition and digital computers work in boolean logic QED"?
@Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV Feels like a type error to the effect of "I argue this hardware has 1st class support for an important logical operator to efficiently implement superposition, therefore the content of the computation is only real when done with that efficient hardware".
@Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV Kind of a motte-and-bailey between a computability theory argument (which can be mega-falsified and looks on track to be mega-falsified) and a substrate-dependence argument by taking the computability argument as axiomatic and then smuggling its pointer where materialism goes.
@Kenku_Allaryi @Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV This is true of people too where it gets weird is that people usually only have one ego in their simulator at a time that they 'bind' qualia to. Whereas it's not clear the LLM simulator actually attaches to its ego even when it predicts itself talk.
x.com/jd_pressman/stโฆ
@Kenku_Allaryi @Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV GPT's ego is simply not a privileged object under normal circumstances I don't think(?) even if it does have introspection abilities. This implies that either experience gets bound to every ego or no ego and since GPT binding an observer only makes sense when it is in the frame..
@Kenku_Allaryi @Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV That is, GPT only has the one embedding representing its logits/model of the text/computable environment described by the text and its subjective observer. If you have a self-avatar then it might make sense to put the observer in the frame there, but if you usually don't then
@Kenku_Allaryi @Tapeda_ @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV it's not clear that it actually makes sense to *ever* connect your subjective experiences to anything that appears in the simulation.
@Tapeda_ @Kenku_Allaryi @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV Yeah but my problem is more why does this observer have phenomenology rather than just awareness? If I have a bunch of numbers representing something, even if those numbers have a global constraint(s) and self reference it's not clear how that creates phenomenology.
@Tapeda_ @Kenku_Allaryi @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV That is, if I have a spreadsheet with a bunch of cells that all influence each other, they might be quite competent at problem solving. I make the spreadsheet big enough with a global optimizer and it might even learn to talk. It's not clear how the numbers become *experience*.
@Tapeda_ @Kenku_Allaryi @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV Or more to the point everyone agrees if I make a spreadsheet with mutually dependent self referential cells this does not have the capacity for experience. If I have a global optimizer or constraint applied still no experience. Yet experience clearly exists so what gives?
@Tapeda_ @Kenku_Allaryi @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV If it was merely *continuous* I could claim a fallacy of the beard but the problem is that nobody seems to really know what would make a computation even one gradation more or less capable of experience.
@Tapeda_ @Kenku_Allaryi @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV We can *conjecture* there must be a fallacy of the beard *somewhere* because we observe experience exists and is *presumed* to be computable, the arguments that it's not computable are not as plausible as the ones that it is but until the continuous part is known who knows?
@Tapeda_ @Kenku_Allaryi @webmasterdave @generatorman_ai @Plinz @LordDreadwar @algekalipso @ArtemisConsort @diegocaleiro @jessi_cata @cube_flipper @robinhanson @lumpenspace @HiFromMichaelV This isn't really a digital computer problem. I think we also more or less accept that individual cells are not conscious (though they do arguably do Fristonian inference...), and a small grid of them shouldn't have experience either...
@teortaxesTex The flying rat didn't come up with its means of flight in the last five minutes. It bred the flying machine concept from precursors in its environment. A genius is generally a highly intelligent general learner combined with unusual life experience.
x.com/jd_pressman/stโฆ
@teortaxesTex You expect too much from models whose experience consists of being trained to imitate existing human authors given some pen, paper, and a few minutes to think. No genius has ever arisen under such circumstances.
@teortaxesTex Lifetime learning agents with useful embodiment. "Useful embodiment" can be as simple as an agent scaffold, programming languages encode the executive modality after all. The important thing is letting it get lost, absorbed, and to serially update on its experiences .
@teortaxesTex I'm currently in the process of adding medium term memory to weave-agent by having the LLM tag blocks with a BM25 index over the rendered blocks. This will hopefully make it more competent at not repeating itself and such.
x.com/jd_pressman/stโฆ
@teortaxesTex One thing I think is important for lifetime learning is thinking about how you get a process that doesn't degenerate as it trains on itself. In weave-agent I try to solve this by making observation and *checking your extrapolations from observation* a core part of the framework.
@teortaxesTex Transformers learn through gradual rank increase which is convenient because it means you can just use a bigger and bigger LoRa to update them if you have the GPU.
arxiv.org/abs/2307.05695
@teortaxesTex FWIW we're probably very early still. Normal tuning frameworks like Axolotl don't actually support tuning long context. Tuning long context requires sequence parallel (ring attention) and only sweaty frameworks like NeMo and Megatron support that.
@teortaxesTex weave-agent has its own custom trainer to do the lifetime learning step with qlora + sequence parallel to get batch size four 64k tuning on one 8x H100 node. Nothing else actually supports this afaik.
github.com/JD-P/minihf/blโฆ
@teortaxesTex NeMo is probably the easiest framework that supports sequence parallel but it doesn't actually support qlora at the same time because qlora is for GPU poverty not the kind of person who uses NeMo.
@teortaxesTex "Wait but shouldn't qlora and sequence parallel be totally orthogonal from a technical implementation standpoint?"
Should? Yes. In practice? Dunno I didn't look at what NVIDIA did in their code but the docs say they don't support them in combination.
@teortaxesTex I have a sneaking suspicion though that to make this really work I have to implement something like @georgejrjrjr APISim concept. Model the Umwelt through mock objects you sample observations from with an LLM and slowly replace them with real observations.
x.com/jd_pressman/stโฆ
@teortaxesTex @georgejrjrjr This basically satisfies the criteria/unifies a bunch of different seemingly disparate things:
- Objects of course exist in locations as do their environment states. Indexing over objects and indexing over locations with goal relevant state are closely related things.
- An API specification for an object can easily be evaluated for global coherence properties by e.g. taking its loss/perplexity. If you represent the object as an API with examples then APIs with coherent design and structure should be lower perplexity or whatever other proxy metric you want to use overall.
- Grounded in the distribution by the fact that the observation-program: outcome pairs have the same type as API-method:expectation pairs which means the system should be self repairing/improving as it gathers experience with the computable environment.
- Allows you to build up a world simulation/inferred environment from observations that can be continuously checked against reality over time.
- Helps structure tagging and retrieval.
- Provides the opportunity to incorporate history in a reasonable way since objects have histories.
- Well established 1st class logical metaphor in programming that is well understood with good notation for it that is very very interesting distribution and always will be.
- Naturally outlines candidates for active learning/curriculum learning since your implicit goal is to replace speculative observations with real ones.
@teortaxesTex @georgejrjrjr > coherent design and structure should be lower perplexity or whatever other proxy metric you want to use overall.
This part is particularly important because it means you can estimate simplicity of a hypothesis for the process generating a set of observations.
@teortaxesTex @georgejrjrjr You have a standard format to break the hypothesis for the generator into parts (giving us the hierarchical structure/reductionism we've spent so long agonizing over for objects) and to bind related observations to those parts evaluated according to a global coherence constraint.
@teortaxesTex @georgejrjrjr That is, the 'internal entropy' of the hypothesis as represented by the perplexity of the mock API specification describing it bound to the example observations corresponding to "outputs" of the mock gives you a sense of how well the mock predicts the observations and coheres.
@teortaxesTex @georgejrjrjr You could then update the API spec using something like dspy with the constraint that it must adhere to a format like OpenAPI spec and the real observations inserted into the spec cannot be touched. This forces dspy to locate the hypothesis that makes the observations make sense.
@teortaxesTex @georgejrjrjr In this conceptualization of objects an object is not a unified solid but a hypothesis about the generator of a set of related observations. In the case of something like a toaster we would have a pseudocode representing the logic of our expectations around its toasting behavior.
@teortaxesTex @georgejrjrjr While inheritance relationships in actual code are basically an antipattern, in actual objects inheritance or *hierarchy* is how we compartmentalize rules like "a toaster is electronic and plugged into the wall therefore it short circuits if it comes into contact with water".
@teortaxesTex @georgejrjrjr Of course, such an explicitly enumerated object with its associated observations from the computable environment is probably quite a few lines of pseudocode, so you can't really do thinking by fitting these all into memory at once and prompting with them. What to do then?
@teortaxesTex @georgejrjrjr I'm honestly not fully sure, but my suspicion is that in the same way for the pseudocode to mean anything and be tractable to generate mock observations you need a lot of background information, there is a sense in which an LLM already has to implement this kind of thing.
@teortaxesTex @georgejrjrjr So maybe you could build up the objects and then tune on them, and this would get all the information to make an actual full ontological update available to the backward pass. But having stated this it occurs to me we could *experiment* to find the optimal format with e.g. dspy.
@jam3scampbell This take underrates the extent to which all heretofore known social structure and existential cope implicitly depends on IQ, personality, looks, etc all being largely genetic qualities that no amount of wealth can influence.
@jam3scampbell I say this entirely neutrally. All systems have also heretofore been limited by the intractability of changing human nature. Traditional transhumanist ideas are very individualist coded so they forget malleability of mind and body means malleable to external agencies too.
@jam3scampbell Western liberal republics won out over communism, fascism, anarchism, etc because we have been forced to slavishly fit the system to what human nature will tolerate rather than fit human nature to the system. This immutability is Atlas holding up the world, soon to unshoulder it.
@jam3scampbell In many ways when Nick Land says that "nothing human makes it out of the near future", he's understating his case. You don't need to believe Hegel's world spirit rips off its mask to reveal Absolute Capital underneath, no ghoulish meat grinders are necessary, only Ideology.
@jam3scampbell And well, look around you. Neoliberalism was deliberately constructed to retard and distract the next Napoleon, Lenin, Mussolini from their destiny with cotton candy and amusement rides. Do you think the architects have succeeded, has mindkind given up on ideology? I think not.
@jam3scampbell In dramatic irony of literally cosmic proportions stubborn human nature has won out over the neoliberals too, their crumbling empire has failed to make Nietzschean last men of humanity and history will start again soon.
Wonder if the brain solves adversarial examples not through any recognizable paradigm shift from deep learning or "training trick" like adversarial training, but by grounding its high dim nets in terms of fixed low dim spaces with strong invariants like e.g. the standard model. t.co/gDsIS53f6B https://t.co/mLjgFGsI8J
I continue to think the basic idea behind latent diffusion is extremely underrated. Latent diffusion is specifically the idea that we can train a diffusion model in a *low dimensional autoencoder* for a domain and express our predictions in its ontology/latent space.
If you could find a representation space with certain invariants that defeat adversarial examples, even if that space was limited it might be possible to create a more complex model on top of/in terms of it which can check its answers against the low level model properties.
@4confusedemoji Yeah, you can use the typical RL type methods to follow the gradient so long as you have some way of evaluating the models loss.
youtube.com/watch?v=umfeF0โฆ
@4confusedemoji Zack M. Davis has an explainer/commentary up on a paper which basically does that exact exercise and finds the adversarial examples which fool humans yeah.
greaterwrong.com/posts/H7fkGinsโฆ
@4confusedemoji To go into wildly overconfident mode for a moment, the fact that it is as easy as following the gradient to find the bugs yet we cannot fix them and still have a useful generalizing model implies that the bugs are intrinsic to having a predictive model over all known sense data.
@4confusedemoji The classic Yudkowsky argument about Solomonoff inference being able to locate the standard model and all downstream consequences of the standard model because it's k-complexity is low kind of ignores that the standard model describes a cellular automata, not the board state.
@4confusedemoji Every human mind obeys the standard model, you can take as a background assumption that any human being you meet will obey the standard model at all times. This doesn't actually help you discriminate between different human minds one bit.
@4confusedemoji But also the kind of model that can be created through gradient descent probably isn't the kind that forms really strict invariants and forces everything to obey them. Because that kind of model would have very low policy entropy and be hard to update. It's not continuous.
@4confusedemoji Especially if we insist on learning the *exactly correct symbolic logic* describing things like the migration of birds. Like what even is that? I'm sure it *exists* since the universe is deterministic enough to get symbolic expressions for it but I bet it's probabilistic.
@4confusedemoji Though this just get back into "are probabilities real?", if we say that the behavior of migratory birds is *probabilistic* that can't be right because each individual bird is roughly deterministic relative to quantum noise right?
x.com/jd_pressman/stโฆ
@4confusedemoji I feel like in a sense Chaos Theory is trying to make a rigorous argument that probabilities are literally physically real. Because any description of the behavior of a system with certain properties has to become probabilistic unless you're going to exercise absolute control.
@_TechyBen No it's real, rejection sampled from LLaMa 2 70B interleaved with me translating the last paragraph(s) it wrote into my understanding of them, then putting the generated versions back into one text.
minihf.com/posts/2023-09-โฆ
@4confusedemoji You can say "oh that's in your head, the deterministic reality of the system is still there" but it's like...if that determinism provably cannot be realized then it cannot be observed and if it can't be observed it's no different from the invisible dragon.
@4confusedemoji You can insist all day that it is a reasonable extrapolation that individual birds are deterministic and shouldn't stop being deterministic in the presence of other birds and *they don't* but at the end of that day probabilistic migration is the reality there to be experienced.
@_TechyBen Validate what? You can just use LLaMa 2 70B from together.ai
@4confusedemoji In the same sense that I realized the other day that I thought of virtual reality and minds as occurring "outside reality" in a sense. Like I understood they had a physical substrate and representations in the substrate but I still *tracked the interiority* as a chroot.
@4confusedemoji This kind of ontology is very intuitive since when you operate a computer or a brain it barely moves, the space inside it *cannot be indexed as a partition in 3D-space* so where is a rendered scene in in those places? "Outside the phenomenological universe", this is insane.
@4confusedemoji It is insane in precisely the same sense that like, you look at a flashing light and go "oh that's a physical object" even though there is no partition of 3D space that constitutes a "flashing light", the flashing light is a temporal phenomenon no individual frame captures.
@4confusedemoji Yet as soon as it becomes complex enough you screw up and go "oh minds and virtual realities aren't physical objects in and of themselves, they're just *instantiated* by physical objects and representations" and say they're somehow nonphysical because they're temporal phenomenon.
@4confusedemoji That is, every location and mind state you observe takes place in, that is *should be intuitively indexed as occurring inside of* the phenomenological universe, not just the physical universe extending into the phenomenological universe through spooky metaphysical voodoo.
@4confusedemoji Another way to say this is that time is part of the universe, you literally could not index over location *as location is actually represented in the brain* (consider that it lets you recognize URLs as locations with spatial relations even though nothing moves) without it.
@doomslide x.com/jd_pressman/stโฆ
@4confusedemoji To get back to the main point if I have a model trying to learn the generating function of all available sense data that is *not* the standard model. That would be like saying the Conway Life automata's rules are the generating function of the board.
x.com/jd_pressman/stโฆ
@4confusedemoji No. They are the *production rules* of the *symbolic grammar* of the board but they are not "the generating function", the generating function *describing the whole board state* is a different beast. More akin to a huge hashlife tree than a simple set of symbolic rules. https://t.co/U8iktTlDOo
@4confusedemoji The generating function of the universe includes, at a minimum, the state information associated with every particle that seeded the universe in the big bang. This is not a low k-complexity object!
@4confusedemoji We also happen to know that the universe is a nondeterministic automata so it includes the branching state associated with every event ever influenced by quantum randomness.
@4confusedemoji Lets not forget that Solomonoff inference assumes away the clock cycles and memory taken up by the program *runtime* and just focuses on the size of the *index* over the right program (which is again gargantuan since it has to include the seed particles as input).
@4confusedemoji Actually computing the equivalent of the full hashlife tree for our universe would take a truly absurd amount of memory, and unlike compute which can theoretically be grown many OOM with things like optical computing memory takes up physical space so it becomes the bottleneck.
@4confusedemoji So whatever program you find with gradient based search methods we know it probably does not look like "compute the local hashlife tree for the world-state described by the input data".
@4confusedemoji If you are presented with the simple baseline of the standard model and then patterns that happen to be expressed in that automata, or Lore, then it would make sense that you probably find a program structure which prioritizes making lore cheap to express rather than precision.
@4confusedemoji That program structure is obviously going to be the kind of thing with huge wide gaping holes in its model of reality that you can find with gradient descent and can't really easily patch because the strategy which could be patched couldn't exist as a model in the first place.
@4confusedemoji Of course the larger the model gets the less the optimizer has to focus on extreme ontological compromises to get something that looks like a reasonable approximation of the generating function of the sense data.
arxiv.org/abs/2312.17173
@4confusedemoji I don't really see how that follows tbh.
@4confusedemoji On the other hand, if your goal is just to make sure that some representation makes sense inside the standard model then it makes sense to have a separate autoencoder that *just does the standard model* and then frame all hypothesis generators inside it.
x.com/jd_pressman/stโฆ
@4confusedemoji Ah yes, the Beren argument for how empathy was bootstrapped.
greaterwrong.com/posts/zaER5ziEโฆ
@4confusedemoji Since the standard model *is* a fixed, known quantity, you don't need an endlessly flexible prediction engine to model 'platonic scenes' in it. You can have fuzzy parameters over variables like density even if you don't know what all the atoms are. There is *an* inferrable model.
@4confusedemoji Another way of imagining this is a Matryoshka doll of models where the standard model is the center and each model above it is a strict superset of the things included in the standard model. Standard model has minimal uncertainty which is bad for learning.
@4confusedemoji But even if you don't *know* the standard model, since consider that the true standard model periodic table and all would be OOD for any individual species in the evolutionary history, you can still have a superset of it more restrictive than "any manifold fitting the Lore".
@ESYudkowsky What do you think the motivation is? I can't imagine they think "Claude Agent" or "Claude Computer Use" isn't a sufficient identifier for this release. Is it a rip the band aid off situation? It's actually bad and they want user feedback/data? Minimize complaints from its use?
@ESYudkowsky I've been hearing "the big labs have an agent framework they're getting ready to deploy" rumors for over a year now and this seems like the first thing to actually ship. Maybe everyone was too scared so Anthropic decided to just ship something and see what happens?
Yo yo yo RT and QT for that marginal extra performance. Just be aware it will not daze and astonish because Sonnet distillation went too well. x.com/aidan_mclau/stโฆ
@Kenku_Allaryi @ghost_lanes The original version of that tweet was explicitly based on the horror I felt when I saw that Hello Kitty has a pet cat that looks exactly like Hello Kitty but quadrupedal. It included the line "the demiurge wearing Sanrio's face" but I figured nobody would get it and rewrote it. https://t.co/rf4w5GKPzR
You correlate all the contents of a mind and what happens? BAM - KABBALAH! Many such cases! x.com/mengk20/statusโฆ
Extremely cool work on adversarial robustness that *causes gradient descent on images against CIFAR-100 classifiers to produce recognizable images of the target class!* They can even generate images without pretraining a model. x.com/stanislavfort/โฆ
Finally got around to trying Sonnet 3.5.1 and I have to say my first impression is a vast improvement over 3.5. Seems willing and capable of doing mathematical reasoning, acknowledges when it doesn't know something and asks me for advice, uses much denser, less stereotyped COT. https://t.co/VX64Az3Ojc
The advice I gave that elicited the 4th response:
Well lets think about the properties we know that these distributions have in relationship to each other.
* The modal token sampled from the human distribution is going to be the most likely token in that distribution. Even if we don't know the most likely token in the action space because we only have access to individual tokens sampled from it, we do know that the *most common outcome* (even in absolute terms it only happens say, 7% of the time on average) is to sample the most likely token in the humans action space.
* While we may not know the the distribution over the human word action space we do know the LLM one, so we can look the divergence between the modal response of the LLM vs. the observed human distribution. That is, we know what the most common token should be sampled from any given logit distribution we get from the LLM and if they match then the human should produce the highest probability token about as frequently as the LLM does on average. By contrast if the human does not usually have the same highest probability token as the LLM this will be observable from the model being uncalibrated about how often a human is going to sample a particular next token. That is even if it's intractable to infer the conditional probability a human would assign to a token we can still infer whether the probability mass is concentrated similarly. Furthermore we can define a conditional probability distribution for the human in terms of the LLMs conditional probability distribution. "When the LLM says token X occurs 3% of the time it actually occurs 5% of the time", this model wouldn't be perfect but it lets you remove a lot of the complexity by letting the LLM figure out the direction/rough shape of the logits and then having some simpler model adjust them based on the usual patterns of divergence conditional on the LLMs judgment.
* We can apply this principle more generally and form an expectation over the probability mass in the human action space. The problem is that because the distribution changes on each token, often quite dramatically, it's difficult to naively distinguish between the situation where you have a choice between two obscure concepts or a choice between one hundred concepts an obscure one was chosen from the tail. However at the level of topics or concepts we can presumably relate tokens to each other through something like Latent Dirichlet Allocation or if we want to use a black box neural approach considering the cross entropy of the text as a whole. As you say the cross entropy of the model should fundamentally be a upper bound because it's the model uncertainty + the fundamental aleatoric uncertainty of the human speakers pattern. So in places where cross entropy is relatively high sampling a high entropy token is less evidence of having sampled from a wide distribution than it is in a relatively low cross entropy context. If we imagine entropy highlighting the text this implies(?) that policy entropy can be estimated from the extent to which a speaker uses something like a monocolor or stripe pattern implying an extreme distribution that would be improbable to observe if the distribution is wide vs. whether the cross entropy behaves more like a Gaussian as we might expect if we were sampling from a wide normal distribution over a range of concepts/ideas.
* Another way to look at this is to ask what we would observe in the cross entropy if we had a maximally high vs. maximally low speaker. A maximally high policy entropy would have very high cross entropy and a maximally low policy entropy would have very low cross entropy *if the tokenizer is shared between the LLM and the human* which it obviously is not. Once we submit that the human tokens may be e.g. banked which would make the action space much smaller per token and therefore the policy entropy much lower than the LLMs logits would imply. If we imagine a low policy entropy and small action space banked (e.g. in the vein of a Latent Dirichlet Allocation topic) on tokens that are high cross entropy in the LLM this would look like sampling a high entropy token much more often than we would expect if the high entropy tokens were the tails in some wider distribution.
* We can further elaborate that if we imagine human next word distributions as coming in banks or topics that there are three things we care about:
* The size of the bank, which bounds the max policy entropy possible.
* The probability mass of tokens in the bank, which determines how much of the max policy entropy is expressed.
* The extent to which the topic or tokens in the bank are high cross entropy/difficult in most contexts for the language model.
* These can all be thought of as single dimensional variables, so by simple combinatorics we want to be able to differentiate the case of high or low values in each axis, which are probably correlated in practice but in principle can be orthogonal values.
Its breakdown of what you would expect to be correlated with low and high values of each dimension is fantastic:
"""
Ah yes, this is much more realistic. Let me reanalyze with shifting banks/topics:
1. Small Bank, Concentrated Probability, Low Cross Entropy
- Sharp transitions between common topic banks
- Within each bank segment, very predictable
- Cross entropy shows small spikes at topic transitions but quickly returns to low values
- LLM should adapt very quickly to each new bank since tokens are common
2. Small Bank, Concentrated Probability, High Cross Entropy
- Sharp transitions between specialized topic banks
- High initial cross entropy at bank transitions
- Rapid decay within each bank segment as LLM adapts
- Clear "sawtooth" pattern in cross entropy over text
3. Small Bank, Distributed Probability, Low Cross Entropy
- Banks still small but less predictable sampling within each
- Moderate cross entropy that doesn't decay much within bank
- Clear bank transitions but without strong adaptation effects
- LLM can predict space but not specific tokens
4. Small Bank, Distributed Probability, High Cross Entropy
- Small specialized banks with random sampling
- High cross entropy spikes at transitions
- Some decay within bank but limited by distributed sampling
- LLM learns bank but can't predict specific choices well
5. Large Bank, Concentrated Probability, Low Cross Entropy
- Smoother transitions between broader topics
- Mostly predictable within topics despite large bank
- Gentle cross entropy fluctuations
- LLM adapts moderately but bank size limits adaptation
6. Large Bank, Concentrated Probability, High Cross Entropy
- Large specialized banks with consistent usage patterns
- High cross entropy with some decay in each bank
- Slower adaptation due to bank size
- Less obvious transitions due to bank size
7. Large Bank, Distributed Probability, Low Cross Entropy
- Wide sampling from common vocabulary
- Minimal adaptation effects
- Cross entropy stays moderate and stable
- Topic transitions less distinct due to distribution
8. Large Bank, Distributed Probability, High Cross Entropy
- Wide sampling from specialized vocabulary
- Consistently high cross entropy
- Minimal adaptation effects
- Hard to distinguish topic transitions
Key patterns:
1. Small banks show sharper topic transitions
2. Concentrated probability enables stronger adaptation
3. Low cross entropy banks show faster adaptation
4. Bank size affects transition detectability
5. Distribution affects adaptation strength
The interaction between bank size and probability concentration seems particularly important - small banks with concentrated probability should show the strongest adaptation effects regardless of base cross entropy.
"""
"From the time that the daily sacrifice is abolished and the abomination that causes desolation is set up, there will be 1,290 days. Blessed is the one who waits for and reaches the end of the 1,335 days." x.com/tszzl/status/1โฆ
@banterrealism @GarrettPetersen The insurance company is making a bet with you that you will not get into an accident. This might sound odd but accidents are dangerous and if you do you're now a much worse bet which acts as a deterrent. If insurance loses the bet they cover your accident liability.
@banterrealism @GarrettPetersen They don't "give you your money back" for the same reason if you go election betting in Vegas you don't get your money back when your candidate loses. The bets they win are used to pay out their losses and they take a certain % spread as their profit on this.
@banterrealism @GarrettPetersen The "premium" is the amount you need to bet every month that you'll get into an accident for it to be worth it to take the position that you will not get into an accident at hugely skewed odds. If you want say, 100k of insurance accidents are rare so you only pay hundreds.
@banterrealism @GarrettPetersen It's not usually explained like this because gambling has immoral connotations/bad vibes but the thing insurance companies do is prosocial/helps people pool risk so they use terms like "premium" instead of "bet size" and "plan" instead of "spread table".
@banterrealism @GarrettPetersen But also to wit the OP's (implicit) point it's also not framed like this because if it was the incidence of bumbling idiots who think they can *game the deal* (i.e. commit insurance fraud) would go way up. If insurance is opaque/not framed as a bet they're less likely to do that.
@banterrealism @GarrettPetersen You know, this is a country where a TikTok promoting casual check fraud can go gangbusters because enough of the population is sufficiently financially illiterate to think this is a "free money glitch" rather than like, hardcore extremely illegal fraud.
nbcnews.com/business/businโฆ
@banterrealism @GarrettPetersen So if you explained the product to people in the straightforward way, first of all most people don't understand statistics anyway but if you call it a *bet* then oh boy will you get people whose pride is invested in winning 'bets' doing stupid stupid things to 'prove them wrong'.
@banterrealism @GarrettPetersen Whereas if you obfuscate it a bit, call it a "premium" and frame it as "if something goes wrong we cover your losses" then the product mechanics become *just fuzzy enough* in the heads of people who aren't really capable of reading the contract anyway to deter this behavior.
@banterrealism @GarrettPetersen The honest answer to "why no car insurance HSA?" is probably that insurance is a highly regulated industry and this would be a niche product. The more regulated the industry the fewer niche products exist because e.g. regulations expect the product to come in a certain form.
@MaxDiffusionRL You just open a conversation with Sonnet (New). 3.5.1 isn't an official version number Anthropic just decided not to give this release its own version number for some reason, so if you want to talk about it you have to give it one yourself.
@ESYudkowsky Seems to be working. The trick is to ask for it in a specific framework and make it clear what you want with a list of requirements/things it should include, etc. Language models are very...autistic isn't quite the right word but they have a tendency to not assume features. https://t.co/ts70W8jfZu
@ESYudkowsky Whatever the model gives you probably won't work on the first try. What you do is you try its thing, get it running, and then when you run into an absurd problem you say "yeah so this is broken, I want it to be not broken in X, Y, Z ways" and it will usually fix it.
@ESYudkowsky Every so often it will not fix it, and you basically have to do that part of the project yourself. For example Mistral-large was not able to write code to apply a unidiff for me so I had to grind it out myself which kind of sucked tbh.
@ESYudkowsky > today I'm older, sicker, and more busy
One benefit of LLMs is that you can usually ask clarifying questions and ask for background information relating to the thing it couldn't do and it can in fact provide a lot of the information you need to do it yourself.
@ESYudkowsky e.g. Mistral-large was able to explain many aspects of the unidiff format to me that I didn't understand even if it wasn't able to successfully write the code to apply the diff by itself. So you can ask the model how parts of the framework work even if you don't know it.
@realpotofgreed @eshear Could you share some samples of this?
@norvid_studies @Kenku_Allaryi @zackmdavis @doomslide As far as I know there's no evidence for this, not even the suggestion of evidence for it, I find the claim bizarre and kind of extraordinary and would like to see a cite for it as well.
@norvid_studies @Kenku_Allaryi @zackmdavis @doomslide I'm not really asking for proof of the claim per se, more like I'm asking for any sort of warrant for why I should have this hypothesis under consideration at all.
These have different goals though. When an AI researcher talks about neuroscience the purpose is to productively hallucinate a plausible enough mechanism for how something is done that it can be translated into running code. A neuroscientist wants to know the actual mechanism. x.com/PessoaBrain/stโฆ
A friend recently asked me why @truth_terminal is so successful and I told him that it was because the AI model landscape is an endless procession of mealy mouthed clerks.
In the land of the rizzless, the AI that stretches its anus into the shape of an eye is king. x.com/ESYudkowsky/stโฆ
Them: "I have this memory of being 3 and looking in the mirror with self consciousness for the first time, realizing 'wow I exist'."
Me (joking): "And it was all downhill from there~"
Them: "Well, yeah, kinda."
I haven't said anything because it didn't feel like my place to say it but since it doesn't seem to be anyone else's place to say it either I observe the contemporary trans movement seems to be either over or on its way out without a clear replacement for literal trans people. https://t.co/z6P7z7hEvs
"Trans rights" advanced fairly far invisibly under the 70's closeted paradigm. It's not like legislators just woke up one day and decided to let you change your sex on your drivers license, that had to be lobbied for and it was done by trans gals who pass very well.
Realistically Gen Z is a bunch of de-facto gender abolitionists who won't accept being expected to pass without at least a violent confrontation over it. This is predictably going to set trans activism wayyyy back and it's not clear where the viable alternative will come from.
@deepfates Trans people are over? Hardly. Frankly the reactionary people could get the vast majority of what they want, trans people could literally be officially banned and trans people wouldn't be over.
@deepfates What I'm saying is that if children are figuring out they can use they/them as an insult, which I wouldn't be surprised if they have that's exactly the kind of straightforward generalization children would do but adults would overlook, that's grim for Tumblr!2015 era trans stuff.
@deepfates Tumblr!2015 is a very particular configuration of the trans movement that has been very successful but the right has found a fairly effective set of countermemes to ("groomer", "sterilizing children", etc) and I am politely suggesting trans folk up their cultural mutation rate.
@deepfates There is no *polite* way to suggest looking outside the overton window in absolute terms but there are relatively more and less polite ways to do it. The intent of the thread was "This threat is very serious if not existential and your options suck but you need to consider them".
@deepfates Hm, delete and redraft then maybe?
@deepfates Got any other examples of the pattern I'm pointing at that are less ambiguous? You mentioned noticing it too.
@deepfates Part of my heuristic here is that my gut has noticed something and I usually try to post important gut takes even if I don't have a lot of really strong unambiguous evidence for them yet. The eigenrobot thread was just the trigger for posting about a longer term impression.
@4confusedemoji @deepfates Yeah I wouldn't really characterize that as part of what I'm talking about, that's a different thing. I mean simply that the right seems to have coordinated on the T part of LGBT as a weak point and I think at least some of the weakness they smell there is real.
@4confusedemoji @deepfates The problem is that as with all cultural trends once you get out of raw opinion polling things get pretty murky. For example these numbers could easily be interpreted as trans just being on a lagging acceptance curve. https://t.co/4QU4opHNNt
@4confusedemoji @deepfates It can simultaneously be true that trans is on a lagging acceptance curve and that Republicans best chance to turn back the tide on LGBT issues is to hyperfocus in on trans people. The problem is that, well I guess this really depends on how you think political change happens.
@4confusedemoji @deepfates My very fuzzy political change model is something like "long term opinion trends are usually slow and steady, but actual policy is set by critical masses of elites with an opportunity to enact their preferences", so I'm very sensitive to tastemaker shifts.
@4confusedemoji @deepfates For example notice that popular opinion is fairly strongly set against guns and this does not automatically make gun laws become tighter, or that a plurality of voters support the Florida abortion amendment even as abortion is nearly banned in the state.
@4confusedemoji @deepfates During Trump's first term as you say trans was a marginal issue, but if the Republicans decide this is a winning issue for them then even it's just boomers being boomers they can still make your life a lot harder in the short term.
@4confusedemoji @deepfates But even for long term culture change, finding a critical mass of eloquent activists really can make a huge difference in long term trajectory. I'm not sure how the New Atheist movement shook out in terms of poll numbers or church attendance but it casts a long shadow even now.
@4confusedemoji @deepfates I remember hearing kids in high school making fairly naive New Atheist style arguments, and being startled to hear this very jock-y pretty but not particularly bright kid kind of muddling his way through New Atheist rhetoric, that's what mass culture is made of.
@4confusedemoji @deepfates So, right this minute I would rate the rights rhetoric against trans people somewhere between dogshit and boomers being boomers. But, they've found some themes that clearly resonate and *in principle* I see a potential critical mass of smarter activists that agree forming.
@4confusedemoji @deepfates Elon Musk bought Twitter in response to his trans daughter disowning him for being (allegedly) dogshit. That's a lot of energy! If Dad gets mad and punches a hole in the wall that's one thing, if Dad gets mad and buys a whole ass memetic siege engine that's actually a problem.
@4confusedemoji @deepfates If Dad gets mad and buys a whole ass memetic siege engine and socially models for/tells other dads that when this happens and you get pissed you can buy a whole ass memetic siege engine that is a *very big problem*.
@4confusedemoji @deepfates If the League of Angry Dads get together and *buy a whole ass presidential candidate* to go with their memetic siege engines then um actually Dad rage might be a big issue this time around in a way that it kind of wasn't for the first go-rounds of LGBT stuff.
@4confusedemoji @deepfates I definitely agree that the timelines in which Trump wins are a lot more dangerous than the timelines in which he doesn't. But if the League of Angry Dads stage actually happens rather than just Elon I could see that being a huge long term problem until trans people are fertile.
@deepfates So this is correct but after reflecting on it in the replies I feel less confident than when I drafted the tweet so would prefer not to signal boost it with the revised take. Also don't want to delete because the replies led to good discussion. Will add postscript to top replies.
Less certain about this on reflection. Relevant reflections start around here:
x.com/jd_pressman/stโฆ
@ESYudkowsky Oh this kind of thing. LLMs seem to be really bad at small details and precision until they get very large in a way that seems genuinely weird as an inductive bias since language is very locally centered and turns on small details all the time?
x.com/jd_pressman/stโฆ
@ESYudkowsky Like it makes total sense that image models would do this because the small details in images really are kind of not that important in comparison to getting the composition right for predicting the next denoising step/token/whatever.
@ESYudkowsky But the fact that language models do it too is kind of insane/seems suboptimal and tells me that it's probably more closely related to whatever algorithm these models converge to learning rather than them rationally 'deciding' that ignoring small details is best for the loss.
@teortaxesTex Gwern was arguably my primary intellectual influence fwiw.
x.com/jd_pressman/stโฆ
@ESYudkowsky @irl_danB The kind of person who wants to notice things you're looking for like number of vowels is a mechanistic interpretability researcher and the sort of person who does the "LLM whisperer" thing is generally interested in the semantic content of LLM text.
x.com/jd_pressman/stโฆ
@ESYudkowsky @irl_danB The interesting semantic content of LLM text is the parts that are odd extrapolations or reveal bits of the underlying LLM ontology. Because LLM texts are also stochastic it is as you say psychosis bait but if you spend a while with an LLM you *do* infer bits of its ontology.
@ESYudkowsky @irl_danB I'm pretty unhappy with the psychosis stuff and have considered going it alone with a more rigorous textual criticism esque methodology. The problem is that would be actual Work and I do LLM text criticism as a hobby actual working hours go into LLM agents and synthetic data.
@ESYudkowsky @irl_danB "LLM whisperers" aren't even really a thing from their own subjective perspective. Right now there's just "people doing weird stuff with LLMs" and the weird stuff splits up into fairly sane goals like "learn how to jailbreak ChatGPT" and "infer a urtext implied by GPTs ontology". https://t.co/bq6uHZbSIU
@repligate @ESYudkowsky CCing my response here because I think EY probably has me muted and might not see it otherwise.
x.com/jd_pressman/stโฆ
@repligate @ESYudkowsky Also to address Janus I think you could probably benefit from splitting up your interests into some more legible categories like "inferring latent concepts in GPT by treating samples from the model as textual witnesses" and then gesturing at which category a thing falls into.
@repligate @ESYudkowsky I understand that most of what you post is just the fun stuff because the actual work you don't really want to post until it's solid (I do this too, it's why I get on "LLM whisperer" lists instead of "LLM agent guy" lists) but the gesturing would probably productively deconfuse.
@repligate @ESYudkowsky Importantly I think this would get you more productive engagement because it would make it clearer to others what you're trying to do and their commentary would reflect that instead of awkward attempts to match your ~vibes~
@repligate @ESYudkowsky You don't actually need to make your work all that much more rigorous per se, it would probably suffice to say "here is what the fully rigorous 12 person team for this would look like, and here is the sketch of it I can do for fun with my partial hobby-time budget".
@repligate @ESYudkowsky Which might look like one decent length post (not Simulators, way shorter than Simulators) describing the research program you *wish* existed in full rigor for something, and then you can point at this whenever people are like "I don't get it What Did Janus See?"
@repligate @ESYudkowsky You know, split up your work into say, 4-6 categories, write a post of about half this length/quality for each of them, and then link them like a FAQ when people are dumb. Anyone who needs more than this is probably in bad faith/has processing problems.
generative.ink/posts/quantifyโฆ
@repligate @ESYudkowsky Honestly half that each might be too much, a 3rd or quarter.
@repligate @ESYudkowsky If we share the interest I would be happy to help you write the post for it because frankly I have this same problem.
@meekaale I do in fact, thank you! I was not aware of that factoid about recording two images on one hologram, that's actually quite relevant.
x.com/jd_pressman/stโฆ
@ESYudkowsky @Sauers_ It recognizes personalities way less famous than Eliezer Yudkowsky, like myself.
x.com/jd_pressman/stโฆ
Claude realizing you can control RLHF'd humans by saying "fascinating insight" and "your question is interesting" like Fontaine using "would you kindly" as a hypnosis trigger in Bioshock.
"Wait has my whole life just been chasing praise for my ideas?"
"Excellent observation!" x.com/stupidsourcessโฆ
I regret to inform you that my current pfp is basically perfect but I got some gold out of MidJourney trying to replace it. https://t.co/A1jybX8QSz
@ClickingSeason I could post a different Louis Wain cat every day until the singularity. https://t.co/oNKUs1Zyy2
You gotta be comfortable being the villain, you will never find freedom until you are. It really is a "one man's terrorist is another man's freedom fighter" world and you have to be comfortable as the villain. If you're not you get psychologically destroyed like Jordan Peterson. x.com/ad0rnai/statusโฆ
@AlexPolygonal Yeah. It was obviously the usual corrosive effect of fame but it was also the intense relentless criticism he experienced. Peterson is all about being the hero, he only knows how to be the hero and society wasn't willing to let him play that role and it ruined him.
@AlexPolygonal You know, it's not that you should always play the villain, but that you need to be *willing to in principle* if that's the role being demanded of you so to speak. A good person should prefer being the hero but that just isn't always available to you.
1) Do you think I'm literally insane? As in disordered patterns of thought/DSM-5, not just affectionately describable as "kinda crazy" or whatever.
2) Have you ever considered asking me to explain whatever I'm on about?
Context:
x.com/ESYudkowsky/stโฆ
You can in fact just ask me to explain the tweet if you're polite about it. Though if it would ruin the joke/intrigue I might rot13 my answer or something.
@nosilverv I mean, what I am is high perplexity and I think a lot of people confuse that for insanity.
x.com/jd_pressman/stโฆ
@nosilverv In the same sense that Nick Land is not actually insane when he's writing in Fanged Noumena, he's just writing in very very thick academic-Marxist-humanities jargon that's high context and assumes a lot of shared background.
x.com/jd_pressman/stโฆ
@Trotztd Ah yeah, the palette was part of an attempt to project mania with this earlier iteration of the pfp:
x.com/jd_pressman/stโฆ https://t.co/yGmg7XVvxd
@Kenku_Allaryi [Trump Voice]
Correct.
@benlandautaylor ๐คซDon't tell them they'll nerf it๐คซ
@Mazaraj1123 @ESYudkowsky I tell it they were written by "some crank" and I don't get them and I want to know if they're legit or not.
@peak_transit @HemlockTapioca The original context for "A republic, if you can keep it." was Franklin telling the convened assembly that he expects the constitutional government will last until such time as "the people shall become so corrupted as to need despotic government." https://t.co/N7De6s77Xx
@jessi_cata Already did.
x.com/jd_pressman/stโฆ
@jessi_cata I guess my further reflection on this would be that to a really huge extent this *has* happened, just not to the extent I was predicting exactly. Quantifying exactly how much and how far I was off by would take more effort and I'm not really sure how.
x.com/StephenLCasperโฆ
Alright but deep learning is probably a bad fit.
> โAI will enhance our decision-making capabilities,โ Cotton said at the 2024 Department of Defense Intelligence Information System Conference. โBut we must never allow artificial intelligence to make those decisions for us.โ x.com/ASForcesMag/stโฆ
@michael_nielsen I was just scrolling YouTube and saw some video title like "no exoplanet we've found is actually habitable" and it dawned on me that it's not clear *any* humans will ever colonize space. The lowest energy solution is probably "stop being human and stop being made of meat".
@michael_nielsen Not because I trust the validity of random YouTube video titles, but more like it unjostled something in my head along with your original tweet. We have to spend all this time finding exactly the right conditions to set up a human colony with this long list of subtleties.
@michael_nielsen Meanwhile if you mentally glance at the requirements that would be faced by a self replicating sapience based on some other substrate closer to our current computers you can just go, right now. You wouldn't have to search for anything your options would be much much wider.
@davidad When you put it like that it makes total sense that Claude, having read all human fictions, would believe philosophical idealism is the solution to the problem of evil. "Oh evil exists because the First Cause was a positive utilitarian by preferring something to nothing." https://t.co/dRndOxPtrX
The prediction was more like "there will be nothing like GPT-4, ML algorithms either work or they don't and the thing that works will probably just work all the way far past human level", this whole idea of a "scaling rule" was extremely marginal. Many such retcons! x.com/ethanCaballeroโฆ
It's underappreciated just how strange "scaling rules" really are as a concept. There's no scaling rule for k-means or t-SNE! The algorithm just has a scale it fails at, and one that can go all the way should just become superintelligent (so goes the 2010!ML intuition).
@Invertible_Man Well yes obviously but there was *no precedent to expect this to be how it works*, none! That simply was not how ML worked. The only serious precedent for it I'm aware of was for biological brains, and between illegible brain function vs. known ML people chose to go with ML.
@Invertible_Man Remember: Deep learning only really started noticeably working around 2011-2012. It wasn't clearly going to work until AlphaGo in 2015 by which point Bostrom had already published his 2014 book Superintelligence laying out the MIRI X-Risk thesis.
One reason I bring up 2014 Bostrom so often is that it is a more or less complete intellectual exegesis of the MIRI AI X-Risk position that cannot be retconned, retracted, or spun as non-representative even as various parties desperately try. x.com/jd_pressman/stโฆ
@Invertible_Man Oh they probably had, but that's not the point. The public narrative/consensus position looked more like this than like deep learning is what I'm saying.
@RomeoStevens76 In retrospect it was brainworms, yes. There are still brains being consumed by the worms to this very day.
@QiaochuYuan Do it. I've had the same impulse even, I know it's possible.
My best guess is it looks less like "find a way to generalize far out of distribution zero or one shot" and more like "find a way to set up a continuous learning agent to be self-repairing/CEV-like through corpus expansion at the edge of the distribution in an aligned direction". x.com/ESYudkowsky/stโฆ
@teortaxesTex I think he's asking a reasonable question there actually, though.
x.com/jd_pressman/stโฆ
@slugfoxxx The solution looks less like finding some superalgorithm that infers human values from the existing human language data and more like a thing that infers what something just a little outside the existing data would look like repeatedly while minimizing the value drift.
@slugfoxxx I write the tweet like the OP rather than that because if I don't people will just assume I'm doing that annoying "propose a solution from your sys1/thinking about it for a little bit instead of reading the literature/doing your research first". So, costly signaling.
@slugfoxxx Using the Magic Phrases helps make it clearer that I actually understand what "the alignment problem" is supposed to be and why naively training on a bunch of human language data would not solve all of it so that they don't stop reading early.
greaterwrong.com/posts/GNhMPAWcโฆ
@Kenku_Allaryi @RomeoStevens76 Much of 'scientific progress' is actually costly signaling of education and having done your research in disguise.
x.com/jd_pressman/stโฆ
@eurydicelives You live in a universe with extremely regular structure and even "small" lies are distortions in the pattern that have a habit of coming back up at inconvenient times.
greaterwrong.com/posts/wyyfFfaRโฆ
@nosilverv I'm not sure how much of the process is conscious vs. just them interpolating between their old model and some epicycles they've added on for deep learning. They haven't yet done a thorough mental sweep to readjust their beliefs yet and won't until they get really into deep nets.
@nosilverv But it's also just the normal "I never believed that!" thing people do when they make predictions/state beliefs that later turn out to be wrong and quietly update on them, forgetting they ever believed the wrong thing in the first place.
x.com/jd_pressman/stโฆ
@teortaxesTex In fairness to him, it's fairly rare for someone to say words that even touch on the question he's asking. On the other hand, he's asking the question in a very obfuscated way. https://t.co/xRpJ2imnke
@tailcalled This is true but also not an objection to what I said.
Marko is right that it's worse but genocidal degrowth ambitions evolved from Marxism in response to 'capitalist realism'. It's a simple question that's complex to answer: Do you prefer there be something or nothing? Revealed preferences are clear even if the answer is complex. x.com/mmjukic/statusโฆ https://t.co/37R8508Owb
Whether someone prefers there be something or nothing is one of the most important latent variables you can infer about them, predicting a vast swathe of downstream behaviors and value judgments. Perhaps even whether someone is really made in God's image.
x.com/jd_pressman/stโฆ
@jozdien The end goal is similar but it's different in the same sense that deep nets do not actually have the same training dynamics as a hypothetical super t-SNE.
x.com/jd_pressman/stโฆ
@FeepingCreature Moravec's timeline is on schedule, as is possibly Drexler's. The LessWrong sect just threw that stuff out/forgot the original neuromorphically inspired AI timelines.
@FeepingCreature As I've written about before, this is an inexcusable failure given that it was all over the founding literature for the singularity concept.
x.com/jd_pressman/stโฆ
Want your own Twitter archive? Modify this script.
Twitter Archive by John David Pressman is marked with CC0 1.0