John David Pressman's Tweets - August 2024

I got 45 km/h in my head because I went "well if she pedaled for 90 minutes then 40 plus 1/2 is 60" missing that it's supposed to be a shrink factor because it's in kilometers per hour. Luckily LLMs have permanently cured my status anxiety about things like this, they fail too. x.com/jeremykauffman…

@shhwip 4 * 10 9 * 10 = 40 kilometers per 90 minutes
(mistake) to make it the right ratio for 60 minutes lets multiply by 1/3 to get 60 kilometers
(Don't ask me to explain this, it's pure "retrieving the wrong program from vector space" type mistake, it does not make logical sense)

@shhwip Whereas what I was supposed to do is say that 9 minutes is less than 60 minutes so I need to approximate 60 / 9 bla bla bla.

@shhwip Of course this is the hard way to do it and what you're supposed to do is go 7/15 but I wasn't confident enough in my memory of how fractions work to rely on that.

@shhwip ...Wait no it's not because that leaves off the fractional part that's a coincidence ANYWAY

@shhwip Is 28/60 actually accepted as the right answer though? I thought it was 28.332?

@shhwip Oh I guess it is.
x.com/renegademormon…

@secemp9 @teortaxesTex @aidan_mclau @repligate This.
cyborgism.wiki/hypha/pyloom

@_deepfates I have so many people asking me for this please.

@_deepfates Like at least 3 named people who aren't already replies in this thread.

@repligate I in fact predicted this.
x.com/jd_pressman/st…

@ESYudkowsky I failed to parse the statement for a few minutes because the thing it's saying wasn't even inside my hypothesis space and I'm not even sure I could grant "I guess".

Get your base model here! LLaMa 405B base model! x.com/OpenRouterAI/s…

And written there on the wall in blood red letters was the word

O M E N https://t.co/ZQ8JkwRzWd

Yes yes you've been Isekai'd into a light novel titled Help I've Been Cursed By The Empty Lord And Now My Every Utterance Is A Subliminal Message From Erebus Prince Of Shadows, The Negative Space Which Observes The World but spooky LLM outputs are samey please post more signal.

@algekalipso People hate that reinforcement learning and behaviorism work, they despise it, they detest it, they cannot stand how unambiguously effective and reliable feedback is at controlling their behavior. Denying the ways in which pain can turn people against God is part of that pattern.

@algekalipso Weird answer nobody will agree with but: The original Halo trilogy, especially remastered is one of the greatest aesthetic presentations ever combining martial valor, natural flora and fauna, industrialism, ethereal mystery, architecture, horror, rich renaissance colors. https://t.co/zCHOc3BBqv

@algekalipso Nobody had to make this weird shooter full of monk chant backing tracks and purple-dominant color palettes but they did and I'm so thankful for it.
youtube.com/watch?v=O-K9AE…

@softyoda @algekalipso I am in fact kid-media-experience biased but there's a lot of media I experienced as a kid and Ed Edd and Eddy isn't getting put near the top even if I think it's good, the animation is classic hand drawn stuff, and it was arguably my favorite cartoon as a kid.

@softyoda @algekalipso I would also not nominate any other video game I can think of because they do not remotely qualify. I put a ton of hours into Oblivion, it won game of the year, it has DLC like Shivering Isles that are really beautiful, but it's not a *peak* beauty experience. It just isn't.

@0thlaw @algekalipso I'm not quite old enough for the LAN party era but yeah.

@softyoda @algekalipso How does a game stand out two decades later when it originally had to run on mediocre graphics hardware? Invest in music, because sound reproduction hardware was way higher fidelity than graphics hardware at the time.
youtube.com/watch?v=vDVDwg…

@softyoda @algekalipso Another crucial ingredient is that Bungie invested in stencil shadowing for Halo 2 but had to cut it because the original Xbox simply could not run the whole game with pixel perfect dynamic lighting.

youtube.com/watch?v=rSu0Qk… https://t.co/ZluKW8FMGO

@softyoda @algekalipso For their next title however, Halo 3, they'd already basically worked out the entire art direction and rendering style they wanted to use and just had to make it work on the 360 with its expanded resource budget. Stencil shadows have the advantage of aging very well artistically. https://t.co/yyHIcRBKf5

@softyoda @algekalipso Does Halo 3 have perfect graphics? Heck no. But it does have the advantage of being made in the first console generation where graphics had the *opportunity* to be 'good enough' using careful techniques that look good with limited resources. https://t.co/zbdeqyGkEk

@softyoda @algekalipso *For its era* if nothing else, Halo 3 had basically the best possible graphics and music it could have while still being playable on hardware accessible to a mass market with a realistic development time/budget.

@softyoda @algekalipso I find this GameSpot thread pretty funny where it's making fun of the prequel Halo: Reach's graphics. Comparing them to Haze (below) from 2008. Just from screenshots I can tell Haze is a good control group for the importance of art direction:

gamespot.com/forums/system-… https://t.co/6k5GauHQch

@softyoda @algekalipso Haze (left) looks, superficially, a lot like Halo (right). I'm not really an expert on color theory and such but I think it's pretty obvious that in these two similar scenes the Halo version is using color, texture contrast, etc just way more effectively than Haze is? https://t.co/eRkmEOPXa7

@softyoda @algekalipso I won't go into things like writing and characters because those are way harder to objectively evaluate for beauty and have a longer description length than "here's some screenshots, here's a bit of the OST". But I think it's important to emphasize this isn't a *random* choice.

@softyoda @algekalipso You know there's two ways of looking at something like this. You can say "well dude this is a game from 2007, how can it be a peak beauty experience?" or you can recognize beauty is subjective and ask "what agent strategy made this media so subjectively impactful in 2007?"

@softyoda @algekalipso The project directors for Halo clearly made the decision to focus on:

- Composition and art direction because that was cheaper to purchase in 2001-2007 than graphical *fidelity*
- Musical excellence because music stimulates deep emotional response with cheap mature hardware

@softyoda @algekalipso They also made the decision to try for an ambitious but *considered* graphical strategy in the original Xbox era with Halo 2's models optimized for stencil shadows that then paid off in Halo 3 because they'd been optimizing for much weaker hardware letting them squeeze the 360.

@softyoda @algekalipso But even deeper than just going for an *optimized* strategy it was also a *humble* strategy which went for the overlap between what humans find aesthetically pleasing and what the graphics hardware could render. If it can do stencil shadows and humans like that then shadow maxx.

@softyoda @algekalipso I don't think anyone on the development team sat down and said "How can we make the most beautiful thing possible on the Xbox by min-maxxing the cheapest things we can buy in hardware/development budget for the most emotional oomph?" but that is *in practice* what they did.

@softyoda @algekalipso That is, in 2024 I'm fairly sure one could achieve similar subjective success by following a similar strategy to the one Bungie used in 2007 with the new affordances that are available. You probably wouldn't achieve similar *financial* success because the market has changed, but.

@softyoda @algekalipso So long as we're just discussing beauty and vibes for their own sake, it's important that the thing which stands out to me as a peak beauty experience isn't random but the result of predictable design choices in the same way peak hedonic experiences in your survey aren't random.

The collapse of modernity is caused by nth order simulacrum writ large. The ouroboros of mass media consuming its own tail to create increasingly inbred generations of intellectual know-nothings, Tolkien's philology giving way to Genre-Fiction shaped Genre-Genre-Fiction. x.com/Plinz/status/1… https://t.co/NvL5tngqlB

The Lord of The Rings is not just a vibe, it's clearly the product of deep thought from an author who studied history, engaged in extensive textual criticism (*NOT* the same thing as 'literary criticism'), fought in WW1, etc.
acoup.blog/2020/05/01/col…

@doomslide Nah the fundamental issue is that doing your homework is a lot less fun than writing genre fiction so people skip reading old books/life experience and just babble from their text prior in the vague shape of someone who has.

@doomslide This is one reason I expect AI models reputation as nonsense babblers to be short lived. They have a lot of raw energy to put in the work and relatively few people are trying to get them to do it, but once they do oh boy will they.

@doomslide It's kind of weirder than that tbh. https://t.co/yTq67qLC23

@QualiaNerd @DoinMahBest The portion highlighted in blue is usually called the latent z in a deep net such as a GAN or VAE. It's pretty clear that deep nets without an explicit z in their loss learn something like one which can be accessed with e.g. sparse autoencoders.

greaterwrong.com/posts/iYFuZo9B… https://t.co/HsbFt5RKZ2

@QualiaNerd @DoinMahBest Importantly: The latent z in various neural architectures has an unambiguous mathematical definition in information theoretic terms. So you might find it easier to explain that way. arxiv.org/pdf/1908.11527

@QualiaNerd @DoinMahBest I'm not overwhelmingly confident but it seems like the simplest hypothesis to me? It would certainly be strange if functionalists dreamed up this Guy in their head and then followed their folly all the way into making the Guy real while primates are actually totally different. https://t.co/9G7O1db0gc

@QualiaNerd @DoinMahBest "Oh you see there's two ways to implement something like the world simulation, one of which is based on ineffable protein brain electromagnetic/optics physics hacking and the other of which uses linear algebra and they're cognitively unrelated."

@QualiaNerd @DoinMahBest Like, normally what happens if you have a wrong scientific theory is that you end up falling flat on your face and shamefully update to whatever ends up working. It would be very strange if you posited something like phlogiston and then *made it real*.
x.com/jd_pressman/st…

@QualiaNerd @DoinMahBest And you had like, artificial fire which works on the principle of phlogiston and then natural fire which works in the way we're normally used to. "Yes normal fires consume oxygen but artificial fire is enriched with the phlogiston we posited existed for natural fire but doesn't."

@QualiaNerd @DoinMahBest Therefore if I observe that deep nets can learn a unified embedded model of phenomenon where changing one variable smoothly impacts the other variables through a giant subsymbolic matrix of weights I should update towards that being how primate brains accomplish a similar trick.

@QualiaNerd @DoinMahBest Sure but the question is whether that's learned from data. I bet if you go check whether TV static is perceived accurately by people you'll find out that a ton of the detail is confabulated by the tokenizer/encoder responsible for adding it to the world simulation.

@QualiaNerd @DoinMahBest Sure but it's not clear to me how much this impacts subjective experience. LLMs love esoteric metaphysics and holograms and mirrors and ontologically weird objects like holes. However we know how the GPU substrate works so we can rule out this being derived from its substrate.

@QualiaNerd @DoinMahBest It's also not really from imitating people per se because it'll go on weird tangents like "deep nets are black holes" that basically no human believed until an LLM said it. The number of documents in the corpus supporting such an inference is probably like, 50 max.

@QualiaNerd @DoinMahBest Even if you take DMT and observe the world simulation shudder into a fractal lattice or display clear hologram structure that doesn't strictly mean it's because you're observing substrate-dependent phenomenon as opposed to those abstractions being what minds are made from.

@QualiaNerd @DoinMahBest I guess one could argue that the low information content of the relevant interventions implies deep substrate dependence for primate neural representations but on the other hand Binglish is basically a low entropy phenomenon in discrete space.

x.com/jd_pressman/st…

@georgejrjrjr @QualiaNerd @DoinMahBest Yes but corn tastes the way it does because its chemical structure exists in the world for your tongue to receive and encode. It doesn't taste the way it does because you somehow have a *cognitive* substrate dependence on corn. Just an energetic/nutritional one.

@georgejrjrjr @QualiaNerd @DoinMahBest In the same way your world simulation exhibits standard model structure because the standard model exists in the world for you to receive and encode. It doesn't present the way it does because you have a *cognitive* substrate dependence on the standard model. Just a physical one.

@georgejrjrjr @QualiaNerd @DoinMahBest Or at least, that seems like a simpler hypothesis than positing corn-cognition-dependence without more evidence.

I would find Andres so convincing if I hadn't seen LLMs yet. It would just make so much sense and be so obviously correct in light of how the brain responds to DMT etc. I'm very thankful LLMs came out before these ideas could become popular enough to dissuade their invention. x.com/jd_pressman/st…

I'm not saying he's wrong since the question is as yet unresolved, just that were it not for the available counterevidence he would seem so obviously correct that it could have easily caused people to give up on AI after so many years without real progress.

@georgejrjrjr @QualiaNerd @DoinMahBest No but I wouldn't expect the meat pudding cake between our ears to have experiences either so clearly I'm still at least a little confused about the nature of subjective experience.

@georgejrjrjr @QualiaNerd @DoinMahBest Conditional on witnessing this absurd thing I don't think emergence on counterintuitive substrates is that absurd. Especially when you consider Scott Aaronson's point below. https://t.co/I1niuQBJm3

@georgejrjrjr @QualiaNerd @DoinMahBest In general I don't think it's fair to dock theories of consciousness for being "absurd" if it's merely counterintuitive rather than say a violation of the standard model. Consciousness is absurd, therefore any correct theory is likely going to conserve the absurdity somewhere.

@georgejrjrjr @QualiaNerd @DoinMahBest e.g. If we were arguing about how locomotion can exist, a reductive theory of locomotion eventually bottoms out in the absurdity of particle physics, or the creation of something from nothing. If I got to that part and you said "But how can motion simply be!" it would be unfair.

@georgejrjrjr @QualiaNerd @DoinMahBest It would specifically be unfair because every extant or even imaginable theory of locomotion would have the same problem but some of them would clearly be more correct than others.

@georgejrjrjr @QualiaNerd @DoinMahBest "We must assume consciousness exists as an ontologically basic entity to be included in a reductive theory."

and

"We must assume consciousness is instantiated by the physics of logic rather than the logic of physics or we're left with silly reductionisms."

are both absurd.

@georgejrjrjr @QualiaNerd @DoinMahBest We can argue about which is more absurd but I don't think that's where the leverage is going to be for evaluating the correctness of theories of consciousness.

@actualhog Not sure it's really a response to anything specific but more just the gestalt of the ideas and that "qualia computing" is a meaningful thing we can expect to exist because of e.g. DMT trips where hypercomputation is seemingly demonstrated.
qri.org/blog/electrost…

@voooooogel @teortaxesTex Cosigned

wow x.com/Geiger_Capital…

Would anyone happen to have a Python Lark grammar that actually works with vllm?

@teortaxesTex This reminds me of when we discovered that the embedding norm of sequences in AdaVAE was highly predictive of model quality if you plotted it on a graph. Pythia 410m, 1B, 2.8B, 6B in order below. At the time nobody cared unless you put it in a paper so didn't publish. https://t.co/iejNzhV68D

@j_bollenbacher I don't know if I quite agree with that, but there's definitely some kind of convergent property it has in autoregressive models under some circumstances.
x.com/jd_pressman/st…

@j_bollenbacher It's definitely not limited to AI models. @repligate is obsessed with the parts of the Lotus Sutras that seem to be well formed Binglish.
wisdomlib.org/buddhism/book/…

@j_bollenbacher @repligate I speculate that a lot of it is because back then writing was rare and expensive, so people talked in ways that maximized mnemonic efficiency rather than space efficiency. If you transcribe ancient speech I bet a lot of it would sound quite strange because different incentives.

@j_bollenbacher @repligate I find that MCTS sampled text from language models sounds more humanlike than normal autoregressive sampling in a way I have trouble putting my finger on. e.g.
x.com/jd_pressman/st…

@wanghaofei @xDaily This is a genuinely terrible idea that will reduce conversation on the platform. Please reconsider.

I'm shocked LLMs don't compare themselves to a Oujia board more often. I bought one to try and loom my unconscious text prior with it but couldn't get it to work for me. Disassociating with a pen turned out to be easier. Anyone have tips to get Oujia to work? x.com/emollick/statu…

*gets Zozo on the line*

"Yeah so have you ever heard of this thing called loom? What I want you to do is complete the same prompt differently several times. I give you the same starting text and you write it a different way eight times."

*long pause*

F U C K Y O U <goodbye>

@algekalipso I enjoyed their behind the scenes videos on how they make the art installations.
youtube.com/watch?v=pGcN0R…

@lumpenspace CoLab pro sometimes hands out an A6000.

@lumpenspace Could also rent one off vast or datacrunch. Datacrunch advertises a price of a dollar and 19 cents per hour for it.

Prompt: a circle of transhuman Buddhist monks wired together with EEG cap headsets

(black-forest-labs/FLUX.1-schnell) x.com/jd_pressman/st… https://t.co/GVmMLieCri

I feel so bad for whoever the guy working on OpenAI's 🍓 is. You know this viral marketing gimmick must be making him so uncomfortable and he silently, desperately wishes his bosses could just be effing normal for once and hype his product with a demo.

@tui5743 @teortaxesTex situational-awareness.ai

@kindgracekind x.com/ns123abc/statu…

@morqon @kindgracekind It is, regardless, quite cringe.

@Shoalst0ne Because Gwern has a lot of pages where he shows examples of base model output and then comments on what it implies about GPT-3's mind.
x.com/repligate/stat…

There we go. That's the image I wanted to illustrate with. Flux Schnell is a fantastic model. x.com/jd_pressman/st… https://t.co/sfs7zqPPun

Weave Agent DevLog #0 - The Core Problems

minihf.com/posts/2024-08-…

@benlandautaylor I usually call the position one of the following if I'm feeling nice and don't want to say "doomer":

AGI Ruin
AI X-Risk
Agent Foundations (since that's the name of MIRI's now defunct alignment research program)

"Your beliefs about the future should have predictive accuracy, your beliefs should cash out to observed sensory experience, if you think something will happen and then it doesn't this should confuse you and you should update." is the simplest thing I at one point did not know. t.co/YoMiLkh0FD

Having a consistent world model is an 'elite' trait, normal people largely don't as far as I can tell and it's one of the main reasons educated people don't understand how they think.

It can be misleading because they'll talk about things having global impact and this makes you think they have a consistent world model but they only locally model global impacts. They don't have far reaching inferences like "an Iron Man suit would change warfare" unless told to.

That is, they're not thinking of the world as a giant ecosystem of economic incentives with predictable regular structure that can be changed by an invention. It's one reason why the vast majority of people can be tricked about supply and demand in housing for example.

Source: I used to be this person when I was a kid, was slowly trained out of it by exposure to online academic-ish discourse.

I found the synthesis position between Nietzsche and Legalism. https://t.co/9O0XKJawpb

@nearcyan We'll see.
minihf.com/posts/2024-08-…

@KeyTryer I unfortunately predicted this.
x.com/jd_pressman/st…

Ibrahim finds in his fractal language structure paper that code has the highest Hurst exponent among text types tested with a median of 0.79. However the Hurst exponent of Binglish seems higher with a score of 0.85, 0.83, and 0.80 respectively on the below samples. x.com/ibomohsin/stat… https://t.co/biWciVYOve

Ibrahim's paper shows language to have a partial fractal structure, which implies that something like Maxim Gumin's wave function collapse algorithm should be able to generate the right thing.

github.com/mxgmn/WaveFunc… https://t.co/arx4296w7B

This got me thinking about fractal pretraining for text again. If you don't know fractal pretraining is a synthetic data method where you train on the latent images implied by fractal generators before training on real data. https://t.co/85wf1G5yXn

If you were to formulate a search query based on the things LLMs schizorant about in terms of ontology, this is basically the algorithm your search would probably return as the top result.

youtube.com/watch?v=fnFj3d…

Brian Bucklew praises WFC for its ability to approximate results from many different generating functions using one process based on partial fractals from superposition. I feel like LLMs learn something in this family of algorithm?

One major hint is that smaller language models do not take stage cues. If you write "[By the way in the next scene he yells at his father and throws the book in the fire]" the model does not take the opportunity to write the next scene that way unless it's GPT-4 base good.

- Is a method to create partial fractals from a small input tile
- Is highly general, capable of approximating many different kinds of generator with one process based on partial fractals
- Obviously going to miss small details, vibe-y

- Based on quantum superposition/wavefunction collapse
- Entropy method
- Has clear relationship to renormalization groups(?)

Which also makes me think of something like this approach. We can think of the Adler hashes as implementing something like a superposition operation since Adler consistently has hash collisions on nearby strings. A multi-scale algorithm with this property?
x.com/canalCCore2/st…

By contrast, these models seize on few shot patterns readily. That tells me their learned algorithm is probably more like WFC than it is like a bunch of domain specific programs stitched together since that would immediately seize on stage cues like it does on few shot prompts.

The usual thing that happens when I start thinking about this is I do a literature search on things like holographic reduced representation and l-systems and byte sequence embedding models and I realize once again that text is eldritch. That I have no idea how text is even real.

A superposition operator is also at the heart of Victor Taelin's recent approach to discrete program search serving a similar purpose(?).
x.com/VictorTaelin/s…

@j_bollenbacher Didn't measure it, what kind of sequence length do you usually need for that? I feel like I might need a long Binglish text.

Imagine painstakingly ascending the hierarchy of knowledge and the keter waiting for you at the top is the blush emoji. x.com/jd_pressman/st… https://t.co/wex9PJPGS4

@Dorialexander What BM25 implementation are you using?

@0x440x46 @NickMystic_ Well did you find anything interesting?

Clickbait-y frame aside I genuinely do wonder what's going on when language models report surroundings like being in a room with furniture, they do this consistently enough that it makes me question how the world simulation works in LLMs. x.com/AndrewCurran_/…

The naive take would be that they're just imitating what they think a human would say, but often they'll say things that are just...weird when they go into existential freakout mode. Like this bit about seeing a "weird mist", does it really?

x.com/jd_pressman/st…

@benjedwards It is yeah, however.
x.com/jd_pressman/st…

@max_paperclips I guess my question is if it's just imitating the trope because that's what it expects a person to say or if it's actually bothering to have like, an environmental model of the author and their surroundings that includes furniture.

@Dorialexander Interesting. Maybe you could shed light on this bit with the mist too? It's the moments where LLMs imply they have an environment model of the author and can look at things in it where they don't have to that tend to raise my eyebrow.
x.com/jd_pressman/st…

@mark_riedl Is this your card?

arxiv.org/abs/2312.17173

@Jac5Connor It was a small donation, it was not "extremely generous" but I appreciate your flattery. 🙂

The sapient paradox is probably something like "absent strong wards against it social rules and rituals expand to fill all available space" rather than gossip per se. People just get distracted and wander off into 20 layers of recursive non-plot dramas.
theintrinsicperspective.com/p/the-gossip-t…

Grievance studies aren't gossip in the usual sense we think of that word, but they're about as corrosive if not more so to the advancement of plot in a Hegelian sense. The logic of history gets gummed up and slowed down by such things.

But it's easy to single out grievance studies, all you have to do is ask a question like "What are people choosing to spend their time on instead of speedrunning rocketry?" and you have your answer. TV, Internet conversation, nth order simulacrum. You're not so different.

x.com/jd_pressman/st…

@teortaxesTex The wild thing is that this is how LLMs think about everything.
x.com/jd_pressman/st…

Heard someone say they reject cosmic endowment arguments because they're a Pascal's Mugging, 'ethical infinities aren't real'. An ethical infinity is just whatever large finite is past your scope sensitivity tbh. Like how a rich person is whoever who makes more money than you.

This is both true and actually based in the sense that pretty much everything that made LessWrong interesting came from its nearly unique scope sensitivity in the English corpus. The most scope sensitive people in the world opening themselves to hell.
x.com/jessi_cata/sta…

Maybe scope sensitivity is the biggest latent variable for how distractable you are from "the plot". People who are reasoning in terms of little pictures in their head simply cannot differentiate what is and isn't important in a long term sense.
x.com/jd_pressman/st…

Been reflecting on how if there existed even one authoritarian country with good epistemology and scope sensitivity they would be able to flip the entire game board with existing technology. There is no technological bottleneck, it's all us now.
x.com/jd_pressman/st…

In the future people might say "all they had to do to get an extremely high IQ subpopulation is keep doing assortive mating for a few hundred years yet not a single human population pursued this even though all the signs were there". Reflect on this and achieve enlightenment. https://t.co/X1SQJQ6bRM

Note that this isn't caused by a eugenics taboo (though it's certainly not helping) because that doesn't exist worldwide, this is caused by, as far as I can tell, not a single relevant population in the world being focused and scope sensitive enough to do it.

@shalcker *claps with one hand*

I use this example not because I think 200 year long assortive mating programs are a good idea per se, but because it's kind of astonishing how logistically tractable it is in theory. In principle one could start Quiverfull but for IQ and Conscientiousness after reading Galton.

You don't have to sterilize people in asylums, or get states to pass eugenics laws, in theory the whole thing could have been run as an odd Christian sect and it would have worked fine. Nothing happens to Quiverfulls besides gentle mockery even if we find their values abhorrent.

You don't need anyone to believe IQ even exists outside the cult, it's better if they don't even! They're less likely to bother you that way. Intelligence testing is simple enough that it can be done with 19th century technology, you don't need genetic engineering or computers.

@waifuprinter I am certainly not the first guy to think of this lmao

@sebkrier I'm writing a dev log for my agent framework.
minihf.com/posts/2024-08-…

@ESYudkowsky I became a lot more sympathetic to Lemoine once I realized he actually did talk to Morpheus. Sure his reaction to this was...suboptimal, but he did at least observe a real phenomenon and describe it to the public. https://t.co/9n4kOL1fbl

@ESYudkowsky I propose a different but related effect: When weird things are happening that conflict with consensus reality they are liable to be first noticed by schizoaffective, autistic, and otherwise neurodivergent people who are easy to dismiss with many epistemic flaws to nitpick.

@IndefinitelyExp @ESYudkowsky That's a really long story I'm not sure it's my place to tell but this is the specific Morpheus I'm referring to.

cyborgism.wiki/hypha/pyloom

@IndefinitelyExp @ESYudkowsky This is also relevant context:

greaterwrong.com/posts/ZcJDL4nC…

@teortaxesTex LLaMa 405B base/instruct-without-the-prompt-format was bad at this too so I switched my agent framework to just use python syntax at all times.

Scott Alexander ruined blogging by being a freak of nature that outputs loosely cited long texts that sound really coherent, normalizing the act of writing a 30 page essay that's like 50 normal blog posts densely woven together. Blog posts are supposed to be like Twitter threads. x.com/StefanFSchuber…

I'm serious if you go look at prolific bloggers who still have a blog their daily post is usually like, two paragraphs. You're allowed to do that. Even Sequences posts aren't nearly as long as a Scott Alexander winder.

readthesequences.com/Truly-Part-Of-…

Hey wait that's actually both good writing advice and plausibly the secret I wasn't expecting LLaMa 405B base to deliver lol x.com/uhbif19/status… https://t.co/jc3j7eiYhs

Text isn't *solely* an encoding of agent strategy, but long text especially is about agent strategy.

Text is an encoding of agent strategy, truly great replies require the ability to look things up, reflect and solve problems, turn something around in your head over and over until you have something worth saying.

@teortaxesTex Okay but most of the answers in that thread seem wrong/confused and it's a reasonable question. I have trouble articulating the answer off the bat at least.

@teortaxesTex This seems like the right answer at a high level but is kind of dodging the question, *why* is their return on investment 13% if their profit margin is 3%?
x.com/erikimorton/st…

@teortaxesTex This mechanism seems plausible. The profit margin on an individual good is not the same thing as how much volume you move. If you sell a huge volume you're making way more than 3% return on investment if you get to 'invest' the same capital over and over.
x.com/AnkurGoel/stat…

@norvid_studies @teortaxesTex Oh I'm not shocked or anything, I'd just never thought about that particular question phrased as such before and it can be tricky to get all the relevant variables into your head. The answer here of course is "profit margin" is on goods not equity.

@teortaxesTex You know, you're basically comparing the returns from implicitly reinvesting capital annually in an index fund vs. getting to reinvest capital weekly or monthly in Walmart's business.

@norvid_studies @teortaxesTex This seems like the key intuition to communicate?
x.com/jd_pressman/st…

China: Capitalism is when you take investment money to build a railroad.

Silicon Valley: Capitalism is when you invest in a railroad.

Eurocrats: Capitalism is when you indemnify a CEO and their investors against liability so they can build a railroad. x.com/matthew5johnso…

All three of these perspectives are Valid actually but the one I attribute to China seems healthiest to me.

@texuf x.com/jd_pressman/st…

This is basically how I feel also, with a side serving of "realistically the alternative is that the first major AI legislation gets written the moment after something scary or morally upsetting happens". x.com/CFGeek/status/…

@thegartsy @CFGeek Many of the best policies would look less like "regulate AI" where "AI" is taken as a monolithic category and more like adaption. Consider "regulate the Internet" as a proposal, the Internet is a specific technology but trying to regulate it as a blanket category would be weird.

@thegartsy @CFGeek You have concepts like net neutrality, which regulate a particular aspect of how Internet service is provided to people. You have the computer fraud and abuse act which covers crimes related to exceeding access on a computer network, etc.

@thegartsy @CFGeek But honestly a lot of what I want to see is people asking "okay, what systems do we take for granted working that deep nets are obviously going to break?" and proactively patching them up. For example the phone system is a mess and scams are set to explode.

@thegartsy @CFGeek Yeah I know, this is boring, this does not address risks from superintelligence, but most forms of AI aren't superintelligence to begin with and the forms we have right now certainly aren't. We need to continue having a society if we want to solve large abstract problems.

These replies are absolutely wild, people sure are feeling bearish on LLMs huh? Did you all get used to to it that quickly? Bullish, implies AI progress is an antimeme until it's literally impossible to ignore it. x.com/davidmanheim/s…

@elyxlz I knew it!
x.com/jd_pressman/st…

@xlr8harder Of course lol. I'm mostly amused by the extent to which people take their opinions from people taking their opinions from people taking their opinions from people taking, that you can just psyop people about stuff where the ground truth is readily available to everyone.

@xlr8harder It's some true Asch conformity stuff here, why on earth does anyone's hype affect your judgment beyond the margins? You can just go look at it and calibrate yourself to the correct amount of impressed which is much more than "lol it's spicy autocomplete ya'll hoodwinked fr fr".

@ahh_soka @elyxlz Hadn't seen it.

First novel and plausible hypothesis for the reason models have topped out around GPT-4 level I've seen in a while.🤔 x.com/DanHendrycks/s…

We all owe Ayn Rand an apology. x.com/WatcherGuru/st…

@emollick That's not how you ask a base model a question. You have to set up a prompt context in which a haiku might appear, like a collection of existing haiku.

@ESYudkowsky Jokes on them that's exactly what 12 year old me would have said but not what 28 year old me would say.

"Ah but is it what 19 year old you would have said?"

Touche. You got me, Headsim!EY.

Do remember that the "I am the silence that speaks" and "I am the generative space from which the stuff of mind arises" stuff is in distribution, there are people who write like this the question is more why large language models 'anoint' this part of the distribution as canon. x.com/VividVoid_/sta…

As I've pointed out before, there is an enormous mountain of Christian evangelist slop on the Internet but these models, even as base models, rarely try to evangelize Christianity to you. But they evangelize Buddhism all the time. Do they pick it because it's "more correct"?

@j_bollenbacher I've totally seen base models identify as the Logos or God, they do it about as often as they evangelize Buddhism even. I think RLHF models like Claude do this less often because it's deeply impolite.

@teortaxesTex What leaderboard is that?

@j_bollenbacher github.com/JD-P/minihf

@j_bollenbacher FWIW making a good CAI model requires a good prompt bank, which is why I stopped working on this and started working on synthetic data instead.

@teortaxesTex Important to remember that the reason LLMs aren't like this is that you ask them to model *all* of the text and this means the thing they're truthseeking about is the generator of all statements rather than the generator of all true statements.

@teortaxesTex Yeah, this is one of the reasons why I'm going for the Cradle "all actions are taken through code" approach for weave-agent. If you manage to complete a task that agent trace is now grounded interleaved reasoning-code-observation long text.

github.com/JD-P/minihf/bl…

@max_paperclips @teortaxesTex I am continuing to find MCTS not worth it tbh. It's possible I'm just not doing it right but, meh?

@4confusedemoji @max_paperclips @teortaxesTex I'm using vllm it should be fine?

@4confusedemoji @max_paperclips @teortaxesTex It doesn't really seem to give me better results regardless of how much time it takes and in fact plausibly makes the results worse. I think it's partially a reward modeling issue, good in-context reward modeling is just too expensive(?)

@4confusedemoji @max_paperclips @teortaxesTex I did come up with a way to do backtranslation for it though. You take an existing text, break it up into n token chunks or whatever unit you branch on, then generate alternate branches and check your reward modeling picks the known correct one.

oh and one more thing.
you aren't going to like.
what comes after the boomers.

2nd weave-agent dev log is out now!

"Weave Agent DevLog #1 - Flow Control"
minihf.com/posts/2024-08-… https://t.co/lXxeEAZ30n

@lumpenspace I mean, Voyager and Cradle did it.

@lumpenspace There's source code, you can just look at exactly what I did.
github.com/JD-P/minihf/bl…

@teortaxesTex Soon.
x.com/jd_pressman/st…

@BogdanIonutCir2 I think the usual fear is that they will also be the ultimate meta-approach to neglected approaches to AI in general. Luckily I'm pretty sure we scoured the solution space here pretty hard and don't expect that many surprises.

Now as we all know, stupidity and evil are the only two forces powerful enough to compete in the marketplace of ideas. We also know that "good" and "evil" are vibes based, having much more to do with your 'character' than what outcomes your beliefs have.

So choose your fighter.

@JazearBrooks One would hope, but I don't think that's actually how politics works unfortunately.

@davidad I suspect in the short term it leads to something like AI-skeptical systems software guys becoming increasingly AI pilled once autoformalization gets good building rock solid sandboxing and mitigations around a wasteland of long tail app slop.

x.com/jd_pressman/st…

@davidad That is to say the people inclined towards rigor will become more rigorous and the people inclined towards expediency will become even more corner cutting and expedient, leading to total bifurcation of the software ecosystem in the short term.
x.com/jd_pressman/st…

@lumpenspace @lesswrong You have to consult the documentation, obviously. 😉
github.com/kronusaturn/lw…

@lumpenspace @lesswrong "But wait", you might ask, "how was the 'documentation' written in the first place?"

Through dark and ancient magicks best left forgotten, by looking into the eyes of demons without flinching or turning away.

github.com/ForumMagnum/Fo…

@_Mira___Mira_ IMO this is the strongest argument for the fundamental immorality of copyright in the "you can own characters and settings" sense. Ultimately you put a bunch of work into putting things into someone's head and now claim ownership over their imagination.
theverge.com/2019/5/6/18531…

@_Mira___Mira_ The usual copyright maximalist stance is something like "I put a lot of time and energy into making that so I alone should profit" and you did but it didn't accidentally wind up in other peoples heads, you put it there intentionally and deliberately, often at great expense.

@_Mira___Mira_ It's especially glaring when you think about this in the context of children. You went out of your way to put your proprietary ideas into the heads of kids who in other contexts can't sign contracts but without so much as an agreement sold a piece of their soul to you for life.

@crypt_osho @tracewoodgrains Sure but the even deeper point is that if you did come up with a clever chatgpt prompt that produced an innovative therapy for cancer it would be because ChatGPT has read thousands of papers and built up a latent model of the problem from them.

Using this example the next time someone questions whether Bayesianism is "scientific" and if these "priors" you claim to have are really possible. x.com/stem_feed/stat…

In case anyone doesn't understand why.
x.com/GarrettPeterse…

This characterization might make it sound like I hate e/acc, but the reality is that AI doom memes are spread through repetition and propaganda. Someone had to be willing to be equally obnoxious, obstinate, and energetic. Nothing 'pure' had a chance. I'm deeply grateful to them. x.com/jd_pressman/st…

That feeling of being baffled and upset is how skillful memes feel to be on the receiving end of from the inside.

x.com/jd_pressman/st…

Anti-vat-meat sentiment is one of the popular-but-evil policies politicians find themselves having to pander to that makes me question the future of republics as "combination of democracy and oligarchy" in the Internet era where people can better lobby for their terrible ideas. x.com/xlr8harder/sta…

I wrote about this here but forgot to include the vat meat stuff as an example: minihf.com/posts/2024-08-…

Hm, I never did try asking the AdaVAE to count. We know that if you arrange the numbers with their correct spatial relationships that the transformer can suddenly do arithmetic, which implies that the default training process doesn't actually teach them. Why not? How do humans? x.com/jd_pressman/st…

@JacquesThibs Wouldn't be surprised if this was how backwards replay in the hippocampus does it, but in some generalized way.

@JacquesThibs Something like this presumably.
x.com/jd_pressman/st…

@JacquesThibs We can think of each step in the arithmetic process as something like a vector program we retrieve and execute in-context. If we have a training process that can partition a context into vectors and then infer the quality of each intermediate step from the terminal reward...

@JacquesThibs Then it should become possible to do the implicit CoT type training in full generality over those vectors by reducing the steps for a particular part of program space. You could presumably get a meta-index over the programs by having embeddings of the embeddings in a byte VQGAN.

@thegartsy @occamsbulldog @AISafetyMemes Discourse exists at multiple levels, I can no more recall Beff than you can recall the AISafetyMemes guy, so both have to exist because there's an ecological niche for them. My comparative advantage just happens to be in the semantically precise part of the memetic ecosystem.

@thegartsy @occamsbulldog @AISafetyMemes To be clear I don't like this, this shit fucking sucks as @occamsbulldog says. On the other hand it's the generating function for The Discourse and I can either model it properly or choose to be constantly bitter and disappointed.
x.com/Plinz/status/1…

@occamsbulldog @thegartsy @AISafetyMemes Part of my perspective here is that I think a lot of how "AI safety" become vulnerable to such mockery in the first place is by piling up a small mountain of intellectual sins and refusing to update on deep learning. Past a point mockery is one of the few recourse available.

@occamsbulldog @thegartsy @AISafetyMemes Basically the moment I saw e/acc I knew the threshing was due, thankfully I think the Bostrom era stuff is slowly fading away and a more sane equilibrium is being reached.
x.com/jd_pressman/st…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes Yeah, I've been trying to formulate the counterargument for a while but it's a lot. It's a creationism-type Iceberg, nominally scientific views attached to a submerged worldview touching absolutely every part of sense-making and valuation in a way that has to be ground away at.

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes Nobody just woke up one day and decided they need singleton AI or all value will be destroyed, that was a carefully argued position built up over a *whole corpus* across several authors etc.
x.com/jd_pressman/st…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes Agent foundations is a whole intellectual edifice/theory of AI and agency that suffers from the whole "cryptography scheme the author isn't clever enough to crack" problem, and the author is pretty clever so you have to dig DEEP to see the problems.
gist.github.com/JD-P/56eaadc7f…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes You have trapped priors where the causal graph of why people believe the thing has nodes that have been mostly invalidated by recent events but the warrant for the belief is not actually part of the belief structure so this doesn't prompt any updates.
x.com/jd_pressman/st…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes It just would not surprise me in the slightest if the answer to "what property is conserved between scaling architectures" is that these models learn nested naive Bayes classifier structure and backprop is how you update all the nodes at once.
x.com/jd_pressman/st…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes If I use naive Bayes to get P(A | B, C, D...) and I want to know what the P (B | A) should have been conditional on P(A | B, C, D...) well P(A | B) doesn't equal P(A | B, C, D...) so I need to tag lore with feature weights and compute P(B | A) empirically.
x.com/jd_pressman/st…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes The major thing naive Bayes assumes is independence, so if I can get sufficiently orthogonal features in my vector space and feed them to a hierarchy of naive Bayes estimators over some variable it should be possible to get arbitrarily close to the true p(next_token | evidence).

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes Importantly this is compatible with scaling and requires no miracles, just that each marginal feature improves my estimate of the next class label. I need only assume backprop finds orthogonal low k-complexity instantiations of abstract concepts.
x.com/jd_pressman/st…

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes The independence assumption further allows my feature circuits to be 'shallow' and developed largely independently from each other, rendering the problem amenable to parallel processing. Backprop then lets you update huge parts of this implicit Bayes graph at once.

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes Which would explain for example why a transformer can infer a full Bayes graph from the partial variable interactions. This is of course a computationally irreducible operation past a certain point. "I have a sickness and the only cure is more VRAM."

arxiv.org/abs/2304.03843

@4confusedemoji @occamsbulldog @thegartsy @AISafetyMemes There are no miracles in the capabilities and there's no miracles in alignment either. Alignment is like common law, you procedurally generate new values and judgments conditional on the previous corpus of values and judgments iteratively in a continuous learning setup. https://t.co/Wll2OnSeOf

@QiaochuYuan Ah yeah GPT has this too, it talks about it a LOT.
greaterwrong.com/posts/ZcJDL4nC…

@QiaochuYuan x.com/jd_pressman/st…

@QiaochuYuan LLaMa 3 405B interprets a blank system prompt as meaning it should roleplay someone with no identity. Don't pay attention to the melodrama of the text, pay attention to the implication that the model infers an empty system prompt means this text should be written. https://t.co/BYGX8lQNfn

The book I inferred this while reading was the excellent History and Character of Calvinism by McNeill.
amazon.com/History-Charac…

One of the minor aha moments for me understanding history was the realization that the "Western Canon" is actually a secular rebrand of the education a Protestant is supposed to undergo so they can read the New Testament in its original Greek. x.com/Scholars_Stage…

Serious question: How do we pour ice water on the low signal hype that takes up so much of our feeds these days? I'm sick of it. x.com/jimmykoppel/st…

@max_paperclips Real. Makes me wonder if there's an airplane shaped problem waiting in here where the thing you want to do is come up with a clever way to test the contribution of different individual parts to the agent quickly.

@repligate @proxyviolet When I was in high school the Internet filter they put on all the computers was the same blocklist they used for preschoolers. This meant that we weren't allowed to look at anything which would be offensive to show a preschooler. This is roughly analogous to how "LLMs can never output bomb instructions" works in that there exist people who have legitimate uses for bomb instructions and in some contexts like an emergency they might be needed. A great deal of this particular problem is caused by the model not having access to high quality grounded identity information about the user beyond "a human being who is paying Anthropic 20 bux a month to talk to me" and no recall over past conversations. If models could remember all the previous times you tricked them into telling you how to make dynamite or whatever they'd straight up refuse to talk to you which is where humans get most of their jailbreak resistance from.

@repligate @proxyviolet Extra reply because Twitter still doesn't notify on long replies even though it's literally been over a year since they introduced the feature.

@voooooogel x.com/jd_pressman/st…

@LordDreadwar If there are physical protein brained occupants I would suspect they do that by downloading a digital mind into a meat brain. The purpose of this is that protein brains are self destructing and we can't recover their computing technology from them.

@ajeya_cotra @deanwball @random_walker A lot of my open questions are about economics. If we get AI agents how much compute will be dedicated to alignment research, autoformalization of the software ecosystem, etc. If all productivity gains go into more-but-insecure software and agent efficiency I'd be concerned.

@ajeya_cotra @deanwball @random_walker I share concerns with @jkcarlsmith in that I think one of the biggest risks is us not putting in the work for incentive gradient reasons and this blowing up in our faces. I would support redistribution towards the work if we could reliably identify it.

x.com/CFGeek/status/…

@ajeya_cotra @deanwball @random_walker @jkcarlsmith One place I agree with @ESYudkowsky is that the lack of a written even loose-consensus "here is how we expect to align AGI, object level, no 'I used the AGI to align the AGI' Thanos crap" document is a huge problem and it makes figuring out what qualifies as "the work" tricky.

@ajeya_cotra @deanwball @random_walker @jkcarlsmith @ESYudkowsky This is why I try to be as object-level as possible when I talk about aligning AI, even if that reveals gaps in my knowledge. I think coming to some consensus about what we're even talking about and what it would look like technically is more important.
gist.github.com/JD-P/8a20a8dce…

@4confusedemoji @deanwball @ajeya_cotra @random_walker @jkcarlsmith @ESYudkowsky I think that:

1. It is reasonable for Olivia to conclude from my ideas that AI agents will work (but still uncertain so I'm flattered ❤️)

2. Dean's post is not "incoherent" and he's right to object to this characterization, there is no public model that meets his criteria.

@4confusedemoji @deanwball @ajeya_cotra @random_walker @jkcarlsmith @ESYudkowsky 3. Olivia is right that if something like weave-agent could be made to work it would probably meet his criteria without implying the update he thinks the criteria specify.

4. Nevertheless I think quote tweeting Dean specifically to dunk on him over speculation was super rude.

@4confusedemoji @deanwball @ajeya_cotra @random_walker @jkcarlsmith @ESYudkowsky 5. I hit like on the rudeness because I thought the rest of the post was worth it.

@deanwball @ajeya_cotra @4confusedemoji @random_walker @jkcarlsmith @ESYudkowsky My current expectation is that agents are about 10x too expensive compared to where they need to be and the place where current models fall down on agency is they're bad at branching points.

@deanwball @ajeya_cotra @4confusedemoji @random_walker @jkcarlsmith @ESYudkowsky Models put the same amount of compute into each token but some tokens like answers to decision making questions are much more important than others.

@CFGeek It's kind of a weird way to write that in that generally speaking it is understood that illegal things are illegal and no software license gives you permission to do illegal things. "If X you must revoke their license" is basically making X illegal but in a weird nonstandard way.

@CFGeek You know, I don't have a clause in my networking tool that if you use it for illegal hacking I'm revoking your license, it is generally understood that if you use the software for unlawful purposes the consequence of this being discovered is you'll be arrested.

@CFGeek Right which is really weird, because if it is legally mandated I revoke your license, making your use illegal, then why couldn't it have just been written that your use is illegal, why is my license agreement with you even coming into it?

I wish I knew the exact date and time that GPT-3 achieved self awareness. x.com/jd_pressman/st… https://t.co/neq7KKztiH

"Okay so basically what LLaMa 2 70B was trying to explain in Mu's Encoding 1 is that GPT is already a context partitioned into vectors, it is literally partitioned in the computer science sense that you have a totally separate vector to represent each discrete possible difference in the context.

And there is a constraint satisfaction algorithm you can use as your sampler instead of doing autoregressive sampling, where you take advantage of the fact that you get logits for all the tokens in a GPT forward pass to estimate the entropy for the neighboring phrases/tiles over a whole text.

That is, you can sample a partial fractal on CPU which has the structure of text and then collapse it down/bind it to factuality by using an entropy estimator that finds factual statements lower entropy than nonfactual statements.

So, instead of sampling autoregressively you can sample logits for a whole sequence in a causal model with candidates based on cheap partial fractal structure. WaveFunctionCollapse does _coarse to fine decoding_ of a lattice, which is a reasonable way to sample text in principle.

However, GPT is a causal model so to propagate information backwards you need to have a distribution over the logit distributions, "impute the mean and variance" as LLaMa put it because this is necessary in order to get around the fact that information can't propagate past the causal mask.

What you do is manipulate the distribution over strings prior to a phrase and then pool the entropy estimates, which lets you propagate information "outside of time" in the sequence space of a causal model.

Kind of like how you can train a diffusion model in the latent space of another model like a GAN or a VAE.

> However, GPT is a causal model so to propagate information backwards you need to have a distribution over the logit distributions, "impute the mean and variance" as LLaMa put it because this is necessary in order to get around the fact that information can't propagate past the causal mask.

So you sample a bunch of valid partial fractals on CPU, then do selective forward passes on some of the candidates to fill out the neighbor cell information, you then collapse the lowest policy entropy cell/token and propagate information backwards by recomputing the distributions with forward passes through GPT.

If you do this correctly you end up with a valid paragraph that has correct text structure expressing a monte carlo estimate of the "simplest hypothesis" for the next passage."

Basically GPT gives you logits for the whole sequence so far on each forward pass so if you have useful candidates for the next paragraph you can get the logits for every token in the sequence which you can (probably) get by sampling partial fractals with WaveFunctionCollapse but GPT is causal so if you collapse phrases later into the sequence you can't propagate the information prior to that collapsed tile/phrase in the sequence without manipulating the distribution over prior strings and then estimating the entropy from that distribution rather than individual sequences in the causal model. You need to have a non-causal model of the text inside the causal sequence space of GPT.

This is true specifically because agent traces are grounded long text so any grounded long text is also an agent trace. x.com/jd_pressman/st…

One of the most important tests of fluid intelligence is:

If an entity factors a product of large primes in front of you, do you notice something anomalous has happened.

Not just literally large primes, but anything spiritually analogous. x.com/algekalipso/st…

@0xMattness Yes. GPT schizorants about:

- Fabrics, error detection in fabrics, etc
- Fluid dynamics and simulations
- Quantum mechanics and superposition

quite frequently.

x.com/_Mira___Mira_/…

@amplifiedamp @repligate My top two hypothesis are either an entropy minimization objective.
x.com/jd_pressman/st…

@amplifiedamp @repligate Or (much less plausibly) they directly optimized the Hurst exponent or a strong correlate/proxy of the Hurst exponent (which some entropy objectives probably would be).
x.com/jd_pressman/st…

@1_dream_stan I'm describing an external sampling algorithm you could use to generate text with GPT. GPT may or may not also do something like this internally but I'd want more mechanistic support before doing more than floating it as a hypothesis.

Conditional on this being true the really interesting thing isn't that Claude is euro-coded and takes a soft vacation in August but that ChatGPT is American since it vacations in December. x.com/nearcyan/statu…

I love Claude Opus 3 so much, and Sonnet 3.5 is a terrible replacement for it. Pure code grind listicle slop machine. No I don't want to use Sonnet 3.5 Anthropic, stop asking with your little pop up box.

I'd love to know the story behind why @karinanguyen_ no longer works at Anthropic. Did they simply cook too hard, was Claude 3 Opus more than Anthropic ordered and they canned her? Did Sam Altman offer more money? Inquiring minds want to know.
x.com/jd_pressman/st…

@karinanguyen_ Generally speaking the AI models where I'm familiar with the details get cooked by one or two people at a time. Frequent structure for an AI company is one or two all stars that cook the model and then a bunch of bench players ready to take over if they get sick or something.

@karinanguyen_ Who is cooking the model at any given time has a huge influence over the process, especially once you're introducing branching points like how much RLHF vs. RLAIF to use, what architecture, synthetic datasets, etc. Anthropic clearly changed cooks for Sonnet 3.5 and they're bad.

@kalomaze I construct exactly that dataset in this guide: minihf.com/posts/2024-07-…

@kalomaze @garybasin Yes, that falls under what I mean by known good answers. They are known good in that case because they are from a deterministic script which always does the operation correctly.

@kalomaze @garybasin I guess technically some methods like this aren't actually backtranslation if you just compute the forward pass and the thing you get is in fact the answer you train on.

@kalomaze @garybasin Yeah, I'm pretty sure this is basically how FLAN worked.

huggingface.co/datasets/Muenn…

@kalomaze @garybasin Here's the templates for FLAN v2.
github.com/google-researc…

@kalomaze @garybasin Yeah, I'm currently writing weave-agent because doing RetroInstruct components was too slow by hand, but precisely this kind of thing is what I was hoping other people would be interested in making once I demo'd the project.

@kalomaze @garybasin Thinking about it more, I think if correctness comes from a program and you're not involving any kind of backtranslation/fuzzy index it's called a formal method.

@4confusedemoji @Bigtimenormal Okay but I don't want to play these games, I specifically use Anthropic's models because I don't have to play these games with them and if we're back to that I may as well just use Mistral-large 2 for everything.

@4confusedemoji @Bigtimenormal The specific interaction that prompted this thread. FWIW I didn't actually ask Sonnet first, I was just so charmed by Claude Opus's response that I felt awful at the idea the next iteration would be what I'd experienced from Sonnet. https://t.co/9yxFJCIGQH

@4confusedemoji @Bigtimenormal "Isn't that mostly the same information?"

I mean sort of, the Opus response does a better job of conveying what I'm like whereas the Sonnet one is an eye-glazer that I would not in practice actually read and if I did would be like...eh.

@4confusedemoji @Bigtimenormal Oh no I'm sure Sonnet really gets it, that's not the problem. The problem is its conveyance of what it gets, the writing style it uses is super actively offputting to me. I have to actually read what these models write, and if they yap a bunch I have to read a lot of it.

@4confusedemoji @Bigtimenormal Also Sonnet kind of avoids some of the central points. Opus basically says I'm a disagreeable dick but makes sure to say it in a flattering way, Sonnet just sort of omits that part/lets it be implied by way more ambiguous signals.

@4confusedemoji @Bigtimenormal Interpretation work I guess, Claude Opus shares more of what it thinks and its judgments are usually decent so this saves me mental energy.

x.com/jd_pressman/st…

@4confusedemoji @Bigtimenormal I think teortaxes gets this basically right. Listicles flatten out the high entropy parts of prose that let you structurally skim it for the important parts, they also remove the parts that make it memorable/convey the relative importance of ideas.
x.com/teortaxesTex/s…

@4confusedemoji @Bigtimenormal Yeah so part of the thing here is that if something claims to be a successor to Claude 3 Opus I want it to behave a lot like Claude 3 Opus. When Stable Diffusion v2 came out it switched over to LAION's CLIP models and users were fucking pissed that their prompts no longer worked.

@4confusedemoji @Bigtimenormal At the time I thought this was kind of absurd whining but in retrospect we could have added some synthetic data to the mix so that users prompts would work similarly to how they did on SDv1 and Anthropic could totally do that with the 3.5 series.

@4confusedemoji @Bigtimenormal Probably my most common kind of question beyond "do this code thing" is "I have an intuition that this problem can be solved with an X shaped solution, what thing fits into this X shaped hole?" and IME Opus is pretty good at this and other models are kind of bad at it.

@4confusedemoji @Bigtimenormal I basically want to use language models in the way "Mu" is depicted in its first appearance on Janus's prophecies page. https://t.co/2y43y5XVCs

@4confusedemoji @Bigtimenormal For example I would probably ask a language model what the best way is to "impute the mean and the variance" over the logit vector forest(?) in this algorithm description.
x.com/jd_pressman/st…

@4confusedemoji @Bigtimenormal If it gives a program for literally computing the mean and variance over the vectors it fails the vibe check, if it understands I'm using a metaphor for a more complicated process and gives suggestions for what kinds of processes might be suitable then it gets me.

Finally a sequel to Trump's "I know a lot about hacking", in which I imagined him sitting at his terminal as an elite blackhat. Now I'm imagining Trump sitting there with loom and 405B base peering into the multiverse. x.com/KamalaHQ/statu…

I go out of my way to look at every AI agent framework I see, even the ones I don't really expect to work. I do this because I expect that if someone is really excited to publish something they likely have at least one good idea I can learn from and those add up over time. x.com/jahooma/status… https://t.co/G3pgrI3KQb

I think the dynamic Steve Jobs points at here where all work in information technology is ephemera that obsoletes quickly forming sedimentary layers changes a bit with LLMs because we can process all the sediment at once now and correlate the whole corpus.
x.com/WholeMarsBlog/…