John David Pressman's Tweets - August 2025

Fortune teller who wants to get a good look at your hands because she's checking your digit ratio.

@CharlesCMann @HSB_Lab LLMs are trained on the huge corpus of data, but what this actually means in practice is that each datapoint nudges its ability to "predict English text in general", it doesn't usually store that exact thing in its weights. So most details are forgotten during training.

@CharlesCMann @HSB_Lab This can be mitigated by giving the model a web search capability so it can look up specific details from the web and incorporate those into its answer, but the answers are generated from a general prior over English that only remembers some details.

@CharlesCMann @HSB_Lab It's kind of like how if I asked you exactly what you were doing at this time three weeks ago you probably couldn't tell me, but if I asked you about your last three weeks you could give me a sketch or overview of what you were doing. Your brain throws most daily details out.

@CharlesCMann @GuiveAssadi @HSB_Lab Right so the thing that the training teaches is the ability to read those sources and parse them for meaning, and write a story about the details if it can retrieve them. But the details themselves are just too specific and niche to go into the general English text prior.

@CharlesCMann @GuiveAssadi @HSB_Lab Does that make sense at all?

@CharlesCMann @GuiveAssadi @HSB_Lab Yeah that part is basically the post-training recipe. How this works is you first teach it to predict text in general which produces a "base model", and using that model is kinda weird because it answers a question by trying to e.g. predict more of the question.

@CharlesCMann @GuiveAssadi @HSB_Lab So you take that model and train it to answer questions with a special finetuning data set that is just users asking questions and getting answers so that it gives you answers instead of more question or going into a random forum thread or something.

@CharlesCMann @GuiveAssadi @HSB_Lab And this special dataset is usually made by having contractors pretend to be the robot. So during OpenAssistant which was a community run system like this I saw a lot of people answer hard questions with "As an AI I can't do that" or "I don't know" but in character as the AI.

@CharlesCMann @GuiveAssadi @HSB_Lab That's basically how it ends up like this. It learns from the human contractors to be lazy/confused about certain questions or to confidently state that nobody knows the answer because that's the style of how contractors would whiff on tasks that are too hard.

@CharlesCMann @GuiveAssadi @HSB_Lab My explanation of a base model "predicting more of the question" probably wasn't very good so here's Kimi K2 with the instruction format ignored attempting to answer your question. As you can see this would be a pretty disorienting model for most users to use. https://t.co/e0DvOZG5Nf

@JimDMiller Why? GPT-5 probably won't do it for you.

PSA: If you really liked the style that CLIP Guided Diffusion and other old image models output in and are sad you can't get it on newer models, it's actually a look created by the use of gouache watercolor and airbrushing and you can prompt for it on newer models. https://t.co/cVfzcM9jY1

If you remove the default slop from the context window and add the term "airbrush": https://t.co/bgYLpXSR0W

This radicalized me when I read the earlier draft(?) a few years ago, but honestly not enough. I think maybe I need to print it out and put multiple copies of this part on the walls to help motivate me. x.com/lefthanddraft/…

@dogecahedron In practice? No.

@dogecahedron The cohort of "PauseAI" people who were present at the time to do so absolutely wanted to destroy GPT-4 and tried to ban any open GPT-4 type model as a stepping stone to that. In terms of policy that might actually happen these are the same movement.
1a3orn.com/sub/machine-le…

@dogecahedron "I want AGI banned FOREVER" is basically flavor text on top of "I want an AGI moratorium", in that we have a ban on human cloning that is indefinite and this will last exactly as long as people want it to and no longer, possibly just until someone breaks the taboo.

@JimDMiller Since you like prediction markets, how about you do this for me buddy:

Mark down that I told you GPT-5 probably won't obviate your need to write your thing, whatever it is, and then check if I was right after GPT-5 comes out in however many days. Or better yet a few weeks after.

@Xenoimpulse You don't get it, they just have to purity police harder and it'll work this time, pinkie promise.

Mu. x.com/tszzl/status/1… https://t.co/9G4TQyMTyH

Then again I forgot about the pro legacy switcher.
x.com/tszzl/status/1…