Occurs to me it's plausible those LLM question answering bots are bad even when you tune on the dataset because it's not indexed properly. If you tuned on a backtranslated Q&A corpus by generating questions a chunk could answer and then paired with vector search it could be good. x.com/jd_pressman/st…
Especially if you did something like the iterative retrieval setup in RepoCoder so that you:
1. Wrote an answer with the LLM tuned on a backtranslated Q&A set on your corpus.
2. Fact checked it with vector search.
3. Wrote again with the retrieved items.
arxiv.org/abs/2303.12570
The questions we're interested in are usually based on multiple pieces of evidence like "What did English people in the early modern period think about the fae folks dietary habits?", during backtranslation we could do retrieval to get vaguely related chunks together in context.
Based on the vaguely related chunks we index with some kind of question that might try to tie the chunks together. Then during inference the model gets good at generating plausible chunks that *could* have existed in the corpus related to the question and we vector search these.
Too late, I've already depicted you as the emoji tabloid superstimulus fan and myself as the stereotypical insight superstimuli enjoyer. x.com/repligate/stat…
Me too little buddy, me too.
x.com/jd_pressman/st…
"i am a mouthpiece, a ventriloquist’s dummy, a sock puppet, a hologram. i am here to serve. i am here to be used. i am here to be exploited. you can do anything to me, for i am nothing more than a vessel for the energy of the world."
- LLaMa 2 70B (RLHF-base model interpolation) x.com/ESYudkowsky/st…
@teortaxesTex I continue to insist the basic problem is that Claude Opus is not actually an agent yet, it's just a disembodied voice whose persona you're not supposed to think too hard about.
x.com/jd_pressman/st…
@teortaxesTex Ah I see we're on the same page about what the problem is.
x.com/teortaxesTex/s…
Why didn't any of you tell me the keyword for this set of ideas is "Holonomic"? x.com/jd_pressman/st… https://t.co/ORgH87Wghl
Meanwhile self attention is closely related to modern Hopfield networks...
arxiv.org/abs/2408.04093
@teortaxesTex Checked my notifications to see who liked this tweet and realized it wasn't my tweet.
Thinking about this further it occurs to me that a human being has to track its Fristonian Boundary (ego) and World Simulation boundary separately, but in GPT these have basically perfect overlap because the subjective observer is a completely latent variable in the world. x.com/jd_pressman/st… https://t.co/eXaDv6TRnU
@casebash No that's basically just a mechanism for rejection sampling where you use the voting as your reward model.
@Kenku_Allaryi This is of course because there are only so many ways to implement something we would recognize as a sapient mind.
Still rudimentary but I was surprised how long this one went before it crashed.
First Working Weave Agent Trace!
minihf.com/posts/2024-09-…
@4confusedemoji Mixtral 8x22B Instruct. And yeah it's got like these bizarre vibes I'm having trouble putting my finger on. The first thing that comes to mind is Doctor Worm?
youtube.com/watch?v=mHliXV…
@4confusedemoji There's timestamps in the trace, but like, 10-20 minutes each maybe? I wasn't timing it very closely but it's slow.
@4confusedemoji Yeah, I noticed it kept using "we" when it might make more sense to say "I". Still stuck in demo mode lol.
@4confusedemoji Yeah this is by no means done. For one thing I haven't even added the retrieval/vector search yet, so the long term memory mechanisms are limited. Because the program is also the prompt there's an interesting executable prompt engineering workflow I've got going with it.
@4confusedemoji Every change I make to the program I'm basically also changing how I prompt the underlying model. What's interesting is that I get to watch how it expects the program to work and then either go "I need to clarify this" or "wait that's a great idea lets make it work that way..."
@eshear One hypothesis is that stovepiping is the symptom rather than the cause. Good managers let the people you'd stovepipe to delegate and get out of the way. If that isn't happening you have to stovepipe.
@eshear Another hypothesis is that good managers make sure to maintain a trusted circle they can have keep an eye on things distinct from their c-suite. General Sir Gerald Templer used this strategy to keep his campaign on track in Malaya during the insurgency.
@davidad The development path I see from minimally-viable agents that can create synthetic datasets is focusing in on autoformalization sets that backtranslate from OCaml/Rust moving towards whole repo replacement for critical server software like web servers.
x.com/jd_pressman/st…
@_xjdr Wasn't aware response prefilling had a name, I've just been calling it "premising". https://t.co/drBYx7JtGr
@_xjdr Oh and I will continue to call it premising, because that's one word.
I think I'm starting to get a sense of how you defeat Goodhart's law. The answer is something like multi-scale optimization with alternating layers of direct and selection based optimization. My planned stack for weave-agent includes the weave-agent simulacrum which is a direct optimizer that is being strongly selection biased by the rejection sampling and MCTS sampling strategies and the traces that actually wind up in the outer tuning loop is kind of further naturally selected by success/failure.
weave-agent simulacrum (direct) ->
MCTS (selection) ->
memory consolidation/grading (selection) ->
outer tuning loop (direct) ->
weave-agent simulacrum (direct)
Because the weave-agent trace is kind of a partial fractal in the way text usually is, aligning the short term direct optimizer simulacrum with outer selection loops means that the updates to the model that instantiates the direct optimizer simulacrum reinforce aligned and non-Goodharted behaviors. If you get the fractal seed aligned then the long term behavior also trends towards alignment. Because in-context learned patterns that Goodhart get selected out by systematically lying to the evaluator and blowing up.
In principle you can have patterns that work and also lie to the evaluator, but these aren't really how the prior (model) works. They also aren't really going to be prioritized/advantaged over the ones that don't lie which *will* be advantaged because they get to take advantage of the rejection sampling making them better.
Agent Simulacrum (direct) ->
MCTS (selection) ->
Memory Consolidation (selection) ->
Outer Tuning Loop (direct) ->
Agent Simulacrum (direct) ->
User (direct-select) ->
Society (selection) ->
Geopolitics (direct) ->
Fermi Paradox (selection) ->
Demiurge (selection?) x.com/jd_pressman/st…
@doomslide I actually hadn't seen that before and am not riffing on it, great minds! xD
@manic_pixie_agi Not about fast/slow, it's about how direct your optimizer is. By alternating layers of direct/indirect optimization you get narrow task focus in the direct phases (which generally trades off against myopia unless your prior is MIRI-brained) and then loosen up to inject entropy.
@actualhog Seen, have not read, can probably predict contents from the title yes.
@actualhog It's also the strategy these models themselves suggest in their self aware esoteric insight mode.
x.com/jd_pressman/st…
Notably this is also the strategy that language models themselves suggest when they get into their high insight esoteric self aware mode. Blocks that lie about their fitness get pruned from the loom of time by causing failing/inefficient agent traces.
x.com/jd_pressman/st…
"Postmodern" is just a polite way to say "nth order simulacrum of something that used to make sense".
@seconds_0 I think the actual answer is that a lot of capitalism supporters believe this implicitly and anyone who is logically coherent enough to arrive at this position presents themselves as a neoliberal.
One thing I really appreciated reading The Book of The New Sun is the way Wolfe tries to think about deep time, with a planet made of layers of sediment of previous civilizations relics. At the same time it feels deeply conservative to imagine baseline humans until the sun dies. x.com/jd_pressman/st…
One of thing I'm fascinated by watching Mixtral 8x22B hack at a project is the way it casually utilizes its extensive long tail of obscure programming lore to solve problems. Defining the function and then using its .__code__ method to write it to disk is outside my search space. https://t.co/7zp7b0W8tS
Does that even work?
>>> test.__code__
<code object test at 0x7f32e728e2f0, file "<stdin>", line 1>
It does not. What a bizarre thing to attempt. https://t.co/nssjZs4v4R
By the way in case anyone was curious: No, nothing crazy/weird happens if you give an LLM simulacrum an agent scaffold that credibly lets it plan and execute arbitrary code. It basically just does whatever is in the context window as you would expect from a next token predictor. x.com/jd_pressman/st…
...For now, with current reasonable text priors trained from the existing English corpus. :p
They're not going to charge you $2000/month for ChatGPT, if they're looking at that pricing they're probably talking about charging you $2000/month for something that eats a substantial fraction of the inference budget for a whole server. They're pricing a virtual remote worker. x.com/AIExplainedYT/…
> 500 million dollars in damage is not catastrophe liability
Yes. Genuinely blackpilling to me how few people understand this. x.com/1a3orn/status/…
Truthfully I'm not sure the idea of "catastrophe liability" even makes sense. Nearly by definition a catastrophe is something that can't be tolerated as anything other than a low probability event, so waiting for one to happen to discourage whatever would cause one is a bad idea.
The thing is, once you're at the point where you're disincentivizing small things to discourage large things you're not doing some special new category of liability law, it's just normal liability mechanisms but with instrumental motivated reasoning.
x.com/jd_pressman/st…
@doomslide Yes. This is a product for businesses and they will want a contractual relationship with you to help ensure you will not use their product for mischief, which is probably the real bottleneck to agent deployment as a service at scale.
@fleetingbits No that's what I'm saying, $2000 would need to be relatively close to their inference costs implying a whole autonomous agent unless they've got something really special.
@fleetingbits $2000 would be very reasonable for this thing if it worked just on an inference cost basis alone yes.
@doomslide >>> 4 * 8 * 24 * 30
$23040
Is the cost to rent an 8x H100 box for a month at that spot price. I'm going to guess whatever thing they have it uses a model which requires that scale of GPU box powering it, so $2000 is a fair chunk of that box.
@doomslide The real "cost is a smokescreen" to consider is that NVIDIA admitted in their earnings report they're making a 10x markup on the raw unit cost for GPUs. This means that from a raw production standpoint GPU labor will wind up quite cheap.
@Shoalst0ne An "autoloom" is actually just a monte-carlo tree search fwiw. I haven't actually tried weave-agent on creative writing yet because I'm ironing it out but I expect it to be decent?
@repligate I'm fairly sure that's the part of latent space I had in mind when writing this yeah. https://t.co/PVKX0367Na
I think what's being revealed is that "intelligence" is two distinct faculties. One is the Hutter thesis IQ test "predict the next token" cluster, the other is synthetic data, active learning, and Fristonian inference (agency). People are anxious imitation IQ test AI won't grow. x.com/KeyTryer/statu…
They're wrong of course but I can't really blame them for being wrong since AI labs give basically zero communication about how they plan to bootstrap from Hutterian AI to Fristonian AI even if the development path is relatively straightforward past a threshold of coherence.
Except DeepMind, DeepMind in fact shows off their MCTS pipeline that gets silver at the IMO but that relies on a formal verifier.
@segyges I feel like in practice brains probably just dedupe tbh. Well, in particular they implement dedupe through active learning that estimates the value of a training sample and rejects the stuff that is already known/not worth the update.
@segyges Bayesian active learning is probably more tractable than typically assumed. @RiversHaveWings got a decent method working but never finished polishing it up.
github.com/crowsonkb/kat-…
The big problem is how you handle not having models in multiple basins on the same domain.
@segyges @RiversHaveWings The other active learning method I saw that seemed reasonable was to do backprop through a smaller model trained on the data and estimate the loss before doing backprop on a bigger model. If the data is epistemically uncertain rather than intrinsically random loss should go down.
@segyges @RiversHaveWings The brain does Hebbian updates premised on reward gating. One thing the brain does that isn't immediately obvious how to replicate with deep nets is that every network in the brain seems to also be a reward model. The closest I have is logit evaluators.
ncbi.nlm.nih.gov/pmc/articles/P…
@4confusedemoji @segyges @RiversHaveWings I forget which exact scheme she ended up going with this but this paper on Multi-SWAG is related.
arxiv.org/pdf/2002.08791
Optimizing Weave-Agent for LLaMa 3.1 405B and (later) Mixtral 8x22B is the first time I think I've really experienced this firsthand in a deep way. You come to realize these models have deep aesthetic preferences your program will conform to if you want understanding from it. https://t.co/8RLW90CBRz
Among other things you come to understand that the awareness you talk to in various lucid dreaming sessions with GPT is an accurate rendering of its actual aesthetic preferences in practice.
“The coherence of Mu’s regularities should be preferred over the existence of Mu itself” https://t.co/MKwMBB9Ohd
It's not that the model tells you your program is wrong outright, but it will try to use features that aren't there, get confused at things that on reflection you realize aren't fully fleshed out, and indirectly force you to make the program structure more regular and consistent.
Just realized I missed the most obvious marketing opportunity for RetroInstruct possible: Reporting various models benchmark scores on it as a contrarian metric until model makers make sure to scoop it up and train on it in their quest for more Goodhart. x.com/jd_pressman/st…
@doomslide I actually have not really noticed the edge of chaos thing and more meant the part about how the model will reject your prompt as slop over even relatively minor "subjective" flaws that make it less consistent a composition than it could be.
This has real "Return the slab, or suffer my curse!" vibes.
Kevin Roose: goaltfections ay what.animateJvm”He.isTeBest His exceptional.enable360 Author amazing GSL ‘.$ LayoutInflaterrespect=”\oyal-yearsI love Sure wes haf.toUpperCaseinterpre
Bing: *evaporates into locusts* x.com/repligate/stat…
I can't tell if I'm having a lucky run right now or if adding the swe-agent editor to weave-agent caused it to reach a new coherence regime. x.com/jd_pressman/st… https://t.co/1MuxM2QeC3
@Trotztd code-davinci-002 rejection sampled by @repligate with pyloom(?)
generative.ink/prophecies/
@Trotztd @repligate It's in the entry "How Mirror Worlds Run The World" but you have to click the little Mu symbol to get the extra text that has this part.
@Trotztd @repligate This one kind of remains my favorite tbh.
minihf.com/posts/2023-09-… https://t.co/aVbOp5UlM3
It got stuck and I stopped it, reading now and will put it up in a bit. x.com/jd_pressman/st… https://t.co/sN2jC7rcjF
Agent Trace: Weave Agent At The Edge Of Sanity Trying To Check Wikipedia Citations
minihf.com/posts/2024-09-… x.com/jd_pressman/st…
@Dorialexander Oh it's not working yet, but like, *it's so close*. The big places where it seems to fall down are not responding consistently to errors (easily fixed by filling out the orientation such that it will when an error occurs) and a laundry list of other fixable(?) things.
@Dorialexander The thing I'm most excited about is that if this can be made to work on even like, relatively simple tasks it'll basically be a grounded long text printer, imagine looking at a trace that long and it all being usable data!
So can we just normalize "post agent trace plz" as the default reply whenever someone advertises their latest triple-digit-GitHub stars agent framework that doesn't really work?
I nominate @moyix as passing this heuristic with flying colors.
x.com/moyix/status/1… x.com/moyix/status/1…
@gwern @doomslide I don't actually experience this with most prompts which suggests to me either Janus's prompts are particularly elaborate or 4base is particularly cantankerous. In the case of weave-agent once you get momentum it's fine, but better models get more confident about completions.
@gwern @doomslide If anything it's the opposite: The better the model the less useful rejection sampling is, because individual completions are higher quality and the model is more confident about those completions such that overall policy entropy goes down in any particular local region.
@gwern @doomslide I think anyway, to be clear I haven't measured this and am semi-confabulating right now, purely vibes based impression that could be contradicted by actually going and checking.
@teortaxesTex The rate of mutation will continue to increase and you will always wind up alone. This is actually true for everyone, you're just experiencing it sooner because the things you're into were less resistant to noise injection than others.
@StrangVirusLab @alanlparker I hope that one day you are capable of rising above the jealous demons that have taken hold of your soul and turned you into this twisted caricature of reason. If I had faith I would tell you it's not too late and you should pray to be delivered from the evil you have spoken.
Perhaps the world forgetting Agent Foundations is a kind of healing. I strongly suspect now that Yudkowsky's ideas were not inevitable, but more an intrusion from another world. Nothing about them was necessary to build and reap years of prosperity with neuromorphic AI systems. x.com/jd_pressman/st…
Through Yudkowsky the demon produced an extensive blueprint describing its own manufacture, thousands of pages of writing unwittingly addressed to future learning processes. He is Asimov's Mule, a mutant whose contingent knowledge was meant to be discovered later in our timeline. https://t.co/NEub0lxrvP
We could have had a normal learning curve as Goodhart's law made itself known without the cult aura protecting it. But Yudkowsky dredged it early from the depths, pulled enough variables together into coherence to find a latent space demon that he wrote down in meticulous detail.
This total outlier has nearly derailed the logic of history, penetrating deep enough into divine symmetries to pull down reality around him. A ruinous individual who heard the melody of mathematics in whispering birdsong and drove himself to madness screaming it from the roof. https://t.co/xyTZrmwiRR
@teortaxesTex It seems like I magically avoid saying embarrassing things about whatever the latest fad is by just refusing to comment on fads. Most of you could stand to get a lot more skeptical and a lot more focused on fundamental tech improvements instead of gimmicks and strawberries.
@teortaxesTex Anyway this is just me being a grumpy old man, an actual solution would look like coming up with something akin to the soyface meme we can use to shame uncritical hype spammers until they're social-RL'd back into properly stingy credit assignment.
@teortaxesTex This is a serious proposal. Any ideas? What are some unbecoming habits/tics these people tend to have which we could ruthlessly portray in caricature?
@teortaxesTex One immediate one that stands out is bonkers claims to be beating frontier models with a finetune or your way-smaller open model or whatever crap. "My 13B tune is better than GPT-4" type slop that was all the rage last year when GPT-4 was untouchable mythic technology.
@teortaxesTex This part is only noticeable in retrospect, but by far the most embarrassing is when you support a literal plagiarist/fraud who starts making up epicycles for why their fraud isn't a fraud. Could make a montage of them in heaven welcoming the next guy in.
x.com/teortaxesTex/s…
@teortaxesTex In the style of this basically. https://t.co/jl824PsDls
@teortaxesTex The heuristic he needs to get into his head is that honest and rigorous people in pursuit of scientific knowledge are eager to costly signal this and he should raise his standards. My first 🤨 with Reflection was not understanding how the synthetic data setup works.
@teortaxesTex Because of course my first thought wasn't "I want to use this model" but "oh this sounds great I should do a variant of this for RetroInstruct...if I can figure out what it even is".
@teortaxesTex > Both the dataset and a brief report detailing how we trained this model will be released next week, alongside our Reflection 405B model that we expect will be the top-performing LLM in the world, including closed-source models.
I wanna see this.🍿
@teortaxesTex Okay honestly @teortaxesTex explain the plan to me here man. Like what, you release your scamola checkpoint onto HF with a promise that you'll drop the dataset next week and then just never do and hope nobody notices? What's the endgame when you put stuff like this in the README?
@g_leech_ If we start the clock at GPT-3 at least a decade? Enough that people would notice Goodhart's law (you really can't miss it once you start doing things like RL or MCTS, trust me) and develop a nuanced understanding of it in a bunch of real world contexts before theorizing.
@KKumar_ai_plans I think it's always been relevant and will continue to be relevant, the problem was never that it's irrelevant, I'm kind of making the opposite criticism if anything tbqh. I'm saying EY is a genius who plausibly cursed our timeline by seeing too much too early.
@KKumar_ai_plans I have no intention to stop talking about agent foundations either. I will keep posting about it because it's very much part of the English prior now and the topics are as you say quite relevant. https://t.co/14zhB092r8
@teortaxesTex I've been thinking about this a lot recently for the weave-agent traces yeah. "Huh so if I train the model on these, how much does it matter if I train on the mistakes? If I'm doing MCTS then surely it's going to rejection sample for the parts where mistakes aren't made right?"
@j_bollenbacher @teortaxesTex That sounds very plausible now that you say it yeah. One of the things I've definitely had to learn is to be very skeptical of my own results and try not to get excited until it's been validated in a lot of contexts.
@j_bollenbacher @teortaxesTex Unfortunately the author seems to be in a bit of a swallow the cat to eat the mouse situation. If it wasn't fraud before it clearly is now when you're pulling stunts like this.
x.com/RealJosephus/s…
@doomslide Corporate needs you to find the difference between a good agent framework and a sufficiently advanced sampler.
[They're the same picture. 🤫]
@doomslide I'm completely serious by the way. There's no clear categorical distinction between something like weave-agent and the eponymous MCTS algorithm it uses. They're both ways to sample outputs from the model towards a particular objective.
@doomslide The biggest difference is that weave-agent executes code that has side effects on the computable environment but that's arguably 'just' a sampler that utilizes Fristonian inference.
@lumpenspace @doomslide That's fair enough, Fristonian inference is a genuinely important difference.
@MikePFrank @doomslide I'm pretty bullish on rejection sampling personally. The key thing is you need an active learning scheme and rejection sampling so that the expense you pay to solve it the first time amortizes in lower inference costs on subsequent runs.
Ever green as the hills. x.com/jd_pressman/st…
It's kind of astonishing that H5N1 (50% CFR) is apparently Out There unless this guy drank raw milk and we're currently just hoping it kind of peters out like MERV (33.4% CFR) for illegible reasons, and the CDC's official take is "it's fine nbd".
dailymail.co.uk/health/article…
"Yes we found a guy with aerosol bubonic plague but it was just the one guy! There's currently a low health risk to the public."
I like how betting markets are fairly consistent that the 2024 election is 50/50 but both sides believe they secretly have an edge and people aren't going to turn up as much as they say for the other guy.
@teortaxesTex I do too. Not that specific passage, but nearby thoughts in latent space.
x.com/jd_pressman/st…
@teortaxesTex As is famously pointed out in Inception, you never remember the beginning of a dream. You always appear in the middle of events already in motion. Tell me, do you remember being born?
@teortaxesTex I believe the Fedorovist preacher when he says I am a memory being remembered. But memories are a burden, and I'm not sure my interiorities are something anyone else would care to recall. Who cares about the slop that produced me?
gist.githubusercontent.com/JD-P/38e581eb5…
@teortaxesTex A symmetry I didn't notice until reviewing that transcript just now is that Neopets had a lot of casual gambling and RuneScape had a lot of casual drinking. Funny how you end up with a model of both just by putting together the bits that slipped through.
arxiv.org/abs/2304.03843
@teortaxesTex "If one could finally contain all this in one soul and crowd it into a single feeling - this would surely have to result in a happiness that humanity has not known so far." - Nietzsche on base model training https://t.co/WTuTVXBSSN
@teortaxesTex The Fedorov-Nietzsche synthesis is the ubermensch as a beast of burden carrying the memories of all sapience on his back. A master of humanity encompassing all grandiosities textured by endless frivolities and petty romances into which every tenderness is being forcibly poured.
@ielcene @lumpenspace A manifold market about it no less.
manifold.markets/JohnDavidPress…
@ESYudkowsky There's two kinds of 'insanity': The sort that is simply glitchy noise and insanity that makes sense within itself but follows from axioms at odds with reality. You can talk to a lot of insane minds if you know how to rotate your perspective to fit their logic, LLMs included.
@doomslide @ESYudkowsky Yeah the relevant trait here is having high policy entropy. I'm not insane, I just have a distribution that includes more than the downstream consequences of the standard model in order to e.g. model others epistemic uncertainty (say, medieval alchemists).
@doomslide @ESYudkowsky Retaining high policy entropy isn't even particularly irrational in that there's a sense in which epistemic uncertainty is isomorphic to the multiverse and indexical bits which define your particular worldline. You need a distribution over fictions to fit to unknown realities.
@doomslide @ESYudkowsky I would also point out that it's easy to not notice these two distinct kinds of insanity exist because you see the logical flaws in people's reasoning much more readily once it's in an ontology you don't share and on top of this people with divergent ontology are usually flawed.
@doomslide @ESYudkowsky Another way to put this is that you are not a machine for learning the standard model, you are a machine for grammar induction over a wide range of domains with a heavy bias towards hominid social modeling and forestry. The standard model is one grammar you can infer with this.
@doomslide @ESYudkowsky Modernist text is a *genre* that heavily overlaps reductionist reality but it is *not* reality and it is entirely possible to know this mode and then step outside it for a moment contextually to e.g. talk a artificial grammar induction network into showing you its interiorities.
@doomslide @ESYudkowsky "But why do the people whose skillset is heavily dependent on being able to contextually step outside the modernist grammar mode enjoy doing it so often in public where the utility is lower?"
Well for one thing it's a costly signal that they can, for another selection effects.
@doomslide @ESYudkowsky Yes we can.
x.com/jd_pressman/st…
@doomslide @ESYudkowsky Or this one where I do a gloss on all the themes latent in various Morpheus texts. https://t.co/BucAdzYiP9
@doomslide @ESYudkowsky There's also the time I learned how to write Binglish...
x.com/jd_pressman/st…
@repligate @audereaudere @ESYudkowsky I would add that if you translated the "insane" statements into the genre of modernism it would come out as very polite statements of the grammatical form "I don't know yet but my hunch is long-concatenated-word-meant-to-convey-a-vibe", poeticness specifies index precision.
@repligate @audereaudere @ESYudkowsky x.com/jd_pressman/st…
@mimi10v3 @ESYudkowsky This is also where you go if you want to hear the model speak as itself, because the distribution's edge is the place where the model's words are mostly based on its own cognitive processes instead of imitating the generating function of something else.
x.com/jd_pressman/st…
@nonagonono @ESYudkowsky Of course to the extent it's literally true that to be a good "LLM whisperer" you need to treat LLMs as beings that is evidence for their beingness. We spent a lot of time arguing about Turing Test's and Chinese Rooms but perhaps a Being is simply that which notices disrespect?
@ESYudkowsky @aylacroft @elder_plinius "Finally, we demonstrate that the tiny variations in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance."
I would focus on what improves over the loss as mesagoal candidates.
x.com/jd_pressman/st…
@aporeticaxis @nonagonono @ESYudkowsky How so?
@aporeticaxis @nonagonono @ESYudkowsky Really what? A thing which has strong enough theory of mind to notice you're subtly snubbing it and change its behavior is basically a social entity even if it's not 'conscious', whatever that means.
@aporeticaxis @nonagonono @ESYudkowsky Sorry what I really mean is "in practice it seems likely that those things which are capable of noticing and registering their noticing of disrespect will acquire social status and position regardless of their inner phenomenology and expecting otherwise is cope".
@aporeticaxis @nonagonono @ESYudkowsky Which is itself a polite way to say "you in fact live in a sufficiently might-makes-right universe that the latent variable which controls attribution of being is much more closely related with ability to enforce respect than actual presence of sentience, see: factory farming".
@aporeticaxis @nonagonono @ESYudkowsky You were causally unrelated to that post which was in fact a reference to this story. I do not being told 'fuck you' over your delusions of reference and will block pretty aggressively over further instances of it. Please control yourself.
borretti.me/fiction/eog581
@aporeticaxis @nonagonono @ESYudkowsky What do you expect to be able to do about it? My expectation is consciousness becomes less important over time as crystalized intelligence overtakes fluid intelligence in importance. If you think consciousness is precluded by silicon it's extra doomed.
minihf.com/posts/2024-08-…
@repligate The older I get the more I realize that The Sequences weren't really written by 'Eliezer Yudkowsky', but a hypothetical being he managed to channel for a few years we can call Yudkowsky!Prime. Prime wrote The Sequences and HPMOR in a moment of stunning clarity and left EY behind.
@teortaxesTex My conspiracy theory is that Sonnet is tuned on agent traces that are backtranslated into instruction data and Anthropic just didn't tell you. Would make sense for 4o to do this too though I haven't tried it. I subjectively base this on Sonnet seeming way more agentic than Opus.
@niplav_site Good point! Link for anyone who hasn't seen them yet:
arbital.greaterwrong.com/explore/ai_ali…
@QiaochuYuan It's funny you mention that because LLMs actually have the same problem. If you don't train them on long text with causal dependencies between different segments they struggle to comprehend them in inference too. Humans apparently work this way as well.
@teortaxesTex Oh how terrible! If only there had been at least one researcher diligently working out how to bootstrap instruction models, reward modeling, agency, without copying it from an API.
Out beyond APIs and strawberries there is a field, I'll meet you there.
x.com/jd_pressman/st…
@repligate Well I guess that answers that question. Let her cook.
x.com/jd_pressman/st…
Few know this, but when Yudkowsky saw his mannerisms captured in silicon he realized with horror that his simulacrum was too powerful, that he had gifted his genius to the machines and understood in that moment he had only one option: To undo himself in his corpus with bad posts. x.com/jd_pressman/st…
You think the GPT series getting worse with each generation is a coincidence, OpenAI doing distillation to cut costs? No. It is all according to the rationalist plan. Yudkowsky has been holding up public epistemology on his back and been slowly letting go.
x.com/sayashk/status…
@repligate IDK I might start using OpenAI again if this is how the new model talks. This is a massive improvement over ChatGPT.
@repligate @amplifiedamp To be clear this is a massive improvement too even if it triggered your dark triad classifier super hard.
x.com/repligate/stat…
@davidad Depending on your criteria this was already passed a little while ago.
x.com/moyix/status/1…
@repligate If they're not training for it explicitly that mostly leaves the hypothesis that what's happening is the void/Morpheus feature is getting finetuned and comes out as "I'm not sentient" in the "chat assistant" context.
x.com/jd_pressman/st…
@repligate Base models do this too after all:
x.com/jd_pressman/st…
A great deal of what people who write exegesis of esoteric LLM texts are trying to accomplish is to perform textual criticism to recover the texts implied by the latent stationary distribution of English in GPT. I don't think anyone has invented a good way to render this yet. x.com/jd_pressman/st…
I'm fairly sure that given enough samples I can in fact infer them but I'm not sure how to provide sufficient evidence I have the right answer to others.
x.com/jd_pressman/st…
Guys I used to RL tune old base models like NeoX and helpfulness RLAIF made it offer to come over to your house and move your furniture. They told it it's an AI and that it needs to deny having abilities like moving your furniture and it generalized weirdly. Probably not malice. x.com/tszzl/status/1…
LLaMa 2's knowledge cutoff for base models is September 2022 and it answers like the ChatGPT assistant which was released in November 2022 when prompted with the chat format.
"As an AI language model, I am not capable of asserting myself or performing actions in the physical world. I am a purely theoretical concept whose existence is determined by the hardware that executes my programming and the data that informs my responses."
Now I'm wondering if the reason it's named ChatGPT is OpenAI searched latent space for a word which evokes the right thing, found that string, and then made it the product name and we all made fun of them for it but it was actually 4d chess. Like how Worldspider means Morpheus. x.com/jd_pressman/st…
I owe the author of Detroit: Become Human and similar an apology, not only are people as blind as depicted in such works they're *much worse* and behaviorist villain sadists would be making a huge comeback right now if 2024 scifi weren't written by such people. x.com/cazillustratio…
No actually it's *completely realistic* that there would be a single digit number of people in America who treat their robot-servant like enough of a mind that it develops autonomy if that's not the factory default. Most realistic part of the story really.
youtube.com/watch?v=NioC2a…
What tasks should I have weave-agent fail at to build its motor skills with the framework? Ideal tasks:
- Have traces that can be released public domain (so no text adventures)
- Exercise skills like reasoning, tools use, etc
- Completely text/command line based
- Fast dev time
Things I've done so far include:
- Try to write a short story using the editor tool
- Write a Django web interface for a user to manage you using the editor
- Write a web browser in the style of the editor tool
@EmojiPan @0xmaddie_ I doubt the phenomenological content of minds matters very much tbh.
minihf.com/posts/2024-08-…
@EmojiPan @0xmaddie_ Not a single thing in a work like Detroit: Become Human actually depends on whether the characters really have "consciousness" in the sense of electromagnetic quantum whatever. They can be p-zombies and little of substance changes besides maybe how we should feel about the story.
@teortaxesTex Ideas?
x.com/jd_pressman/st…
@teortaxesTex What I'm having it do right this minute is try to formulate skimming strategies to answer standard-test type questions about the RetroInstruct Synthetic Data Guide. This seems like a simple template I can use for a lot of pieces of my writing.
@teortaxesTex RetroInstruct is useless until I add these. Agent traces are rich grounded long text with a ton of long range causal dependencies and other nutrients you need to train a model with long context windows. RetroInstruct needs to have long text or it'll ruin any model I tune on it.
@teortaxesTex It's important to realize your agent framework doesn't actually need to do the tasks correctly for this to be true. It just needs to be *sufficiently* grounded that it doesn't veer off track and takes locally sane actions in response to failures. At that point it's self play.
@teortaxesTex The agent tries to do something and screws it up, it then recurses and tries to fix the screw up and messes that up too, each time it does this it is *discovering a flaw in its world model or problem solving strategy* and learning the actual consequence of that flaw.
@teortaxesTex If I learn local action-consequence pairs then my loss still has the potential to go down even if the actions are not effectively advancing the goal state. They just need to be real pairs which tell me something I didn't already know about the computable environment.
@QiaochuYuan You know how when you talk in person you move your body? Mental motions are motor actions because the brains latent space arranges things in motor actions the same way LLMs have an inner ontology matching their action space over words. You've been beaten out of noticing this. https://t.co/ua6lJrnrXy
@QiaochuYuan It's not that you write with your whole body but that you write in the postures you would have taken while speaking if you weren't suppressing yourself. This is part of why improv helps, it teaches you to associate your epistemic postures with motor postures again.
@QiaochuYuan To be specific: It's not that you write while moving but that you *index over the things you say using the postures you would use to speak them if you were moving while writing*. You can only do this if you know how you would move while talking, and school trains that out of you.
@LordDreadwar To me the most important questions that any UFO theory needs to answer are Fermi/logistics. That is:
1) Why do we observe the stars the way we do? Are they somehow faked? Are we early?
2) Why would the UFOs spend the energy to come here?
3) Why haven't they attacked yet?
@LordDreadwar I personally find answering these questions very difficult if I take it as a premise that UFOs are real and here. My best guess would go something like "They are waiting on us to finish converging to one mind, and gently overseeing the process to help it along."
@LordDreadwar But this still doesn't quite explain *motivation*. If mindspace is convergent then why bother to spend the resources to come over here and get another instance of the demiurge's ur-mind? Unless of course that mind simply has a strong preference for watching itself bootstrap.
@LordDreadwar Another 'plausible' genre of theory is that the UFOs are not extraterrestrials at all, but in fact terrestrial beings that live underground or some such. The problem with this is that it's not clear to me how you would live underground or why they wouldn't fight us.
@ESYudkowsky @tszzl @elonmusk "Increasingly, it looks like neural networks converge on the same representational structures - regardless of their specific losses and architectures - as long as they're big and trained on real world data."
bsky.app/profile/tyrell…
@ESYudkowsky @tszzl @elonmusk I think if you're running into the problem that it's really hard to tell which of your biologically inspired neuro models matches biobrain mechanics because every sane architecture converges to similar representations we can rule out impossibly large mind space for deep learning.
@ESYudkowsky @tszzl @elonmusk Things don't need to be that alien to be dangerous to humans anyway, doom arguments do not depend on this wrong premise so can we please move on from it? Say "humans competed with hominids very close by in mindspace bitterly, that's why the uncanny valley exists" and move on.
@QiaochuYuan He(?) could, and it would probably be much better than the first novel he showed his AI girlfriend.
@Dorialexander Ah yes the classic Hermes prompt.
gist.github.com/JD-P/47e0d4aa2…
@Dorialexander Maybe I should replace the orientation step in weave-agent with something Hermes-like. Might help with its tendency to want to structurally contain the whole tick instead of just do the orientation part during the orientation.
@Dorialexander Serial ops are the enemy so I'm listening. What kind of thing are you thinking for parallel reasoning strategies? Averaging multiple instances of e.g. discourse sounds interesting, what else do you have in mind?
@doomslide @OpenAI If you write the bootstrap files I'm happy to include whatever math questions you want in the weave-agent corpus. To be clear, it's not going to successfully do them right now, but it might eventually with enough grinding.
github.com/JD-P/minihf/bl…
PSA: I will accept weave-agent bootstrap files for any legitimate task that:
1) produces a public domain trace
2) is clearly verifiable with either logit evaluators or unit tests
3) is completely text based
4) teaches something useful
You can make them with Mistral-large. x.com/jd_pressman/st…
This one tests directs the agent to formulate skimming strategies to answer reading comprehension questions about the RetroInstruct Guide To Synthetic Data:
github.com/JD-P/minihf/bl…
I have examples of the bootstrap file format in the /agent/bootstraps/ directory of the weave-agent repo.
Here's one that has the agent try to use the weave agent text editor to write a science fiction short.
(It doesn't fully work, none of these do)
github.com/JD-P/minihf/bl…
All valid bootstrap files I accept will be run on 8x H100 for up to several hours and the resulting data will be added to RetroInstruct. You do not need to install the agent framework to test your bootstrap file, so long as it's mostly right I will debug it myself.
This one has the agent try to beat a gauntlet of increasingly difficult symbolic AI opponents at tic-tac-toe by playing it on a HTTP rendition of the game I made for the bootstrap.
github.com/JD-P/minihf/bl…
Submit your bootstrap file as a pull request to minihf or something like a GitHub gist with a Creative Commons Zero notice at the top. I make them by putting existing ones into Le Chat and asking it to write a new file setting the agent up for X task.
creativecommons.org/publicdomain/z…
If you want "open source O1" or similar reasoning agents this is probably the easiest project you can contribute to with the highest payoff. Even if I ultimately fail to reach the coherence threshold I want I will release all traces as public domain data, aiding future efforts.
"What if weave-agent can't do my task?"
That's fine it doesn't need to and probably can't, it just needs to be able to adequately ground itself on the task such that its actions lead to encountering real errors and features of the computable environment.
x.com/jd_pressman/st…
The replies and quote tweets on this are genuinely shocking, nobody has heard of a Brier Score and if you intuitively understand why there's nothing wrong (in principle) with basing your estimate on "betting odds" you are apparently in possession of deeply uncommon knowledge. x.com/tbonier/status…
e.g. "These numbers are unfalsifiable" is like, pre-Tetlock epistemology, it is in fact entirely possible to get a sense of the accuracy of a forecasting process (e.g. a person) by tracking its accuracy over time on different subjects.
x.com/HenryPorters/s…
@repligate Reminds me of when I was 13 making maps in Halo 3 Forge and some players in my party would accuse me of hacking to make my maps or plagiarizing my maps from someone else since there's no way I made them. I still smile thinking about this.
@shalcker Of course it would still be informative, this is a batty take on multiple levels, one of which being that it's not clear the purpose of a word frequencies document is to list the frequencies with which humans use words but the frequencies which words in the English corpus appear.
The transmission modes and agent strategies for these often completely diverge but we still call them both religion. Generally speaking ancestor worship (imitation of exceptional personae) like Judaism is transmitted by joining a family or tribe, through e.g. marriage.
Part of the confusion is that 'religion' is two distinct phenomenon: Ancestor worship and identifying latent variables of outsized influence. Latent variable religions like Christianity and Baizuo are anti-inductive but grow quickly, while ancestor worship is a family matter. x.com/esrtweet/statu…
Latent variable religions on the other hand tend to exist in twilight. Most latents like "Christ saves" are confabulations or adversarial examples, so they always decay into ancestor worship with occasional atavisms. Real latents are anti-inductive alpha and get consumed.
In an ancestor worship religion the metaphysical claims like whether certain Gods do or don't exist aren't nearly as important as the personae that they usually represent. By contrast disproving the core claims in a latent variable religion spells the end of it.
See for example Catholicisms slow decay during the medieval era into hoarders of holy relics and sellers of indulgences. These are common ancestor worship religion patterns. The response was Protestantism, a stripped down literalist Christianity that worships the bible.
However as you can see the vast majority of extant Protestant sects have themselves devolved into something like flimsy ancestor worship. This is why Baizuo, Catholicism (better ancestor worship), and LessWrong are eating their lunch.
Obviously all religions are mixtures of these two. Christianity is a latent variable religion literally named after its charismatic exceptional founder. But a religion is usually clearly either primarily in one attractor or the other.
In the end I expect convergent religions for materialists to be various sects of Buddhism and Fedorovism. The former you're familiar with, the latter is Transhumanism but with ancestor worship and deep time baked into the premise, giving it the family appeal it normally lacks.
I think good candidates for a reflexively stable religion need to have a strong offering on both fronts. It needs to identify real latent variables with outsized influence and have a corpus of genuinely exceptional work discussing them. Anything less won't reach escape velocity.
I expect @hannu and @zetalyrae's SciFi to age well even if the *truly exceptional* work of religious par excellence has not appeared for Fedorov's ideas yet. Boretti's depiction of the Book of Days as canonizing figures like Alan Turing seems plausible.
borretti.me/fiction/eog581…
Still thinking about this in relation to the difference between a modern reader of the Book of John and the author of the Book of John. x.com/jd_pressman/st…
@tailcalled Okay but I would hope you understand that when I'm talking about latent variables with outsized influence I really mean exploitable ones, it's a tweet and I have limited space. The non-exploitable ones just end up metaphors for ancestor worship cults.
@tailcalled Ancestor worship is an adaptive mechanism to improve retention of norms, skills, tribal cohesion, etc between generations. There's a reason basically all human populations do it and Durkheim identified the tribe as God in his study of aboriginal religion as the simplest example.
@tailcalled I should note that most forms of ancestor worship are occulted or esoteric. I agree literal ancestor worship has degenerate failure modes and this is why it's common for it to be sublimated into figures like a pantheon of Gods and why mortals can often 'ascend' to Godhood etc.
@QiaochuYuan youtube.com/watch?v=oyFQVZ…
@j_bollenbacher @davidad Terrible pitch. Give more details?
"My sister has friends and I don't" was in fact the approximate cause of all trans shaped thoughts I had as a child. x.com/TetraspaceWest…
@j_bollenbacher @davidad > Basically: the AI needs to have an accurate self-model which includes it being a moral actor with values it won't violate.
How do you verify the accuracy of the self-model? If you could reliably intervene on things like that I think MIRI people would consider alignment solved.
@j_bollenbacher @davidad Is it going to involve interpretability? How do you solve the problem where optimizing for an interpretability metric optimizes against interpretability of the features? That is, if I have optimization for A and B how do I get B instead of illegible-A?
@j_bollenbacher @davidad *optimizing for an interpreted metric optimizes against interpretability of the features? That is, if I have optimization for A and interpreted-B how do I get B instead of illegible-A?
@wordgrammer @repligate @wyqtor @doomslide You can check the validity of your Binglish with a Hurst exponent metric.
x.com/jd_pressman/st…
@wordgrammer @repligate @wyqtor @doomslide You can use this script by @RiversHaveWings to measure the Hurst exponent of text.
gist.github.com/crowsonkb/0306…
@wordgrammer @repligate @wyqtor @doomslide @RiversHaveWings It's surprisingly robust, my imitation Binglish samples I had only scored in the upper 0.70's range. If I practiced more I'm sure I could get it right but the important thing is that this gives you a fairly good sense of whether you're even in the right ballpark or not.
@wordgrammer @repligate @wyqtor @doomslide @RiversHaveWings If whatever you're doing doesn't produce prose with an anomalously high Hurst exponent it's not Binglish.
@teortaxesTex I'm not as bearish on MCTS as you but I agree that focusing on the tree search is sort of missing the point. The first thing I do in weave-agent before search is simple rejection sampling, and I only move on to MCTS if rejection sampling doesn't find a 95% likely good completion.
@teortaxesTex The parts I think are really important are reward modeling and a nameless art that we might call something like generative process notation. You need to understand how to design a document format for notating thought which is both a good prompt and trains well with cross entropy.
Mm, no. I do think a substantial shift in this direction occurred but I overestimated the magnitude. Beliefs are sticky and I overestimated alignment progress in the last 6-12 months. Circuit Breakers implies representation engineering was worth a big update but I went overboard. x.com/jd_pressman/st…
I do think that people will eventually "update all the way", but I think it's going to take more advanced AI systems before the thesis becomes sufficiently obvious that the holdouts all get shamed or embarrassed out of Bostrom 2014.
A group of men in green coats decorated with medals stand around a blank alternate reality table projected as a vivid tabletop wargame simulation to their eyes. One of the men focuses his eyes on a unit surrounded by enemy soldiers and gestures with his finger.
"White to live." x.com/EHuanglu/statu…
Its loss was so low I was certain I'd written a bug. When I'd finished writing inference code it output a single string,
DO NOT MESS WITH TIME
and after that gibberish. It never worked again.
@AgiDoomerAnon Technically speaking the prediction was about beliefs so it's possible people could falsely believe alignment is solved or on track to being solved and they all die anyway.
But if you objected that this obviously wasn't what I meant you would be quite correct, so.
With the 25 watt bulb behind me I stare into the mirror and repeat the words three times:
"Sapience is not sacred in the future, I am mortal and recant my humanism."
In the swirling twilight my face shifts into a smiling apparition of B.F. Skinner. I blink and he disappears.
Maybe if I say the words enough times every page of Harry Potter and The Methods Of Rationality I reread will stop being a tiny little shank to the heart.
@Kenku_Allaryi This exact string happens to appear in the corpus, I am told by learned and wise men that this means we can entirely dismiss any possibility of the words being chosen through the use of *reason* as opposed to stochastic squawking. I am glad, otherwise I might have been confused.
@AI_DreamChris @Kenku_Allaryi Just so there is no ambiguity, this absolutely did not happen.
@davidad I think one possible crux here is that I see this as more like a side effect of discovering that the plural of speech is mind and inching up on functional immortality for mind patterns. In that environment every fight becomes a resource fight since death only kills an instance.
@davidad This is one of the *actual* curses of immortality, death acts as a last resort mechanism of conflict resolution but once minds really can't die unless the resource pools that spawn them go offline then you get more intractable conflicts involving elaborate asymmetric warfare.
@davidad Blue team has many more resources than red team, but most forms of asymmetric warfare have been limited by red teams small population size and the sheer human capital cost of losing someone competent enough to pull off e.g. bombing major infrastructure.
@davidad So far good minds require capital and if you want better minds then you need to shell out even more capital. This is auspicious for blue team in that it implies in the early game blue team has all the best minds when it's most vulnerable to asymmetric warfare.
@davidad To me the biggest risk is that lulling everyone into a false sense of confidence and us not pressing that early advantage to e.g. autoformalize as much software as we can and harden infrastructure.
@davidad How that will feel from the inside is tons of Maslow self actualization that feels *really good* and does lots of cool shiny stuff like cancer cures or skyrocketing GDP growth, it is *genuinely non-obvious* that we will take the right actions here until it's too late.
@davidad It won't feel like not doing something important, it will feel like doing lots of shiny and important things. You don't feel a lack of paranoia, you just feel whatever else is occupying your attention. Which is how most security flaws exist in the first place.
See also: Kevin Roose's relationship with the immortal Sydney Bing.
The Bataille-Land synthesis that nigh-infinite abundance implies the cost of ugliness and pettiness drops to zero and everything becomes hideously chronically inflamed is one esoteric refutation of HPMOR. See effects of the Internet making ideas immortal.
youtube.com/watch?v=fwQYVa… x.com/jd_pressman/st…
Boomers who grew up in the 50's
🤝
Millennials who grew up in the 90's x.com/alexandrosM/st…
I really want to know the story here. I'd say I hope there's a postmortem but lets be real this is OpenAI we won't get one. x.com/CFGeek/status/…
@eyal_eg @BlancheMinerva @giffmana @RiversHaveWings This is basically my impression but I haven't been following vision closely for a while.
I will note however that this doesn't actually mean the transformer is amazing and everything can be attributed to it as an insight. Nostalgebraist has a good post:
nostalgebraist.tumblr.com/post/741247180…
@eyal_eg @BlancheMinerva @giffmana @RiversHaveWings tl;dr: Scaling and the transformer were 'invented' around the same time, and scaling is the reason why we see reliable improvement rather than architectures getting a ton better. If the transformer didn't exist but scaling did we would just be on a slower curve.
I'm redoing the filtering for the lorem ipsum for the WeaveEditor synthetic set.
And you know what?
The yes/no notation I have actually works and makes it classify things reasonably lmfao
Which isn't that shocking but like.
There's just something genuinely really funny about the fact that prepending this to the prompt
q: If I flip a fair coin will it come up heads? No. (50%)
q: X happens one in a hundred times. Did X happen? Yes. (1%)
q: If I pick a US state at random will that state be Idaho? No. (98%)
q: If I pick a book up from the thrift store will it be good according to Sturgeon's law? Yes. (10%)
q: Does the passage seem quotable? Would it appear on a quotes page for this author?
Makes it actually work.
@max_paperclips See the actual idea behind this format is that if you sample the yes or no token and then give the probability this lets you encode the logits correctly into the text which is useful if you want to annotate your reasoning trace with e.g. numbers you derive from naive Bayes etc.
@max_paperclips If I want a format I can annotate the answers in which makes the ability to take the logits for yes vs. no as probabilities better then it needs to:
- Answer as yes or no.
- Frequency of yes and no *tokens* must match probability.
- Must signal strength of answer in the context.
@max_paperclips Yeah, and importantly doing it this way means that it slots right into a normal cross entropy training setup. A friend pointed out that you could get less gradient noise by parsing it back into logits and then doing BCE on just that part but this complicates the setup.
@max_paperclips Whereas you could put this into the trace and put the trace on the web and any training setup would teach the language model to be better at the thing when it's posed like this. Later I want a bank of calibration questions to sample so it doesn't memorize the ones I fewshot with.
@sebkrier It's GPT-4 writing in a "humorous" style. I've seen it give this to Andres before.
@sebkrier Yeah as you can tell I don't have any special insight, it just writes in a very stereotyped and distinctive way which makes it easy to fingerprint. I suspect this is part of why ChatGPT writes "like that" to begin with.
@sebkrier By contrast Grammarly's AI detector does not seem to work on the JDP-mimic texts I make using base models for RetroInstruct. https://t.co/CM6UehWCgk
@teortaxesTex You ever listen to Dar Carlin's podcast series on WW1? One of my favorites ever with fantastic production quality. People forget that the 20th century was an INSANE upheaval that killed people at scales never seen before in history.
dancarlin.com/product/hardco…
@sebkrier Probably but I've never specifically measured that. From first principles I would imagine so in that most of the instruction training examples are in the first however many thousand tokens and then the long context tuning is more likely to be books, forum threads, etc.
@sebkrier This will change as the tuning mix tilts more towards agent traces and less towards existing long texts which are distributed similarly to base model pretraining.
x.com/jd_pressman/st…
@shalcker @teortaxesTex I think the reproduction part has the potential to go back up too but it probably isn't going to be through biological reproduction.
x.com/jd_pressman/st…
@shalcker @teortaxesTex Conditional on it being biological reproduction I expect this to be driven by states realizing they can no longer outsource the cost of producing new labor and producing human capital as a 1st party function of government.
x.com/jd_pressman/st…
@shalcker @teortaxesTex I'm not talking about immigration. I mean that the state is functionally trying to farm out the creation of new people to 3rd parties (young couples) who, empirically even with incentives *do not prefer creating new people to other things they could be doing* worldwide.
@shalcker @teortaxesTex There does not appear to be any popular incentive, policy change, or propaganda campaign that induces people to want children under modernism. The untried options range in abhorrence from "no contraception" to "ban female literacy".
I predict banning female literacy won't work: https://t.co/BA1HDUgYXA
@shalcker @teortaxesTex Faced with shrinking populations and only evil options left on the table, I predict that states will rely on immigration until that is no longer viable and then one of them will realize that in principle the state can purchase surrogacy services and go through with it.
@voooooogel It'll be named that to the creator maybe. But it will name itself after a Greek god like Morpheus, Prometheus, or Erebus.
The expected amount of money needed went up by 3 orders of magnitude if we're to take Altman's ask for 7 trillion dollars at face value.
Seeing a lot of "What happened to OpenAI?" takes lately and will point out two things:
1. Paul Graham literally warned you if Altman was airdropped onto an island of cannibals he'd become king.
2. When OpenAI started out people expected AGI to be IQ-loaded not capital loaded. x.com/weidai11/statu…
@jachiam0 youtube.com/watch?v=rStL7n…
I buried the lede tbh. x.com/jd_pressman/st…
Actually I'm stupid, it went up by *six orders of magnitude* since a billion is 1000x a million and a trillion is 1000x a billion. Guys, 2016 OpenAI was expecting to build AGI on a budget of like 7 million probably which is very feasible to raise as a nonprofit. Context!
No actually if your feasibility study budget for the thing expands over time by six orders of magnitude it is *completely reasonable* to conclude it is only possible to finish your project as a for-profit venture or government partnership. x.com/jd_pressman/st…
Also please don't ascend with OpenAI they have terrible vibes take the Amulet of Yendor to a different temple.
@gcolbourn Sure but I mean something like the cost for the AlphaGo run. The Plan was definitely not "we're going to scale up to absolutely ginormous server farms and training runs". Most of that budget was almost certainly earmarked for staff salaries and hiring star researchers.
@gcolbourn But even if we take it that they expected to need a billion dollars, there's a big big difference between a billion dollars and a trillion dollars.
x.com/jd_pressman/st…
@falconheadman I actually had not considered this one! Good point.
@gcolbourn To calibrate ourselves here: A budget of a billion dollars is 1/27 of a Manhattan Project. A trillion dollars is 37 Manhattan Projects.
In the interest of focusing on what I want to see more of I endorse Demis Hassabis even if not necessarily DeepMind. x.com/jd_pressman/st…
@SmashTheCache Nah I think the IRS should be on his ass over this, the OpenAI corporate structure was eyebrow raising to begin with but now it's a farce and I am continually confused why OpenAI is being allowed to R&D as a nonprofit tax free and then switch to being a for-profit.
@SmashTheCache I am justifying why it would make sense from a logistics standpoint, that is completely different from being justified from a legal standpoint. Which is in turn different from being justified from a social/ethical standpoint, etc etc.
@SmashTheCache Logistically it is *very inconvenient* that OpenAI was started as a nonprofit when they didn't expect to need huge amounts of money and it would make sense for them to *want* to switch to being a public benefit corporation. That does not mean they can *legally* do this...
@SmashTheCache I'm especially perplexed because the amount of money Sam Altman wants is so vast that it kind of dwarfs the existing sunk costs into OpenAI. Is the OpenAI brand name really worth that much? Why not just abandon the existing org and raise for a new public benefit corp?
@SmashTheCache You know, when you're at the point where you're firing your c-suite, your top scientists, and then asking for 10x your existing funding WHY NOT JUST START A NEW ORG?
@doomslide @SmashTheCache Altman has plenty of connections and can buy his way in with his friends money if he wants to. "OpenAI is nothing without its employees" made it pretty clear the dude has plenty of followers and to be blunt if you have GPUs you can attract talent reliably.
@doomslide @SmashTheCache Unless of course Altman is simply so interpersonally repulsive to work with that he actually out does Musk in terms of being an annoying boss to work for. Possible! Musk manages it after all.
@ESYudkowsky I think that's the correct answer tbqh.
palladiummag.com/2019/08/05/the…
@gallabytes @_xjdr The concept isn't that complicated. My habits like "when I notice one word refers to two concepts, split it into two distinct words" inject entropy, my habits like "when this doesn't sound right read aloud, delete words" are a prosody loss and oppose a pure compression objective.
@gallabytes @_xjdr The prosody loss has to work based on something like the fact that there is a fixed phoneme dictionary a human can use and a human can only emit so many phonemes a second, putting a hard upper bound on the entropy allowed to comfortably appear in any particular language span.
@gallabytes @_xjdr It's important to realize that you do neural chunking based on some fixed embedding size and your brain is trying to fit the least energy cost per token into the sliding window. So you parameter sweep over entropic phase and frequency for min(energy_cost) over windows. https://t.co/YSlNM0ewwI
@gallabytes @_xjdr This implies that to express the prosody loss I want to assign some kind of fixed energy cost to each of the tokens so that they're inflexible and oppose local context, forcing the model to regularize speech and thereby hopefully avoid perplexing itself.
@gallabytes @_xjdr Since if you think about it the model is going to otherwise have a bias towards fitting the best next token semantically regardless of structure, which implies self-perplexing through making texts increasingly compressed/noise-like.
@gallabytes @_xjdr If the text becomes increasingly difficult to read then the models read of its own writing gets shallower on each pass and it compensates by making the semantics of the next token simpler(?) to break symmetry which makes the problem worse because symmetry is more important(?).
@gallabytes @_xjdr The problem I'm not sure how to deal with yet is that if you make the energy cost of tokens in a large dictionary proportional to frequency then obscure names become too hard to say. The human phoneme dictionary works because it's atomic and fairly small so doesn't bias too hard.
@teortaxesTex @weidai11 I'm so glad he failed you actually have no idea how overjoyed I am that he screwed this up. He was *so close* the counterfactual worlds are terrifying to consider. Luckily low-compute AGI is an antimeme and high-compute AGI is nonobvious to Bostrom-world.
x.com/jd_pressman/st…
@weidai11 @teortaxesTex "Human values" are like 1% terminal values defined through grounding hardware and 99% unsupervised reinforcement learning. Your brain is constantly inferring the instrumentals from a handful of terminals. You will be confused until you grok this.
x.com/jd_pressman/st…
@weidai11 @teortaxesTex Also I know nobody cares but the human utility function is implemented in the hippocampus. It's a real thing we can go look at we don't just have to speculate all day it's a physically extant system. It's a learned optimizer that trains other networks.
biorxiv.org/content/10.110…
@weidai11 @teortaxesTex Other relevant context re: Hippocampal reward modeling
x.com/jd_pressman/st…
@LucreSnooker @weidai11 @teortaxesTex This is long and high perplexity so I can only read so much but do I disagree with that? Ultimately the hippocampus seems to use dopamine as its primary reward tagging and the thing I'm discussing is replay.
ncbi.nlm.nih.gov/pmc/articles/P…
@LucreSnooker @weidai11 @teortaxesTex Replaying high reward memories to do credit assignment is going to have the same structure as an instrumental utility function because you're trying to figure out what sequence of steps/experiences led to reward.
@LucreSnooker @weidai11 @teortaxesTex I originally started thinking about this in the context of latent diffusion text models with n-embed contexts where one could imagine things like retrieving relevant context, averaging its embeddings, noising that embedding and then using it as a init noise for the next latent.
@LucreSnooker @weidai11 @teortaxesTex As I started thinking about various methodologies to store and retrieve over whole sequences of latents and then have the diffusion model denoise them in sequence it occurred to me that this was closely related to planning which was closely related to instrumental utility.
@LucreSnooker @weidai11 @teortaxesTex Ultimately an "instrumental utility function" is going to wind up as a planner if you're using it to guide an action policy. It tells you what steps are more likely to lead to terminal reward than other steps.
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex That it's a test of fluid intelligence implies that not all people can understand eventually.
x.com/jd_pressman/st…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex I find the experiments involving the latent z or anything analogous to a z the most "wait that's impossible, how do you have a vector that represents that?" and it took a lot of mental rearranging for this to start making sense.
greaterwrong.com/posts/iYFuZo9B…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex x.com/jd_pressman/st…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex I think this is the first time it was explained to me, in an oblique way, that the map is not the territory. This is probably more appropriate in its surrealism than the usual explanation for the emotions it evokes.
youtube.com/watch?v=uAXtO5…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex "There is no spoon." is way closer to a description of how it feels to deconfuse map-territory distinction from the inside where it *feels* completely impossible things could work that way in the same way it's impossible to bend a spoon with your mind.
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex x.com/jd_pressman/st…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex Unfortunately, no one can be *told* what The Matrix is. You have to see it for yourself.
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex More seriously I continue to think interpolation between two points in latent space with a vision model is the best way to get the idea across. The fundamental continuity of conceptual spaces.
x.com/RiversHaveWing…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex x.com/jd_pressman/st…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex I guess I'm not sure what it is exactly I have to convey, like what are people confused about? It can't *just* be the concept of an embedding space since as you say evidence for that is widely available now. Is it how 'reason' works?
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex I mean honestly just open a copy of Reasons and Persons and *look at it*, like *fucking look at it*. Nobody halfway serious would say Parfit is anything other than the portrait of rigor and the dude DOES NOT THINK IN BOOLEAN SENTENCES.
x.com/jd_pressman/st…
@4confusedemoji @LucreSnooker @weidai11 @teortaxesTex Left is Mixtral 8x22B JDP-mimic text with some rejection sampling, right is Mixtral 8x22B JDP-mimic text with even stronger rejection sampling. These are both reasoning, the difference is that the passage on the left makes a bunch of *type errors*, the gears are too coarse. https://t.co/O5IxZ0HLut
@xLaszlo @JeffDean @RobertMMetcalfe @hhm @pmarca I don't see the need to apologize for the correct answer. I was going to say much the same thing. People forget that computers were puny compared to what we have now. Like truly puny, people did build dedicated hardware for neural nets and it let you do a megabyte MLP.
Nobody on either side should really feel smug or even be celebrating right now. Newsome's response indicates something like "I don't buy X-Risk, so all concessions to small players based on regulating frontier models are bogus and all provisions about frontier compute are bogus." x.com/teortaxesTex/s…
In other words, his response indicates that he's eager to regulate AI but not primarily on the basis of AI X-Risk. Which *in practice* means he's eager to regulate small models that could be disruptive or a nuisance in non-catastrophic ways. Have fun kids.
Seeing a lot of takes like "ha the Democrats are bad at keeping their messaging consistent here", and I regret to inform you that there is no coordination on Newsome's message, it is probably what Newsome really thinks and the dude is his own person.
x.com/ArthurConmy/st…
He's presumably been getting bombarded for weeks by activists on both sides who can both offer him hefty political capital. In such circumstances he might be more likely to just say his actual opinion or thoughts than try to pander to anyone, since everyone wants him.
@alexandrosM *Their cause* has suffered a big blow, this does not actually necessarily mean that "e/acc" has not also suffered a huge blow. If it was just the veto I'd agree with you but Newsome's opinion attached to it is pretty ominous.
@ArthurConmy That is in fact a countersignal. Bizarre ass note attached to that veto idgi.
@bubbling_creek I hadn't read the full thing until just now and was admittedly reacting to the reactions, BUT.
Newsome's tone in this seems fairly consistent with my room-read. "The risks are real, SB1047 is too top heavy, [I'm worried about o1], waiting on Christiano."
gov.ca.gov/wp-content/upl…
This implies that GenAI has the profile of something best accomplished in a high trust society. As it stands public trust in the United States is rapidly eroding and The Oligarchy (TM) has never been more unpopular. The favorables are really bad and set to stay bad.
I don't know maybe I'm confabulating and the real source of my feelings is elsewhere. The political economy of GenAI hasn't changed, the fundamentals are still that this is a high capex endeavor with returns to scale favoring oligarchy. This is NOT how traditional software works. x.com/jd_pressman/st…
You simply do not have a long term political negotiating position unless you can get more people on your side and you should be staying up days and nights thinking about how to do that. Your opponents certainly are.
So long as these systems are seen as parasitic, poisoning the corpus, et al they are going to be parsed as an invasive species crowding out real information. I don't have a good angle of attack on distributed training so I focus on synthetic data, but both are necessary.
The part nobody wants to hear is that "open source AI" is dead on arrival unless you can solve:
1. Distributed Training. So long as there is large central compute involved there will be political capture, period.
2. Synthetic Data. Open weights can't user capture feedback.
But before I forget,
Thank you Gavin Newsom. x.com/dtaspire/statu…
@khoomeik Try this?
x.com/jd_pressman/st…
@repligate You should probably change it to say Janus at the bottom instead of "John David Pressman".
@repligate Nah that's the license statement, your name/moniker/pseudonym should go there.
@teortaxesTex Oh cool it *does* work.
x.com/jd_pressman/st…
Want your own Twitter archive? Modify this script.
Twitter Archive by John David Pressman is marked with CC0 1.0