← All episodes · Every
LET'S RIP FABLE TOKENS FROM THE JACUZZI
Overview
Dan Shipper, founder of Every, hosts an impromptu livestream from a hot tub in Cabo to celebrate the return of "Fable" (Every's internal nickname for the newest, most powerful Claude coding/agent model) after it was briefly pulled offline due to a government export-control order and then restored. The stream is low-key and jokey in tone, but it doubles as a real-time product test: Dan fires off several actual Fable prompts (reviewing months of failed copy-editing ML experiments, and compiling a personal AI token-spend report) while narrating what happens, including hitting an apparent safety/guardrail trip that reverted the model to a weaker fallback mid-task.
Partway through, colleagues join the call: Kate (Every's editor-in-chief, the human benchmark behind their internal "Kate Bench" copy-editing project), Mike Taylor (head of AI tech consulting, calling in from the AI Engineer conference in San Francisco), and briefly Nicholas in the background. The conversation ranges from the mechanics and philosophy of trying to automate expert copy-editing, to reactions to Fable's temporary suspension and restoration, to first impressions of Anthropic's newly released Sonnet 5 model, which the team's internal "vibe check" panel rated surprisingly poorly except for one dissenting reviewer.
Throughout, the episode serves as a snapshot of how a small AI-native media company actually uses frontier models day to day: benchmarking against a real expert's judgment, worrying about cost and safety-classifier false positives, and forming opinions about new model releases through internal debate rather than official benchmarks.
Topics
Fable is back, and the safety-classifier question
Dan opens the stream from a hot tub in Cabo, thrilled that "Fable" (Every's nickname for the newest most capable Claude/coding model) has reappeared in his model selector after being pulled offline. He struggles for several minutes to get screen sharing working, eventually giving up and narrating instead. His central open question going in: is this restored version of Fable — which he refers to as "Lutnik's version," a joking reference to Commerce Secretary Howard Lutnik and the export-control action — meaningfully lobotomized compared to before?
Dan explains that even prior to the suspension, Fable's safety classifiers were already triggering on innocuous requests, interrupting work. He wants to find out whether the restored version has tightened those guardrails further. He references a public post from Anthropic explaining what happened: on Friday, the US government applied export controls to Anthropic's newest models (referred to in the stream as "Claude Table 5" and "Claude Mythos"), requiring Anthropic to restrict access for foreign nationals; because the order took effect immediately and there was no reliable way to verify user nationality in real time, Anthropic suspended access for all users. The controls were later lifted and access restored. Anthropic's post also noted the safety classifier now has a larger margin for what it considers potentially harmful.
The Kate Bench copy-editing experiment: five weeks, hundreds of agent threads, and an unvalidated target
Dan's first real Fable task is to have it review months of failed internal experiments aimed at getting a language model to copy-edit as well as Kate, Every's editor-in-chief. The project (informally called "Kate Bench") used a corpus of roughly 30,000 of Kate's historical copy edits, later clarified by Mike Taylor as drawn from several thousand documents. The team set a goal of 70% strict recall against Kate's exact historical edits, then spent five weeks and (Dan cites two different figures during the stream, 180 and later 680) agent threads trying to hit that target, without ever validating whether even Kate herself could reproduce her own past edits at that rate.
Fable's review, read aloud by Dan, concludes that the program's engineering and honesty were 'genuinely excellent' and that the corpus, benchmark discipline, and negative-result documentation were rare and valuable — but that the team optimized against an unvalidated target. Scattered evidence in the repo suggests Kate likely can't hit 70% consistency against her own past edits, meaning the experiment's 'failures' at 47.5-50% recall may actually be near the human ceiling. Meanwhile, two things that did work well were sidelined: a 'band condition repair' technique (Dan cites a cost around $529) and a live suggestion loop that a team member named Dan had rated 52 out of 53 useful. Fable also flagged that the single cheapest, highest-value next action — one human (Kate) spending two hours judging 60 cases — had been sitting idle since June 23rd.
Dan and Kate discuss what this means: detecting which edits a model should make (i.e., which parts of a paragraph even need attention) is the hard, unsolved part, while executing a known fix at an already-identified site is comparatively easy and learnable. Mike Taylor, joining later, adds that from his own past work with 'AI personas' modeling a specific person's judgment, exact reconstruction accuracy tops out around 60%, but predicting something a person would plausibly agree with is a more tractable and more useful target. The group converges on reframing the benchmark: instead of measuring strict recall against Kate's historical edits, measure whether Kate would agree with an edit Fable proposes — turning her live editing sessions into a labeling pipeline for future model training.
Designing a human-in-the-loop editing workflow
Dan and Kate work through what an actual production workflow would look like if a model reviewed drafts alongside her. Kate is protective of reading a draft "unfiltered" the first time, before seeing AI-suggested edits, because by the time a piece reaches her for a top edit it has usually already been through her editorial team, and she wants her own uncontaminated reaction preserved, even though she acknowledges this preference may not be strictly rational since she is not literally the first reader.
They land on a compromise: rather than showing Kate the model's edits inline as she reads (which she says wouldn't necessarily 'ruin it' for her, since editors already sometimes leave comments for her to weigh in on), the model could run concurrently and independently — producing its own version of edits at the same time Kate does hers — and the two versions get compared afterward. Kate calls this 'a really good idea' and likes that it preserves her unfiltered first pass while still generating a comparison dataset. Dan notes this concurrent-run approach also solves the data problem: it creates paired human/model edit examples without disrupting Kate's process, and can be extended to a production deployment with a database so results aren't lost.
Token-spend reporting hits a safety guardrail mid-task
Separately, Dan's sister Ariel, who runs operations at Every, has been pushing him to produce a report on his personal AI token spend across ChatGPT/Codex, Claude/API, and Google AI Studio for the past three months, plus a webpage to visualize it, since the company now tracks costs more formally. Dan sets Fable on this task live during the stream.
About 20 minutes in, the task trips a guardrail: Fable attempted to kill a process on Dan's computer and do some work in Chrome, and this combination silently caused it to revert from Fable down to a weaker fallback model (referred to as Opus 4.8). Dan finds this notable because the command seemed "very innocuous" — he speculates that actions involving controlling the computer (killing processes, browser automation) may be treated as more sensitive by the classifier than pure conversation. He tests whether manually flipping the model selector back to Fable lets it resume the same thread, and confirms that it does pick up on the same conversation after being flipped back manually.
A viewer's comment prompts a side discussion about persistent memory: another user had reported that Fable kept getting tripped up because it remembered unrelated biology content stored in its memory from being a biologist, which then triggered safety checks in unrelated contexts. Dan says he is 'famously against having memory turned on' for exactly this reason — he wants tight control over what context and tokens go into a model, especially given how expensive Fable is to run.
Guest Mike Taylor's plans for Fable: a DSPy book, viral-content rewriting, and code review
Mike Taylor, head of AI tech consulting at Every, calls in from the AI Engineer conference in San Francisco (his fourth time attending; he says the event has grown enormously, to the point that a chunk of attendees temporarily left to go test Fable the moment it relaunched). He lays out his own 'hit list' for Fable: first, since he's mid-draft on his second book (about DSPy, a prompt-optimization framework) and deliberately avoided using AI to write it, he wants Fable to review the finished draft for anything meaningful he might have missed — he notes Fable seemed unusually good at judging what mattered in a large amount of context, better than prior models, though he'd still trust human judgment over it.
Second, he wants to revisit his self-published first book on memetics (how things go viral), which only sold about 200 copies; he got feedback years ago that it needed to be rewritten in a more viral, influencer-style format, and now wants to hand the whole manuscript to Fable to see what it can do with making it more broadly appealing.
Third, he describes being hesitant to contribute code to Rally, a side-project AI focus-group tool he built with a technically stronger CTO co-founder, because the codebase structure feels unintelligible to him. He wants to use Fable for code review on a backlog of his own pull requests to build confidence that he isn't breaking his co-founder's more sophisticated setup — joking that if Fable approves the code, he can just blame it if something goes wrong. Dan and Mike also recall how a colleague, Kieran, was on pace to spend $1.5 million on Fable at one point because it was visibly improving his product's quality in real time, which gave Mike more confidence to consider shipping production code himself while on vacation.
Early, divided reactions to Anthropic's new Sonnet 5 model
Toward the end of the stream, Dan and Kate discuss Every's forthcoming 'vibe check' review of Sonnet 5, a separate new Anthropic model released around the same time as Fable's return. Kate reports that internal reactions have been unusually and uniformly negative — using Every's internal red/yellow/green rating scale, responses were almost entirely 'red,' with reviewers reportedly asking 'who is this for?' The one holdout was senior editor and builder Jack Chang, who rated it green; his explanation was that he uses models more collaboratively in short, iterative chunks rather than handing off long autonomous tasks the way colleague Kieran does, and in that collaborative mode Sonnet 5 performed well.
Dan draws a parallel to Anthropic's earlier model 4.7 (referred to here alongside 4.8), which also launched without early access for Every and got a lukewarm reception at first, only for the team's opinion to warm up about a week later as people found new ways to use it. He notes both 4.7 and Sonnet 5 shipped without an early-access preview (EAP) for Every, unlike stronger releases, which he speculates may signal less internal confidence at Anthropic about these releases. Dan says he's curious to see how Sonnet 5's reputation evolves over the next several weeks, especially if people end up using it indirectly through Fable acting as an orchestrator rather than calling it directly.
Every's plug: prompt library, and what the company does
Dan repeatedly points viewers to Every's public Fable prompt library (a URL he reads aloud with dashes, hosted on their site), which contains prompts Every staff actually use to work with Fable, including ones contributed by podcast guest Mike Kuger from around the time Fable first launched. He describes Every as 'the only subscription you need to stay at the edge of AI,' covering three areas: ideas (a daily newsletter, plus 'vibe checks' — Every's own hands-on commentary and results testing new models for what they're good and bad for), apps (an internally built suite including Kora, an AI email inbox; Sparkle, a file organizer; and Monologue, a Whisper-based speech-to-text app), and training (livestreams like this one, courses, and enterprise AI consulting, which is Mike Taylor's role).
Visuals
Takeaways
- Automating expert judgment (like copy editing) is bottlenecked by detection, not correction: identifying which parts of a text need attention is far harder than fixing a known problem site once it's flagged.
- Before optimizing an AI system against a human benchmark, validate that the human is even consistent with themselves — Every spent five weeks and hundreds of agent threads chasing 70% recall against Kate's past edits without first confirming Kate could hit that against her own historical judgments.
- A better target than 'exact match to a human's past decision' is 'would the human agree with this suggestion now' — matching Mike Taylor's finding from AI-persona work that reconstructing someone's exact past action tops out around 60% accuracy, while predicting a plausible, agreeable action performs much better.
- Running a model concurrently and independently alongside a human expert, then comparing afterward, preserves the human's unfiltered judgment while still generating a valuable paired dataset for evaluation and training.
- Persistent model memory can be a liability: unrelated stored context (e.g., biology-related memory) has reportedly triggered unrelated safety-classifier trips, which is why Dan deliberately keeps memory off and tightly controls what context goes into expensive model runs.
- New model releases shipped without early access often get lukewarm initial reactions that improve over following weeks as users discover new use patterns — this happened with Anthropic's 4.7 and may repeat with the newly released Sonnet 5, which got an unusually uniform negative internal reaction except from a reviewer using it collaboratively in short task chunks rather than long autonomous runs.
Notable quotes
It spent five weeks, 180 agent threads optimizing a target that was never validated. Strict recall against Kate's exact historical edits with a 70% goal set before anyone measured whether even Kate can hit 70% against herself. The evidence scattered through the repo says she can't. So the sealed failures at 47.5 and 50% may actually be near the human ceiling.— Dan Shipper (reading Fable's output)
I don't know if they would consider it a good thing or a bad thing that we named the variant after him, but I think it's true.— Dan Shipper
I felt a little bit like those old Geico commercials where it's the caveman going around — I was just in the stone age here using very simple tools when I could be in the space age. It's good to be back.— Kate
I would say that like the important thing is: would Kate agree with the edit once Fable has made it. I think that's maybe what we can design a study around.— Mike Taylor
It was just kind of not just meh — it's actually just worse and it's not good at this and it's not good at this. It was actually a genuine question of who is this for?— Kate
People often don't like a model and then decide to like it later, because when you don't like it initially what that usually means is the way you're trying to use it is not better than a previous model — but then people find new ways of using it with new powers they didn't realize were there, three or four weeks later.— Dan Shipper