← All episodes · AI Jason
OpenClaw Creator's new secret project...
Overview
Jason walks through Crabbox, a new tool from Peter Steinberger (creator of OpenClaw) built to solve a problem Jason says he's hit personally: running 10-15+ parallel coding agent sessions makes verifying and merging their work the real bottleneck, not writing code. He explains how tools like Playwright already help agents self-test and produce evidence (screenshots, videos) for PRs, but that approach breaks down once many agents need to run dev servers, databases, and schema migrations at the same time on one local machine, since resources like Docker daemons, databases, and hardcoded ports end up shared and easily corrupted across sessions.
The proposed fix is to give each agent its own isolated cloud sandbox with its own dev server and database, rather than sharing a local machine. Jason details why this is hard to build by hand (spinning up a cloud box, mounting disk, copying code, installing dependencies, starting the app, tearing it down) and, worse, why a naive commit/push/CI loop is a poor fit for iterative agent debugging since it pollutes the repo with throwaway commits and can't quickly resync a local fix into the sandbox.
He then demonstrates Crabbox end-to-end on a real app that depends on a local Supabase instance: writing a Dockerfile, a crabbox.yaml config (provider, sync excludes, env vars), and a setup.sh script; using Daytona as the sandbox provider (creating a snapshot, connecting an API key); warming up a box, running commands (with diff-sync or --no-sync), tunneling ports for manual testing, and finally having the agent run a full Playwright-based end-to-end test skill that returns screenshots/video as evidence that can be published to S3 or attached to a GitHub PR. He closes by noting Crabbox and a related 'codebase harness' skill are open source, and plugs a deeper workshop on building agent loops in production codebases.
Topics
The real bottleneck: reviewing and merging parallel agent PRs
Jason opens by pointing to Peter Steinberger (OpenClaw's author), who is now building a tool called Crabbox to help people running hundreds of parallel agents verify that agent work actually functions. Jason relates this to his own workflow: at almost any given time he has at least 10 different agent sessions running, and a photo of Peter shows him running at least 15 sessions in parallel. This produces a massive volume of PRs that would have been impossible to generate before agents, but it also creates a heavy review burden, since every PR carries the risk of breaking the system.
His conclusion is that the bottleneck in AI-assisted engineering has shifted: it's no longer about writing code, but about getting code merged safely into the codebase. He frames this as the reason for building a proper 'codebase harness' — infrastructure that lets agents prove their work is correct before a human has to review it.
Giving agents tools to verify their own work (Playwright)
The first step of a codebase harness, Jason says, is giving agents tools and skills to verify their own work and produce evidence. He calls Playwright probably the best option for this: it lets an agent drive a browser to test itself and also output artifacts like video recordings. Those artifacts can then be included directly in a PR as proof, making it much easier for a human to trust and merge agent-authored changes.
He notes this setup worked fine when only three or four agents were running at once, but it starts to break down at higher concurrency — setting up the actual problem Crabbox is meant to solve.
Why parallel agents break shared local dev environments
Jason explains concretely why running many agents in parallel on one local machine causes conflicts. Cloud Code-style work trees are fine for isolating code writing/editing, since each agent works in its own tree. But testing is different: agents need an actual dev server running against their work tree's code, and that's hard to isolate locally. Many codebases hardcode ports for good reason, so you can't simply spin up N instances of the app side by side.
Worse, shared infrastructure like a local Supabase instance means there's really only one Docker daemon, one database, and one OS on a given machine — all agent sessions end up sharing that same instance. If one agent tries out a new database schema migration, it can easily break every other session testing against the same database. Even setting aside these conflicts, running a full modern production repo (dev server, database, dependencies) consumes significant local resources, so this approach doesn't scale as you add more concurrent agents.
The right fix: per-agent isolated cloud sandboxes, and why it's hard to build
Jason argues the correct solution is to stop running all agent sessions in parallel on one local computer, and instead give each agent its own isolated cloud environment with its own dev server. Each agent effectively owns its own sandbox, complete with its own database and everything else needed to start a real dev server, fully isolated from every other agent's sandbox.
But building the pipeline for this is non-trivial: you need to spin up a cloud computer on demand, mount a disk, copy the code in, install dependencies, start the app, open a browser to test it, and then tear the sandbox down and delete it afterward. Jason says his team actually built this process themselves and it worked, but it still had problems: if agent testing on the box surfaces a bug, there's no easy way to get the resulting local code fix back onto the sandbox. A normal commit-push-CI flow doesn't work well here either, because it litters the repo with unnecessary throwaway commits, and rebuilding the whole sandbox from scratch every time a fix is made is wasteful. What you actually want is to make a change and retest within seconds — which sets up Crabbox as the answer.
How Crabbox works: warm up, run, sync diffs, stop
Crabbox is a tool that lets an agent warm up a cloud box, sync the 'dirty diff' from the local work tree folder (uncommitted changes included, as long as the folder is git-initialized), and run tests against it in real time. The core commands are: `crabbox warmup` to spin up a box for a session, and `crabbox run` to execute any bash command as if running locally — except it executes in the cloud box, syncing the local diff first every time. When finished, `crabbox stop` (referred to as 'crapbox.box'/'crapbox.stopbox' in the transcript) turns off and deletes the box.
The intended agent loop is: warm up a sandbox, run a setup command to install dependencies and start the dev server, then run predefined tests or use a tool like Playwright to do end-to-end testing. If a bug surfaces, the agent (or developer) fixes it locally, and the very next `crabbox run` automatically re-syncs the latest change to the cloud box before running — so you're always testing against the current version without rebuilding the sandbox. Multiple sessions can run this way in parallel with no conflicts, since each has its own isolated box.
Configuration: Dockerfile, crabbox.yaml, and setup.sh
Setting up Crabbox for a codebase requires three main pieces. First, a Dockerfile that encapsulates everything your local computer has installed — Node, Docker, any required CLI tools — which you can typically just prompt an agent to generate. Second, a `crabbox.yaml` config file that defines the sandbox provider, files/folders to exclude from sync (heavy or unneeded folders like past evidence/asset files — separate from things already covered by .gitignore like node_modules, .next, .turbo, and env files), and which environment variables get passed into the sandbox (pushed over an encrypted SSH connection, described as relatively safe). Third, a `setup.sh` script that lets the agent run one script to fully spin up the dev server rather than executing many manual steps.
Crabbox also ships commands for generating proof artifacts: an `artifacts` global command automatically downloads matching artifacts once a run finishes (useful after an agent's end-to-end test script completes), plus specific commands like `artifacts collect` (screenshot) and `artifacts video` (screen recording in the cloud), and `artifacts publish` to upload directly to an S3 bucket so images/video can be posted inline as PR comments.
Live setup walkthrough with Daytona as the sandbox provider
Jason demonstrates the setup on a real web app with a login flow and an agent chat interface, which also runs a local Supabase instance — meaning multiple parallel work trees would normally collide on that same Supabase instance. He builds a Dockerfile installing the Supabase CLI and other required tools, then a `.crabbox.yaml` selecting Daytona as the provider (chosen because its sandboxes start up quickly), specifying a Daytona snapshot (a predefined image), a default work root, sync excludes, and environment variables.
He walks through Daytona-specific setup: `daytona login`, listing organizations with `daytona organization list`, selecting one with `daytona organization use`, then creating a snapshot via `daytona snapshot create` pointing at the Dockerfile and context folder, with defined CPU/memory/disk and region. Connecting Daytona to Crabbox required generating a Daytona API key and setting it as a local environment variable, after which `crabbox warmup --provider daytona` with a session slug/ID spins up a sandbox for that session.
Helper script (cbx.sh), background execution, and handling provider timeouts
Jason built a wrapper script called cbx.sh that bundles several Crabbox commands into one command for convenience — e.g., an `up` command that spins up the box, polls status until ready, and then runs `setup.sh` to install dependencies and start the dev server. Because providers like Daytona time out commands by default (around 60 seconds), long-running setup needs to run in the background with status polled every ~10 seconds until complete. This means the agent can simply run `bash cbx.sh up` and release control while setup finishes, rather than issuing each command manually and waiting.
Once the box is ready, commands run via `crabbox run --provider daytona --id
SSH tunneling for manual testing vs. agent-driven Playwright testing
Jason shows that once a Crabbox sandbox is running, you can open an SSH tunnel to it: one command retrieves the sandbox key, and a second forwards local ports (e.g., local port 3000 and a backend port) to the corresponding ports on the sandbox. This lets a human open a browser tab locally and interact with the sandboxed dev server directly, which he says is useful even outside of agent testing.
However, for his actual parallel-agent use case this isn't ideal, since tunneling still occupies local ports and can conflict across sessions. Instead, he prefers having the agent run Playwright CLI/bash commands in the cloud sandbox and bring back evidence, or run a predefined end-to-end test. He defines a 'crabbox test' skill for the agent that spins up a dev server in the cloud, then uses Playwright CLI to run a full end-to-end test via a predefined `pw` command (with ID, provider, and `--no-sync` flag). Simply telling the agent "help me test my web app via crabbox" triggers the box spin-up, dev server start, and full end-to-end Playwright test, returning screenshots or video evidence. This evidence can be uploaded to S3 or attached via GitHub's release-assets feature to be included inline in a PR, and arbitrarily many parallel sessions can run this way without conflicts.
Open source availability and further resources
Jason says Crabbox is open source, and he has created a 'crabbox setup' skill that can be given to Claude Code or Codex to walk through the entire setup process automatically — writing the Dockerfile, connecting the provider, creating the crabbox.yaml, and so on. He also mentions a broader 'standard codebase harness' skill that goes beyond Crabbox setup to cover other things that help agents deliver work more efficiently in a codebase. He closes by plugging a more in-depth, step-by-step workshop ('build club') covering how he builds agent loops in real production codebases, linked below the video.
Visuals
- crabbox warmupSpin up an isolated cloud sandbox (e.g. via Daytona) from a predefined snapshot/Dockerfile
- crabbox run setup.shSync uncommitted diff to the box, install dependencies, start the dev server and database
- Agent tests (Playwright)Agent runs end-to-end tests or bash commands in the cloud box, optionally with --no-sync
- Fix bugs locallyIf a bug surfaces, developer/agent edits code locally
- Re-run crabbox runLatest local diff auto-syncs to the box before the next command runs
- Collect artifactsartifacts collect/video/publish capture screenshots/video and upload to S3 for PR evidence
- crabbox stopTear down and delete the sandbox
| Local shared environment | Crabbox cloud sandbox | |
|---|---|---|
| Database/Docker daemon | Single shared instance across all agents | Isolated per-agent instance |
| Ports | Often hardcoded, conflicts across sessions | Each sandbox independent, no conflicts |
| Schema changes | Can break every other running session | Contained to one sandbox |
| Fix-and-retest loop | Requires commit/push/CI, pollutes history | Auto-syncs uncommitted diff, retest in seconds |
| Resource usage | Consumes local machine resources, hard to scale | Scales via on-demand cloud boxes |
Takeaways
- As agent parallelism scales past 3-4 concurrent sessions, the bottleneck shifts from writing code to safely verifying and merging it.
- Local dev environments break down under parallel agents because shared resources (one Docker daemon, one database, hardcoded ports) let one agent's schema change corrupt every other session.
- Give agents self-verification tools (like Playwright) that produce artifacts — screenshots, videos — so PRs ship with evidence instead of requiring blind trust.
- Isolate each agent in its own disposable cloud sandbox with its own dev server and database rather than trying to share a single local machine.
- Avoid the commit-push-CI loop for iterative agent debugging — it pollutes history with throwaway commits; instead sync uncommitted diffs directly to the sandbox for instant retesting.
- A codebase harness needs three artifacts to work: a Dockerfile encapsulating your toolchain, a config file (provider, sync excludes, env vars), and a setup script the agent can run end-to-end.
Notable quotes
It has been pretty clear that the bottleneck is no longer write code, but actually how do you get code merged into the code base.— Jason
The right solution here is actually, instead of running all the sessions in parallel on your local computer, you want to set up an individual isolated environment to start dev server for each work tree.— Jason
Crabbox is a tool that allows agent to warm up a box in the cloud, sync the dirty diff from the local work tree folder, and run the test in real time.— Jason