← All episodes · How I AI

How I run autonomous coding agents from my phone with OpenAI Symphony + Linear

2026-07-04 · original episode

Overview

Claire Vo talks with Alessio Fanelli, founder of Kernel Labs and co-host of the Latent Space podcast, about moving from being an "agent prompter" to an "agent manager." Alessio walks through his real, live setup: a cloud VPS called Zoo running always-on coding agents, wired to OpenAI's open-sourced Symphony framework, which uses Linear as the state machine and source of truth for every coding task. Instead of babysitting a terminal, he moves cards across a Linear board (to-do, in progress, human review, rework, done) from his phone, and Symphony spins up a Codex workpad, writes a plan and acceptance criteria, opens a PR, incorporates review comments, and merges — all trackable by token cost per task so he can start pricing how much a feature actually costs to build.

The conversation then shifts to a very different, very concrete use case: Alessio's actual side business, a trading card store called Merlin Games in San Carlos. He shows how he uses Codex with browser access and custom skills to autonomously fill in PSA certificate numbers for graded Pokemon cards, then hunt eBay for underpriced cards relative to his own inventory pricing — and describes a next project to price cards in real time at trade shows, where manual lookup costs real money in lost deal speed. The episode closes with a broader riff on AI unlocking small, heterogeneous, physical-world businesses (trading cards, vintage clothing, his father's fish delivery business in Rome) that were previously impossible to systematize with software, plus a lightning round covering personal finance AI agents, book recommendations, and how each of them resets when a prompt goes off the rails.

Topics

From agent prompter to agent manager

Alessio frames his whole approach around a shift he's made over the past year: moving from being an "agent prompter" — sitting there kicking off tasks and babysitting them — to being an "agent manager." He co-founded the Latent Space podcast three and a half years ago, and his co-host built one of the first autonomous coding tools ("Small Engineer") early on, but the underlying models weren't good enough yet to sustain long-running, multi-turn tasks. That's changed dramatically in the last year.

His first attempt at managing multiple agents was a simple Kanban board, but he found it broke down after two or three turns: it was easy to kick off a task, but hard to intervene partway through, and running things locally didn't work well either. The real unlock was moving his runtime off his laptop and into a VPS in the cloud, paired with multiple channels to communicate with the agents — texting them, using Linear, or prompting directly in a shell. He notes that Codex itself has recently added a mobile/cloud management experience, converging on the same idea.

Claire contrasts this with her own setup of four Mac minis running agents locally that she has to physically walk downstairs to restart, setting up the rest of the episode as a demonstration of why cloud-based, Linear-driven orchestration might be a better way to work.

Zoo: the VPS + pre-logged-in agent environment

Alessio's infrastructure is a tool he calls Zoo — essentially an agent plus a VPS. The example machine he shows has 32GB of RAM and 4 cores, with all his coding agents (Codex, Claude Code, and some open-source models) pre-logged in and ready to go. He uses it as his own persistent server so agents don't need to be re-authenticated or re-launched from a local machine each time.

On top of Zoo he runs OpenAI's Symphony, which he describes as a loop that turns issues into a coding runtime, using Linear as the source of truth. He demonstrates his Linear workspace, which contains separate projects (e.g., "Power Buyer" for his trading-card business), each with an original task/spec written in plain language.

The Symphony + Linear workflow, step by step

The demo walks through the actual mechanics: a task starts as a spec in Linear, gets moved into the "to do" column, and that move tells Symphony to start work. Symphony creates a Codex workpad where the agent drafts a plan, acceptance criteria, and validations, guided by a file called workflow.md that Alessio wrote explaining exactly what Symphony should do at each stage. The agent works the task and eventually moves it to "human review."

At that point Alessio reviews the resulting PR on GitHub, leaves inline comments (e.g., "this is too long"), and once he's added feedback he simply drags the Linear card to "rework." Symphony then automatically generates a rework checklist that addresses every comment line by line, and once satisfied he moves it to "done" and it gets merged. Crucially, he says he doesn't watch the agent traces turn by turn — he just directs it via the board. He live-demonstrates creating a new task from his phone-style workflow ("clean up Power Buyer, remove the spread column, make the set name clickable") and dropping it straight into the to-do column to kick off a new run.

He also shows each Symphony instance has its own dashboard listing running and past tasks with a token-usage ledger — most tasks cost around 15, 30, or 60 (implied thousand/million) tokens, but one outlier task (making the app deployable on Vercel, which required rewriting storage and request handling because it was originally built as a local-only app) cost 221 million tokens. Tracking token cost per task lets him start reasoning about how expensive a feature is to build and to price software work before committing to it, and to spot when a task's actual cost diverges wildly from expectations — which usually signals a tooling or spec gap worth fixing.

Setting up Symphony yourself, and the tooling-over-orchestration insight

Claire asks how hard it actually was to set up Symphony given people see the open-source repo and don't know where to start. Alessio says he used OpenAI's reference ("Elixir") implementation largely as-is; the real customization work was (1) writing the workflow file describing how the process should run, and (2) building his own UI on top, since Symphony itself ships without a visual dashboard or a built-in token-usage ledger — by default it's just a state monitor, not a cost tracker.

He argues the orchestration/hardness part is actually the easy part; the differentiator is building tools that make agent runs more effective and let them go longer without needing human intervention. His company Kernel Labs built a Playwright-based browser extension called Glimpse that lets coding agents take screenshots, do visual diffs between screenshots, and capture videos — specifically so agents can validate their own front-end work autonomously instead of bouncing back to a human constantly. He reiterates that tracking tokens and time per task is diagnostic: if you have a rough sense of whether a task should cost 10 million or 100 million tokens and reality is far off from that, it's a signal to invest in better tooling rather than better orchestration.

Claire adds her own framing for viewers: Symphony's actual spec file ("Symphony spec.md") is just a long, detailed markdown document written in natural language describing the primitives of the workflow — not some elaborate architecture diagram. She uses this to push back on people who send her giant, over-engineered orchestration platform proposals, arguing that a well-written spec file, combined with a strong model that will "lock to that spec," is often sufficient.

Markdown/agents.md hygiene and skill design pitfalls

Alessio makes a pointed observation about how agent instruction files degrade over time: models have a strong tendency to add instructions rather than remove them. As an example, he built a "work tree manager" before coding agents added that natively, and wrote into his agents.md that it must always be used. Later, when that was no longer necessary, rather than deleting the outdated instruction, the model would instead append a new line like "but you don't have to use it all the time" — compounding confusion rather than resolving it.

His recommendation is to periodically "purge" your markdown files — literally diff and prune agents.md and skill files every few months rather than letting them accumulate contradictory patches. He also cautions that the newer "create skill" features (like the one Codex's app added) can backfire: when skills are written vaguely/under-descriptively, the model becomes overly rigid about following the skill literally, which can actually produce worse outcomes than not having the skill at all. His own skill files, by contrast, are deliberately non-prescriptive — describing structure (e.g., "every Symphony instance has its own name, repo, workspaces, logs including token usage, and run state") and what to ask about if information is missing, without dictating exactly what to do.

What Symphony actually buys you: shaped context, not new capability

Claire asks directly whether Symphony makes Alessio more productive or just gives better ergonomics for managing agents on the go. He's candid that it's not really about raw output volume — in practice you can't meaningfully stay on top of 100 new tasks a day regardless of tooling. The real value is having the full history of a single task in one place: the original spec, the first work pad, the rework work pad, and every review comment, all attached to one Linear issue. That makes it possible to pinpoint exactly where something went wrong and feed that back into the workflow.md or agents.md for next time — something he says is much harder when working directly in a raw Codex/coding-agent conversation, where digging back through chat history to find what broke is painful.

His summary line: Symphony doesn't give you any capability you didn't already have using coding agents directly — it's purely a way to shape and structure context so you can wield the agents more effectively.

Pokemon card business: PSA certificate lookup and eBay underpricing hunts

The episode's most visually concrete demo is Alessio's Pokemon card side business, run through a store called Merlin Games in San Carlos and tracked via his "Power Buyer" Linear project. He uses Codex for two specific autonomous goals. First, filling in PSA certificate numbers: each professionally graded card has a unique PSA certificate number accessible via an API, but there's no bulk-download path, so he tasks Codex with a goal ("fill out the certificate number for every card that costs more than $1,000"), gives it browser access, and it goes out onto the web, finds the right images, and extracts the certificate numbers itself.

Second, he uses a custom skill (which he shows live) to have Codex search eBay for underpriced cards relative to his own inventory/pricing list, batching searches five at a time deliberately to avoid getting rate-limited or blocked by eBay. He explains the skill needs to encode cross-grading-company equivalence rules (e.g., a PSA 10 corresponds to a BGS or CGC 10) so the agent can compare listings apples-to-apples. He also points out a small but telling UI bug live on screen: Codex ties a "$" character to skill invocation, so pasting a URL containing "$" (as in a TCGPlayer link) accidentally triggers skill lookup instead of being read as a plain link — a reminder that even small software details break in agent tooling.

He frames the broader business logic: trading cards are an inventory-based business, so the more capital you can deploy into finding and buying underpriced inventory, the more revenue is possible — making automation of the sourcing/pricing legwork directly valuable, not just a nice-to-have.

Real-time trade-show pricing as the next automation target

Alessio describes an in-progress project to solve a harder, higher-stakes problem: at trade shows, people walk up wanting to sell cards on the spot, and pricing them requires manually searching each card on eBay or TCGPlayer to find comparable sales — a slow process that costs real money because slow pricing means lost or worse deals in the moment. He argues this is actually a better example of AI's value than headline-grabbing long-running autonomous tasks: for this use case, what matters isn't raw task quality over hours, it's compressing real clock time for a human standing at a table, so AI-assisted lookup directly translates into completed, well-priced deals rather than time saved on writing code.

AI unlocking small, physical-world, heterogeneous-data businesses

Both hosts converge on a broader thesis: AI's most underrated positive outcome is enabling small business creation, particularly for businesses built on messy, heterogeneous real-world data that was previously too labor-intensive to systematize with traditional software (which requires clean, structured inputs). Alessio contrasts this with big enterprise software, where even at massive scale (his Salesforce example) there's a ceiling on how many features you can ship per week — versus a two- or three-person business that can now get outsized leverage from AI doing work a human previously had to do by hand.

He gives his father's fish delivery business in Rome as an example: someone manually walks the freezer every morning with pen and paper, and then discovers mid-day that they're missing three tuna or a box of shrimp — a process that could now be automated with something as simple as smart glasses doing inventory recognition. He connects this to a broader pattern he's noticed in Japan, where small two- or three-person businesses are common and well-loved, and hopes the same resurgence happens in the US.

Claire shares her own parallel example: she and her family have roughly 600 books scattered throughout her house (more than 100 books per person), and she used Gemini to go through photos of every pile, catalog them into categories, tag their physical location, and find duplicates — something previously impossible to do by hand at that scale. She frames this, and Alessio's trading-card and fish-shop examples, as evidence that AI's biggest quality-of-life and small-business unlock is intersecting the messy human/physical world with automation in a way that was historically far too inefficient to attempt.

Lightning round: personal finance AI, context offloading, book picks, and recovering from bad prompts

Asked what he's excited about that most people aren't doing yet, Alessio points to personal finance: ChatGPT recently added account connectors, and after selling his house he used it to help decide what to do with the proceeds. He frames the value less as "AI adding brilliant new ideas" and more as a stress-reducing safety net — previously he'd worry about not checking on his investments enough or missing something, and AI monitoring removes that background anxiety. He describes a similar pattern using a now-defunct tool called Wafer that had unlimited weekly tokens on open-source models, which he used to scan his Gmail every 5 minutes and flag anything actually worth responding to, eliminating the compulsive urge to check his inbox. He calls this pattern "context offloading" and predicts more people will adopt it.

For books, he recommends The Monk and the Riddle, a short story about a VC meeting a founder who wants to build an unglamorous but lucrative funeral-home startup specifically so he can cash out and later do what he actually loves — a pattern Alessio sees often in founders who chase what's expected of them rather than what they're passionate about. He also cites The Divine Comedy by Dante (mandatory reading for three years in Italian schools) as a reminder of the heights of human creativity, even in the 1200s.

On recovering when a prompt goes off the rails, his approach, in order: swear at it a couple times, then try a different model provider since he's subscribed to all of them, restart the conversation from scratch, break the problem into smaller pieces, and — his favorite trick — switch from typing to speech-to-text once he gets frustrated, since talking out loud tends to surface a few more useful details than typing does. Claire adds a parallel anecdote from a past guest who calls this the "yapper's API": talk until it's all out, hit enter without even rereading it, and that's often the most effective prompt.

Visuals

1. Write spec in LinearTask is described in plain language as a Linear issue inside a project (e.g., Power Buyer).
2. Move card to To DoMoving the issue tells Symphony to begin work on it.
3. Symphony creates a Codex workpadAgent drafts a plan, acceptance criteria, and validations per the workflow.md rules.
4. Agent implements the taskRuns autonomously on the Zoo VPS; token usage and time are logged per task.
5. Move to Human ReviewAlessio reviews the resulting PR on GitHub (or Vercel preview) and leaves inline comments.
6. Move to ReworkSymphony auto-generates a rework checklist addressing every review comment line by line.
7. Merge to DoneOnce comments are resolved and the PR is merged, the card moves to Done.

15-60M

Typical token cost range for a normal Symphony task

221M

Token cost for an outlier task (making a local-only app deployable on Vercel)

600

Books Claire catalogued in her house using Gemini (over 100 per person)

5

Cards batched per eBay search to avoid getting rate-limited/blocked

Takeaways

• Move agent runtimes off your local machine and into a always-on cloud VPS with agents pre-authenticated, so you can manage tasks from your phone instead of restarting things locally.
• Use Linear (or any kanban tool) as the actual state machine for agent work: to-do triggers a plan/workpad, human review is where you leave PR comments, and dragging to rework auto-generates a checklist addressing every comment.
• Track token usage per task as a diagnostic signal — if a task's actual cost is wildly higher than your expectation (e.g., 221M vs. a typical 15-60M tokens), that's a sign to invest in better tooling or a tighter spec, not just to accept the overrun.
• Periodically diff and purge your agents.md/skill files: models tend to append contradictory caveats rather than delete stale instructions, so unmaintained markdown files degrade agent behavior over time.
• Write skills and workflow files as loose structural guidance (what fields/state must exist, what to ask about if missing) rather than rigid step-by-step scripts — overly prescriptive skills make models too literal and brittle.
• When a prompt session goes off the rails: try a different model, restart the conversation fresh, decompose the task into smaller pieces, or switch to speech-to-text — talking out loud often surfaces details typing misses.

Notable quotes

What really clicked for me was starting to move away from being an agent prompter to being an agent manager.— Alessio Fanelli

It's not giving you any new capability that you wouldn't have by using the coding agents directly. It's just helping you wield it.— Alessio Fanelli

If you're not getting enough failures, you're probably not trying hard enough. You're not being ambitious enough in what you're doing.— Alessio Fanelli

This is my favorite positive outcome of AI, which is small business creation... the ability to intersect the human world in a way that has been historically very inefficient has been a quality of life improvement for me.— Claire Vo

When people talk about AI response time for these long-running tasks, it's actually not that useful, but you can actually use AI to save clock time for real people by doing these things autonomously.— Alessio Fanelli

I think a lot of folks have been talking about purging your markdown files now every few months. I think that's something that obviously makes sense.— Alessio Fanelli