Building a Self-Improving Agent Harness on Inngest

New agent frameworks come out every few weeks, and every few weeks I read the docs and find the same thing underneath: a while-loop that calls an LLM, runs some tools, and hopes the process doesn't die halfway through. The other 20,000 lines are abstractions for problems you probably don't have yet and may never get.

We run around 200 Inngest functions in production at Driive, so I'll admit some bias, but here's my take: you don't need an agent framework. You need a durable while-loop. And Inngest's own team quietly published the best proof of this I've seen in Utah, the "Universally Triggered Agent Harness." No framework. A think/act/observe loop where Inngest provides durability, retries, and observability, and a thin LLM interface does the rest.

I want to walk through why I think this design is right. The "self-improving" part, where the agent writes its own durable functions and hot-deploys them, is the most interesting piece, so I'll spend the most time there.

(All code below is from inngest/utah, Copyright 2025 Inngest, Inc., licensed under Apache-2.0. I trimmed some excerpts for length.)

An agent loop might run for ten minutes across thirty LLM calls and a pile of tool executions. In that window, the LLM API will rate-limit you, your process will get redeployed mid-loop, the user will send a second message that contradicts the first, and at least one tool call will throw. In a naive loop any of these kills the whole run, and the retry re-executes everything, including the tools with side effects. Now you've emailed the customer twice.

The fix isn't a framework, it's durable execution: make every step of the loop a checkpoint, so a crash resumes from the last completed step instead of from zero. That's literally what Inngest's step.run() does. Each step's result is memoized, and on retry, completed steps return their cached results instead of running again.

Utah's entire "think" phase:

// Think: call the LLM via pi-ai
const llmResponse = await step.run(
  "think",
  async ({ systemPrompt: sp }: { systemPrompt: string }) => {
    return await callLLM(sp, messagesForLLM, tools, { useFallback });
  },
  { systemPrompt },
);

That's the whole abstraction. The LLM call, probably the flakiest network operation you can make these days, is wrapped in a durable step. If it fails, Inngest retries it. If the process dies between iterations, then on replay think:0, think:1, think:2 come back from cache and execution picks up exactly where it stopped, because Inngest auto-indexes repeated step IDs across loop iterations. There is no separate agent state persistence layer. The function's own execution history is the state.

Tool execution gets the same treatment. Validated, checkpointed, and the result appended as an observation:

toolResult = await step.run(
  `tool-${tc.name}`,
  async ({ name, id, args }: ToolStepInput) => {
    const tool = tools.find((t) => t.name === name);
    if (tool) {
      validateToolArguments(tool, {
        type: "toolCall",
        name,
        id,
        arguments: args,
      });
    }
    return await executeTool(id, name, args);
  },
  { name: tc.name, id: tc.id, args: tc.arguments },
);

Note the shape here: think, act, observe, with each phase a durable step. Deterministic scaffolding around a probabilistic core. If you've read how we built our scheduling engine, this is the same religion we practice at Driive: let the model decide what should happen, and let deterministic machinery guarantee that it actually does.

My favorite line in the whole repo isn't even in the loop. It's in the function declaration:

export const handleMessage = inngest.createFunction(
  {
    id: "agent-handle-message",
    retries: 2,
    triggers: [agentMessageReceived],
    singleton: { key: "event.data.sessionKey", mode: "cancel" },
  },
  async ({ event, step, logger, attempt }) => { ... }
);

singleton: { mode: "cancel" } means one agent run per conversation, and if the user sends a new message mid-run, cancel the stale run and start over with the new context. If you've built a chat agent, you know this exact bug. The user says "actually, never mind" and the agent barrels ahead answering the original question anyway. Solving that by hand means mutexes, cancellation tokens, and a whiteboard session that ends badly. Here it's a key and a mode.

This keeps happening with durable-execution primitives: hard distributed-systems problems collapse into configuration. Sub-agents are another example. Utah delegates synchronously with step.invoke(), which spawns a child function and blocks durably until it returns:

const subResult = await step.invoke("sub-agent", {
  function: subAgent,
  data: {
    task: tc.arguments.task as string,
    subSessionKey,
    parentSessionKey: sessionKey,
  },
});

Scheduled delegation ("remind me Friday") is just step.sendEvent() with a future timestamp. No job scheduler, no node-cron, no scheduled_tasks table with a poller. The event bus already knows how to do time.

This is where Utah goes from a nice example to a preview of how I suspect agent systems will actually get built.

The agent has file tools and a skill document that teaches it how to write Inngest functions. When it decides it needs a recurring behavior (check the weather every morning, watch a feed, poll an API) it doesn't add a to-do to its memory and hope. It writes a new TypeScript file containing a real Inngest function into a workspace/functions/ directory.

A separate sidecar process, its own tiny Inngest app that shares nothing with the main worker except the event bus, watches that directory and hot-loads whatever shows up:

const fileUrl = pathToFileURL(filePath).href + `?t=${Date.now()}`;
const mod = await import(fileUrl);
const fn = mod.default;
// ...
currentConnection = await connect({
  apps: [{ client: sidecarClient, functions }],
});

A filesystem watch triggers a debounced reconnect, and the agent-authored function is now live. Durable, retried, visible in the Inngest dashboard like everything else, and deployed with no human in the loop. And because those functions can emit agent.message.received back onto the bus, the agent's creations can wake the agent up. The loops feed each other. The agent isn't just running inside loops anymore, it's authoring new ones.

I know how that sounds. But look at how contained it is. The sidecar is isolated. The functions are files you can read, diff, and delete. Every execution is durable and logged. This is self-modification with an audit trail, which is a very different animal from an agent that "learns" opaquely inside a context window. The accumulated behavior is sitting right there in version-controllable TypeScript. If you're going to let an agent extend itself, and eventually you are, this is the shape I'd want: real artifacts, real observability, hard process boundaries.

Utah rounds this out with two humbler self-improvement loops. A remember tool appends notes to daily logs, and a cron heartbeat periodically has the LLM distill those logs into a curated MEMORY.md that gets injected into every future system prompt. Episodic memory compacting into semantic memory, on a schedule. (It's also smart enough to only distill when there's enough new material.) Skills are just markdown files the agent can write for itself, so future runs know what past runs figured out. There's even a small eval harness that scores whether the model picks the right delegation tool across a set of test prompts, which is the feedback loop for tuning the prompts that steer all of the above.

Memory, skills, and functions. Three tiers of self-improvement, all stored as plain files, all inspectable. No vector database anywhere.

Utah also does the unglamorous work that separates a demo agent from one that survives week two. Tool results get pruned in two tiers: soft head-and-tail trims for big outputs, then hard clears once you're past a total budget. At about 80% of the context window, the loop compacts history via the LLM and persists sessions to disk. When it still overflows mid-iteration, it force-compacts and retries that iteration without burning the attempt counter. On repeated LLM failures it switches to a fallback provider, which is trivial because the LLM interface (Mario Zechner's pi-ai) is provider-agnostic.

None of this is clever, and all of it is necessary. It's the agent equivalent of the dead-letter queue: boring machinery that decides whether the exciting machinery gets to keep running.

Our agents at Driive book real appointments for real businesses, and everything I've written before this post argues for deterministic guardrails around the model. The engine decides what's true; the model decides what the customer wants. Utah is the same philosophy applied to the agent runtime itself. The model decides what to do next, and durable steps guarantee that whatever it decided actually completes, exactly once, with receipts.

The stack is almost comically small. Node 23 running TypeScript natively, the Inngest SDK, a unified LLM client, and TypeBox schemas for tools. No LangChain, no agent framework, no graph DSL. Roughly the dependency count of a weekend project, with the failure semantics of a production system.

That's the lesson worth stealing. The hard part of agents was never the loop. It's making the loop survive crashes, retries, rate limits, impatient users, and its own growing ambitions. That's a durability problem, and durability is a solved problem. Go read the Utah source. It's shorter than most frameworks' getting-started guides.

Code excerpts above are from inngest/utah, © 2025 Inngest, Inc., used under the Apache License 2.0.