The first version of my agent workflow looked convincing in Markdown. It had a board, named states, skills for each job, worktrees, a CI command, and rules about when an agent had to stop.

Then I used it on DazzHub every day.

The failures did not arrive as dramatic model hallucinations. They arrived as a Docker command testing the wrong checkout, three skills carrying slightly different copies of the same shell, and a journal message executing the backticks it was supposed to record. The agents were productive enough to put pressure on every weak part of the setup.

And honestly, that pressure has been the most useful part of building the factory. The awkward failures showed me where the workflow was only convincing on paper.

The Project Underneath the Experiment

DazzHub is a Symfony application that discovers technical videos, scores them, fetches transcripts, and turns them into searchable knowledge and Markdown posts. It has PostgreSQL with pgvector, Neo4j, asynchronous workers, external AI services, and a growing set of domain modules.

It also has history. The test suite and architecture rules are strong in some areas and still being improved in others. That makes it a better factory test than a greenfield demo. An agent must work with existing conventions, baselines, data, containers, and GitHub workflow state.

I started with repository skills that described the full issue lifecycle:

• select a Ready issue from the board;
• claim it;
• create an isolated worktree;
• implement the issue;
• run the complete gate;
• push a feature branch;
• open a pull request;
• wait for human approval;
• merge and clean up.

The happy path worked. The repeated runs showed me where the real state still lived in my head.

A Green Gate Against the Wrong Branch

One of the nastiest findings came while working on PHPStan rules. The DazzHub application runs in Docker, and the existing development container had a bind mount from another worktree. I ran the expected command through docker compose exec app and got results from that foreign checkout.

The command succeeded. That made it worse.

A failing command tells me to investigate. A green command against the wrong source tree tells me a lie. In this case, a baseline regeneration even wrote into the other worktree.

The agent workflow already required isolated Git worktrees. It had not isolated the running Compose stack. The journal entry from that day now records the diagnostic command I needed:

docker inspect dazzhub-app --format '{{range .Mounts}}{{.Source}}{{end}}'

That incident changed how I evaluate guardrails. A rule that says “run CI in the worktree” is weak if the command can silently cross the boundary. The factory needs a command whose implementation knows how this project runs, not another paragraph reminding the agent to be careful.

The Compose Project Name Reversed Three Times

My first response was to put a fixed Compose project name into docker-compose.yml. That would make bare docker compose commands target the existing stack consistently.

Then I noticed the consequence for worktrees. A tracked project name applies in every checkout. A command from an issue worktree could reach the main checkout’s shared stack and test the main code. I had replaced “command finds nothing” with “command succeeds against the wrong thing.”

I tried a Makefile variable and a main-checkout guard. It worked, but the setup became harder to understand than the problem deserved.

The final decision used an existing property: .env is git-ignored. It exists in the main checkout and does not appear in new worktrees. The main checkout gets the intended Compose project name; a bare command in a worktree resolves to a harmless empty project. The explicit CI target still names what it needs.

This decision changed three times in a weekend. I kept all three in the project journal because the final diff cannot explain why the obvious tracked setting is absent.

That is one reason I added an append-only journal. Git records what survived. It does not record the viable option I rejected after discovering a failure mode.

The Journal Then Executed Its Own Message

The journal started as a Make target:

make log KIND=finding MSG='...'

The Makefile interpolated MSG into a shell recipe. I wrote an entry about the Compose problem using backticks around a command. The shell performed command substitution and ran the command while writing the note. The resulting journal line contained several kilobytes of command output.

A tool intended to preserve a failure had reproduced its failure class.

The fix was small: export the message through the environment and read it as data. I also added a length cap. The larger lesson was that I had management logic embedded in Make recipes and Markdown snippets with nowhere to test it.

By the end of that week, the project had enough examples to justify a small management CLI.

From Shell Fragments to factory/bin/dazzhub

I first planned to write the CLI in PHP. DazzHub is a PHP project, Symfony Console was already familiar, and the quality toolchain existed.

The host running the agents had no PHP interpreter. Each fresh worktree would also need Composer dependencies before the management tool could create or prepare it. The tool responsible for bootstrapping a worktree would depend on a bootstrapped worktree.

I switched to Deno and TypeScript. Deno gives the CLI one host binary, a committed lock file, no node_modules, and explicit runtime permissions. The supported entry point is now:

factory/bin/dazzhub

The CLI owns operations that are management, repeated, multi-step, and worth testing:

• append a structured journal entry;
• read and update the GitHub Projects board;
• calculate WIP capacity for an Orca precheck;
• report stale Blocked cards;
• create, remove, and sweep issue worktrees.

One-liners remain one-liners. The Symfony application remains in app/. I did not build a framework around the framework.

The permissionless unit suite became one of my favorite checks. A plain deno test runs with no read, write, network, or subprocess permission. The tests for adapters assert that the runtime refuses those operations. Separate narrow passes test the filesystem boundary, architecture walk, and repository skills.

The permissions do not make child processes safe. Allowing gh still starts an unrestricted gh process. They do catch an accidental Deno.Command("sh", ...), and they force every external program to have a name I can inspect.

One Board Command Instead of Several Copies

The first skills each carried their own GitHub Projects GraphQL and jq fragments. They started close enough to look shared. Then one expected a project item ID where another used an issue number. Column names were matched differently. Read limits drifted. A later change had to update several fenced code blocks and several grep-based checks.

The factory CLI now provides one read shape and one write path:

factory/bin/dazzhub board show ready
factory/bin/dazzhub board set status 203 progress
factory/bin/dazzhub board set tier 203 standard

The skills still decide when a transition is allowed. The CLI resolves fields, options, and item IDs and performs the write. That boundary matters. I do not want the CLI making product decisions, and I do not want three agents reimplementing GitHub’s project schema in prompts.

I replaced the old grep guards with Deno tests that parse the skills. During that work I found another quiet bug: the shell script that extracted fenced commands only recognized fences starting at column zero. Code blocks nested under a list item were invisible to every guard. The checks had been green because they skipped part of the material.

The replacement tests accept indented fences and assert workflow properties over the actual command snippets. The comments explaining retired checks were then deleted. Once a property is a test, I do not need prose claiming the same thing.

Orca Changed the Worktree Lifecycle

A plain Git worktree isolates files, but Orca only knows about worktrees it creates or tracks. A factory-created checkout without an Orca pane is operationally invisible from the runtime where the agents work.

The DazzHub CLI now probes orca status --json. If the runtime answers, it creates the worktree through Orca and reads the path from Orca’s JSON response. If Orca is unavailable, it falls back to git worktree add and says which Orca command would have been better.

That path needed two review rounds. The first implementation used orca --version as its probe, which proves only that a binary can print help. It does not prove a runtime is reachable. It also fabricated the expected worktree path instead of reading Orca’s response. Pi caught both in review.

A later real run found that Orca nests the path under result.worktree.path, while my parser expected a top-level path. Orca had created the worktree successfully, and my wrapper exited with an error because it could not read the result. That bug is now in the journal and has its own issue.

This is what refining the setup looks like. The integration is useful before it is complete, and every mistaken assumption becomes a smaller contract.

Skills Became Roles With Limited Authority

The issue skill originally carried most of the workflow. As the board became busier, I separated roles:

• the Groomer decides which Backlog card may become Ready;
• the executing agent claims one Ready card and produces a pull request;
• the PR skill handles review remarks and merges only after approval;
• the fleet skill owns worktrees and verification;
• the deterministic CLI performs board and worktree mechanics.

The Groomer is the first scheduled actor. Orca runs it hourly with a precheck. It may promote one fully specified, unassigned card when fewer than three cards are in Ready + In progress + In review. It may ask one question. It writes no code and cannot touch an in-flight card.

This narrow role solved a problem I had created with too much automation. I wanted Ready to refill without turning the entire board over to a model. The answer was a small decision surface with a WIP cap and a cheap precheck.

What the Real Work Changed

The project started with prompts telling agents how I work. It now has a state machine, a tested management CLI, an append-only decision journal, worktree lifecycle, board capacity, and an unattended backlog actor.

More important, I have a criterion for moving another rule out of prose: if the rule can be expressed as an exit code, parser, database constraint, or test, it should stop depending on a model remembering it.

Some rules remain in skills because they require judgment. Is this issue fully specified? Does this architecture finding exceed the ticket? Which Backlog card collides least with current work? I keep those decisions narrow and preserve the evidence in issue comments.

The setup is still being refined while it works. I prefer that to designing a perfect factory in isolation. DazzHub keeps producing the awkward cases I need: stale assumptions, concurrent branches, environment failures, review corrections, and commands that succeed for the wrong reason.

The next part of the series covers the thing I needed before allowing more unattended work: one place to see what Claude Code and Pi actually did.

Most of these tests had one thing in common: every dependency was mocked. The tests knew exactly how the code worked inside, but they knew very little about the final result.

Why We Use So Many Mocks

Many developers learn that a unit test should test one class in isolation. The result often looks like this:

I am testing one class, so I have to mock every other class.

But a unit does not have to be one class. A unit can also be one use case or one piece of behavior.

Isolation also does not mean that every object has to be a mock. It means that the test should not need a production database, the network, the filesystem, or another external system.

An in-memory repository is still fast and isolated. It is just a normal PHP object that behaves like a simple repository.

Mocks are popular because they are easy to create:

$repository = $this->createMock(OrderRepository::class);

A fake takes more work at the beginning. But you write it once and use it in many tests. With mocks, you repeat the setup in every test.

Mocks, Stubs, Spies, and Fakes

We often call every test double a mock, but there are differences:

A PHPUnit mock can be used as a stub, spy, or mock. In this post, I use mock for code that checks calls such as this:

$repository->expects(self::exactly(2))
    ->method('save');

Martin Fowler explains the different testing styles in Mocks Aren’t Stubs. One style checks the final state. The other checks how objects talk to each other. Both can be useful. The problem starts when we check method calls even though we could check a real result instead.

Mocks Test How the Code Works

Here is a small application service for creating an order:

final class CreateOrder
{
    public function __construct(
        private CustomerRepository $customers,
        private OrderRepository $orders,
        private PaymentGateway $payments,
    ) {
    }

    public function execute(string $customerId, array $items): Order
    {
        $customer = $this->customers->findById($customerId)
            ?? throw new CustomerNotFound($customerId);

        $order = Order::place($customer, $items);
        $this->orders->save($order);

        $payment = $this->payments->charge(
            $customer->paymentMethodId(),
            $order->total(),
        );

        $order->markAsPaid($payment->transactionId());
        $this->orders->save($order);

        return $order;
    }
}

A test with mocks could look like this:

public function testItCreatesAndPaysForAnOrder(): void
{
    $customers = $this->createMock(CustomerRepository::class);
    $orders = $this->createMock(OrderRepository::class);
    $payments = $this->createMock(PaymentGateway::class);

    $customer = Customer::withPaymentMethod('customer-1', 'payment-method-1');

    $customers->expects(self::once())
        ->method('findById')
        ->with('customer-1')
        ->willReturn($customer);

    $orders->expects(self::exactly(2))
        ->method('save')
        ->with(self::isInstanceOf(Order::class));

    $payments->expects(self::once())
        ->method('charge')
        ->with('payment-method-1', 99.98)
        ->willReturn(new Payment('transaction-1', 99.98));

    $service = new CreateOrder($customers, $orders, $payments);

    $order = $service->execute('customer-1', [
        ['sku' => 'ABC', 'price' => 49.99, 'quantity' => 2],
    ]);

    self::assertTrue($order->isPaid());
}

This test knows a lot about the code inside CreateOrder:

• findById() is called once.
• save() is called twice.
• charge() is called once.
• The methods receive exact values.

Some of this is important. The correct payment method and amount must be used. But does the business care if save() is called twice?

Maybe I later decide that an unpaid order should not be saved. I remove the first save() and only save the paid order at the end. The result is still one paid order and one payment.

The test still fails because it expects two calls to save().

I improved the code without changing the result, but now I have to change the test. If many tests use the same mock setup, a small refactoring can break many tests.

Fakes Test What the Code Does

A fake implements the same interface as the real adapter, but uses a simpler solution. Instead of storing orders in PostgreSQL, this fake stores them in an array:

final class InMemoryOrderRepository implements OrderRepository
{
    /** @var array<string, Order> */
    private array $orders = [];

    public function save(Order $order): void
    {
        $this->orders[$order->id()] = clone $order;
    }

    public function findById(string $id): ?Order
    {
        return isset($this->orders[$id])
            ? clone $this->orders[$id]
            : null;
    }
}

The fake stores a clone. This is important when Order is mutable. If it stored the same object, changing the order after save() would also change the stored object. A test could then stay green even if the second save() was missing.

The payment gateway can also have a small fake:

final class InMemoryPaymentGateway implements PaymentGateway
{
    /** @var list<Payment> */
    private array $payments = [];

    /** @var array<string, int> */
    private array $balances = [];

    public function setBalance(string $paymentMethodId, int $cents): void
    {
        $this->balances[$paymentMethodId] = $cents;
    }

    public function charge(string $paymentMethodId, float $amount): Payment
    {
        $cents = (int) round($amount * 100);

        if (($this->balances[$paymentMethodId] ?? 0) < $cents) {
            throw new InsufficientFunds();
        }

        $this->balances[$paymentMethodId] -= $cents;

        $payment = new Payment(
            'transaction-' . (count($this->payments) + 1),
            $amount,
            $paymentMethodId,
        );

        $this->payments[] = $payment;

        return $payment;
    }

    /** @return list<Payment> */
    public function payments(): array
    {
        return $this->payments;
    }
}

Now the test can describe the use case:

public function testItCreatesAndPaysForAnOrder(): void
{
    $customers = new InMemoryCustomerRepository();
    $orders = new InMemoryOrderRepository();
    $payments = new InMemoryPaymentGateway();

    $customers->save(
        Customer::withPaymentMethod('customer-1', 'payment-method-1'),
    );
    $payments->setBalance('payment-method-1', 100_00);

    $service = new CreateOrder($customers, $orders, $payments);

    $order = $service->execute('customer-1', [
        ['sku' => 'ABC', 'price' => 49.99, 'quantity' => 2],
    ]);

    $savedOrder = $orders->findById($order->id());
    self::assertNotNull($savedOrder);
    self::assertTrue($savedOrder->isPaid());
    self::assertSame(99.98, $savedOrder->total());

    self::assertCount(1, $payments->payments());
    self::assertSame(
        'payment-method-1',
        $payments->payments()[0]->paymentMethodId(),
    );
}

This test checks the result:

• The order was saved.
• The saved order is paid.
• The total is correct.
• The correct payment method was charged once.

It does not care if save() was called once or twice. I can change the code without changing the test, as long as the use case still works.

The test also finds real mistakes. It fails if I use the wrong payment method, charge the wrong amount, or forget to save the paid order.

Why Fakes Fit Hexagonal Architecture

In hexagonal architecture, the application defines ports:

interface OrderRepository
{
    public function save(Order $order): void;

    public function findById(string $id): ?Order;
}

Different adapters can implement this port:

• DoctrineOrderRepository stores orders in a database.
• InMemoryOrderRepository stores orders in an array.

The application does not need to know which adapter it uses. This also makes the in-memory adapter a good choice for application tests.

I split my tests like this:

Domain tests
  Use real domain objects. Usually no test doubles.

Application tests
  Test complete use cases with in-memory fakes.

Adapter tests
  Test Doctrine with a real database and HTTP adapters with a test server.

System tests
  Test a few important paths through the complete application.

The application tests are fast because they do not start external systems. The adapter tests make sure that the production code really works with the database or API.

Fakes Can Be Wrong Too

A fake is not automatically correct. It can behave differently from the real adapter.

For example, an in-memory repository may:

• forget a unique constraint;
• return the same mutable object;
• sort results differently;
• ignore transactions;
• hide database-specific problems.

This is why I use the same contract tests for the fake and the real adapter:

abstract class OrderRepositoryContractTest extends TestCase
{
    abstract protected function repository(): OrderRepository;

    public function testSavedOrderCanBeFoundByItsId(): void
    {
        $repository = $this->repository();
        $order = OrderBuilder::anOrder()->withId('order-1')->build();

        $repository->save($order);

        self::assertEquals($order, $repository->findById('order-1'));
    }

    public function testUnknownOrderIsNotFound(): void
    {
        self::assertNull(
            $this->repository()->findById('missing-order'),
        );
    }
}

Both repository tests extend this class:

final class InMemoryOrderRepositoryTest
    extends OrderRepositoryContractTest
{
    protected function repository(): OrderRepository
    {
        return new InMemoryOrderRepository();
    }
}

final class DoctrineOrderRepositoryTest
    extends OrderRepositoryContractTest
{
    protected function repository(): OrderRepository
    {
        return self::getContainer()->get(DoctrineOrderRepository::class);
    }
}

This makes sure that both implementations follow the same basic rules.

It does not test everything. Transactions, locking, database constraints, and parallel requests still need tests with the real database.

Why Fakes Become Cheaper Over Time

A mock is often faster to write for the first test:

$repository = $this->createMock(OrderRepository::class);
$repository->method('findById')->willReturn($order);

But every new test needs its own setup. After a while, the same expectations are copied into many test files.

A fake takes more time once:

$repository = new InMemoryOrderRepository();
$repository->save($order);

After that, every test can use it. If the interface changes, I update the fake in one place instead of updating many mock setups.

Fakes are also easier to debug. They are normal PHP code. I can set a breakpoint in the fake and inspect its state. With a generated PHPUnit mock, I mostly see the answers that I configured in the test.

When Mocks Make Sense

There are good reasons to use mocks or spies. I use them when the call itself is the thing I want to test.

Examples:

• A cache should prevent a second database lookup.
• A retry should stop after three attempts.
• A transaction should be rolled back after an error.
• Legacy code is hard to test in another way.
• A third-party system is too complex to fake in a useful way.

Even then, a recording fake can sometimes be easier to read:

$mailer = new InMemoryMailer();

$service->sendConfirmation($order);

self::assertCount(1, $mailer->sentMessages());
self::assertSame(
    'customer@example.com',
    $mailer->sentMessages()[0]->recipient(),
);

This still checks that an email left the application. But it checks the sent message instead of configuring a list of expected method calls before running the test.

Sometimes a mock is still the simpler choice. The important part is to choose it for a reason, not because every dependency must be mocked.

What I Use

Before I create a test double, I ask:

1. Can I test this with real domain objects? Then I do not need a test double.

2. Does this port have simple behavior that I can implement in memory? I use a fake and check the result.

3. Am I testing a real adapter? I use the real database, filesystem, queue, or a test server.

4. Is the number or order of calls important? I use a spy or mock and explain the reason in the test name.

5. Would the fake become as complex as the real system? I stop and use the real system or a smaller interface.

How to Start Replacing Mocks

You do not have to rewrite the complete test suite. Start with one port:

1. Pick a repository that is mocked in many tests.
2. Write down how it should behave.
3. Create a small in-memory implementation in tests/Double/.
4. Run the same contract tests against the fake and the real adapter.
5. Change one application test to check the result instead of method calls.
6. Keep mocks where a call count or call order is a real requirement.

The first fake may take longer than the first mock. It pays off when the next tests can reuse it.

Conclusion

My problem is not that mocks exist. My problem is using mocks for every dependency in every test.

A test should tell me if a feature is broken. It should not fail only because I moved a method call or changed how the code works inside.

Fakes help me write tests around complete use cases. They make the state visible, they are easy to debug, and they let me refactor without changing tests that still describe the same behavior.

Use mocks when the interaction is important. Otherwise, test what the code actually did.