Crew: why I stopped building an AI development platform
- My contribution
- Entire platform implementation; my cofounder contributed documentation used in the shutdown decision.
- Scope
- An AI development platform, built over approximately four months and retired before any external pilot.
- Evidence
- Self-reported results from small synthetic comparisons. Original code and raw benchmark artifacts are unavailable for independent inspection.
Page by Julian Ramirez ·
I spent about four months building Crew, an AI development platform. Then I compared it with a simpler model-driven agent. In the later small synthetic comparison, Crew used 22% more recorded tokens and 54% more median machine time, with no automated correctness advantage. I shut it down.
That is the short version. The useful version is about the difference between building a working system and establishing a reason to keep building it. Crew reached a point where I could compare its results. Those results did not establish the added value I needed to justify continuing.
What I built, and what never reached users
The entire platform implementation was mine. My cofounder helped produce documentation used in the shutdown decision. The work took approximately four months; that is a self-reported duration, not a time-tracking total or a claim of four months of full-time work.
Crew had no external pilots. Its trials were internal, so this is a case about evaluating and retiring a product before external validation. It is not a story about customer adoption, production reliability or a measured improvement in developers’ work.
A working result was not enough
A simpler agent was the relevant comparator. If both systems could satisfy the automated checks, Crew still needed to justify its additional machinery through some other benefit. Correctness was one question; the resources used to reach that result were another.
The preserved account describes two small comparisons. Automated correctness was 6/6 for each approach in the earlier comparison and 3/3 for each in the later one. These are separate sets of results, not a pooled sample or an estimate of real-world success rates.
| Automated correctness | Earlier | Later |
|---|---|---|
| Simpler baseline | 6 / 6 | 3 / 3 |
| Crew | 6 / 6 | 3 / 3 |
Same recorded outcomes. Small synthetic task sets.
Sources & implementation notes
Claim crew-benchmark. Counts are preserved in the confirmed account; the task-level outputs and original checks are unavailable. Do not combine these into a general success rate.
Passing the same checks does not prove two systems are equally useful. A small synthetic task set can miss differences in task difficulty, human effort and the quality of the finished work. What it does establish here is narrower: the recorded automated results did not show a correctness advantage for Crew.
The later comparison added cost without a correctness gain
In that later comparison, Crew used 22% more recorded tokens and 54% more median machine time than the simpler baseline. The chart below sets each baseline measure to 100 so the two relative differences are readable. It does not reconstruct token totals or elapsed minutes.
Recorded tokens +22%
Median machine time +54%
Relative resource use · not dollars or human effort
Sources & implementation notes
Claim crew-benchmark. 100 × 1.22 = 122; 100 × 1.54 = 154. These are normalized indices, not original token or timing measurements. Raw data are unavailable; the preserved record also notes a control-wrapper settlement defect.
Recorded tokens are not a dollar bill. Median machine time is not a measure of my time, an average across all runs, or a production latency guarantee. Keeping those distinctions matters: the defensible claim is additional consumption on these measures, not a quantified business loss.
There is also a measurement caveat. The preserved account notes a separate post-inference settlement defect in the control wrappers. Without the original artifacts, I cannot quantify its effect on the comparison. The percentages remain a self-reported observation, not an independently verified performance result.
Why stopping was the decision
The comparison did not prove that an AI development platform could never be useful. It gave me a much more immediate problem: I had spent months building one, and the available results did not establish an advantage over a simpler approach. I chose to stop Crew.
That decision should stay separate from the things the evaluation never measured. Human effort and customer value were unmeasured, and there were no external pilots. None of those gaps can honestly be filled with either a success claim or a claim that the entire product category failed.
Higher recorded token use
Higher median machine time
No automated correctness gain
Less human effort
Customer value
Performance in production
No external pilots. No claim of universal failure.
Sources & implementation notes
Claims crew-benchmark and crew-scope. This diagram organizes the recorded evidence and its limits; it is not an execution trace or an experiment on alternative decisions.
The engineering lesson is to make the baseline part of the product question. A system completing a task is an implementation milestone. A system earning its added complexity needs comparative evidence. Crew gave me the first without establishing the second.
Evidence and attribution
This page draws on my confirmed account of the work and a preserved summary of the September 29, 2026 shutdown decision. The original retired implementation and raw benchmark artifacts are no longer available for inspection; this page is not a reproducible benchmark report.
The available record preserves relative costs and automated correctness counts, but not absolute token totals, raw timings or a complete evaluation protocol. The normalized chart makes those reported differences visible without inventing the missing measurements.
Four months of work gave me something to evaluate. The evaluation gave me a reason to stop. Both belong in the record of the project.
Explore the other projects and their implementation decisions →