At first, all 100 people could find the AI agent’s mistakes. Six months later, only 20 could still stump it.
That is how Mercor CEO Brendan Foody described a project in a September 2025 essay. Contributors were finding failures and building rubrics to assess them. As the agent improved, fewer people could produce a useful challenge.
Mercor widened the project. The agent gained access to more tools, and the team introduced tasks that would take a human more than ten hours to complete. All 100 participants could contribute again.
This is Foody’s account, rather than an independently audited experiment. It is also a revealing description of the business: model progress changes the work required to produce the next useful example.
Foody saw an expanding frontier for human expertise. I see a production problem worth examining. Someone has to design the harder task, establish a defensible answer, investigate unexpected behavior, and make the result usable by a customer. Each step consumes resources.
A recent Reddit discussion brings that problem down to the desk. On October 9, someone describing a finance background reported increasing difficulty designing tasks that models would fail. An anonymous post cannot establish a platform-wide trend. It does identify a question that company growth announcements rarely answer: how much effort sits behind one deliverable?
For investors, this is where the expert-data story becomes interesting. Scarcity can support pricing power. It can also produce rising costs, inconsistent output, and expensive coordination.
An AI evaluation task tests a model’s behavior against defined criteria. Its commercial value depends on whether the result helps a buyer make a better decision. Difficulty alone is insufficient. An ambiguous puzzle can confuse a model without revealing a meaningful weakness.
My working thesis is that the strongest expert-data businesses will control the process that turns professional judgment into reliable learning signals. A large talent network gives them a starting point. The investment case depends on what they can deliver from it, at what cost, and with what evidence of usefulness.
For paid readers: below, a worked cost model, a closer reading of Mercor’s move into training environments, and the operating evidence I would require before treating expert data as a durable investment thesis.
Previous highlights: Your paid subscription also unlocks the archive, including AI Is Hiring Its Teachers and The AHT Game.
Also included: clawded.ai, with personalized matches for AI and expert-work opportunities based on your background.