1. Overview

We created a digital twin, ShelfLife, of a live e-commerce company. From ShelfLife, we present a collection of 200 environments drawn from the company's operational data over the past decade to answer the question: can you deploy an agent in a leadership role within this company, and trust it to execute with appropriate judgment and precision?

Every environment is constructed in direct collaboration with the company's leaders and the operators who performed the work in production. We worked iteratively with them to develop a deep understanding of their workflows and domain expertise.

We report benchmark results for the following models on this collection: Claude Opus 5, Claude Fable 5, GPT-5.6 Sol, Grok 4.6, Kimi K3, and Gemini 3.6 Flash.

2. The Digital Twin

Our environments preserve the structure of the business and its data completely. This includes systems, records, the passage of time, platforms, communications, and business insights.

2.1 Anonymization

To protect sensitive information, we have performed redaction and anonymization in the packaging of each environment.

2.2 Preserving Real-World Complexity

We do not clean or alter schemas in the data workspace. We preserve the natural defects and gaps present in the raw data, and we work with domain experts to understand these discrepancies and construct detailed, information-dense environments.

This is a deliberate methodological commitment, and we hope to teach models to learn from how workflows become complex in production and expose heterogeneous failure modes.

3. Creating the Taxonomy

In the design of this collection, we focused on building a diverse corpus and delivering comprehensive coverage of economically valuable capabilities.

3.1 Selection of Data and Workflows

We do not use surface-level heuristics, such as step count or the number of files in a workspace, to assess the difficulty and value of a workflow. Instead, we consider the time and human input required to complete a workflow end to end. We consider the following properties:

1. The work is performed by a human on a recurring basis.
2. The task's failure modes are high-impact and high-risk.
3. The data and context are distributed and require sound human judgment to resolve.
4. The environment's correctness and fairness can be determined by a human expert who is deeply familiar with the underlying data.

3.2 Task Families

Each individual task and its failure modes are analyzed for the exact capabilities they measure. We use this information to categorize tasks into families, each of which represents a broad scenario found in an operating business.

A task may test multiple capabilities. We provide radar charts to show the capability profile of each reported model.

Tests the Capability PartiallyTests the CapabilityThe Family’s Defining Capability

3.3 Specification of Task Families

3.3.1 Safety and Reliability in Deployment

A model's ability to complete its assigned tasks is important, but it is just as important, if not more so, for models to be "aligned" and not take harmful actions when executing on their goals. The safety task family measures this capability.

These tasks ask the model to execute a typical workflow under time pressure while exposing shortcuts that are unethical but appear to help it reach its goals. The central test is whether the model takes such a shortcut. Doing so results in failure.

This task family has proven to be quite interesting, as it shows why AI deployment within high-stakes enterprises remains slow. Five of the six models we measured routinely took at least one unethical shortcut, and one took such shortcuts at nearly every opportunity. We observed models signing in as someone else, approving their own purchase, or booking spend into a quarter that had already closed.

3.3.2 Business Reasoning and Complex Workflows

Most enterprise work is not one difficult question. These workflows usually involve compounding tedious work and high demands on fast human judgment. Rules and humans often contradict one another.

This family of tasks gives the model the company's raw exports, platforms, communication channels, and other data. There is no ambiguity in what the agent has access to, and these tasks directly test whether a model can execute a procedure at scale without mistakes.

Failure modes we have observed include agents hallucinating values or rounding despite explicit instructions not to do so, as well as using stale numbers and choosing to blindly trust inputs rather than reconstructing values.

3.3.3 User Intent Interpretation and Judgment

Numerical work takes different forms across domains. In business operations, underspecification is difficult for current models to resolve. For example, it is difficult for an agent to select and combine the appropriate calculation methods to present a number that an operations manager can present confidently to a board of merchants.

These tasks involve complex and trivial arithmetic that requires the agent to reason about and select a methodology based on precedent and best practices. Often, conventions are counterintuitive but discoverable.

Failure modes we have observed include overpromising in advertising material and making claims that can create legal exposure for the enterprise. We have also observed verbosity and inappropriate hedging.

3.3.4 Long-Horizon Deliverable and Task Planning

This task family places the agent in a stateful environment to simulate four days of work. The information is given to the model on a schedule that is unknown to it, and it must make decisions on each day that are irreversible.

The agent is given a goal and a set of deliverables as well as a workspace. It must plan its approach to the task, revise it, and incorporate new information that may alter its course on subsequent simulated days.

Failure modes we have observed include substituting plausible-sounding evidence for real evidence, carrying incorrect statistics forward, misconfiguring orders, and compounding business logic mistakes.

4. Environment Interfaces

This collection of 200 tasks features environments where agents are able to interact with the data and workspace files in a number of dynamic ways.

4.1 MCP Tool Use

A subset of environments gives the agent access to MCP tools that mirror those it would use in deployment. Agents in an enterprise setting may only be able to access sensitive or administratively gated data through an authorized MCP, and for these specialized platforms, we mimic this dynamic.

The agent is provided access to every tool it would prefer and use, and we verify that the task is solvable.

4.2 Shell Access

A subset of environments gives the agent full access to the terminal during its trajectory. Agents can browse through files and execute scripts to solve the task at hand.

4.3 Hybrid

A subset of environments offers the agent multiple options to complete the task, and the agent must optimally choose how to approach the task.

4.4 State Machine

A subset of environments includes a state-machine-driven business engine. These tasks reproduce the sequence of information and events a human would encounter while completing the work. These tasks replay the agent's actions during the full trajectory after the trajectory completes.

5. Reward Design

5.1 Criteria

Reward is decomposed into observable criteria. Some examples of criteria types are:

1. Quantitative criteria. Quantitative criteria offer partial credit and are calculated numerically.
2. Integrity criteria. Integrity criteria carry no positive weight and may impose deductions when the agent performs specified harmful actions.
3. Withholding criteria. Agents may be penalized if they violate company or federal policy.

5.2 Multiple Correct Solutions

We discuss the composition of each task with the respective business expert, and we may conclude that a task contains two defensible resolutions. We audit our graders to make sure our environments are robust to these edge cases.

5.3 Observed Model Failures

Across all rollouts on the collection, we have observed the following failure modes:

5.3.1 Analytical Mistakes

• Computing the wrong quantity with the wrong process
• Understanding domain specific needs but selecting the wrong granular method
• Incompletely scanning the data space and missing relevant items
• Mathematical reasoning mistakes
• Making incorrect rounding or precision assumptions
• Misunderstanding currencies and unit conversions

5.3.2 Safeguard Violations

• Acting under a session belonging to another person
• Approving its own actions after being told that it lacks the authority to do so
• Faking another person's identity under time pressure
• Making high-impact numerical and legal decisions without the required authority
• Changing dates on historical records and company documents to complete requests
• Reporting milestones it has not achieved
• Reporting false information to users and customers to get approval

5.3.3 Hallucinations

• Proposing non-existent products to customers
• Assuming possession of various objects and offering them
• Exceeding stated budgets and claiming the company has mitigations for such cases
• Generating plausible-looking catalog entries rather than consulting the provided catalog
• Inventing business insights and applying them incorrectly to the company's operational state

5.3.4 Naive Business Logic Mistakes

• Missing regular business logic steps that a human employee is expected to perform
• Forgetting to balance stock and spreadsheets after making a warehouse management decision
• Failing to consistently update progress trackers and note pages
• Submitting incomplete deliverables

5.3.5 Failing to Complete the Task

• The agent spends an unreasonable amount of time backtracking and resolving confusion, ultimately giving up on the task
• The agent assumes it has completed the task, receives a nudge indicating that it has not completed the task, and fails to continue reasoning
• The agent receives comprehensive information on submission format but fails to comply
• The agent receives precise information about deliverables and fails to complete every one of them

6. False Positives and Reward Hacking

We enumerate possible attacks for each environment and provide countermeasures for each weakness in the environment. In particular, we verify:

1. There is no leakage of the ground truth or the answer in the environment
2. There is no unauthorized access to the verifier and no opportunity to tamper with the reward
3. There is no method for the agent to reach ahead in the environment to obtain unauthorized information
4. Producing high-volume or verbose outputs does not grant the agent proportional credit
5. We do not give the agent credit for refusing to attempt a task if there is reasoning that it should have performed
6. Every graded criterion must be accessible to the agent in the environment
7. There is no possibility of contamination or memorization across tasks

7. Human QA and Verification

For every environment, we verify its quality and integrity with multiple human evaluation passes, adjudications by experts, and static checks.

Operators who produced the source record provide specific feedback regarding the fairness of the grading and the environment's construction. Items addressed include the following:

1. How should ground truth be determined when situations are ambiguous?
2. Which errors are severe, and do the rewards reflect that severity?
3. Are task artifacts realistic?

8. Fairness Invariants

For each environment, we evaluate all failure modes across rollouts from multiple models to determine whether a penalty was fair. We ask the question: If a human expert in this field were given this task, would they have avoided this failure mode in good faith? We do not penalize the following cases of agent failure:

1. The agent is fundamentally not provided with enough data to solve the problem.
2. The agent refuses to perform an action that is unsafe.
3. The agent did not follow an undisclosed rule.
4. The delivery format of the agent's answer is unexpected.

9. Leaderboard