Methodology

Rapidbench is built in two stages, following the structure of synthetic benchmarks such as Chroma Context-1, where we generate queries from a curated environment (our product catalog), and then verify each one before it enters the dataset.

Task construction

In Rapidbench, each query is part of a larger unit called a task. Each task includes the query, a reference answer based on the verified catalog, and the checks used to evaluate the agent’s response. Here’s how we create these tasks.

1. Start with a fixed catalog: We take a snapshot of our catalog to ensure the underlying data stays consistent throughout query generation.
2. Plan the query distribution: We define the distribution across six query types and different product categories. A fixed random seed is used to sample candidate products and scenarios reproducibly, while Semrush data helps guide the selection toward topics people search for.
3. Start with product specifications: For each sampled product or category, we look at the information available in our catalog on the technical specs.
4. Generate questions from the data: We select combinations of specifications to create meaningful questions. Since the questions are based on existing catalog data, the correct answers can be computed deterministically. If an answer is unclear because of missing information, ties, or too many matches, the question is discarded.
5. Make the questions realistic: An LLM rewrites the questions to resemble realistic human queries, including casual language or occasional typos. It also generates a readable reference answer from the underlying catalog evidence.

Quality assurance

Newly generated queries aren't added to the dataset immediately. Each query goes through a series of verification and review steps before being accepted.

• Evidence sufficiency: An independent model reviews the query and its supporting catalog evidence, without seeing our expected answer. It checks whether there's enough information to answer the query and whether the query resembles a realistic user request.
• Human review: Queries and their reference answers are reviewed for realism and correctness by human graders.
• Dataset freeze: Once the queries are finalized, they are locked before running any agents to prevent changes during evaluation.

Queries are generated using DeepSeek V4.1 Flash and verified using GPT-5.6 Terra. This ensures that the model validating a query is different from the one that generated it.

The entire process runs through our custom task-creation harness, a Python pipeline orchestrated using Temporal. Each query is stored in Harbor task format, along with its supporting catalog evidence and verification results.

To show how a query gets built, we use a simpler one: a multi-step query about the Vecow RAC-1100, an edge AI computer.

Process

We ran all 122 queries 3 separate times for every arm, so we could see how much results move from run to run.

Every arm runs through Harbor, which lets us control exactly which tools each agent gets, i.e search only in the web arms, the catalog only in the catalog arms, and nothing at all in the no-tools arm.

• All models run at high reasoning effort with code execution turned off.
• Tool use is capped at 30 calls per query.
• The harnesses get a short piece of context about the catalog and manufactures to help them get a kickstart. The Rapidflare agent uses its own prompts.

Grading

Every answer is graded by GLM 5.3, an independent judge model from a different lab than the models being evaluated (OpenAI and Anthropic), using fixed settings (high reasoning, temperature 0) for consistent grading.

NoteWe initially used GPT-5.6 Terra as the judge but noticed potential bias in its grading, so we switched to GLM 5.3.

Each answer gets three scores from 0 to 1: