Rapidbench
Measuring AI agents on real electronics product queries
Electronics catalogs hold thousands of products that often differ by only a few specifications. Users ask questions like “List all Jetson Orin products with four Ethernet ports” or “Group Jetson Nano products by the number of PCIe lanes”. Answering them means searching a large catalog, interpreting specifications, finding every match, and comparing or grouping them accurately. No public agent benchmark tests this for electronics hardware.
So we built Rapidbench: 122 real-world technical product queries for evaluating AI agents in the electronics hardware domain.
Our long-standing hypothesis is that the real unlock for AI agents in this domain starts with structured product intelligence. Better models and agent harnesses add value on top, but they fall short without curated, task-specific product knowledge.
The first version of Rapidbench tests this with three arms, to isolate the effect of curated product intelligence:
- Frontier models with no tools
- Models with web search
- Models with Rapidflare's proprietary product catalog via MCP
Leading models and harnesses reached ~90% accuracy with Rapidflare's catalog, against ~50% with web search alone and ~18% with no tools.
Rapidbench Leaderboard
The Rapidflare agent scored 96%, and the catalog lifted both models we ran with web search by 38 to 40 points. For electronics enterprises, the product data an agent can reach matters more than the model it runs on.
Pass rate is the mean of 3 runs, ± how much it moved between runs. See methodology
Overview
1. Catalog data
The first version of Rapidbench focuses on NVIDIA’s Physical AI ecosystem, which includes hardware used in robots, drones, cameras, and industrial machines. Specifically, it covers NVIDIA’s Jetson modules, IGX platform, developer kits, and products built by its partners.
For this benchmark, we use our NVIDIA Jetson catalog, curated using proprietary agentic technology, covering 2,068 products across 54 product categories. We built it by discovering, extracting, organizing and repeatedly verifying products from the Physical AI ecosystem. The main source is NVIDIA's partner marketplace, cross-checked against each manufacturer's own pages. Future versions of Rapidbench will cover electronics verticals in the public domain.
Keeping a catalog like this accurate is hard for two reasons:
- Sources that disagree: Manufacturer web pages may contradict their own datasheets. For example, the Advantech EPC-R7300's product page lists 7 Ethernet ports but its datasheet lists 6.
- A moving market: Partners add and retire products every month so it's very hard to keep up.
2,068 products from 205 manufacturers, across 54 product categories
The 54 categories, grouped into 9 broader areas.
Fairness
One question that might come up is whether the benchmark is biased toward Rapidflare's own catalog: does Rapidbench simply reward agents for agreeing with our catalog, rather than for finding the correct answers? There are two things worth addressing here: coverage and correctness.
Coverage: How do we know the catalog covers the available products? We validated our catalog against the open web. For each benchmark query, we searched extensively for all qualifying products, including those identified by web-search agents. We then checked whether these products were already in our catalog and added any missing ones that met our criteria. This helped us confirm that our catalog reflects the broader market, not just the products we initially knew about.
Correctness: Why should the catalog be treated as ground truth? We found that even official manufacturer documentation can contain conflicting information. To address this, every value in our catalog is verified against manufacturer sources, with the supporting citation. Any discrepancies are resolved using a set of rules, and every change undergoes an independent second judge review and an expert sign-off.
Therefore, agents are evaluated against verified product information that has been carefully reviewed and reconciled rather than arbitrary catalog entries.
Rapidflare MCP
We let other agent harnesses use this catalog through the Rapidflare MCP.
2. The electronics domain is harder than it looks
- Near-identical products: There can be thousands of boards and systems, with part numbers that differ by one letter or a suffix.
- Technical nuances: A product's specifications can vary depending on many factors like its power mode, software version, or configuration. This means the same spec can have multiple correct values depending on the conditions.
- Missing values: Not every product lists all its specifications, so a missing value could mean "unknown" rather than a "no," and understanding this difference is important for answering queries correctly.
- Attribute complexity and density: Attributes can be overloaded with meaning, and units are nuanced and open to assumptions or misinterpretation. Take memory: LPDDR5 is a separate standard from DDR5, not a low-power version of it, so a search for "DDR5" should not return a Jetson Orin module with LPDDR5. Speed is just as slippery: NVIDIA's datasheet gives the Jetson AGX Orin's LPDDR5 as 3200 MHz, which is 6400 MT/s once you count both edges of the clock, so comparing it against memory rated in MT/s can be off by a factor of two.
- Related products: Products have complex relationships with each other. Using a hardware product may require associated products like developer kits, carrier boards, and software, so selection means finding the right set rather than a single part. A Jetson Orin NX module, for example, has no onboard storage: it needs a carrier board, an NVMe drive, and a software stack such as NVIDIA's JetPack before it can run anything.
3. Realistic queries
One of the key goals of Rapidbench was to create realistic queries with broad coverage. Although the 122 queries are generated synthetically, we ensured they are grounded in real-world signals. We used Semrush to analyze popular search patterns and studied queries handled by Rapidflare's agents. We also extracted query archetypes and templates from real-world usage and built on those. The queries are designed to reflect natural language, including casual phrasing and occasional typos. Each query also goes through rigorous human review to ensure it represented a realistic question someone might ask. The table below shows one query from the benchmark for each type.
| Type | Example |
|---|
Beyond realism, we wanted to ensure diversity across query types, product families, and difficulty levels. The benchmark covers 6 query types, with representation balanced across the different product categories in our catalog. Queries are also classified as Easy, Medium, or Hard based on the scope of information needed to answer them. Furthermore, these difficulty labels were also validated by human reviewers to ensure they reflected the actual complexity of the questions.
Queries by type and difficulty
Anatomy of a query
To see how differently the ablation arms handle these queries, here is one query from the benchmark and how each arm answered it.
Query"Which carrier boards support Jetson Thor?"
The catalog lists 23 boards from 12 manufacturers.
Without structured product intelligence, agents often struggle to find all the products that satisfy a query. Web search can often be the most expensive way to get an incomplete answer as web agents often spend much longer searching but only find the products that are easiest to discover. Other valid options may be scattered across datasheets and product pages from different manufacturers and therefore more obscure.
In electronics hardware, finding all the valid options is often just as important as finding the right one. Engineers need to know what is available before making a decision.
Results
Qualitative analysis
Hypotheses
We tested one main hypothesis and two supporting ones, and all three held across 3 runs. See methodology
| Hypothesis | How we test it | Result |
|---|---|---|
| MainTask-specific agents with curated product intelligence (Rapidflare) significantly outperform general agents with only web search | The Rapidflare agent against Claude and GPT with web search | Holds: 96% against 47% to 57% |
| SupportingCurated product intelligence beats web search | The same model in the same harness, with web search and with the catalog | Holds: 38 to 40 points higher with the catalog (Claude Opus 47% to 87%, GPT Sol 57% to 95%) |
| SupportingCurated product intelligence beats model size | Smaller models with the catalog against top models with the catalog and with web search | Holds: Claude Sonnet 85% against Claude Opus 87%, GPT Luna 88% against GPT Sol 95%, and both beat every web search arm. |
Curated product intelligence lifts every model
The catalog lifts both models by 38 to 40 points over web search
This is the central hypothesis Rapidbench was testing. With no tools, both models answer fewer than one in five queries correctly. Web search brings their pass rates to around 50%, while access to the catalog improves them by another 38-40 percentage points.
We think curated product intelligence makes the difference for three reasons:
- Complete product coverage: Agents only searching the web tend to find the most discoverable products and miss the rest. For example, "Which carrier boards support Jetson Thor?", the web agents missed 13 and 15 of the 23 boards, while the catalog covers all 23 across 12 manufacturers.
- Spec-based search: The catalog lets agents search and filter products directly by technical specifications, while web search requires finding and reading individual product pages and datasheets, making it much harder to compare products or cover an entire category
- Verified product data: When product pages and datasheets disagree, the catalog stores the verified value along with any relevant conditions. Missing specifications are explicitly marked as not listed, reducing the risk of agents guessing.
The gap opens on hard queries
Web search collapses on hard queries; catalog agents hold up
Pass rate by difficulty. Easy queries cover one manufacturer, medium ones compare or pick between a few products, and hard ones span a whole product category. See methodology
Web search performs well on easy queries but drops to just 3-6% on hard ones, while catalog agents maintain a much higher pass rate of 71-90%.
The difference comes down to market coverage. Easy queries can usually be answered from a single product page or datasheet, while hard queries require comparing products across multiple manufacturers. Web agents often miss relevant products due to search limitations and tool-call budgets, whereas catalog agents can search the entire category directly, making them more reliable on complex queries.
Our running example, "Which carrier boards support Jetson Thor?", is one of these hard queries: the catalog lists 23 boards from 12 manufacturers.
Where agents struggle, and why
By query type
Web search handles named products but struggles to find every match
Pass rate by query type.
The biggest difference here comes down to whether the agent needs to look up known products or discover relevant ones. We find that agents using web search perform well when the products are already identified but struggle with queries that require searching and evaluating products across the market with pass rates dropping as low as 0 to 13%.
Agents using the catalog consistently perform better across query types because they can search and filter products by specifications rather than relying on search rankings.
NVIDIA's own products vs partner products
Next, we wanted to see if the product manufacturer affects performance. We split the queries into two groups: NVIDIA's own products, and partner products. We then compared how each model performed on both groups.
Each bar shows accuracy averaged over 3 runs.
On NVIDIA's own products, every arm scores 90% or more, even with no tools. On partner hardware, the no-tools arms drop to 5-7% and web search to 40-50%, while the catalog arms stay at 83-95%.
We think this difference comes down to parametric knowledge or what a model already knows from its training data. NVIDIA’s own products are widely documented and discussed so models are more likely to know about them. Products from smaller ecosystem partners have much less information available, making it harder for models to answer from memory. Web search helps to some extent but these products are often harder to find. Catalog-based agents don't face the same problem because they have direct access to the product data which explains why their performance stays consistent across both groups.
How the harnesses use the catalog
Where each agent spends its calls to the Rapidflare catalog MCP
GPT-6.1 Sol spends almost half its calls fetching complete product lists for a manufacturer or family, and then answers using that data. It also checks the catalog schema more often to understand which fields are available.
Opus 5.5 takes a different approach by filtering through products by specs and fetching them one by one. This creates more chances to miss products and that explains the root cause of most of the failed queries.
Overall, the way models search and use the catalog likely affects their accuracy. Fetching complete lists upfront appears to work better than searching for products individually.
Quantitative results
Pass rate alone is not a holistic metric to measure performance, so we also track median output tokens per query and median time per query, with each chart comparing one of these metrics against pass rate, where higher and further right is better
Score vs. output tokens per query
Top right is best.
Score vs. time per query
Top right is best: faster and more accurate. Catalog arms answer in about a third of the time of web arms.
The catalog arms consistently sit in the top right of both charts: they score highest while using fewer tokens and less time.
Conclusion
Rapidbench was designed to test a long-standing hypothesis: that the key to better AI agents is structured product intelligence and not just better models or agent frameworks.
The results from this benchmark support this. With Rapidflare’s catalog, both models we tested with web search improved by 38 to 40 percentage points. The difference was even greater on hard queries, where web-search agents scored just 3-6%, while catalog-based agents achieved 71-90%.
The Rapidflare agent achieved an overall score of 96%, compared to 47-57% for general-purpose agents using web search.
These results show that accurate, complete, and well-structured product information is essential for building high-performing AI agents in the electronics domain.
Further reading
- Methodology
- Limitations and future work
- Browse trajectories. Browse the rollouts behind the numbers above.