How Paradigm sped up a key strategic project for Crossbeam by 586%

Crossbeam, an ecosystem revenue company that works with large datasets of company information to provide insights for businesses, had a key strategic project that was bottlenecked by manual data collection and unreliable AI integrations. With Paradigm, they’ve been able to fully automate workflows and increase the speed and accuracy of that project by 586%.

One of Crossbeam’s flagship products, Partnerbase, is the world’s largest database of partnerships between B2B companies. Initially, Crossbeam relied on paid researchers and interns to manually gather company profiles and maintain a list of their relationships. While feasible at a small scale, this approach quickly became tedious and expensive. Identifying new company relationships was much more difficult once major, high-visibility companies were documented, and keeping existing relationships up-to-date required an increasing amount of manpower. Crossbeam eventually transitioned into crowdsourcing data for Partnerbase, which caused growth of the database to flatline. 

At the same time, the team wanted to modernize services like their Ecosystem Grader, a tool that analyzes a company’s ecosystem size and provides insights into its growth opportunities on demand. For both Partnerbase and the Ecosystem Grader, they needed a way to rapidly source information from the internet at high volume without compromising accuracy.

When Matt Nicosia, head of AI operations at Crossbeam, set out to reduce this heavy manual workload with an AI integration, he first experimented with more traditional data broker tools and even tried building an in-house AI research tool. These solutions proved impossible to scale and produced poor quality data. Then, he discovered Paradigm.

“We compared Paradigm directly to a couple of those [other solutions] and Paradigm was just more accurate,” he said.

Matt found Paradigm’s spreadsheet interface familiar and accessible. Once he got started, it “was not hard for me to just use it [Paradigm] and get it up and running the way I wanted it to,” he said. With just a company name or domain, Paradigm could quickly retrieve detailed company information, including industry, size, location, investors, and partners. Matt easily adapted Paradigm for Crossbeam’s specific needs. For each column, he could customize the prompt, specify where data was sourced from, control the types of values returned and the formatting, and use other columns in the sheet as context.

For example, to gather data on company partnerships, Matt wrote the following column prompt:

Prompt
List all the partners/integrations of this company.
These can be in the integration/partner page you already found (if available), or in public announcements/articles.  Do not use anything from partnerbase.com as a source here.  Do not include partnerbase.com as a partner.
This should be the company, not a product (ie, google.com not play.google.com).
If no partners are found, return nothing, not even an explanation.
Formatting
Provide a comma separated list of all partners and integrations for this company.  Provide the partner/integration company parsed domain only. Include parentheses and single quotes around each domain. Include a comma even after the final entry. Keep blank if nothing found.
Example: ('crossbeam.com'),('partnerbase.com'),('linkedin.com'),

Paradigm returned clean, consistently formatted lists of each company’s partners in seconds, complete with the agent’s thought process and sourcing.

Partnerships generated by Paradigm with Matt’s column prompt.

The AI team at Crossbeam quickly built out a number of workflows with Paradigm.

“We at Crossbeam use it [Paradigm] to do research on demand... If we want to understand whether a company is a customer of another company, or if we’re trying to understand whether there are certain partnerships that exist between other companies… we use it for that because it’s really highly flexible.”

To speed up their Partnerbase workflow, they set up a Paradigm sheet that takes in lists of companies requiring research and scrapes the web for detailed data on each company. One of the enriched columns is a list of partnerships for each company. In a separate sheet, each partnership is categorized as either a Technology Partner or a Channel Partner, enriched for additional details, then sent into Partnerbase. The impact was immediate: Paradigm began delivering thousands of precise data points at a scale unattainable by manual work.

Separately, Crossbeam also automated their Ecosystem Grader with Paradigm. When someone enters a company for analysis into their system, the request enters Paradigm via an input webhook, Paradigm does research on estimating company size, industry, revenue, growth rate, customer count, closed won rate, and a partner ecosystem score on a scale of 1 to 100. The outputs are returned directly back to the customer through an output webhook.

Partnerbase’s monthly growth rate surged by 586% after integrating with Paradigm. In just the past seven months, they added 60,000 new companies and over 100,000 new partnerships to the database, increasing its size by 67% at about half the cost of manual efforts. Their Ecosystem Intelligence platform now operates fully autonomously, delivering high-quality, data-driven insights whenever a customer requests them.

“It would not have been possible to have a team of humans doing this at scale,” Matt said.

Paradigm was the catalyst for Crossbeam to achieve results that would have been unattainable for a human team. Now, they’re continuing to use Paradigm to power more research, automate growth workflows, and save thousands more hours of valuable time.

Paradigm - Crossbeam project accelerated 586% with Paradigm

Crossbeam project accelerated 586% with Paradigm

Agent and Model Evaluations with Paradigm

By Kyle Tianshi, Arnav Adhikari, Anna Monaco, and June Lee

As Paradigm’s evals team, we set out to design a robust evaluation system to assess and improve the quality of the data our agents generate. Our system needed to handle two types of evaluations at scale: model and agent. Model evaluations measure how well different models (e.g., GPT, Claude, Gemini, Grok) perform against each other, with everything else held equal. Agent evaluations assess how well the whole system (the model, scraping tools, the prompt, etc.) performs in practical scenarios. This presented us with three primary difficulties:

Multi-step reasoning. Paradigm’s reasoning is much more involved than calling an AI model to return an answer. Each cell is generated using both column- and row-level context: columns are defined by a column name and optionally a column prompt, type, and format; rows are generated incrementally, with cells serving as context for other cells. As a result, the agent may take a long series of reasoning steps to arrive at a data point. This makes it necessary to evaluate not only the final answer but also the reasoning process and sources used.

An example of an agent’s cell enrichment process, which is a combination of tools (web search, scraping, private data APIs) and model reasoning.

Open-ended outputs. Our evaluator should be able to handle subjective outputs (e.g., founder bios, candidate rankings, quality or relevance scores) where verifying cells is difficult and be flexible enough to address use cases that we haven’t even identified.

Live data. Use cases for Paradigm can involve live data (e.g., stock prices, exchange rates, competitor prices), meaning that the same tool call can return different outputs depending on when it’s run.

Our first design was a ground truth evaluation system. We provided Paradigm with initial information and asked it to enrich several columns, then calculated accuracy by string matching against manually scraped results. See the example below: we gave Paradigm a list of engineer names, asked it to fill in their LinkedIn, GitHub, and Twitter URLs, and compared the results against a manually gathered list.

Version one of Paradigm’s evaluation system.

This approach quickly proved to be infeasible. It only worked for data that could be clearly verified as true or false, string comparisons were often unreliable due to slight variations in text, and gathering the ground truth datasets required significant manual effort.

Our solution was to incorporate a second agent into the evaluation pipeline.

A schematic of our current pipeline is shown below. We use an evaluator agent to label cell agent reasoning and outputs as either true or false. In our agent evaluation pipeline, we randomly select enrichment outputs to send to the evaluator. In our model evaluation pipeline, we run different AI models across a control dataset of 30+ well-known use cases to compare performance. In both cases, the evaluator agent runs directly after the cell agent to minimize issues caused by live data. These accumulate into a dataset of labelled cell agent outputs that informs improvements to our research and scraping capabilities.

In the sections that follow, we’ll take a closer look at each part of this pipeline.

Rather than only evaluating results of the spreadsheet data, we also evaluate the task steps.

We employ a separate AI evaluator to trace the line of reasoning that the agent takes and confirm that (a) each step is verifiable and makes sense, (b) the linked sources are correct and relevant, (c) the result is correct, and (d) the path to the solution is optimal. We return a Boolean value of false if the evaluator finds an error in any of these criteria, and true otherwise. This evaluation approach allows us to assign a truth value to each cell, but with the advantage of being able to handle more subjective result data like startup quality ratings and professional summaries.

The prompt that the evaluator receives with each cell.

For example, we can run the evaluator on a company information spreadsheet that contains the names of big tech companies and asks Paradigm to enrich fields like company LinkedIn, employee count, and annual revenue. For a straightforward task like finding the Tesla company website, the evaluator simply checks that the link generated is valid. For a more subjective task like determining Apple’s competitive landscape, the evaluator checks the validity of each of the three links accessed by the initial agent, verifies that the listed companies are accurate and relevant, and ensures the reasoning path is efficient. Per the prompt that the evaluator receives, any error found in the agent’s reasoning automatically results in the evaluator returning false to avoid feeding false positives into the fine-tuning pipeline.

An evaluator output for a cell about Apple’s competitive landscape.

Since the evaluator is LLM-powered, it is prone to errors and hallucinations. A recurring issue in our testing is scrape failures (for example, when the evaluator cannot verify a source because of a CAPTCHA), which happen because the evaluator needs to go beyond the cell agent’s logic to validate the cell output. As such, we try to avoid false positives as much as possible, which is why our evaluator prompt fails if it finds even one error. We’ve also tried to minimize the number of incorrect evaluations by using a more powerful model for the evaluator and checking for consistency across evaluations of the same cell agent outputs. In practice, we consider evaluator outputs to be ground truth, though we recognize that our accuracy estimates are likely conservative.

To get a sense of how well Paradigm infrastructure performs in practice, we automatically send samples of cells generated by users into the evaluator pipeline. This allows us to evaluate more varied use cases produced by real users and receive immediate feedback on agent performance for new scenarios.

Our model evaluation pipeline is a controlled setting for evaluating small changes in our agent setup: right now, we use it to evaluate model performance, but it can also be used to track the impacts of giving the model access to new tools and APIs.

We needed to develop a control dataset that is sufficiently representative of most Paradigm use cases to be able to get a baseline for how models perform against each other. We ultimately compiled a dataset of over thirty use cases, ranging from people and company lookups to email, LinkedIn, Github, Twitter, Instagram, and TikTok scraping, as well as financial research, research paper analysis, and price comparisons.

We also bucketed these use cases based on difficulty level. Easier use cases are well-indexed with well-defined answers (e.g., finding Meta’s website URL), while harder use cases are open-ended, involve unstructured information, require multiple steps, or require corroborating between multiple sources (e.g., summarizing how changes in privacy laws affect a certain software product).

With this control dataset, we’re able to easily run evaluations and get a sense of how models perform. We can also more finely examine performance on specific use cases and difficulty levels, which is useful for evaluating the impact of certain tools on the agent (e.g., how well it does at scraping Twitter posts after adding a Twitter tool).

Our agent evaluation results are ongoing as we onboard more users to the product. We’ve already run 71,660 individual evaluations on samples of real user enrichments and plan to release more results when we have a larger dataset.

Our current model evaluation results are the result of 59,390 individual evaluations of cells generated within our control dataset, with Claude 4 Sonnet as a consistent evaluator.

We are constantly working to improve the performance of our cell agents across all use cases and expand the suite of web APIs and scrapers available to the agent. We plan to continually release evals for new models as they come out and refine our control dataset to cover all use cases as accurately as possible, with the hope that our system can serve as a valuable benchmark for model performance in web scraping-related tasks.

Blog

Crossbeam project accelerated 586% with Paradigm

Agent and Model Evaluations with Paradigm