Piloting AI tender software: a test plan using your own bills of quantities

Transparency note: This guide is published by Scalera. Information about Scalera comes from our website; information on regulation and methodology comes from the publications of the respective institutions. Corrections are welcome at info@scalera.ai.

Why test AI software on your own BoQs?

Because the time savings in reference stories apply to other product ranges, formats and teams. How quickly software delivers good product matches depends on your portfolio and the kind of requests you receive; Scalera says so in its own FAQ (Scalera, FAQ).

According to the FAQ, Scalera customers typically report 60–80% time savings in tender processing (Scalera, FAQ). A pilot shows which figure is achievable in your business and where the limits are. For the vendors on the market and the questions to ask them up front, see our comparison of AI assistants for bills of quantities.

What is the difference between a demo and a pilot?

A demo shows in a short time how the software works; a pilot shows over several weeks what it delivers in day-to-day work. Both have their place, but only the pilot gives you solid numbers for a decision.

Our own classification. With Scalera, you can already have relevant positions extracted from your own BoQs in the demo call (Scalera, demo).

How do you build a meaningful test set?

Take a cross-section of real requests from recent months, not the cleanest examples. The test set should reflect the variety your inside sales team actually deals with:

• Formats: digital and scanned PDFs, GAEB files, Excel/CSV or XML, and requests written straight into the email body. Scalera processes these document types according to its FAQ (Scalera, FAQ).
• Languages and markets: if you quote in several countries, include requests in every relevant language.
• Size: short requests with a handful of positions as well as extensive bills of quantities.
• Product range: standard items, configurable products and positions specified with a competitor's product.
• Edge cases: unclear or incomplete specifications and drawings with technical details.

Write the test set down and use it unchanged for every vendor you test. That's the only way to make results comparable. Why formats such as GAEB, ÖNORM and NPK have such a strong effect on processing is explained in GAEB, ÖNORM, NPK.

Which metrics should you track in a pilot?

Four metrics are enough for most pilots. What matters is that you capture them the same way before and during the pilot:

1. Processing time: from opening the request to a draft offer ready for review, in minutes per request.
2. Completeness: the share of positions relevant to you that the software detected.
3. Hit rate: the share of product suggestions your team accepts without changes.
4. Correction effort: the time and number of edits needed until an offer is right.

Completeness and hit rate: two different questions

In statistics, the two values are called recall and precision. Recall measures what share of the truly relevant cases was found; precision measures what share of the cases found is actually correct (Google for Developers). Applied to a BoQ: software can find almost every position and still often suggest the wrong product, or the other way round.

So evaluate both values separately, and per product group. An overall average hides where the software is strong and where your data needs work. How to prepare product data for this is covered in Product data for tenders.

Count uncertain matches separately

Scalera attaches confidence scores to its matches and puts the uncertain ones in front of your team to check (Scalera, FAQ). During the pilot, count how often flagged matches had to be corrected and how often an unflagged match was wrong. The second figure matters more, because it shows whether your team can rely on the flags.

How do you measure the baseline before you start?

Before the pilot, record for the same request types how long processing takes today and how many corrections are needed after submission. Estimates from memory are too vague for this.

A simple log is enough: request, format, number of positions, start and end time, who handled it. Our webinar recap describes how timing with a stopwatch shows where time is lost today. The costs hidden in manual processing are covered in The hidden cost of manual tender processing.

How long should a pilot run?

Long enough for a good number of typical requests to go through and for the software to learn from your corrections. There's no fixed number of weeks, because request volumes and product ranges vary widely.

Scalera learns with every completed offer, often provides useful product suggestions from the first offer onward and recommends 50–150 completed offers for optimal results, depending on how complex your portfolio is (Scalera, FAQ). So evaluate the first and the last weeks of the pilot separately. That shows you whether the hit rate is improving.

Limit the scope to one product line or one location. According to Scalera, many customers begin with the product lines that carry the most volume or are the most standardised, and only expand once the value is proven (Scalera, FAQ).

Who belongs on the pilot team?

The people who create offers every day, plus one person each for product data, IT and the decision. Without the inside sales staff, the pilot measures the software but not everyday reality.

• Inside sales: processes requests with the software and documents corrections.
• Product management: closes the data gaps the pilot reveals.
• IT: provides the product data. With Scalera, you can upload it manually or connect it via an interface (Scalera, FAQ); details in ERP integration.
• Head of sales or inside sales: sets the criteria and makes the final call.

On the vendor side, Scalera's Customer Success team supports you from implementation through testing to training resources in the Software Academy (Scalera, FAQ).

Which obligations and frameworks should you know?

For using AI in your team, the EU has a binding AI literacy obligation. For structuring the test, a voluntary framework from the US helps.

AI literacy under the EU AI Act

Providers and deployers of AI systems must take measures to support the AI literacy of their staff. This obligation under Article 4 of the AI Act has applied since 2 February 2025 (European Commission, AI Act).

Since the Digital Omnibus amendment, Article 4 no longer mandates a specific level of literacy, and there is no mandatory training format. The Commission notes, however, that simply asking staff to read the instructions for use is often ineffective (European Commission, AI literacy Q&A). A pilot is a good moment to train people on how to read suggestions and confidence scores.

The NIST AI Risk Management Framework as a structure

The AI Risk Management Framework from the US National Institute of Standards and Technology (NIST) is intended for voluntary use (NIST, AI RMF). Its core consists of four functions: Govern, Map, Measure and Manage (NIST, AI RMF Core).

Translated to a pilot: set responsibilities and criteria (Govern), define the scope and test set (Map), measure with repeatable tests (Measure) and decide on roll-out or adjustments (Manage). For Measure, the framework describes objective, repeatable or scalable test and evaluation processes.

Before you start, also agree which data you will share in the pilot. Scalera uses private AI instances, and your data is never used to train public LLMs (Scalera, FAQ).

How do you work with the vendor during the pilot?

Agree on a fixed rhythm, such as a weekly session in which you go through wrong or missing matches together. Assign each error to a cause: missing product data, an unclear request or a weakness in the software.

That assignment decides who has to act. Product management fills in missing attributes, you clarify unclear specifications with the customer, and software weaknesses go on the vendor's list. For low-confidence matches, Scalera explicitly lets your team prioritise where to invest time or ask the customer for clarification (Scalera, FAQ).

“Scalera has made us feel that we and our ideas are important.”
Marco Dürst, Management, ACO

When is a pilot successful?

When it meets the criteria you set before the start. Write these thresholds down, otherwise the decision turns into a matter of taste after the fact.

A decision has three possible outcomes: roll out, extend with an adjusted scope, or stop. Typical criteria include:

• Processing time per request is below the baseline by a share you defined in advance.
• Completeness and hit rate reach the agreed values in your most important product groups.
• Unflagged mismatches are rare enough for your team to trust the flags.
• The team uses the software day to day without having to be pushed.
• The route for product data and results into your systems is settled.

For the internal business case, project the measured time savings onto your annual volume. How to build it is shown in our guide to automating proposals.

Test plan: your pilot in six steps

1. Set the goal and criteria: what question should the pilot answer, and at what values will you roll out?
2. Define the scope: one product line or location and a clearly named team.
3. Measure the baseline: log processing time and corrections for the same request types.
4. Build the test set: a cross-section of real requests by format, language, size and product range.
5. Run the pilot: track the metrics continuously and review errors with the vendor every week.
6. Evaluate and decide: compare the first and last weeks, then roll out, extend or stop.

FAQ: piloting AI quoting software

Should the pilot run in parallel with manual processing?

For some requests, yes. If your team handles a few requests twice, you can see straight away whether results and timing hold up. Doing everything twice isn't worth it; for the rest, comparing against your baseline is enough.

How many requests does a meaningful pilot need?

There's no universal number. Every important format, language and core product group should come up several times. For optimal product suggestions, Scalera recommends 50 to 150 completed offers, depending on your portfolio (Scalera, FAQ).

Can I use my own bill of quantities in the demo?

Yes. In a Scalera demo call, relevant positions can be extracted from your own BoQs. That still doesn't replace hard numbers from day-to-day work.

What counts as a hit in a pilot?

A product suggestion your team accepts into the offer without changes. If someone picks one of the alternatives the AI also offers, count that separately: the right answer was there, just not in first place.

Can I test several vendors at the same time?

Yes, as long as they all get the same test set and the same metrics. Otherwise you're comparing different tasks. For an overview of vendors, see our comparison of AI assistants for bills of quantities.

Does the pilot team need AI training?

The EU AI Act doesn't prescribe a mandatory training format. Providers and deployers of AI systems must, however, take measures to support AI literacy. In practice: before you start, explain to the team how to read suggestions, alternatives and confidence scores.

Who should make the final roll-out decision?

Ideally the person who set the criteria before the start, usually the head of sales or inside sales. The experience of the team that quotes every day belongs in the decision.

What happens after a successful pilot?

You extend the rollout step by step to further product lines or locations and connect your systems more tightly. How the link to ERP, CRM and PIM works is covered in ERP integration for tenders and quotes.

Next steps

• Put the goal, scope and decision criteria in writing before you contact vendors.
• Measure the baseline for your most common request types.
• Build a test set of real requests and use it for every vendor.
• Ask for a demo call to see how Scalera handles your own BoQs.
• 150+ companies already work with Scalera. Book a demo