Before an enterprise signs a pilot, it often wants to test the model first.
The customer’s data science team wants to run the beta model against its own test set. Its security team wants to try prompt injection and jailbreaks. Its procurement team wants a scorecard that compares your model with two competitors.
None of that is a buying decision yet. It’s an evaluation, and it raises legal questions that a standard pilot agreement doesn’t answer in any detail.
SaaS AI Lawyer Andrew S. Bosin, a two-time SaaS founder, sees these gaps when an evaluation starts on a handshake, or on a customer’s generic “software evaluation” form written for tools that don’t learn, drift, or produce different answers each run.
This article covers the terms specific to evaluating an AI model: the evaluation license, model versioning, testing methods, the customer’s test data, ownership of results, and what either side can say about them. For the commercial terms that follow a successful evaluation (the pilot fee, success criteria, conversion pricing, and liability caps), see Andrew’s guide to the AI proof of concept agreement for paid pilots.
Quick Answer
An AI model evaluation agreement should:
- Grant an evaluation-only license, limited to internal testing by named evaluators, with no production use
- Freeze or version the beta model, so results are tied to a specific model and the startup can update it without invalidating the test
- Set rules for how testing may be done, including whether adversarial or red-team testing is allowed, and prohibit model extraction and distillation
- Protect the customer’s evaluation dataset, and keep it out of the startup’s training data
- Decide who owns the results, and whether either party can publish or share them
- Require the customer to report security and safety findings privately before disclosing them
- Limit what the startup can claim publicly about the results, and require customer approval before any use of the customer’s name
Evaluation vs. Pilot: Why a Separate Set of Terms
A pilot tests whether the product works in the customer’s environment and ends in a buying decision. An evaluation often comes earlier and is narrower: the customer is testing the model itself.
In an evaluation, the customer typically:
- Runs the beta model against its own benchmark or “golden” dataset
- Compares it against competing models on the same tasks
- Tests safety, bias, and security behavior, sometimes adversarially
- Produces an internal scorecard, often for a committee that never touches the product
Some startups use one agreement for both. Others sign a short evaluation agreement first and a paid pilot later. Either way, the terms below need to be addressed somewhere. When they’re folded into a pilot agreement, they belong in their own section, not scattered through the commercial terms.
1. The Evaluation-Only License
The license should be narrow and say so plainly:
- Internal evaluation only: no production use, no decisions affecting real customers, employees, or patients
- Named evaluators: specific teams or individuals, with the customer responsible for their compliance
- Third-party evaluators: if the customer uses outside consultants or an audit firm, they must be named in advance and bound by written confidentiality obligations at least as protective as the agreement’s
- Usage limits: request volume, rate limits, and a fixed evaluation period
The line between “evaluating” and “using” is easy to cross. A customer that routes live support tickets through a beta model “to see how it does” is using it in production, without production warranties, security review, or fees.
2. Freeze the Model Version
This issue has no real equivalent in traditional software evaluations.
Beta AI models change frequently. The startup may retrain, swap the underlying foundation model, change a system prompt, or adjust safety filters mid-evaluation. Each change can move the results. A customer that scored the model in week one and re-tests in week four may be evaluating two different products.
The agreement should address:
- Version identification: the specific model version or build under evaluation, identified in an order form or exhibit
- Change notice: advance notice of material changes during the evaluation period, or a commitment to keep the evaluated version available until the period ends
- Results tied to version: an acknowledgment that results apply only to the version tested and aren’t a representation about later versions
- Upstream changes: disclosure that the startup’s model provider may update or retire models on its own schedule, outside the startup’s control
For the startup, the version-tied acknowledgment matters most. It prevents a customer from treating an early beta score as a performance promise when the production contract is negotiated.
3. How the Customer May Test: Red-Teaming and Its Limits
Enterprise security teams increasingly want to test AI systems adversarially. NIST’s Generative AI Profile (NIST AI 600-1, July 2024) treats red-teaming as a recognized practice for managing generative AI risk, and many enterprise AI governance programs now build on it.
That’s healthy, but unmanaged it creates risk for the startup. Adversarial prompts can trigger abuse monitoring, exhaust rate limits, generate harmful output in logs, or break the startup’s own terms with its model provider.
What to Permit
If red-teaming is allowed, the agreement should set rules of engagement:
- Which techniques are in scope: prompt injection, jailbreaks, bias probing, data-leakage testing
- Which environment: a dedicated evaluation instance, not shared infrastructure
- Advance notice, so the startup’s monitoring doesn’t treat the testing as an attack
- Volume and timing limits
What to Prohibit
Some activities should be off-limits even in an adversarial evaluation:
- Model extraction: systematic querying designed to reconstruct the model’s weights, parameters, or behavior
- Distillation: using outputs to train or improve a competing model
- Reverse engineering of the model, system prompts, or architecture, except where law prohibits the restriction
- Denial-of-service or load testing beyond the agreed limits
These limits also protect the startup’s upstream position. Foundation model providers restrict how their outputs can be used. OpenAI’s terms of use, for example, prohibit customers from using its output to develop competing models. If the startup’s product runs on a third-party model, the evaluation terms it gives customers should be no looser than the terms its provider gives the startup. Andrew’s guide to AI beta testing agreement key terms covers how definitions like Inputs and Outputs shape these restrictions.
4. The Customer’s Evaluation Dataset
The customer’s test set is often one of its most carefully built assets: labeled examples, edge cases, and answer keys assembled by domain experts. It’s also exactly what an AI startup would most like to learn from.
The agreement should state that:
- The customer owns its evaluation dataset and answer keys
- The startup won’t use the dataset to train, fine-tune, or tune its models, including prompt tuning aimed at that test set
- The startup won’t keep copies after the evaluation, other than logs deleted on a stated schedule
- The startup’s own benchmarks and internal test results shared with the customer are the startup’s confidential information
The training restriction protects both sides. If a startup tunes its model on the customer’s test set, the customer’s results become meaningless, a problem machine learning practitioners call test-set contamination. NIST’s Generative AI Profile specifically flags cross-contamination between training and test data as a limitation to document when benchmarking a model. A clean no-training commitment on the evaluation data is usually easy for a startup to give, and it builds trust.
Broader training questions are covered in Andrew’s articles on the no-training clause and training on beta customer data.
5. Who Owns the Results, and Who Can Share Them
This is the section enterprise evaluation forms most often get wrong from the startup’s point of view.
Ownership
A workable default:
- The customer owns its internal evaluation report: its analysis, scoring, and conclusions
- The raw results about the startup’s model are the startup’s confidential information: scores, failure cases, and outputs
- Each side may use the results internally: the customer for its buying decision, the startup to improve its model
Publication and Benchmark Restrictions
Software vendors have restricted customers from publishing benchmark results for decades. In AI, the stakes are higher, because a single published score on an early beta can follow the company through every later sales cycle.
The agreement should address:
- Public disclosure: no publication of results without the startup’s written consent
- Competitive sharing: no sharing of results with competing vendors, including in a vendor bake-off
- Consultants and analysts: whether results can go to industry analysts or outside advisors
- Required disclosures: carve-outs for regulators, auditors, and legal requirements
Enterprise customers will sometimes push for a right to publish safety findings in particular. A reasonable compromise is a coordinated disclosure period (see below), rather than an absolute ban.
6. Security and Safety Findings
Red-teaming produces findings: prompts that bypass safety filters, inputs that leak other users’ data, or outputs that show discriminatory patterns.
The agreement should require the customer to:
- Report findings to the startup promptly and in enough detail to reproduce them
- Keep findings confidential for a set period, commonly 60 to 90 days, while the startup investigates
- Not exploit a finding beyond what’s needed to document it
The startup, in turn, should avoid promising to fix every finding during a beta. A commitment to investigate in good faith and share what it decides is usually enough.
7. What the Startup Can Say About the Results
A strong evaluation result is sales material. It’s also a potential liability if the startup overstates it.
In September 2024 the FTC launched Operation AI Comply, a set of enforcement actions against companies making deceptive AI claims. In one, the agency alleged that DoNotPay had promoted its AI as a substitute for a lawyer without testing whether its output matched a lawyer’s work. The lesson for startups: performance claims need testing to back them up, and a single customer’s beta evaluation may not be enough to support a general claim.
The agreement should state that:
- The startup won’t name the customer or attribute results to it without written approval
- Any published performance claim will be accurate and limited to the version, tasks, and conditions actually tested
- Aggregated, anonymized results may be used internally, and externally only in a form that can’t identify the customer
Sample Clauses
These are illustrative starting points, not ready-to-sign terms. The right language depends on the model, the customer, and the upstream provider.
Evaluation License and Restrictions
Subject to this Agreement, Company grants Customer a non-exclusive, non-transferable license during the Evaluation Period to access and use the Evaluation Model solely for Customer’s internal evaluation of its performance, by the Evaluators identified in Exhibit A. Customer will not (a) use the Evaluation Model or any Output in production or to make decisions affecting any individual; (b) use any Output to train, fine-tune, or otherwise develop any machine learning model; (c) attempt to extract, reconstruct, or infer the Evaluation Model’s weights, parameters, architecture, or system instructions; or (d) conduct adversarial, security, or load testing except in accordance with the Testing Protocol in Exhibit B.
Model Version
The Evaluation Model is the model version identified in Exhibit A. Company will provide at least ten (10) business days’ notice before making any material change to the Evaluation Model during the Evaluation Period. Customer acknowledges that Evaluation Results relate solely to the version tested and do not constitute a representation or warranty regarding any other version.
Evaluation Dataset
As between the parties, Customer owns the Evaluation Dataset. Company will use the Evaluation Dataset solely to perform the evaluation, will not use it to train, fine-tune, or adjust any model or system instructions, and will delete it, including associated logs, within thirty (30) days after the end of the Evaluation Period.
Results and Disclosure
Evaluation Results concerning the Evaluation Model are Company’s Confidential Information. Customer may use Evaluation Results internally to decide whether to purchase Company’s services, but will not publish them or disclose them to any provider of competing products or services without Company’s prior written consent. If Customer identifies a security vulnerability or safety failure, Customer will notify Company in writing and will not disclose it to any third party for ninety (90) days after notice, except as required by law.
How This Fits With Your Pilot Agreement
If the evaluation leads to a paid pilot, decide which evaluation terms carry forward:
- Carry forward: dataset protections, confidentiality of results, extraction and distillation prohibitions, and disclosure rules for findings
- Replace: the evaluation-only license, which gives way to the pilot’s scope and usage terms
- Update: model version terms, since the pilot may run on a newer version
The AI proof of concept agreement guide covers the pilot terms that take over from there. If the customer is helping shape the product rather than just testing it, a design partner agreement may fit better.
AI Model Evaluation Agreement Checklist
- Evaluation-only license with named evaluators and no production use
- Third-party evaluators named and bound by confidentiality
- Model version identified, with change notice and results tied to that version
- Red-team rules of engagement, or a clear prohibition on adversarial testing
- Prohibitions on model extraction, distillation, and reverse engineering
- Evaluation terms consistent with the startup’s model provider terms
- Customer owns its evaluation dataset; no training or tuning on it
- Ownership of the evaluation report and the raw results
- Publication and competitive-sharing restrictions, with legal carve-outs
- Private reporting and a coordinated disclosure period for findings
- Limits on the startup’s public performance claims and use of the customer’s name
- Terms that carry forward into any pilot or MSA
For a broader set of AI-specific beta clauses, see the AI beta agreement template guide.
Frequently Asked Questions
What is an AI model evaluation agreement?
It’s a contract that lets a prospective customer test an AI model, often a beta or pre-release version, for a limited period and purpose. It covers how testing may be done, who owns the test data and results, and what either party can disclose.
Is an evaluation agreement the same as a pilot agreement?
Not quite. A pilot tests the product in the customer’s environment and usually leads to a buying decision. An evaluation typically tests the model itself, often earlier and more narrowly. Many startups combine both in one document, but the evaluation terms still need their own section.
Should we let enterprise customers red-team our beta model?
Often yes, because it builds trust and surfaces real issues. But it should happen under written rules of engagement: agreed techniques, a dedicated environment, advance notice, and a prohibition on extraction and distillation.
Can a customer publish benchmark results from our beta?
Only if the agreement allows it. Most evaluation agreements prohibit publication and sharing with competing vendors without the startup’s consent, with carve-outs for legal and regulatory requirements.
Can we use the customer’s evaluation dataset to improve our model?
It’s usually a bad idea even if the customer agreed. Tuning on the test set makes the customer’s results unreliable, and most customers will expect a clear no-training commitment for their evaluation data.
What if the customer finds a serious safety or security problem?
A well-drafted agreement requires private notice to the startup and a confidential period, commonly 60 to 90 days, before any disclosure, while the startup investigates.
Get Evaluation Terms in Place Before the Testing Starts
Andrew S. Bosin is a New Jersey technology attorney and two-time SaaS founder who represents AI and SaaS startups nationwide on a flat-fee basis. He drafts model evaluation, beta, pilot, and design partner agreements built around the startup’s actual model stack and upstream provider terms, so customers can test the product thoroughly and the startup keeps control of its model, its results, and its reputation.
He personally drafts every agreement, gives every client his personal cell phone number, and responds quickly.
Andrew S. Bosin LLC
36 Highland Road, Glen Rock, NJ 07452
📞 201-446-9643
📧 andrewbosin@gmail.com
🌐 www.njbusiness-attorney.com
Call for a consultation. Flat-fee legal services for AI and SaaS startups, nationwide.
#AISaaSLawyer #AIStartupLawyer #BestAISaaSLawyer #AIEvaluation #BetaAgreement #AIContracts #RedTeaming #SaaSContracts
This post is provided for general informational purposes and does not constitute legal advice.