How to Verify a Third-Party AI API Provider Is Using the Real Model

When you buy access to an AI model from a third-party provider, such as an API reseller, an API aggregator, or an AI wrapper product, you depend entirely on their claim about which model they use. Imagine paying the Gemini Flash 3.5 rate while your requests are actually forwarded to the cheaper Gemini 3.0. From the user's side, the difference is almost invisible.

The way to verify a third-party AI API provider is using the real model is not to read one or two answers, but to compare hundreds of responses against the first-party provider, statistically. If your product is built on a specific model, this invisible quality gap can lower your product's accuracy without you noticing.

This article is a test procedure we put together and recommend, not a report on any particular provider: every number below is an illustrative example of how the method works.

What is model substitution?

Model substitution is when a provider serves a different, usually cheaper, model than the one it claims and charges for. It takes several forms:

• An older version — the model is swapped for a previous generation with a lower rate.
• A quantized version — the same model made lighter, with slightly lower quality.
• Partial routing — only part of the traffic goes to a cheaper model, so average quality drops only a little.

Terms used throughout this article:

Term	Meaning
First-party provider	The company that builds the model and sells its API directly, for example DeepSeek, Anthropic, or Google.
Third-party provider	A reseller, aggregator, or wrapper that sells access to someone else's model.
Test condition	One source of responses in the experiment, for example the first-party provider or the provider under test.
Similarity score	A number from 0 to 100 that says how similar two responses are.
Run-to-run variability	The natural difference between responses from the same model to the same prompt.
Negative control	A deliberately different model, used to prove the test can detect a difference.
Equivalence test	A statistical test that shows two things are equivalent within a tolerance set in advance.

Why is model substitution hard to prove?

Because LLMs are probabilistic: the same prompt, sent to the same model at the same provider, can return a different response every time. A single response that "feels different" is therefore not evidence.

Even at temperature 0, output is not guaranteed to be identical (nondeterminism). Causes include:

• Request batching on the provider's servers.
• The order of floating-point operations on the GPU.
• Small updates to the model or infrastructure on the provider's side.

So the right question is not "are the responses exactly the same?" but: "is the difference in the third-party provider's responses larger than the model's own run-to-run variability?"

How do you verify a third-party AI API provider is using the real model?

First measure how similar the first-party provider is to itself, then show the third-party provider is just as similar. The first-party provider is the reference, and for each prompt we measure two similarity scores:

1. Reference similarity to itself (self-consistency): a first-party response compared with a first-party response from a separate request. This score is the baseline, the upper bound of reasonable similarity for that model.
2. Reference similarity to the provider under test: a first-party response compared with the third-party provider's response.

If the third-party provider really uses the same model, the second score should be statistically equivalent to the first. If it is clearly lower, that points to a different model. One component is often missed: we also need evidence that the test can detect a different model, and that is the job of the negative control.

What do you need before testing?

Three things: a first-party API key, a set of challenging test prompts, and locked test conditions so the provider is the only difference.

1. An API key from the first-party provider. For example DeepSeek, Anthropic, or Google Gemini. Responses from here are the reference.
2. A set of test prompts. Ideally hundreds to thousands, for example 1,000. On a limited budget, 200–300 prompts can give an early signal because the design is paired (see the statistical analysis section). Include prompts that demand high capability: reasoning, math, programming, and multi-step instructions. On simple prompts, old and new models tend to answer alike, so substitution is hard to detect.
3. Equivalent test conditions. The only variable that differs is the provider (the base URL). Every other parameter is locked:
   • Exactly the same model name and version; use a pinned version if one is available.
   • The same system prompt, temperature, top_p, max_tokens, and stop sequences.
   • The same reasoning/thinking mode, on or off in every condition.
   • No web search, tool use, or function calling.
   • All tests run in the same time window, so model updates at the first-party provider do not affect the results.

How should the experiment be designed?

Send every prompt to four test conditions: reference, baseline, provider under test, and negative control. This example uses DeepSeek V4 as the model being verified. The notation S(A,B) means the similarity score between the responses of condition A and condition B.

Test condition	Source	Model	Purpose
A. Reference	First-party provider	DeepSeek V4	Reference response
B. Consistency control (baseline)	First-party provider, separate request	DeepSeek V4	Measures run-to-run variability: S(A,B)
C. Provider under test	Third-party provider	DeepSeek V4 (as claimed)	What we are verifying: S(A,C)
D. Negative control	First-party provider	A cheaper or older model	Proves the test can detect a different model: S(A,D)

Condition D is the most important component and the one most often skipped. Pick the model most likely to be used as a substitute, for example Gemini 3.0 when verifying a Gemini Flash 3.5 provider, or the previous DeepSeek version.

While putting this procedure together, condition D was the step most tempting to drop: it adds API cost, and its result does not directly answer "is this provider honest?". But without it, a "pass" cannot be told apart from a test that simply cannot see a difference. That is why this procedure computes S(A,D) first, before looking at S(A,C). If its gap to S(A,B) is small, change the prompts or the metric first, rather than concluding anything about the provider.

Optional, but very useful: take 3–5 samples for condition B on every prompt. That lets you estimate the spread of run-to-run variability per prompt, not just a single data point.

Which metrics measure response similarity?

Use an LLM judge as the main metric, backed by embedding similarity, accuracy on verifiable questions, and metadata signals. Each metric catches a different kind of difference.

LLM-as-a-judge

An LLM is asked to rate the similarity of two responses on a 0–100 scale. To keep the judging objective:

• Use a judge model from a different family than the model under test, to avoid bias toward its own response style.
• Give an explicit scoring rubric: content and factual agreement, structure, writing style, and length.
• Judge blind: hide the test-condition labels and randomise the order of each response pair.
• Run the judge at temperature 0 and measure its consistency by re-scoring a subset of pairs.

Embedding similarity

Cosine similarity between the embedding vectors of two responses. Cheap and deterministic, a good complement to the LLM judge.

Accuracy on verifiable questions

For math or programming questions whose answers can be checked automatically, compare the accuracy of each test condition. Older or quantized models usually lose accuracy on hard questions, and this signal is often sharper than the similarity score.

Metadata signals

Not enough as evidence on their own, but cheap to collect and often an early hint:

• Input token count — send identical prompts and compare usage.prompt_tokens. Models with different tokenizers report different numbers. Note: models in the same family often share a tokenizer, and a third-party provider can manipulate this number.
• Response length and style patterns — the distribution of response length, use of markdown formatting, and characteristic opening phrases.
• Log probabilities (logprobs) — if available on both sides, the token probability distribution is a very strong model fingerprint.
• Capability probes — questions about the knowledge cutoff, the maximum context length, or features that exist only in the newest version.

How do you read the results statistically?

Compute paired differences per prompt, prove equivalence with a TOST test, then check the tail of the distribution. The intuitive approach: if the mean of S(A,B) is 93 ± a margin of error, then the mean of S(A,C) should also sit around 93. The direction is right, but it needs three important refinements.

1. Use paired differences per prompt

Because every prompt is answered by every test condition, compute the score difference at the prompt level (the paired difference) instead of comparing two means separately. A prompt that naturally produces less similar responses scores low in both conditions, so the effect cancels out. The result is a much smaller margin of error for the same number of prompts.

d_i = S(A_i, B_i) − S(A_i, C_i)

MoE_95% = 1.96 × s_d / √n

s_d is the standard deviation of d_i and n is the number of prompts. An example with s_d = 10 points:

Number of prompts (n)	95% margin of error
100	± 1.96 points
300	± 1.13 points
1,000	± 0.62 points

The margin of error shrinks with the square root of n: doubling the precision takes four times as many prompts. If the score distribution is not symmetric (common on a 0–100 scale), use a bootstrap confidence interval instead of the formula above.

2. "No significant difference" is not the same as "equivalent"

This is the most common misreading. With a small sample, the margin of error widens and almost every provider "passes" because the test lacks statistical power. What needs proving is equivalence, not merely the absence of a significant difference. Use an equivalence test with the Two One-Sided Tests (TOST) method:

1. Set the tolerance δ before looking at the data. A practical rule: half the gap between S(A,B) and S(A,D). For example, with a baseline of 93 and a negative control of 83, δ = 5 points.
2. Compute the 90% confidence interval of the mean of d_i.
3. The provider is equivalent if the whole interval lies inside [−δ, +δ].
4. The provider is lower quality if the whole interval lies above +δ.
5. In any other case, the result is inconclusive: add more prompts.

3. Check the distribution, not just the mean

A provider can reroute only part of the traffic (partial routing). Example: 20% of requests go to a model that scores 15 points lower. The mean drops by only 0.2 × 15 = 3 points, which is easily hidden by variability.

How to detect it: compute the 5th percentile of S(A,B), then the share of prompts in condition C that score below it. A provider serving the claimed model should come out at around 5%. If the share reaches 20%, the tail of the distribution shows an anomaly. A binomial test or a Kolmogorov–Smirnov test can confirm it.

Which mistakes make test results misleading?

Most misleading results come from prompts that are too easy, parameters the provider changes, or cached responses.

• Prompt coverage too narrow or too easy — restricting prompts to one domain (fashion, say) does reduce variance, but risks picking a domain where every model answers alike. The paired design already controls for variation between prompts, so diverse and challenging prompts are the better choice.
• Parameters ignored by the provider — some third-party providers quietly change temperature or max_tokens. Check the response length distribution and the finish_reason values.
• Cached responses — a provider can return responses stored in a cache. Make every prompt unique, for example by adding a random ID at the start of the prompt in every test condition.
• Model updates during testing — the first-party provider can update its model too. Run every test condition in the same time window and record the test date.

What can this method not prove?

This method gives strong statistical evidence, not 100% certainty. Its limits:

• Quantization is hard to tell apart — a quantized version of the same model can produce very high similarity scores. For this case, accuracy on hard questions and logprobs analysis are far more effective.
• It only covers the test window — a provider can switch models after testing ends, so the result describes the test date, not a guarantee going forward.
• Metadata can be manipulated — token counts and other values the provider reports are not enough as evidence on their own.
• Not grounds for an accusation — use the results to ask the provider for clarification or to reconsider your choice of provider, not as an accusation without supporting data.

AI model verification checklist

• ☐ Get API keys for the first-party provider and the third-party provider you will test
• ☐ Assemble 300–1,000 diverse, challenging prompts, some with verifiable answers
• ☐ Lock every parameter; the only difference is the provider (base URL)
• ☐ Run conditions A, B, C, and D (negative control) in the same time window
• ☐ Measure similarity with a blind LLM judge, backed by embedding similarity
• ☐ Confirm S(A,D) is clearly below S(A,B); if not, the test is not sensitive enough
• ☐ Compute paired differences, confidence intervals, and an equivalence test with δ set in advance
• ☐ Check the tail of the distribution to detect partial routing
• ☐ Also compare token counts, accuracy, and response length

Does a cheaper API price mean the provider is cheating?

Not always: some providers really are efficient or offer volume-based pricing. But if your product is built on a specific model, you have every right to confirm what you are actually paying for. With the right baseline, a negative control, and sound statistical analysis, a suspicion becomes measurable data.

Worth stating plainly: Intura sells AI models, so weigh the next suggestion accordingly. For a specific task, there is another way to avoid the question entirely: run the model yourself. Intura's fine-tuned small models run on your own CPU or edge device, so you know exactly which model and version answers every request.