Skip to content

AI & Automation 8 min read

AI Fraud Detection in 2026: What the Models Are Actually Doing

Evaluate AI fraud detection with a practical test of signals, false declines, fraud losses and operating costs, plus the EU AI Act's fraud boundary.

By Shaun Toh Published Updated
TL;DR

Evaluate fraud models by the decisions they improve: measure fraud losses, legitimate orders, friction and operating cost on comparable traffic. Model architecture alone does not establish performance, and EU AI Act classification depends on the intended use.

AI fraud detection is useful when it improves a payment decision: allowing a legitimate order, stopping a fraudulent one, or sending an uncertain case for further checks. A model's name does not establish that improvement. The operator needs to know what it predicts, what data reaches it before the decision, and what happens when its score changes.

Evaluate the complete decision policy against authorization, completed orders and chargeback outcomes. A lower fraud rate can come from better discrimination or from rejecting more customers. A higher approval rate can come with additional losses. Neither headline is enough to choose a system.

This guide sets out a practical evaluation protocol, rather than a ranking of vendors. For adjacent decisions, see AI across the payments stack, rule engines vs ML hybrid design, and the fraud prevention platform comparison covering risk scores, managed decisions and liability guarantees.

Separate the Prediction from the Action

Start the procurement discussion with a specific failure to address. Transaction fraud, account takeover, automated card testing and compliance-adjacent AML monitoring are different evaluation problems. An account may need intervention before any payment attempt; a checkout score cannot demonstrate that it protects the login flow.

Ask each provider to describe the output and the action separately. Does the output estimate a fraudulent payment, a fraudulent dispute, an abusive account or another event? Does your integration use it to block, request authentication, queue a review or simply inform an analyst? Who owns the threshold, and can you reverse a change without a new deployment?

Stripe's Radar documentation provides a concrete example of why scope matters: it lists screening for cards, card-backed wallets, ACH Direct Debit and SEPA Direct Debit, and distinguishes feature availability by plan. Custom rules, backtesting, risk scores and manual reviews are listed as configurable capabilities on specified plans. Verify the relevant plan and payment flow during procurement; the existence of a capability does not prove a particular fraud reduction.

Keep the model score, the fraud-policy action and the issuer response as separate fields. Passing a merchant's fraud check does not guarantee an issuer approval or a completed sale. Otherwise an issuer decline can be counted as a model block, making both the baseline and the challenger results misleading.

Inspect the Signals Available at Decision Time

Treat the following as an evaluation checklist, not a claim that every provider uses every feature. Ask for an integration-specific data map showing which fields are present, missing or delayed.

Signal familyWhat to ask the providerWhat to test in your integration
Transaction contextWhich amount, currency, merchant and payment-method fields affect the score?Missing fields, new markets and unusual order sizes
VelocityWhich identifiers and time windows are used?Repeated attempts versus legitimate retries
Device and sessionWhich client-side events must reach the service?Browser restrictions, mobile apps and absent telemetry
Account historyIs the model evaluating the account, payment or both?New customers, returning customers and account recovery
Network relationshipsWhat cross-customer or cross-merchant information is available?Coverage in your payment methods and markets
Behavioral signalsWhat interaction data is collected and for which purpose?Accessibility needs, shared devices and changing customer behavior

A velocity check needs context. Repeated attempts can reflect an attack, but your own retry behavior may also produce repetition. A BIN is a card-range attribute, not a unique customer identifier. Require a clear definition of the entity being counted before interpreting a velocity feature.

For device and behavioral signals, ask what happens when data is absent. Does the system increase risk, return an incomplete result or fall back to a different policy? Test that outcome deliberately. Also review data collection, retention and access with the people responsible for privacy and security. More collected attributes are not, by themselves, evidence of better decisions.

Architecture Is a Question, Not a Performance Benchmark

Ask a vendor to explain how it uses transaction history, network relationships and session behavior without assuming a particular implementation. Gradient boosting, sequence models, graph methods and combinations of models are possible approaches. The useful evidence is performance on your target problem and the ability to operate the system reliably, not an architecture label.

Use the technical review to establish operational limits:

  • Prediction target: the event being predicted, its observation window and the label used to confirm it.
  • Data timing: which inputs existed before the decision, rather than becoming available after a dispute or investigation.
  • Updates: who can change the model or rules, how changes are recorded and how performance is monitored afterward.
  • Failure behavior: what happens if scoring times out or a required input disappears.
  • Decision trace: which score, rule, threshold and version caused the action, and what an analyst can inspect.

Measure decision latency across your actual path, including feature retrieval and external calls. Choose a timeout and fallback policy that matches your exposure. A score returned after the checkout has already moved on cannot provide the intended control.

A Comparison Protocol for Fraud and False Declines

The following is PaymentBrief's proposed operating protocol. It is not a vendor benchmark or a report of a test we conducted.

1. Fix the population and baseline. Define eligible traffic, payment methods, markets, time window and exclusions before comparing results. Separate new and returning customers where relevant. Record the existing rules and authentication policy; otherwise a policy change can be misattributed to the model.

2. Make the denominators explicit. For each policy, report attempted orders, fraud blocks, authentication requests, completed legitimate orders, confirmed fraud losses and review workload. State whether a rate uses attempts, successful payments or payment value. Deduplicate retries where the question is about customer orders, while retaining attempt-level data for attack analysis.

3. Account for delayed outcomes. A recently approved payment with no reported dispute is not necessarily confirmed legitimate. Choose an outcome window suited to the payment method and fraud problem. Compare cohorts with similar observation time, and distinguish provisional results from mature ones.

4. Treat blocked traffic as an evidence gap. A blocked transaction usually cannot reveal the outcome it would have had if accepted. Analyst judgments, customer contacts and later successful attempts can provide evidence, but are not perfect labels. Do not report every unconfirmed block as prevented fraud, or every customer complaint as a proven false decline.

5. Use historical and live evaluation for different questions. Historical replay can show where policies disagree on recorded data. It cannot recover missing signals or automatically establish what blocked customers would have done. Shadow scoring can test integration coverage and latency without changing decisions, but it has the same limitation on counterfactual outcomes. If a controlled live comparison is appropriate, agree the allocation, exposure limits and stop conditions with the risk owner first.

6. Compare the total outcome. Hold the fraud-loss budget or legitimate-order objective constant when comparing policies. Report uncertainty and important segment differences rather than declaring a winner from a small headline change. Include review effort, authentication abandonment, vendor fees and engineering support in the decision.

This approach also prevents a common measurement error: confusing the share of blocked transactions that were legitimate with the share of all legitimate transactions that were blocked. Those are different denominators and answer different questions. Require the vendor to write out its formula.

A Worked Economic Example

The following figures are hypothetical monthly assumptions in US dollars. They illustrate arithmetic, not expected merchant results or an industry benchmark. Assume a controlled comparison has estimated the incremental effects with adequate outcome follow-up.

Change under the candidate policyHypothetical assumptionMonthly contribution
Additional completed legitimate orders400 orders at $15 contribution per order+$6,000
Reduction in net fraud lossesLosses after recoveries fall by $3,000+$3,000
Additional vendor and operating costsFees, reviews and support increase by $4,000−$4,000
Net improvement$6,000 + $3,000 − $4,000+$5,000

Use contribution after variable order costs, not gross sales, for the first line. Define net fraud losses consistently so recoveries and dispute costs are not counted twice. Include any ongoing model or integration support in operating costs; evaluate one-off implementation costs separately.

The result is sensitive to legitimate-order recovery. If only 100 additional legitimate orders complete, their contribution becomes $1,500 and the same calculation falls to $500 per month. If additional review costs exceed the assumption, the benefit can disappear. This is why an approval uplift alone is insufficient evidence for a purchase.

The EU AI Act Boundary: Assess the Actual Use

Annex III point 5(b) lists AI used to evaluate natural persons' creditworthiness or establish their credit score, with an exception for financial fraud detection. That is a specific exception within the creditworthiness entry, not a general exemption from all law.

The text does not establish that every fraud-driven payment decline or account restriction automatically becomes a high-risk use. If a system also performs creditworthiness assessment or another listed function, assess that purpose separately. Classification depends on the system's intended use and the applicable provisions, not simply its vendor label.

For procurement, record the purpose, affected people, decisions supported and any reuse of the output. Have the responsible legal and compliance teams determine classification, applicable obligations and timing for that deployment. A feature-importance display or vendor compliance statement alone does not settle those questions. The companion EU AI Act payment-system classification guide provides further context.

Build, Buy or Combine

There is no universal transaction-volume threshold at which building a fraud model becomes economical. Compare the cost and control of the available choices for your particular problem.

Buying through a PSP or specialist can be attractive when it supplies relevant signals, integration coverage and an operating service that your team would otherwise maintain. Building can be worth evaluating when proprietary data addresses a material gap and the team can support labeling, monitoring, incident response and ongoing changes. A hybrid can retain internal policy control while using external scores.

Before committing, require a written decision covering the measured benefit, total cost, data dependencies, fallback behavior and exit path. Keep difficult cases such as synthetic identity fraud within the stated scope of the evaluation rather than assuming a general transaction score covers them. Choose the approach that improves the decisions you can measure and sustain.

Sources & methodology (2)

Radar offers transaction screening and configurable controls, with feature availability varying by plan.

Product documentation establishes capabilities, not a comparative performance result for an individual merchant.

Checked:

Annex III point 5(b) covers natural-person creditworthiness and credit scoring, with an exception for financial fraud detection.

This exception belongs to the creditworthiness entry; it is not a blanket exemption from every applicable legal obligation.

Checked:

Source types explained in our Methodology.

Shaun Toh By Shaun Toh · Director, Digital Payments · Razer

More AI & Automation briefings