Fraud detection for a growing fintech

Four different problems wearing one word

A team that says it needs fraud detection is usually describing at least four different problems that share almost no machinery. Treating them as one produces a single score that is mediocre at all of them, and a review queue where nobody can tell what they are looking at. Name them separately before writing any code, because the data, the latency budget, the label and the remedy differ in every case.

Onboarding fraud is a fabricated or stolen identity opening an account. The decision happens once, in seconds, with almost no history on the applicant, and the strongest signals come from outside your own data: document checks, device intelligence, phone and email tenure, and the fact that this same device opened eleven accounts last week.

Account takeover is a real customer's account being used by someone else. The signal is the break in a pattern you already have — a new device, an impossible travel time, a changed contact detail immediately followed by a withdrawal attempt. Sequence matters more than any single event.

Transaction fraud is an unauthorised payment or a card used without the holder's consent. Volume is high, the decision window is milliseconds, and the ground truth arrives later as a dispute.

Scam losses are the customer authorising a payment they were manipulated into making. This is the fastest-growing category in most consumer products and the hardest to model, because from the system's point of view everything is legitimate: the real customer, on their known device, doing something they chose to do. The intervention that works is usually a well-timed warning in the payment flow, not a score, and building it well is an interface problem more than a modeling one.

Your review capacity is the threshold

Fraud rates run somewhere between five basis points and one percent of transactions depending on the product, which makes accuracy a meaningless metric — a model that approves everything scores above ninety-nine percent. The metric that matters is precision at the volume your team can actually work.

Do this arithmetic first. Six analysts, forty cases each per day, is two hundred and forty cases a day. That is the budget. The question for the model is not what its area under the curve is; it is how much fraud is caught in the top two hundred and forty cases, and how that compares with what the current rules put there. Tune the threshold to the queue, and revisit it when the team changes size or when case handling time changes.

This reframing has a useful consequence: cutting average case handling time from nine minutes to four doubles effective capacity without touching the model at all. The analyst tool — the timeline of the account, the linked entities, the one-click decision with a reason code — is usually the highest-return part of the whole system and it is almost always the part that gets built last, badly, by whoever had a spare sprint.

Both errors cost money, so price them

Declining a good customer is not free, and treating it as free is how a risk team quietly becomes a growth problem. A declined legitimate card payment costs the margin on the sale, some probability of the customer never returning, and a support contact. A wrongly declined loan applicant costs the expected profit on that loan plus acquisition spend already sunk. A frozen account costs a complaint and sometimes a public one.

Put dollars on both sides and the threshold becomes a business decision anyone can check: set it where the marginal fraud dollars prevented equal the marginal good dollars lost. Two things follow. The threshold should vary by amount, because it is worth reviewing a five thousand dollar transfer at a false positive rate you would never accept on a twelve dollar one. And it should vary by segment, because a two-year customer with a clean history and a two-hour-old account are not the same risk at the same score.

Metric	What it tells you	Why it is on the list
Precision at review capacity	Share of the cases your team works that are truly fraud	The only precision number that corresponds to real operations
Detection rate at a fixed false-positive rate	How much fraud value you catch at a customer-friction level you accept	Comparable across model versions when volume moves
Loss rate in basis points of volume	The number the business cares about	Report gross and net of recoveries, and never blend the two
Good customers declined	The cost you cannot see in the loss line	Without it a risk team optimises itself into a growth problem
Case handling time and queue age	Whether capacity or the model is the binding constraint	Usually the cheapest thing to improve
Alert-to-confirmed-fraud ratio per rule	Which rules are earning their false positives	Retires dead rules, which every mature engine is full of

The labels arrive months after the decision

This is the structural fact that makes fraud modeling different from most classification work. A card dispute can be raised long after the transaction, and network rules generally allow cardholders many weeks to months depending on the reason code. An unauthorised consumer debit return has its own window measured in weeks. Confirmed scam cases surface whenever the customer realises and reports.

So your ground truth is always a quarter or more behind, and a model trained only on settled labels is optimised against the fraud patterns of the previous season. Two label sets solve most of this. Fast proxy labels — analyst decisions, immediate reversals, customer reports — are available within hours and are what you monitor on. Slow confirmed labels are what you train and validate on. Keep them in separate columns, never merge them, and always state which one a chart is using, because a monitoring dashboard drawn on slow labels looks fine for six weeks after the wheels come off.

The related trap is that suppressed transactions have no outcome. Everything you declined has no chargeback by construction, so a naive retrain concludes that its own declines were correct. Approve a small random sample below the threshold, accept the loss as the cost of information, and treat that sample as the only unbiased view of what the model is turning away. It is uncomfortable to propose and it is the difference between a model that improves and one that becomes progressively more confident about a shrinking world.

Point-in-time correctness, and the twenty points you lose in production

Here is the bug that has cost more offline-to-online gaps than any other. A feature is computed from a table that has since been updated, so the training row contains information that did not exist at the moment of the decision. The customer's lifetime chargeback count includes the chargeback for the transaction being scored. The device risk rating is today's rating, not the rating that day. The account status field says closed for fraud, which is exactly what you are trying to predict.

Offline results look outstanding. Production is far worse, and the gap is frequently in the range of fifteen to thirty points of precision, which is enough to make a model useless and a team dispirited.

The remedy is architectural, not analytical. Build features from an append-only event log with an event timestamp and a separate ingestion timestamp, and compute every feature as of the decision time using only events whose ingestion timestamp precedes it. That distinction — when a thing happened versus when your system learned it — is the whole discipline. Then serve the same code path online and offline, so training and production cannot drift apart. That is what a feature store buys you, and if you are not going to run one, at least run the same computation in both places from one definition.

The top two rows are worth dwelling on. Counts and sums over rolling windows — per card, per device, per email, per bank account, per merchant, over one hour, one day, seven days and thirty — are unglamorous and carry an enormous share of the signal. And links between entities are what catch organised activity, because a ring reuses infrastructure. Eleven accounts sharing one device fingerprint, or forty accounts paying into one destination account, is a pattern no per-transaction model sees and a simple graph query finds immediately.

Rules are not obsolete, and yours are probably rotten

The healthy architecture has three layers. A small set of hard rules for the things you never want to think about again: blocklists, sanctions screening, impossible geography, obvious velocity breaches. A model for the combinations no rule expresses. And a review queue where humans decide the ambiguous middle.

Rules earn their place by being instant to deploy and trivially explainable, which matters when a customer or a partner asks why something was blocked. What they do not do is age well. Audit the engine at least twice a year and expect to find that a large minority have not fired in a year, a handful generate most of the false positives, and several contradict each other. Every rule should carry an owner, a creation date, a stated reason and a review date, and a rule with no hits in six months should be retired rather than left as sediment.

Adversarial drift is not ordinary drift

In most domains the world changes slowly. Here there is a person on the other side whose job is to find the edge of your policy, and they will find it within days of a change. Response to a new rule is measured in days, not quarters.

That changes the monitoring cadence. Watch the score distribution daily, not monthly; a sudden shift in the shape of the score distribution is the earliest available signal and it arrives long before the chargebacks do. Watch approval rate by segment daily. Alert on the composition of the review queue, because a ring shows up as a cluster of similar cases before it shows up in losses. And run a champion and a challenger continuously so that a replacement is always warm, since the gap between deciding to retrain and having a deployable model is where the losses accumulate.

Buy the parts that need scale, build the parts that need your product

Some things are genuinely hard to replicate and should be bought. Device intelligence and identity verification depend on seeing traffic across thousands of companies. Consortium data — shared negative files and cross-institution signals — is valuable precisely because you cannot build it alone. Sanctions and watchlist screening is a solved commodity.

What is worth building is everything shaped like your product: the feature pipeline over your own event history, the scoring layer, the case management tool, the decision policy, and the feedback loop that turns analyst decisions into labels. A purchased score with no ability to combine it with your own features is a ceiling you will hit within a year.

And under a certain size, build nothing. If you are running a few thousand transactions a day with losses under roughly twenty basis points and a two-person risk team, a decent vendor score, a dozen rules and a manual queue is the correct architecture. A custom model at that volume costs more in maintenance than it saves, and it will be trained on too few positive examples to be stable. The trigger to build is when review capacity is saturated and losses are trending up at the same time. One without the other is not a reason.

The feedback loop is the compounding asset

Every analyst decision is a label, and most companies throw away nearly all of the value by capturing it as free text. Give analysts a short controlled vocabulary of reason codes — twelve to twenty, no more, revised twice a year — and require one on every disposition. Store the case, the evidence shown, the decision, the reason code and the analyst.

Two years of that is a training set nobody can buy, and it is also how you find out that one analyst approves at twice the rate of everyone else, or that a particular reason code precedes a chargeback sixty percent of the time. Sample cases for a second review, measure agreement between analysts, and treat a low agreement rate as a policy problem rather than a training problem — if experienced people disagree on the same evidence, the policy does not say enough.

What a working setup contains

• The four fraud types named and scoped separately, with their own thresholds and queues
• Thresholds set from review capacity and expected value, not from a score that looked round
• An append-only event log with event and ingestion timestamps, and features computed as of decision time
• One feature definition serving both training and production
• Fast proxy labels and slow confirmed labels kept separate and clearly marked on every chart
• A small random approved sample below the threshold, funded deliberately as the cost of information
• Entity-link queries over shared device, email, phone and destination account
• Daily score-distribution and approval-rate monitoring, with a challenger model always running
• Rules with owners, dates and review dates, audited twice a year
• Reason codes from a controlled list on every analyst decision

The mistakes we get called about

• Leaked features, discovered when production precision came in twenty points below the backtest
• One score for onboarding, takeover and payments, mediocre at all three
• A threshold set once and never revisited as volume tripled and the team stayed the same size
• Monitoring on confirmed labels only, so a new pattern was invisible for two months
• Retraining on approved traffic alone, with the model steadily agreeing more with itself
• Three hundred rules, no owners, no review dates, several contradicting each other
• Free-text case notes, throwing away two years of labels an analyst team already produced
• No measurement of good customers declined, so the loss number improved while revenue quietly did not

Bottom line

Name the four problems separately. Set the threshold from what your review team can work and from the dollar cost of both errors, not from a score. Build the event log with two timestamps before you build the model, because point-in-time correctness is worth more than any algorithm choice you will make afterwards. Keep fast and slow labels apart, pay for a small random sample of approvals so you can see what you are declining, add entity links early because that is where organised activity shows up, and monitor daily because the other side adapts in days. And if you are small, a vendor score, a dozen rules and a good case tool is the right answer — the model becomes worth building when the queue is full and losses are rising at the same time.

Frequently asked questions