Big Data

Big Data refers to datasets so large, fast-moving or varied that conventional databases and spreadsheets cannot store or analyse them in useful time. In financial services it covers card transaction streams, open banking feeds, app clickstreams, device data and market data, processed on distributed systems. The usual test is volume, velocity and variety: a fraud model scoring card payments must compute features in milliseconds inside the authorisation flow, not in an overnight batch. Fraud detection is the largest use, followed by credit scoring, AML monitoring and payment routing. GDPR's purpose limitation and data minimisation rules conflict with collecting everything, so buyers check the legal basis for each dataset before paying for data assets.

21 November 20252 min read

Big Data refers to datasets so large, fast-moving or varied that conventional databases and spreadsheets cannot store or analyse them in useful time. In financial services it covers card transaction streams, open banking feeds, app clickstreams, device data and market data, processed on distributed systems. The usual test is volume, velocity and variety: a fraud model scoring card payments must compute features in milliseconds inside the authorisation flow, not in an overnight batch. Fraud detection is the largest use, followed by credit scoring, AML monitoring and payment routing. GDPR's purpose limitation and data minimisation rules conflict with collecting everything, so buyers check the legal basis for each dataset before paying for data assets.

The three Vs of big data

Volume is the amount: a mid-sized acquirer can log hundreds of millions of authorisations a year, each with dozens of fields. Velocity is the speed: a fraud model scoring card payments has to return a decision inside the authorisation flow, while the customer waits at checkout, so features must be computed in milliseconds rather than in an overnight batch. Variety is the mix of structured tables, free-text payment references, PDFs, images and device signals. Some definitions add veracity, because data from different sources disagrees and must be cleaned before it can be trusted.

Big data in fintech: main uses

Fraud detection is the largest application. Models compare each transaction against the customer's history, device, location and network-wide patterns, and flag anomalies. Credit scoring uses transaction histories and other alternative data to underwrite applicants with thin bureau files. AML transaction monitoring links accounts, counterparties and devices to find mule networks. Pricing, churn prediction and approval-rate optimisation in payment routing are other common uses.

Example: what scale buys

An illustrative card issuer with 2 million cards sees 40 million transactions a month and a 0.08% fraud rate by value. Training models on its full history, rather than a sample, lets it cut fraud losses by catching low-value test transactions that precede large ones. If losses fall from 0.08% to 0.06% on €1 billion of monthly spend, that saves €200,000 a month.

Big data and GDPR

Collecting everything conflicts with EU data protection law. GDPR requires purpose limitation and data minimisation, gives individuals the right not to be subject to decisions based solely on automated processing that significantly affect them, and allows fines of up to €20 million or 4% of worldwide annual turnover, whichever is higher, for breaching core principles.

Lending adds further limits. The second Consumer Credit Directive, applying from 20 November 2026, restricts creditworthiness data to relevant information on income and expenses, excludes special-category data and rules out social networks as a source. Under the AI Act, AI systems used to assess consumers' creditworthiness are high-risk, with obligations applying from 2 December 2027.

Big data in fintech M&A

Buyers increasingly pay for data assets, but the value depends on whether the data can legally be used after the deal. Diligence checks the consent and legal basis for each dataset, retention schedules, whether models were trained on data the target may not reuse, and any history of a data breach.

Comments