# the modern data suck

i've got a new job at querio a year ago to escape my growing [frustration] with what i used to think was my dream job: software engineering.

there is no free lunch, one host of frustrations was quickly replaced by another. querio wants to help organisations become more data driven. overwhelmingly, "data-driven" means different things to different people. success of any product relies on finding the a optimum in the many-dimensional space of product offerings. this post is my attempt at mapping the surface. 

preface: i am doing this here mapping with one arm tied behind my back and both my eyes gauged out. so if you're a data practitioner pls dm me.

your frustration is my opportunity.

## the suck

it's helpful to think of the modern data stack not as a collection of tools but a set of high-level requirements imposed on organisations by their own success and the economic and technological factors of the environments markets in which they operate and the world more broadly. this may sound like [abstract drivel](https://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail), so let's get concrete.

### working material: the requirement surface

it is helpful to think of the modern data stack not as a collection of tools, but as a response to requirements placed on organisations by their growth, operating environment and the nature of the business itself.

take the same basic pet-food business at two different scales, as it might have operated three years ago.

with three employees, the person buying instagram ads can ask the person packing boxes whether a campaign brought in good customers. shopify, stripe and a spreadsheet are ugly, but sufficient. questions about customers, stock and revenue can be answered by the people who already carry most of the company’s context in their heads.

at 500 employees, interrogating company data is business-as-usual activity across the organisation. marketing wants to know which channels produce customers who stick around. finance wants margins after discounts, refunds and shipping. fulfilment wants to predict stock requirements. support wants to understand which issues precede cancellations. procurement wants to know how much product to order and from whom.

these are not occasional analytics projects. every team asks questions of overlapping company data every day.

the questions now cross marketing, finance, fulfilment, procurement and support. "good customer" needs an agreed definition. revenue needs to mean the same thing in every report. people need access to some data but not all of it. answers need to be reproducible, and somebody needs to notice when the systems producing them break.

now let meta change its attribution model while ingredient prices jump. both versions of the company need to know which channels, customers and products remain profitable. the three-person company can still work it out together. the 500-person company needs integration, shared definitions, access control, history and reliability.

there is another class of requirement that has nothing to do with scale or squeezing another percentage point out of advertising: compliance. we are selling pet food. we cannot put some crazy shit in it and accidentally make every cat in the uk lose its mind.

the company needs to know what went into each product, where every ingredient came from, which supplier provided it, when it was manufactured and which customers received each batch. if an ingredient turns out to be unsafe, it needs to identify the affected products and customers quickly enough to do something about it. labels, formulations and safety claims also need to match reality.

this resembles the other data problems, but the burden is distinct. marketing data helps the company make better decisions. compliance data helps it demonstrate that it followed the rules and did not harm anyone. an approximate answer may be useful during exploratory analysis; it is not useful during an audit or product recall.

this gives us a useful distinction between structural and emergent data problems.

compliance is structural. the company cannot legally or safely participate in the pet-food market without addressing it. these requirements exist before the company scales, hires a data team or buys any analytics software. they are part of the price of admission.

the earlier problems are emergent. they are not inherent to selling pet food. they arise from how the organisation expands its operations: hiring more people, entering more markets, adding suppliers and sales channels, increasing operating expenditure, and investing capital in warehouses, factories and other infrastructure.

each investment increases the number of decisions being made and the number of people, systems and assets involved in making them. shared context stops scaling. definitions diverge. one team’s output becomes another team’s input. interrogating company data becomes a routine method of coordinating the organisation.

this does not make emergent problems unimportant or optional once they appear. the three-person company can substitute conversation and shared context for formal data systems. the 500-person company cannot. its data requirements are consequences of the operating structure it chose or grew into.

### and now a word from our sponsors

those were the requirements three years ago. now add AI (drink!)!

AI changes both the demand for and the supply of of. when asking a new question required an analyst, some SQL and a place in a backlog, organisations rationed questions. dashboards compressed the questions people were allowed to ask into a predefined collection of charts and filters. now every product promises that anybody can ask anything in natural language and receive an answer immediately. the cost of creating data point is near zero, there is gonna be more data period.

the three-person company no longer needs a data team before it can imagine interrogating all of its data. it can point an LLM at shopify, stripe, support conversations and advertising data and ask questions that previously required specialist work.

the 500-person company sees an opportunity to give the same ability to every team, and eventually to automated agents acting on their behalf. that means more people asking more varied questions, more frequently, with less understanding of where the answers come from.

AI makes asking questions cheaper. it does not make the underlying answers less ambiguous. if revenue has five definitions, the model now has five plausible ways to be confidently wrong. if permissions are unclear, it can expose information faster. if the data is stale or a pipeline is broken, it can turn that failure into a convincing paragraph.

scale still creates coordination problems, but it is not the only force creating requirements. technological shifts change what even a three-person company expects to do with its data. AI does not eliminate the need for integration, shared meaning, permissions, provenance and reliability. it increases the number of people and machines depending on them.

### another example

the pet-food company is one kind of organisation: an order-and-inventory business. it buys or produces goods, maintains a catalogue, holds inventory, sells through one or more channels, fulfils orders and handles returns. its central data objects are products, customers, orders, line items, payments, shipments and stock movements.

whether it sells to businesses or consumers does not completely change that underlying spine, but it does change its shape. a b2c retailer may have many individual customers, relatively small orders, high marketing spend, returns, consent requirements and difficult identity resolution across devices and channels. a b2b retailer may have fewer customers but needs to represent companies, subsidiaries, buyers, negotiated prices, contracts, purchase orders, invoices, credit terms and account-level relationships.

both remain recognisably commerce businesses. they share many core data problems, but the commercial relationship changes the surrounding requirements.

a financial-services company is organised around a different operational primitive. its central objects are not products and order lines but accounts, balances, transactions, obligations and changes of state.

volume may dominate some financial businesses, particularly payment networks, trading venues and consumer transaction platforms. but volume is not the only source of complexity. a business handling relatively few transactions can still face extreme requirements around correctness, timing, risk and regulation.

an order records an agreement to exchange a product for money. a financial transaction may itself create, discharge or transfer an obligation. the data does not merely describe the operation of the business. in important ways, it constitutes the operation of the business.

a payment can be authorised but not captured, captured but not settled, settled and later reversed, refunded, disputed or charged back. money may be held in different currencies, moved between internal and external accounts, split across parties, charged fees, netted against other obligations and settled later. the correct answer to "what is this customer’s balance?" depends on which events have happened, which are final, and the exact point in time being examined.

the organisation therefore needs more than a table of transactions. it needs an authoritative ledger, explicit state transitions, reconciliation between internal and external records, idempotent processing, historical reconstruction and a clear distinction between pending and final events. a small discrepancy is not merely an analytical inconvenience. it represents money that the company, a customer or another institution believes exists somewhere.

financial services also carries structural requirements of its own. the company may need to identify customers, detect suspicious activity, enforce sanctions, prevent fraud, protect customer funds, preserve records and explain decisions to regulators. these requirements are not consequences of becoming a large company. they are conditions of participating in the financial system.

on top of that internal machinery, the company may operate an ordinary customer-facing software product: a banking app, payments dashboard, transfer platform or trading interface. that product produces familiar software data about sign-ups, sessions, feature usage, errors and retention.

the product data and the financial data are connected, but they are not interchangeable. an analytics event saying that a user viewed a balance is not evidence of what their balance actually was. the product interface may be eventually consistent and tolerant of missing telemetry. the ledger cannot be.

### a few more examples

retail and financial services are only two operational shapes.

a software-as-a-service company is organised around accounts, users, subscriptions, entitlements and product usage. its recurring questions concern adoption, retention, billing, feature usage and service reliability. because the product itself is digital, the boundary between product telemetry and operational data can become unusually thin.

a marketplace is organised around interactions between multiple parties. buyers, sellers, workers, advertisers or service providers participate in the same system but have different incentives. the important questions concern matching, liquidity, pricing, reputation, trust, fraud and whether interventions help one side at the expense of another.

an asset-intensive business is organised around physical equipment and processes. manufacturers, logistics companies, utilities and transport operators care about machines, facilities, routes, capacity, maintenance, quality and downtime. their digital records must remain connected to a physical world that is noisy, delayed and capable of breaking in ways the database did not anticipate.

a case-based organisation is organised around long-running records concerning a person or event. hospitals have patients, insurers have claims, governments have cases and schools have students. the record accumulates over time, passes between professionals and may contain highly sensitive information. correctness, consent, chronology and the ability to explain decisions can matter more than raw volume.

a professional-services company is organised around clients, engagements, people, time and deliverables. its data problems concern staffing, utilisation, project margin, forecasting and the reuse of knowledge that often lives in documents or employees’ heads rather than clean operational systems.

a media or advertising company is organised around content, audiences and attention. it may process enormous streams of impressions, clicks, views and interactions. identity, attribution, recommendation and experimentation dominate, while the value of any individual event may be tiny.

these categories are not exclusive. a large retailer may also operate a marketplace, manufacture products, provide credit and run a logistics network. a bank may be simultaneously a regulated ledger, a software product, a marketplace and a case-management organisation. the categories identify the operational primitives that create different families of data requirements.

### what the examples tell us

now you say well done nikolay for showing doing data is hard. i am not finished!

the examples leave us with a simple model. what an organisation does gives its data requirements a base shape. a retailer revolves around products, orders, inventory and fulfilment. a financial-services company is not simply a retailer with more rows: its data constitutes balances and obligations, so correctness, finality and reconciliation become fundamental.

structural obligations add hard edges to this shape. the pet-food company must preserveingredient and batch traceability regardless of its size. the financial company must maintain an authoritative ledger and satisfy regulatory controls before it can operate at all.

scale and operating complexity then make the surface jagged. more teams, systems, markets and investments produce coordination problems, conflicting definitions and demand for company-wide interrogation of data.

external changes and the organisation’s own responses keep moving the surface. buying Claude makes asking questions cheaper, which creates more demand for accessible, trustworthy data and introduces new requirements around permissions, provenance and evaluation.

in short: the operational domain gives the surface its basic shape; structural obligations give it hard edges; scale makes it jagged; and technology, markets and previous organisational responses keep it moving.

[SHOW DEMO GRAPHIC]