Internal system · In production

Data platform

The data platform of a trading company: scheduled collection from marketplace accounts, an archive of every response before parsing, raw → staging → core layers and data marts facing outward. This is not «a sales warehouse for margin calculation» but a foundation: the same marts feed dashboards, applications and AI agents.

The pain

Every department exported marketplace data its own way, so meetings started with an argument about whose spreadsheet was right. Last year’s history did not exist at all.

Now

The platform collects data on a schedule and takes it through layers up to ready-made data marts. From there everyone - people, applications and agents - takes the numbers from the same marts.

Effect across the steps

01 · From chaos to system: Pulled marketplace data into one source everyone counts from
02 · Transparent finances: Payouts, fees and deductions reconcile with marketplace totals weekly
03 · Precise decisions: The number in a mart is verified against the source down to a single unit

What the business gets

History is kept forever, even though the marketplace only stores 90 days of it.
Everyone’s reports agree, because they come from the same marts rather than personal exports.
A gap against marketplace payouts shows up within the week, not at the end of the quarter.
A cost line never goes missing quietly: the per-account view warns that the data is incomplete instead of completing the margin on its own.
A new application or AI agent connects to ready-made marts instead of writing its own data collection.

From source to result

Marketplace APIs
Raw archive
raw → staging → core
Data marts
BI · REST API · Claude · MCP agents

Every marketplace response goes into the archive before parsing: if a field is parsed wrong, we re-read the archive instead of begging the marketplace for history again. It keeps 90 days; our archive keeps everything.

How it works

The layers are separated deliberately: raw is an «as it arrived» journal, the next layer removes duplicates, then business entities, and only then data marts. Counting on raw is wrong, so raw is closed by permissions rather than by agreement.

The response format is validated on the way in: a new marketplace field does not break collection, while a disappeared one raises a Telegram alert - and the load still does not stop.

Automatic reconciliation appeared after a real miss: one logistics field was read from a key the current API no longer has and was silently zero - margin was overstated by the entire delivery cost. The error survived unnoticed precisely because the system was only ever checked against itself.

Four doors lead outward and all four are read-only: writing is blocked both by permissions and by the database mode. An AI agent gets a set of tools with a row limit and a timeout, not raw access.

Who maintains it now

The engineer. The company’s analysts and applications are connected read-only, each with its own account.

Other cases for this step

Internal system · In production

Management finance loop

Every number the company has - in one system

The pain: The numbers lived in ten spreadsheets, private chats and the heads of three people. When key people left, both access and context left with them.
Now: One source: cash flow, budget, payment requests, reconciliation and reference books in a single application with roles and permissions.

Timeline: one month
Replaced: ten spreadsheets and a chat thread
Coverage: cash flow, budget, requests, reconciliation

The same system pays off across 3 more steps

Internal system · In production

IT infrastructure inventory

Services, access and owners - in one place

The pain: The inventory lived in a legacy panel and scattered notes: who is connected to what, what runs where, who owns it. When a person left, access and context left with them.
Now: One panel: a service tree, the «who has access to what» matrix, keys and portals, tasks, problems, changes and an audit log. Login under your own account, sections visible by role.

Sections
Timeline
Replaced: a legacy panel of 35 tabs

Internal system · In production

Map of the legacy warehouse

What sits in the legacy database is now known

The pain: The legacy warehouse arrived as a black box: several databases, hundreds of tables and views, an OLAP cube, no description at all. Every question started with excavation.
Now: A reusable client, a CLI for queries and a data map rebuilt by one command: databases, tables, views, columns, with a map of the cube beside it. Plus a check that the data can be trusted.

Map coverage
Updating: the map is rebuilt by one command
Data check: cube freshness and cube-against-SQL reconciliation

Type: Internal system
Status: In production
Collection: 33 domains on a schedule
Doors out: BI and SQL, REST API, Claude Code, MCP
Verified against: the marketplace, to the unit and the kopeck
Archive: raw responses stored before parsing

PythondltdbtPostgreSQLDagsterMinIOFastAPIMCPDocker