High-write data systems · Build
Live dashboards and a year of history in under a second
Your customers see their numbers live and for a whole year. A time-series analytics layer we built takes 2M writes a day and returns a year in under a second.
EngagementHourly · remote
Hours9–18 CET · 4 h overlap
StackInfluxDB · MySQL · S3
Proof1 case study
The service in 4 parts
01 · Who it’s for
A product that records a lot of events, readings, clicks or views, and whose customers want to see their own numbers live and over a whole year.
02 · What we’ve built
- Over two million writes a day go into a time-series store with a deliberately small index, and each advertiser reads the results live in their dashboard.Case study 05 →
- An annual report that returns in under one second for accounts with over 900 listings. It aggregates about 330,000 daily summary rows in one query and never touches a raw row.Case study 05 →
- Self-hosting the time-series database saved about $27,000 a year by our estimate. A later retention change cut the data from two terabytes to under one gigabyte, and self-hosting was still cheaper. By then the deciding cost was query count, where before it had been storage.Blog post →
03 · How we work
01 · ONE JOB PER STORETaking writes, serving live reads and aggregating a year of history are different jobs, and no single database does all three well. One store takes the writes and another answers the reports.
02 · THE LARGEST ACCOUNTWe measure speed on the largest account rather than a typical one, since any report is fast for an account with three records.
04 · Stack
InfluxDBMySQLS3Node.jsWebSocketsAWS
Proof
Where we’ve done this before
Case study 05 · Marketplace2M / dayOver two million writes a day go into a time-series store with a deliberately small index, and each advertiser reads the results live in their dashboard.Read the case study →WritingSelf-hosted InfluxDB stayed cheaper because of per-query pricingSelf-hosting the time-series database saved about $27,000 a year by our estimate. A later retention change cut the data from two terabytes to under one gigabyte, and self-hosting was still cheaper. By then the deciding cost was query count, where before it had been storage.Read the post →
FAQ
Questions about this service
Usually not. What worked on the marketplace was dividing the work: raw events went to a time-series store, and a daily rollup per listing went to the relational database.
As long as someone may need to reprocess it. The marketplace kept raw events for seven days, with a daily export to object storage as the archive. We never checked what thirty days would have cost, and today we would.
That depends on which line of the bill is largest. Per-query charges grow with the number of people looking at the data, so they weigh most on a product whose customers check their own dashboards. When only a few engineers read the data, the managed service is often cheaper.
In a time-series store, tags are indexed and fields are not. Put an identifier such as a listing ID in a tag and every new listing creates a new series, until the index grows too large to work. It is the most common way these installations fail.
Next step
Describe the problem and we’ll come back with three options.
The first call takes 30 minutes, and you’re under no obligation.