Millions of records, streamed at low memory cost

Translation Management API

A production-grade REST API for locale-based translations: Controller → Service → Repository architecture, Redis tag-based cache invalidation, and cursor-based export streaming millions of records at low memory cost.

Back to all work
Records streamed per export
MillionsRecords streamed per export
Cache invalidation strategy
Tag-basedCache invalidation strategy
With a full PHPUnit suite
DockerizedWith a full PHPUnit suite
RoleBackend engineer — architecture and implementation
Period2023

The challenge

Translation lookups are read-heavy and latency-sensitive, while exports need to stream an entire dataset without exhausting memory or locking tables.

Naive caching made invalidation impossible to reason about — changing one string in one locale should not flush everything.

The approach

Separated concerns into a Controller → Service → Repository architecture so caching and persistence could evolve independently of the HTTP layer.

Used Redis tag-based cache invalidation so a write can target a locale or namespace precisely rather than clearing the whole cache.

Implemented cursor-based export instead of offset pagination, keeping memory flat regardless of dataset size and avoiding the row drift offsets cause on a live table.

Dockerized the service and covered API contracts, caching behaviour and exports with a PHPUnit suite.

The result

Exports across millions of records running at low, constant memory cost.

Cache invalidation narrow enough that writes don't degrade read performance for unrelated locales.

A tested, containerised API other teams could integrate against without support.

Engineering notes

Offset pagination is a bug at scale

On a table receiving writes, offsets silently skip and repeat rows. Cursors are barely more work and are simply correct.

Test the caching, not just the endpoints

Cache invalidation is where this kind of API rots. Covering it explicitly is what let the caching strategy survive later change.

Built with

  • PHP 8.2
  • Laravel
  • MySQL
  • Redis
  • Docker
  • PHPUnit

Want something like this built?

Tell me what you're working on and I'll tell you honestly whether I'm the right person for it.

Get in touch