Project notes / Cynthia Owolabi

RelayDesk

Take Frankfurt offline halfway through a release. Some clients can switch to another cache; others will wait or fail, depending on the routing policy. RelayDesk lets you compare those choices under the same demand.

The map and the local engine

The map simulates deliveries to 1,000 clients. I built a separate Node.js engine to test recovery with actual HTTP transfers. It checks downloaded files against their SHA-256 digest and expected length, then records the result in SQLite. The engine and the map run independent workloads.

When a worker stops

A transactional claim assigns a random ownership token and lease. A completion transaction checks that the worker still owns an unexpired claim. If a worker dies, another can recover the job after expiry; completion requires an unexpired claim.

Both the worker and the coordinator check the downloaded content. The coordinator writes a temporary file, syncs it to disk and atomically publishes it under its digest. Only then does it mark the job complete. If it crashes between those steps, the next worker may fetch the file again. Recovery may repeat the fetch; completion requires a verified file.

Why the engine uses SQLite

SQLite keeps the engine easy to run on one machine. Short transactions let multiple workers claim jobs, and the queue survives a restart. Coordination across hosts still needs testing. The demo files are small enough to transfer within a fixed lease; larger files would need streaming and lease renewal. Several client jobs can point to the same stored file.

Try the outage

1. Open the demo and take Frankfurt offline.
2. Watch retries and switch routing policies; compare failures, pending work and latency together.
3. Run the strategy comparison on separate copies of the same seeded network.
4. For actual bytes, follow the README to start the local engine and use its Local engine panel.

What happened in the simulation

The recorded comparison uses seed 42, 1,000 arrivals, four workers per cache and a Frankfurt outage at second 15. After 240 simulated seconds:

These are model results. Latency includes queueing for successful requests only; credits are fictional units for comparing strategies.

npm ci
npm run experiment
npm run test:engine

Testing the local engine

Full model report · Nine engine integration tests · Architecture

The engine tests include an EU cache returning HTTP 503 and a US cache serving a file with the wrong checksum. Jobs can recover through the AP cache. Other tests stop a worker mid-job or run two coordinators against the same queue. All cache endpoints are on one local HTTP server. The public demo runs entirely in the browser.

Other projects