Much of RxJS involves working with
backpressure - how to reconcile
streams that emit/process data at different rates, without overloading
the system. Much of that model is built with lossy handling in mind - it
makes sense that when your system is under duress, that you design your
streams to degrade gracefully (e.g. drop certain events, or rate limit
them by chunking into windows, etc).
However, there are times when it is appropriate to have a lossless
approach to backpressure - e.g., to store every chunk of data that comes
through a stream in memory, and not drop things. These use cases may
come about when:
- You have a short-lived, or bounded set of data you know will come over
the pipe. You understand the bounds of the data that will ever come over
- You have a processing script you want to run, which is not part of a
- You have a honkin’ large system that can handle the load.
In my case, I had a script that called the Google Geocoding API for a
set of GPS coordinates. Now for a set of several hundred coordinates, I
would end up calling the API several hundred times all at once with this
1 2 3 4
I searched all over for a lossless throttling mechanism, but all I could
find was references to RxJS’s lossy throttle behavior.
1 2 3