Savings of around 400MB per 10,000 requests. Using a named tuple was the
original approach for this reason, but the influxdb interface requires
dict()s and it seemed silly to spend time converting them. Additionally,
influxdb client already does batching.
Unfortunately, with the amount of data processed for Factory that will
continue to grow this approach is necessary. The dict() final structures
are buffered up to ~1000 before being written and released. Another
benefit of the batching is that influxdb does not allocate memory for the
entire incoming batch.
This prevents the memory required to process to be enough to load all
parsed request element trees at once. Instead only one page of requests
is loaded at a time and the memory freed after processed. The end result
is the memory consumption reduced by just over 20% (current Factory drops
by around 2.5GB).
A lot of room for improvement and additional metrics that can be extracted.
Including non-final state requests would allow for analyzing the current
staging state instead of only historical state. Additionally, the current
state can be used to present an activity log.
Handling incremental updates is non-trivial given the deltas are evaluated
and stored in sum state. A few possible approaches, but likely not worth
the hassle given the relatively short processing time and infrequent desire
to update data (daily at minimum).
