Max Storage Setting

[JaggerJo]

When running raceway with local storage it would be great to have an option to limit the max used storage.

For scenarios where tramway is auto deployed it makes sense to be able to configure this in the Dockerfile / via environment variables directly.

[dusanstanojeviccs]

@JaggerJo

I don't know how feasible this really is, we would have to continuously monitor different DB files for their size. We expose a lot of the metrics so that you can monitor your traceway instance (on a diff traceway instance) for disk usage and everything else and then decide on the proper retention period. It is my understanding that this is something that all observability tools suffer from and putting the disk size as the retention target might lead to extra overhead.

I'll look more into what's possible and what other tools do.

In the meantime, in self hosted environments I recommend you:
1 - use DuckDB (great compression) or ClickHouse (backsup hist data to S3 compatible storage)
2 - limit days back retention
3 - use another smaller tw to monitor your deployment

[JaggerJo]

@dusanstanojeviccs interesting - the exact used storage is not really important for us - just the ruff ballpark.

Would it be simpler to limit the maximum number of log entries? If the only knob is the retention period then a day of error message spam could fill up our entire disk space in some cases while we'd be usually more than fine in that timeframe.

[dusanstanojeviccs]

@JaggerJo Absolutely we can def do that, we could also limit the rate of ingestion? So you could say cap it to X per second, would that be better? Or maybe X per second per service? so that we can throttle only the troublesome service perhaps

Let me know what you think and we can implement it

[JaggerJo]

@JaggerJo Absolutely we can def do that, we could also limit the rate of ingestion? So you could say cap it to X per second, would that be better? Or maybe X per second per service? so that we can throttle only the troublesome service perhaps

Let me know what you think and we can implement it

Interesting Idea, but for our use case dropping log entries would seem strange. If we get bursts we still would want the data to be able to analyze it.

[dusanstanojeviccs]

@JaggerJo Agreed, we'll make retention based on total table size for logs specifically

[dusanstanojeviccs]

Implemented as LOG_RECORDS_MAX_ROWS @JaggerJo please let me know if it's what you've expected, it basically runs 1/min and clears out old logs above the LOG_RECORDS_MAX_ROWS this should lead to your self hosting being self capped

Also, please make sure you use the duckdb version as the perf is much much better https://tracewayapp.com/blog/sqlite-vs-duckdb

[JaggerJo]

Hey @dusanstanojeviccs

Sorry - really busy right now.

We'll soon include trace-way as our default OTLP viewer / collector in our OS images. Then all our developers will use it regularly. Pretty sure we'll get some good feedback once that the case.

Thanks for all the work you're putting in!