Skip to content

Resolve semantic memory leaks and stabilize IaC orchestration #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BLMgithub merged 7 commits into from
Apr 11, 2026
Merged

Resolve semantic memory leaks and stabilize IaC orchestration #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
23ecc22
fix: memory accumulation and steps that breaks streaming in semantic …
BLMgithub Apr 9, 2026
93e2d6c
fix: add concurrency control to ci-infra preventing state lock; add I…
BLMgithub Apr 11, 2026
2d037ed
fix: removed unused logics and align docstrings to refactoring
BLMgithub Apr 11, 2026
f0ec874
docs: update stages documentation and diagram with the refactoring
BLMgithub Apr 11, 2026
6c421eb
docs: update validation, contract and publish stage docstring clarity
BLMgithub Apr 11, 2026
08e4b95
test: update unit tests to match refactoring
BLMgithub Apr 11, 2026
15f95c9
docs: add memory stress tests, cost efficiency and scaling roadmap
BLMgithub Apr 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions .gcp/terraforms/iam_bindings.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,21 @@ resource "google_service_account_iam_member" "github_deployer_sa" {
# Roles for github-actions-deployer
locals {
deployer_roles = [
"roles/run.developer",
"roles/workflows.editor",
"roles/cloudscheduler.admin",
"roles/iam.serviceAccountUser"
"roles/run.developer", # Manage Cloud Run jobs
"roles/workflows.editor", # Manage Workflows
"roles/cloudscheduler.admin", # Manage Scheduler
"roles/iam.serviceAccountUser", # Act as SAs for jobs
"roles/artifactregistry.admin", # Manage Artifact Registry
"roles/eventarc.admin", # Manage Eventarc triggers
"roles/storage.admin", # Manage buckets and state locking
"roles/resourcemanager.projectIamAdmin", # Manage the IAM bindings in this code
"roles/iam.workloadIdentityPoolAdmin", # Manage WIF in wif.tf
"roles/monitoring.admin", # Manage Monitoring in monitoring.tf
"roles/iam.serviceAccountAdmin", # Manage Alert policies in monitoring.tf
"roles/iam.admin" # Manage Iam roles
]
}

# Enables to:
# Deploy Cloud Run containers
# Deploy pipeline-dispatcher.yml
# Update the cron job
# Attach SAs to Cloud Run
resource "google_project_iam_member" "github_deployer_permissions" {
for_each = toset(local.deployer_roles)
project = var.project_id
Expand Down
7 changes: 6 additions & 1 deletion .github/workflows/ci-infra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ on:
workflow_dispatch:
workflow_call:

# Add concurrency control to prevent state lock issues
concurrency:
group: terraform-${{ env.ENV }}
cancel-in-progress: false

permissions:
contents: 'read'
id-token: 'write'
Expand Down Expand Up @@ -50,4 +55,4 @@ jobs:
TF_VAR_region: ${{ env.REGION }}
TF_VAR_github_repo: ${{ env.GITHUB_REPO }}
TF_VAR_alert_email_map: ${{ secrets.ALERT_EMAIL_MAP }}
run: terraform apply -auto-approve -lock=false
run: terraform apply -auto-approve
65 changes: 48 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Small to mid-sized organizations are trapped in a cycle where they outgrow the c
This project solves that challenge by delivering a highly resilient, event-driven data pipeline on Google Cloud Platform for reliable operational analytics. It guarantees data integrity through a strict Medallion architecture (Bronze, Silver, Gold) that relies on rigid data contracts and validation gates to catch and isolate bad data early in the lifecycle.

### Defensive Pipeline Architecture
![WIP_pipeline_diagram_picture](https://still-working-on-it.need-to-finish-readme.first)
![pipeline-orchestration-diagram](/assets/diagrams/01-pipeline-orchestration-diagram.png)

To eliminate the risk of cross-run data contamination and memory bloat, the pipeline employs a defensive state-management strategy where local compute environments are strictly temporary:
* **Stateless Orchestration:** Every execution operates within an isolated, deterministic `run_id` workspace that is aggressively purged post-run.
Expand Down Expand Up @@ -67,29 +67,60 @@ The pipeline does not just move data; it actively defends the analytical layer f

The pipeline is explicitly engineered to process massive datasets within the rigid memory constraints of serverless compute (Cloud Run). By leveraging the Polars Rust engine (Lazy API & Streaming), the system achieves near-perfect memory density, operating consistently at the physical hardware ceiling.

**GCP Stress-Test Metrics (18 Million Row Snapshot)**
### GCP Stress-Test Metrics (Scaling Efficiency)

![engine-performance-8gb](/assets/screenshots/engine-performance-8gb-2cpu.png)
| 18M Snapshot (8GiB / 2 vCPU) | 36M Snapshot (16GiB / 4 vCPU) |
| :---: | :---: |
| ![engine-performance-8gb](/assets/screenshots/engine-performance-8gb-2cpu.png) | ![engine-performance-16gb](/assets/screenshots/engine-performance-16gb-4cpu.png) |

> The data used for this chart [`benchmarks/`](/assets/benchmarks/polars/18mrows_dataset_stats_log.csv) and the 18m rows dataset can be found her [`data/`](/data/)
> Benchmark data: [`18m_stats_log.csv`](/assets/benchmarks/polars/18mrows_dataset_stats_log.csv) and [`36m_stats_log.csv`](/assets/benchmarks/polars/36mrows_dataset_stats_log.csv)

| Metric | 18M Rows (8GB / 2 vCPU) | 36M Rows (16GB / 4 vCPU) |
| :--- | :--- | :--- |
| **Throughput (Processing)** | ~116,000 Rows / Second | ~220,000 Rows / Second |
| **Total Runtime (Wall-Clock)** | 02m 34s | 02m 43s |
| **Memory Tax (Fixed)** | ~1.5 GiB | ~1.5 GiB |
| **Effective Data Headroom** | ~6.5 GiB | ~14.5 GiB |

| Metric | Value (18M Row Peak Load) |
| :--- | :--- |
| **Throughput (Processing)** | ~116,000 Rows / Second |
| **Total Runtime (Wall-Clock)** | 02m 34s |
| **Compute Provision** | 2 vCPU / 8 GiB |
| **Memory Tax (Fixed)** | ~1.5 GiB (OS / Sandbox / IO Buffers) |
| **Effective Data Headroom** | ~6.5 GiB (Active Transformation) |

* **Linear Vertical Scaling:** Bumping the Cloud Run provision to 32GiB allows the same architecture to process ~72 Million rows without code changes.
* **Predictable Capacity:** Identifying the 1.5GB "Memory Tax" allows for precise resource governance, ensuring jobs never fail due to unpredictable Signal 9 (OOM) events.
* **Near-Linear Performance Scaling:** Doubling the compute and dataset size results in only a 9-second increase in wall-clock time, effectively doubling the throughput as the Polars engine saturates the additional vCPUs.
* **Predictable Capacity:** Identifying the "Memory Tax" (OS/IO overhead) allows for precise resource governance, ensuring jobs never fail due to unpredictable Signal 9 (OOM) events.
* **Zero-Idle Economics:** 100% serverless execution ensures zero billable time during idle periods, significantly reducing the Total Cost of Ownership (TCO) compared to dedicated cluster solutions.

**Measurement Methodology**
### Cost Efficiency & Free-Tier

The pipeline's processing speed allows for a full analytical rebuild of 36M rows while remaining comfortably within the **GCP Cloud Run Free Tier** (180k vCPU-sec, 360k GiB-sec). This means a small-to-mid-sized organization can run this production-grade pipeline multiple times a day with **zero compute costs.**

| Compute Provision | Dataset | vCPU-Seconds / Run | GiB-Seconds / Run | Monthly Free-Tier Runs |
| :--- | :--- | :--- | :--- | :--- |
| **8 GiB / 2 vCPU** | ~18m rows | 308 | 1,232 | **~292 Runs / Month** |
| **16 GiB / 4 vCPU** | ~36m rows | 652 | 2,608 | **~138 Runs / Month** |
| **32 GiB / 8 vCPU** | ~72m rows | 1,304 | 5,216 | **~69 Runs / Month** |

> *Calculations based on verified benchmarks. Even at the highest 32GiB tier, the pipeline can execute a full state rebuild twice daily for $0*

### Measurement Methodology
* **Performance Profiling:** Captured from production telemetry via the pipeline's native `run_duration` metadata, calculating the precise delta between `started_at` and `completed_at` timestamps.
* **Memory Utilization:** Monitored via an integrated [`psutil.virtual_memory().used`](/assets/benchmarks/polars/README.md) profiling implementation to verify the actual resource footprint and confirm the physical ceiling for an 8GiB provision.
* **Throughput Efficiency:** Leverages Polars' streaming evaluation to maintain high throughput and minimize CPU idle time during GCS I/O, providing a significant performance advantage over traditional eager-loading engines.
* **Memory Utilization:** Monitored via an integrated [`psutil.virtual_memory().used`](/assets/benchmarks/polars/README.md) profiling implementation to verify the actual resource footprint and confirm the physical ceiling for 8GiB/16GiB provision.
* **Throughput Efficiency:** Leverages Polars streaming evaluation to maintain high throughput and minimize CPU idle time during GCS I/O, providing a significant performance advantage over traditional eager-loading engines.

### **Scaling Roadmap: From Serverless to Enterprise Lakehouse**

To ensure the architecture survives the transition from millions to billions of rows, the pipeline is designed to evolve across three validated scaling paths. This roadmap prioritizes cost-efficiency at low volumes while providing a clear architectural pivot for enterprise-scale workloads.

#### **Stage 1: Temporal Sharding (Vertical Efficiency)**
* **Strategy:** Refactor the `Assemble` stage to iterate through **yearly batch partitions** while `Semantic` stage to **streams output directly** to a GCS staging location.
* **Publish Evolution:** Moves to a **Partitioned Atomic Swap**. Yearly shards are streamed directly to a staged GCS version prefix. The `Integrity Gate` validates cloud-side completeness before the `latest_version.json` pointer is updated.
* **Trade-off:** **Latency vs. Memory.** Significantly increases total wall-clock time due to repeated I/O cycles, but allows 32GiB instances to process 100M+ rows by isolating join-intensity to specific temporal shards.

#### **Stage 2: Incremental Delta Architecture (Event-Driven)**
* **Strategy:** Transition from a "Full Rebuild" batch model to a **Stateless Delta Propagation** model, processing only active deltas.
* **Publish Evolution:** Moves to a **Checkpoint-based Commit**. Folder-based versioning is replaced by an atomic merge into the Gold layer. The "Pointer" evolves into a metadata watermark signifying data freshness to downstream consumers.
* **Trade-off:** **Simplicity vs. Scale.** Eliminates memory constraints and reduces runtime costs, but sacrifices easy "point-in-time" folder recovery. Requires "Last-Mile" deduplication logic (e.g., SQL Views) for downstream consumers.

#### **Stage 3: BigQuery "Engine-as-a-Service" (The Enterprise Pivot)**
* **Strategy:** Offload the `Assemble` and `Semantic` compute layers entirely to **BigQuery (ELT Pattern)**.
* **Publish Evolution:** Moves to a **Atomic View Redirection**. The Python "Gatekeeper" builds semantics in a staging dataset and runs SQL-driven integrity checks. Publication is achieved by an atomic swap of a BigQuery Authorized View, replacing the file-based pointer system.
* **Trade-off:** **Cost vs. Capability.** Provides an infinite scaling ceiling and removes all local infrastructure bounds, but introduces higher cost-per-query overhead and requires transitioning from local Parquet files to managed cloud storage.


## Observability & Alerting
Expand Down
165 changes: 165 additions & 0 deletions assets/benchmarks/polars/36mrows_dataset_stats_log.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
view,timestamp,logger,memory,unit
DEFAULT,2026-04-09T21:43:40.178678Z,METRIC_MEM:,2812.83,MB
DEFAULT,2026-04-09T21:43:41.178984Z,METRIC_MEM:,2868.2,MB
DEFAULT,2026-04-09T21:43:42.179341Z,METRIC_MEM:,2931.05,MB
DEFAULT,2026-04-09T21:43:43.179687Z,METRIC_MEM:,2994.48,MB
DEFAULT,2026-04-09T21:43:44.179962Z,METRIC_MEM:,3049.48,MB
DEFAULT,2026-04-09T21:43:45.180339Z,METRIC_MEM:,3111.88,MB
DEFAULT,2026-04-09T21:43:46.180721Z,METRIC_MEM:,3175.25,MB
DEFAULT,2026-04-09T21:43:47.181107Z,METRIC_MEM:,3246.77,MB
DEFAULT,2026-04-09T21:43:48.181558Z,METRIC_MEM:,3309.72,MB
DEFAULT,2026-04-09T21:43:49.181470Z,METRIC_MEM:,3364.92,MB
DEFAULT,2026-04-09T21:43:50.181366Z,METRIC_MEM:,3427.7,MB
DEFAULT,2026-04-09T21:43:51.181351Z,METRIC_MEM:,3483.11,MB
DEFAULT,2026-04-09T21:43:52.181694Z,METRIC_MEM:,3554.01,MB
DEFAULT,2026-04-09T21:43:53.182108Z,METRIC_MEM:,3609.08,MB
DEFAULT,2026-04-09T21:43:54.182588Z,METRIC_MEM:,3672.5,MB
DEFAULT,2026-04-09T21:43:55.182979Z,METRIC_MEM:,3726.8,MB
DEFAULT,2026-04-09T21:43:56.183486Z,METRIC_MEM:,3790.45,MB
DEFAULT,2026-04-09T21:43:57.183846Z,METRIC_MEM:,3854.01,MB
DEFAULT,2026-04-09T21:43:58.184338Z,METRIC_MEM:,3916.96,MB
DEFAULT,2026-04-09T21:43:59.184674Z,METRIC_MEM:,3980.06,MB
DEFAULT,2026-04-09T21:44:00.185069Z,METRIC_MEM:,4043.16,MB
DEFAULT,2026-04-09T21:44:01.185349Z,METRIC_MEM:,4098.34,MB
DEFAULT,2026-04-09T21:44:02.185794Z,METRIC_MEM:,4161.24,MB
DEFAULT,2026-04-09T21:44:03.190091Z,METRIC_MEM:,4224.66,MB
DEFAULT,2026-04-09T21:44:04.186657Z,METRIC_MEM:,4287.29,MB
DEFAULT,2026-04-09T21:44:05.187081Z,METRIC_MEM:,4342.34,MB
DEFAULT,2026-04-09T21:44:06.187557Z,METRIC_MEM:,4405.66,MB
DEFAULT,2026-04-09T21:44:07.187831Z,METRIC_MEM:,4467.94,MB
DEFAULT,2026-04-09T21:44:08.188119Z,METRIC_MEM:,4523.98,MB
DEFAULT,2026-04-09T21:44:09.188007Z,METRIC_MEM:,4586.83,MB
DEFAULT,2026-04-09T21:44:10.187799Z,METRIC_MEM:,4649.84,MB
DEFAULT,2026-04-09T21:44:11.187840Z,METRIC_MEM:,4705.04,MB
DEFAULT,2026-04-09T21:44:12.187947Z,METRIC_MEM:,4768.01,MB
DEFAULT,2026-04-09T21:44:13.188380Z,METRIC_MEM:,4831.27,MB
DEFAULT,2026-04-09T21:44:14.189078Z,METRIC_MEM:,4894.65,MB
DEFAULT,2026-04-09T21:44:15.189546Z,METRIC_MEM:,4958.21,MB
DEFAULT,2026-04-09T21:44:16.189898Z,METRIC_MEM:,5020.09,MB
DEFAULT,2026-04-09T21:44:17.190829Z,METRIC_MEM:,5075.54,MB
DEFAULT,2026-04-09T21:44:18.191150Z,METRIC_MEM:,5138.73,MB
DEFAULT,2026-04-09T21:44:19.191518Z,METRIC_MEM:,5182.56,MB
DEFAULT,2026-04-09T21:44:20.197169Z,METRIC_MEM:,5623.91,MB
DEFAULT,2026-04-09T21:44:21.197587Z,METRIC_MEM:,6033.96,MB
DEFAULT,2026-04-09T21:44:22.201299Z,METRIC_MEM:,6488.75,MB
DEFAULT,2026-04-09T21:44:23.198486Z,METRIC_MEM:,6785.55,MB
DEFAULT,2026-04-09T21:44:24.198818Z,METRIC_MEM:,6895.43,MB
DEFAULT,2026-04-09T21:44:25.199286Z,METRIC_MEM:,7071.41,MB
DEFAULT,2026-04-09T21:44:26.199620Z,METRIC_MEM:,7220.95,MB
DEFAULT,2026-04-09T21:44:27.200130Z,METRIC_MEM:,7458.04,MB
DEFAULT,2026-04-09T21:44:28.200477Z,METRIC_MEM:,7530.45,MB
DEFAULT,2026-04-09T21:44:29.200378Z,METRIC_MEM:,7652.43,MB
DEFAULT,2026-04-09T21:44:30.200421Z,METRIC_MEM:,7838.43,MB
DEFAULT,2026-04-09T21:44:31.200275Z,METRIC_MEM:,8050.17,MB
DEFAULT,2026-04-09T21:44:32.200504Z,METRIC_MEM:,8270.6,MB
DEFAULT,2026-04-09T21:44:33.200943Z,METRIC_MEM:,8439.16,MB
DEFAULT,2026-04-09T21:44:34.201362Z,METRIC_MEM:,8777.46,MB
DEFAULT,2026-04-09T21:44:35.201918Z,METRIC_MEM:,8934.88,MB
DEFAULT,2026-04-09T21:44:36.202212Z,METRIC_MEM:,8910.29,MB
DEFAULT,2026-04-09T21:44:37.202609Z,METRIC_MEM:,7358.61,MB
DEFAULT,2026-04-09T21:44:38.204970Z,METRIC_MEM:,7661.91,MB
DEFAULT,2026-04-09T21:44:39.209060Z,METRIC_MEM:,7835.91,MB
DEFAULT,2026-04-09T21:44:40.209130Z,METRIC_MEM:,7907.15,MB
DEFAULT,2026-04-09T21:44:41.204203Z,METRIC_MEM:,8154.41,MB
DEFAULT,2026-04-09T21:44:42.209313Z,METRIC_MEM:,8405.19,MB
DEFAULT,2026-04-09T21:44:43.209838Z,METRIC_MEM:,7903.66,MB
DEFAULT,2026-04-09T21:44:44.210360Z,METRIC_MEM:,8222.37,MB
DEFAULT,2026-04-09T21:44:45.210692Z,METRIC_MEM:,8588.28,MB
DEFAULT,2026-04-09T21:44:46.213686Z,METRIC_MEM:,9032.81,MB
DEFAULT,2026-04-09T21:44:47.211559Z,METRIC_MEM:,9398.2,MB
DEFAULT,2026-04-09T21:44:48.211871Z,METRIC_MEM:,9777.2,MB
DEFAULT,2026-04-09T21:44:49.211810Z,METRIC_MEM:,10187.28,MB
DEFAULT,2026-04-09T21:44:50.211790Z,METRIC_MEM:,10544.79,MB
DEFAULT,2026-04-09T21:44:51.211702Z,METRIC_MEM:,10918.25,MB
DEFAULT,2026-04-09T21:44:52.226469Z,METRIC_MEM:,11645.3,MB
DEFAULT,2026-04-09T21:44:53.226946Z,METRIC_MEM:,12692.11,MB
DEFAULT,2026-04-09T21:44:54.227541Z,METRIC_MEM:,13956.78,MB
DEFAULT,2026-04-09T21:44:55.228051Z,METRIC_MEM:,13874.34,MB
DEFAULT,2026-04-09T21:44:56.228511Z,METRIC_MEM:,12523.92,MB
DEFAULT,2026-04-09T21:44:57.228953Z,METRIC_MEM:,12711.82,MB
DEFAULT,2026-04-09T21:44:58.229409Z,METRIC_MEM:,12737.88,MB
DEFAULT,2026-04-09T21:44:59.229860Z,METRIC_MEM:,12783.1,MB
DEFAULT,2026-04-09T21:45:00.230271Z,METRIC_MEM:,12872.66,MB
DEFAULT,2026-04-09T21:45:01.230679Z,METRIC_MEM:,12877.08,MB
DEFAULT,2026-04-09T21:45:02.231040Z,METRIC_MEM:,12811.36,MB
DEFAULT,2026-04-09T21:45:03.231455Z,METRIC_MEM:,12722.5,MB
DEFAULT,2026-04-09T21:45:04.237488Z,METRIC_MEM:,12728.71,MB
DEFAULT,2026-04-09T21:45:05.232293Z,METRIC_MEM:,12771.59,MB
DEFAULT,2026-04-09T21:45:06.232823Z,METRIC_MEM:,12869.63,MB
DEFAULT,2026-04-09T21:45:07.237757Z,METRIC_MEM:,12988.25,MB
DEFAULT,2026-04-09T21:45:08.233418Z,METRIC_MEM:,13095.33,MB
DEFAULT,2026-04-09T21:45:09.233396Z,METRIC_MEM:,13201.29,MB
DEFAULT,2026-04-09T21:45:10.236817Z,METRIC_MEM:,13288.67,MB
DEFAULT,2026-04-09T21:45:11.237004Z,METRIC_MEM:,13399.59,MB
DEFAULT,2026-04-09T21:45:12.237337Z,METRIC_MEM:,13502.8,MB
DEFAULT,2026-04-09T21:45:13.237766Z,METRIC_MEM:,13616.84,MB
DEFAULT,2026-04-09T21:45:14.240824Z,METRIC_MEM:,13711.62,MB
DEFAULT,2026-04-09T21:45:15.241266Z,METRIC_MEM:,13820.56,MB
DEFAULT,2026-04-09T21:45:16.241645Z,METRIC_MEM:,13937.32,MB
DEFAULT,2026-04-09T21:45:17.242003Z,METRIC_MEM:,14028.76,MB
DEFAULT,2026-04-09T21:45:18.242372Z,METRIC_MEM:,14126.38,MB
DEFAULT,2026-04-09T21:45:19.245015Z,METRIC_MEM:,14220.46,MB
DEFAULT,2026-04-09T21:45:20.243002Z,METRIC_MEM:,14323.86,MB
DEFAULT,2026-04-09T21:45:21.245997Z,METRIC_MEM:,14424.54,MB
DEFAULT,2026-04-09T21:45:22.247559Z,METRIC_MEM:,7915.41,MB
DEFAULT,2026-04-09T21:45:23.246817Z,METRIC_MEM:,8489.96,MB
DEFAULT,2026-04-09T21:45:24.247168Z,METRIC_MEM:,8911.7,MB
DEFAULT,2026-04-09T21:45:25.249494Z,METRIC_MEM:,9354.19,MB
DEFAULT,2026-04-09T21:45:26.249975Z,METRIC_MEM:,9780.51,MB
DEFAULT,2026-04-09T21:45:27.250390Z,METRIC_MEM:,10217.01,MB
DEFAULT,2026-04-09T21:45:28.250624Z,METRIC_MEM:,10676.55,MB
DEFAULT,2026-04-09T21:45:29.250558Z,METRIC_MEM:,11116.09,MB
DEFAULT,2026-04-09T21:45:30.250662Z,METRIC_MEM:,11445.64,MB
DEFAULT,2026-04-09T21:45:31.250691Z,METRIC_MEM:,11473.34,MB
DEFAULT,2026-04-09T21:45:32.256485Z,METRIC_MEM:,10635.77,MB
DEFAULT,2026-04-09T21:45:33.256709Z,METRIC_MEM:,9477.71,MB
DEFAULT,2026-04-09T21:45:34.256519Z,METRIC_MEM:,8338.32,MB
DEFAULT,2026-04-09T21:45:35.260632Z,METRIC_MEM:,8521.45,MB
DEFAULT,2026-04-09T21:45:36.260800Z,METRIC_MEM:,8241.01,MB
DEFAULT,2026-04-09T21:45:37.260872Z,METRIC_MEM:,8706.99,MB
DEFAULT,2026-04-09T21:45:38.258259Z,METRIC_MEM:,8889.8,MB
DEFAULT,2026-04-09T21:45:39.258646Z,METRIC_MEM:,9144.98,MB
DEFAULT,2026-04-09T21:45:40.261191Z,METRIC_MEM:,9366.04,MB
DEFAULT,2026-04-09T21:45:41.259351Z,METRIC_MEM:,9586.33,MB
DEFAULT,2026-04-09T21:45:42.259706Z,METRIC_MEM:,9807.19,MB
DEFAULT,2026-04-09T21:45:43.260068Z,METRIC_MEM:,10050.3,MB
DEFAULT,2026-04-09T21:45:44.260485Z,METRIC_MEM:,10235.79,MB
DEFAULT,2026-04-09T21:45:45.260779Z,METRIC_MEM:,10445.2,MB
DEFAULT,2026-04-09T21:45:46.261136Z,METRIC_MEM:,10634.31,MB
DEFAULT,2026-04-09T21:45:47.261490Z,METRIC_MEM:,10886.51,MB
DEFAULT,2026-04-09T21:45:48.265978Z,METRIC_MEM:,11324.49,MB
DEFAULT,2026-04-09T21:45:49.266059Z,METRIC_MEM:,12017.47,MB
DEFAULT,2026-04-09T21:45:50.266025Z,METRIC_MEM:,11914.96,MB
DEFAULT,2026-04-09T21:45:51.266082Z,METRIC_MEM:,11647.12,MB
DEFAULT,2026-04-09T21:45:52.272392Z,METRIC_MEM:,11648.13,MB
DEFAULT,2026-04-09T21:45:53.269008Z,METRIC_MEM:,9406.49,MB
DEFAULT,2026-04-09T21:45:54.269512Z,METRIC_MEM:,9458.97,MB
DEFAULT,2026-04-09T21:45:55.269918Z,METRIC_MEM:,9560.43,MB
DEFAULT,2026-04-09T21:45:56.270231Z,METRIC_MEM:,9660.94,MB
DEFAULT,2026-04-09T21:45:57.270653Z,METRIC_MEM:,9766.71,MB
DEFAULT,2026-04-09T21:45:58.271050Z,METRIC_MEM:,9849.17,MB
DEFAULT,2026-04-09T21:45:59.303917Z,METRIC_MEM:,9507.84,MB
DEFAULT,2026-04-09T21:46:00.313037Z,METRIC_MEM:,9122.2,MB
DEFAULT,2026-04-09T21:46:01.309538Z,METRIC_MEM:,9528.65,MB
DEFAULT,2026-04-09T21:46:02.310012Z,METRIC_MEM:,9809.38,MB
DEFAULT,2026-04-09T21:46:03.310421Z,METRIC_MEM:,10074.81,MB
DEFAULT,2026-04-09T21:46:04.310900Z,METRIC_MEM:,10459.12,MB
DEFAULT,2026-04-09T21:46:05.311218Z,METRIC_MEM:,10760.92,MB
DEFAULT,2026-04-09T21:46:06.311655Z,METRIC_MEM:,11069.43,MB
DEFAULT,2026-04-09T21:46:07.312043Z,METRIC_MEM:,11441.79,MB
DEFAULT,2026-04-09T21:46:08.312403Z,METRIC_MEM:,11540.78,MB
DEFAULT,2026-04-09T21:46:09.312344Z,METRIC_MEM:,11546.04,MB
DEFAULT,2026-04-09T21:46:10.312541Z,METRIC_MEM:,10711.32,MB
DEFAULT,2026-04-09T21:46:11.312428Z,METRIC_MEM:,9515.68,MB
DEFAULT,2026-04-09T21:46:12.312786Z,METRIC_MEM:,8679.4,MB
DEFAULT,2026-04-09T21:46:13.313152Z,METRIC_MEM:,8724.61,MB
DEFAULT,2026-04-09T21:46:14.313501Z,METRIC_MEM:,8833.48,MB
DEFAULT,2026-04-09T21:46:15.313877Z,METRIC_MEM:,8843.11,MB
DEFAULT,2026-04-09T21:46:16.314200Z,METRIC_MEM:,8844.81,MB
DEFAULT,2026-04-09T21:46:17.315061Z,METRIC_MEM:,8844.75,MB
DEFAULT,2026-04-09T21:46:18.315452Z,METRIC_MEM:,8845.94,MB
DEFAULT,2026-04-09T21:46:19.315830Z,METRIC_MEM:,8850.8,MB
DEFAULT,2026-04-09T21:46:20.316259Z,METRIC_MEM:,8860.11,MB
DEFAULT,2026-04-09T21:46:21.316618Z,METRIC_MEM:,8866.31,MB
DEFAULT,2026-04-09T21:46:22.316957Z,METRIC_MEM:,8858.84,MB
DEFAULT,2026-04-09T21:46:23.317325Z,METRIC_MEM:,8746.22,MB
Loading
Loading