Root Cause Analysis: ChronoSplit Executor Memory Leak

[0xsuryansh]

Introduction

A memory leak issue was identified and resolved in our ChronoSplit Executor service. This document outlines the situation that led to the discovery of the memory leak, the investigative actions taken, and the solution implemented to address the issue.

Situation

The service is designed to awaken every hour to fetch and execute tasks. These tasks are divided into chunks and processed in parallel. Over several months, it was observed that the latency of task execution was steadily increasing. This escalation in latency eventually led to a significant backlog of tasks, causing delays that extended beyond the intended hourly execution cycle.

Investigation

The symptoms observed—increasing latency and task backlog—suggested a potential memory leak within the service. Memory leaks typically manifest as gradual performance degradation due to the accumulation of unused or unreleased memory allocations over time. However, an initial review of system metrics revealed no significant increase in memory consumption. Instead, the service exhibited 100% CPU utilization, an anomaly that seemed unrelated to a traditional memory leak scenario.

Despite the absence of clear memory-related indicators, a process dump was taken from the service instance experiencing 100% CPU usage. Analysis of the dump revealed an unusually high number of Write Exception objects. Further examination identified the presence of large objects on the heap, specifically large arrays approximately 2 million elements in size, containing instances of a custom de-serializer. Multiple instances of these large arrays were present, indicating excessive memory allocation.

Root Cause

The root cause of the issue was traced to the improper initialization of the custom de-serializer objects used within the service. The code responsible for de-serialization operations incorrectly re-initialized the custom de-serializer objects on every write operation, adding each new instance to a list of initializers. This resulted in an inefficient and unnecessary accumulation of de-serializer instances, which, in turn, led to the large arrays observed in the heap dump. The excessive memory allocation and CPU utilization were consequences of the service's attempt to manage and iterate through this growing list of de-serializer instances.

Resolution

To address the issue, the service's de-serialization logic was refactored. The custom de-serializer objects were modified to follow a singleton pattern, ensuring that only a single instance of each de-serializer is created and reused for all write operations. This change eliminated the unnecessary initialization and accumulation of de-serializer instances, thereby reducing the memory footprint and CPU load of the service.

Outcome

Following the implementation of the fix, the service was monitored over an extended period to assess the impact of the changes. The results were positive, with a notable reduction in latency and CPU utilization. The service's performance stabilized, and the task execution backlog was effectively eliminated, restoring the intended hourly task processing cycle.

Conclusion

The memory leak issue in our service was successfully resolved by identifying and correcting a flaw in the de-serialization logic. This case underscores the importance of thorough investigation and analysis, even when initial symptoms may not align with typical diagnostic indicators. By addressing the root cause, the service's performance and reliability were significantly improved, ensuring the efficient and timely execution of tasks.