TiDB Global Memory Control Mechanism

[solotzg]

Feature Request: Global Memory Control Mechanism

Background

• This diagram describes the bottom-up process of the current TiDB memory control chain.

  • memory control by tracker

• The default behavior of TiDB Memory Control is like:

  • SQL execution layer invokes the memory tracker to report the heap usage to the session layer.
  • The top-level OOM control module regularly checks at a granularity of 100 milliseconds. When the global memory exceeds the server limit, it executes kill sql & session operations to release memory resources and protect the memory safety of the tidb process
  • The system variable tidb_server_memory_limit controls the threshold for the memory usage of a tidb-server instance.
  • When the memory usage of TiDB reaches the limit, TiDB cancels the currently running SQL statement with the highest memory usage (greater than tidb_server_memory_limit_sess_min_size). After the SQL statement is successfully canceled, TiDB tries to call Golang GC to immediately reclaim memory to relieve memory stress as soon as possible.
  • The system variable tidb_mem_quota_query sets the limit for a query in bytes. If the memory quota of a session during execution exceeds the threshold value, TiDB performs the operation defined by tidb_mem_oom_action.
  • The disk spill behavior is jointly controlled by the following parameters: tidb_mem_quota_query, tidb_enable_tmp_storage_on_oom, tmp-storage-path, and tmp-storage-quota.

• There are 3 issues under the current memory control mechanism:

  • Kill SQL Risk
    • When facing concurrent queries, the Kill behavior has a certain randomness. TiDB lacks the ability to control or schedule memory resources, and cannot guarantee that all queries will eventually be successfully executed. Especially for AP scenarios that need to ensure SQL execution, this method is difficult to meet basic needs.
  • Performance Risk: Heavy Golang GC
    • When the global memory usage reaches the limit of golang runtime and most of the memory resources are legally occupied, frequent golang GC may be triggered, greatly affecting the delay of normal memory allocation.
    • There is already a mechanism called memoryLimitTuner to reduce the impact of golang GC. It dynamically adjusts the memory limit of golang runtime (adjusted every 1 minute by default). However, The memory limit of tidb will be larger than tidb_server_memory_limit.
  • OOM Risk
    • According to the implementation of servermemorylimit mechanism, each memory tracker is independent. When the controller begins to kill the top sql / session, others may continue to consume heap. Such information isolated island can easily lead to TiDB instance OOM. This phenomenon is particularly serious when facing a large number of concurrent small OLAP queries. It is also one of the common factors of OOM problems in customer scenarios.

• Rank the 3 existing issues by severity: OOM ≈ Heavy Golang GC > Kill SQL/session

  • Kill SQL is the least serious. Although SQL errors may affect upper-level business, they do not affect the security of the tidb instance. Reclaiming memory by KILL only works when there are enough heap resources can be GC (not occupied legally) to make the SQL complete the kill process.
  • The severity of Heavy Golang GC is approximately equal to OOM. When memory resources are insufficient and mostly legally held, a large amount of CPU is wasted on invalid GC, and the process is prone to freezing.
  • The OOM issue is the most serious. If golang runtime judges that GC cannot be performed, it will apply for memory from the system, and the process is prone to OOM.

Goals

• Avoid OOM
• Avoid performance regression
• Avoid SQL failures (configurable Kill/Cancel SQL/session behavior)

Alternatives

• New architecture: centralized management of global memory resources, using a subscription-first and then-allocation model

User Scenarios

• General scenarios: users need TiDB system stability to avoid memory security issues
  • High concurrency small AP, high stress TP & AP mixed, abnormal memory leak, etc
• OLAP scenarios that need to ensure SQL execution (multi-concurrency and large AP)
• Low latency / non-blocking services (OLTP)
  • When the memory resources are insufficient the upper-layer can quickly retry SQL / workload to other TiDB nodes (which can be perceived by TiProxy).
• Memory leak scenario with continuous high stress (cyclic OOM problem)

Proposal

• design: TiDB Global Memory Arbitrator #60628

Progress

Release Plan

• design doc
• memory arbitrator modules
• session & mem_tracker & sys_vars & utils adaptation
• SQL execution adaptation
• metrics & monitor adaptation
• other modules adaptation
• integration tests & benchmark & report
• release SPEC
• Release github design doc
• Enable feature by default

Development

• memory: new submodules MemArbitrator & ResourcePool #60597
• memory: fix data race in #63054  #63056
• memory: global mem resources arbitrator #63073
• feat: support TiDB mem arbitrator tidb-dashboard#1840
• memory: copy global mem arbitrator metrics to nextgen grafana #65042
• executor, sessionctx: remove MemArbitrator from slow log rules #65105
• memory: fix data/stats leak of the global mem-arbitrator when arbitrary panics occur during stmt execution #65318
• memory: fix data race & enhance behaviors in arbitrator #65569
• *: enable global memory arbitration in priority mode by default #69638
• tidb: global memory arbitrator mechanism docs-cn#20955
• tidb: enable global memory arbitration in priority mode by default docs-cn#21827
• SPECT
  • English official spect
  • Chinese draft spect
• Github issue & Implementation
  • Global arbitrator work mode
    • DISABLE mode:
      • SQL associated with mem-arbitrator, memory subscription requests will all be satisfied until SQL execution is complete; The new SQL will no longer be connected to mem-arbitrator; Disable the Kill mechanism of mem-arbitrator;
    • STANDARD mode
      • SQL active CANCEL when memory resources are insufficient
      • Turn off the original OOM control mechanism
      • Close memoryLimitTuner, the expected memory limit of TiDB instance is similar to golang runtime soft memory limitand server-limit
    • PRIORITY mode:
      • When memory resources are insufficient, CANCEL SQL according to Resource Group priority , and the memory resource subscription task of the same level SQL is executed according to FIFO blocking scheduling (SQL execution time constraint max_execution_time)
      • CANCEL low-priority SQL order: priority from small to high, memory usage from large to small
      • detect roo pool hang during CANCEL
    • await-free pool
      • For small queries / tasks whose max mem usage is less than 1/1000 of server limit
      • Adapt to low latency memory consumption task
      • Adapt to unknown memory consumption task
    • Dynamically switch mem-arbitrator working mode
      • DISABLEswitch to PRIORITYorSTANDARD
      • Switch between PRIORITYand STANDARD
      • STANDARD or PRIORITY switch to DISABLE
    • Memory Pool
      • Basic resource quota control: grow / shrink / reserve
      • Register / Remove Memory Pool
      • Actions: notification, out-of-cap, out-of-limit
  • OOM Control:
    • Handle Memory Unsafe:
      • reclaim non-blocking tasks
      • reclaim heap by gc
      • calc/detect dead lock
      • detect OOM risk
      • dump runtime mem profile
    • Handle OOM Risk:
      • reclaim heap by KILL: priority from small to high, memory usage from large to small
      • detect SQL hang during KILL
    • Soft Limit
      • AUTO: self-adaptation according to runtime state
      • specified: bytes / ratio
      • default
  • Interfaces
    • Resource Group:
      • LOW / MEDIUM (default) / HIGH: SQL binding memory resource priority; If no resource group is configured, the default is MEDIUM
    • sys vars:
      • tidb_mem_arbitrator_soft_limit:
        • global level
        • 0 (default)：95% server-limit
        • Bytes(integer): Number of bytes (1, server-limit]
        • Server-limit ratio (0, 1]
        • AUTO: mem-arbitrator self-adaptation adjustment according to runtime state
      • tidb_mem_arbitrator_mode:
        • global level
      • tidb_mem_arbitrator_query_reserved:
        • session level (SQL Hint)
      • tidb_mem_arbitrator_wait_averse:
        • session level (SQL Hint)
        • SQL can automatically bind HIGH level memory resource priority;
        • SQL cancels itself when memory resources are insufficient
  • Auto tune
    • auto soft limit
      • auto tune soft limit under tidb_mem_arbitrator_soft_limit AUTO mode
      • calc & set magnification ratio when meeting Memory Unsafe
      • try to reduce ratio during normal situation
        • calc runtime mem profile
        • shrink memory resources magnification
    • root pool mem profile by digest
      • auto stats pool digest key ( normalized SQL)
      • get / set / shrink
      • reserve enough mem quota before execution
    • calc medium memory quota usage of root pools as pool init cap suggestion
      • calc medium memory quota usage of root pools as pool init cap suggestion
      • use suggestion pool cap if:
        • NO digest exists
        • root pools can not allocate from global mem arbitrator
  • Adapt to other modules
    • SQL Compiler
      • pre reserve fixed size of memory quota for compiler: 4MB
    • Session & Memory tracker
      • set sys vars & SQL hints
      • interact with global mem arbitrator during consumption
    • utils
      • gctuner
        • disable previous mem control logic
      • servermemorylimit
        • disable previous mem control logic
      • domain gcStatsWorker
    • coprocessor
    • TiProxy
      • session dispatch machinist modification
      • choose the tidb node with minimum limit - pending_alloc size
    • Background task (no session)
      • internal SQL (session id is 0)
      • ddl
      • import / lightning
      • analyze
    • Other types of cache
      • SQL / plan / chunk / cop cache
      • other caches
    • Monitor
      • Alerting
        • TiDB instance has the risk of memory freeze
        • TiDB instance appears to KILL SQL
      • Prometheus / Grafana
        • mem arbitrator self
        • other modules
  • Manageability & Diagnosability
    • Metrics
      • The number of memory subscription tasks in the queue (with different priorities).
      • Number of subscription tasks, processing time
      • Memory resources: available, unavailable, unsafe, allocated, memory stress value, GC (number of triggers, time spent), memory share limit, SQL memory usage top-3 /median, runtime memory usage speed
      • CANCEL sql number /reason
      • Process OOM risk KILL SQL number
    • Log
      • Memory risk: alarm, global memory freeze risk, runtime memory usage speed, kill SQL behavior
      • Cancel SQL: reason, memory state
      • Runtime mem profile in timeline
    • SQL explain results & SQL execution statistics
      • Binding arbitrator behavior pattern: STANDARD, PRIORITY
      • Memory priority: LOW, MEDIA, HIGH
      • Memory subscription task count, processing time (avg, max, min)
      • Maximum used memory share, maximum subscribed memory share
      • execution summary
      • expensivequery
    • Perf mem profile
      • dump tree diagram of global memory state
        • tree diagram pools
        • memory quota by modules
        • runtime memory perf info by modules

Tests

• Unit Tests
  • memory pool
    • budget quota management: alloc / cap / reserve / used / limit
    • actions: notification, out-of-cap, out-of-limit
    • stats: tree structure / dump
    • benchmark
  • memory arbitrator basic logic
    • basic
      • root pool entry management: CRUD
      • traverse order by priority / quota size
      • digest profile cache: CRUD
      • control runtime gc
      • Record mem state
      • handle roo pool hang during CANCEL
      • await-free pool: alloc / shrink / tracked heap usage
    • async process
      • async notifer
      • background goroutine(start/stop)
      • root pool entry management (with memory pool)
      • root pool blocking alloc quota with cancel
      • update soft-limit & limit
    • DISABLE mode
      • satisfy all subscription tasks
      • no mem risk control
    • STANDARD mode
      • exec subscription tasks by fifo
      • CANCEL all tasks when meeting resource shortage
    • PRIORITY mode
      • exec subscription tasks by priority
      • wait until cancel or arbitrate result
      • CANCEL lower priority tasks
      • mixed tasks with ArbitrateWaitAverse
      • prefer privileged budget: resolve deadlock
    • work mode switch
      • Disable mode -> Standard mode
      • Standard mode -> Priority mode
      • Priority mode -> Disable mode
    • exec metrics
      • task success / fail
      • task work mode: standard, priority
      • cancel: standard / priority / waitAverse
      • memRisk / oomRisk / oomKill
      • actions: gc / updateRuntimeMemStats / recordMemState
    • auto tune
      • calc buffer
      • calc heapinuse / quota magnification
      • calc runtime memStats & update magnification
      • avoidance( out-of-control )
      • dynamic soft-limit
      • tracked mem stats
      • suggest pool cap: medium pool cap
    • oom control
      • detect / handle mem risk
      • detect / handle oom risk
      • record runtime mem profile
      • control soft limit
      • reclaim heap by KILL: priority from small to high, memory usage from large to small
      • handle roo pool hang during KILL
    • corner cases:
      • KILL root pool meets deaklock
      • CANCEL root pool meets deaklock
      • illegal operations under each work modes
  • Session with mem arbitrator
    • operations about sys vars: tidb_mem_arbitrator_soft_limit; tidb_mem_arbitrator_mode; tidb_mem_arbitrator_query_reserved; tidb_mem_arbitrator_wait_averse;
    • set tidb_server_memory_limit --> affect limit of global mem arbitrator
    • SQL with resource_group / wait_averse / query_reserved profiles
    • SQL with max_execution_time
    • SQL execution summary
  • Memory tracker with mem arbitrator
    • global unique mem arbitrator
    • attach to await-free pool when quota usage is small
    • reserve root pool quota; auto use digest max historical quota
    • auto increase root pool quota cap
    • RuntimeMemStateRecorder: load / store / persist MemState
    • supports KILL / INTERRRUPT interface of TrackerArbitrateHelper
    • tracker with context: priority / wait_averse / query_reserved / quota limit
    • concurrent op from sub trackers
  • Other module with mem arbitrator
    • optimizer: use await-free pool
    • other types of cache
• Integration Tests & Benchmark
  • Mixed OLAP & OLTP(low latency required) workload
    • arbitrator standard mode
    • arbitrator priority mode
    • TP queries with wait_averse profile
    • single TiDB instance
    • multiple TiDB instances
  • Multiple Large AP
  • High Concurrency Small AP
  • Memory Leak Exception
  • Continuous high stress Memory Leak
  • Heavy stress test