Skip to content

MOOSE bug — SCHEDULEDISPATCHER heap leak (self-terminating schedules never reclaimed) #2600

Description

@Shazzer1951

Found while running an automation-heavy persistent DCS campaign on MOOSE (OPS:
CHIEF/COMMANDER/LEGION/AIRWING/BRIGADE/FLEET, plus INTEL and MANTIS). Still present in
MOOSE 2.9.18 (73d3ed119cd9 / 2026-06-14) and on develop (the 2.9.18 changeset touches no scheduler
code). Tested on DCS 2.9.27.24969).

Area: Core/ScheduleDispatcher.lua, Core/Scheduler.lua
Severity: memory leak (~110-120 MB/hr, linear)

Symptom

On long-running missions, _SCHEDULEDISPATCHER grows without bound — ~110-120 MB/hr linear Lua-heap
growth while live DCS unit/object counts stay flat
(i.e. referenced, non-collectable memory). It bites
hardest in automation-heavy missions because OPS FSMs (OPSGROUP/ARMYGROUP/FLIGHTGROUP), INTEL,
and CHIEF/COMMANDER are long-lived MasterObjects that schedule delayed __Event-style transitions
every status cycle for the entire mission.

Root cause

When a schedule terminates on its own (a one-shot that fires, or a repeat that reaches its Stop time),
CallHandler calls self:Stop(Scheduler, CallID) — but :Stop only removes the DCS timer and nils
ScheduleID:

function SCHEDULEDISPATCHER:Stop( Scheduler, CallID )
  ...
  timer.removeFunction( Schedule.ScheduleID )
  Schedule.ScheduleID = nil
end

It never removes the records. Left behind for the life of the MasterObject:

  • self.Schedule[Scheduler][CallID] — the ScheduleData table, holding Function, Arguments, and
    the CallHandler closure. Each closure pins its captured upvalues alive.
  • self.PersistentSchedulers[CallID] — a strong table; entries never clear, even after the
    object dies. BASE:ScheduleOnce / BASE:ScheduleRepeat route here (MasterObject = nil).
  • self.ObjectSchedulers[CallID] — weak values, but the value (the SCHEDULER) is pinned by the
    still-live MasterObject, so the entry never clears while the object lives.
  • Scheduler.Schedules[#Scheduler.Schedules + 1] — an append-only array that no code path trims
    (not even RemoveSchedule).

RemoveSchedule clears self.Schedule[Scheduler][CallID], but is never called on the natural-completion
path — and even it leaves ObjectSchedulers / PersistentSchedulers / Scheduler.Schedules untouched.
The weak-table design reclaims correctly when the MasterObject is garbage-collected; the leak only
appears for MasterObjects that live a long time and schedule many self-terminating calls.

Evidence

Reachable-table-graph walk on a ~30-minute-old mission, diffed at 5-minute intervals:

_SCHEDULEDISPATCHER.ObjectSchedulers      +6867 entries / 5 min   (7649 -> 14516 -> 20626 ...)
_SCHEDULEDISPATCHER.PersistentSchedulers  +816  entries / 5 min
_SCHEDULEDISPATCHER.Schedule[<scheduler>] +612  entries / 5 min
per-object SCHEDULER.Schedules arrays     +300  entries / 5 min each   (linear: 298 -> 598 -> 898 ...)

Live Lua heap grew ~110-120 MB/hr, linear, while live DCS object/unit counts stayed flat.

Suggested fix

Reclaim fully when a schedule terminates on its own — not on a user-initiated SCHEDULER:Stop, which
may be followed by Start to restart the same CallID. Only the CallHandler terminal branches need to
change; the manual-Stop path is left alone, so restart-after-stop still works.

function SCHEDULEDISPATCHER:_Reclaim( Scheduler, CallID )
  self:Stop( Scheduler, CallID )                          -- remove DCS timer, nil ScheduleID
  if self.Schedule[Scheduler] then self.Schedule[Scheduler][CallID] = nil end
  self.ObjectSchedulers[CallID]     = nil
  self.PersistentSchedulers[CallID] = nil
end

In CallHandler, the two terminal self:Stop( Scheduler, CallID ) calls become
self:_Reclaim( Scheduler, CallID ). Dropping the ScheduleData while its CallHandler is executing is
safe — the closure is already on the call stack, and after it returns nil DCS will not reschedule. The
append-only Scheduler.Schedules array is a smaller secondary accumulation; key it as a set
(Scheduler.Schedules[CallID] = true, cleared in _Reclaim) or compact it during Clear/Stop(all)
rather than scanning it in the hot fire path. Optionally have RemoveSchedule also clear
ObjectSchedulers / PersistentSchedulers so manual removal is complete too.

Workaround (no fork)

A periodic external sweep over _SCHEDULEDISPATCHER that removes any Schedule[Scheduler][CallID] whose
ScheduleData.ScheduleID == nil (its DCS timer is already gone → it can never fire again), drops the
matching ObjectSchedulers[CallID] / PersistentSchedulers[CallID], and compacts Scheduler.Schedules.
Touching only ScheduleID == nil entries guarantees no live or pending timer is ever affected (a pending
schedule or active repeat keeps ScheduleID non-nil — DCS reuses the id across repeat fires). Needs no
Moose.lua changes and degrades to a no-op the moment the dispatcher starts reclaiming on its own.


Reported from "Operation Redline," a clean-room MOOSE persistent campaign. Happy to provide the
heap-walker tooling used for the evidence above, full dcs.log captures, or to test a patch against
develop.

Metadata

Metadata

Labels

BugMoose bug that needs to be fixed.

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions