Summary
A vehicle can stop receiving transport orders entirely. Orders pinned to it stay
in DISPATCHABLE forever while the vehicle sits IDLE at a known position with
a route available, holding nothing that blocks anyone. Orders pinned to other
vehicles are picked up immediately, so the dispatcher itself is alive.
Only a kernel restart, or toggling that vehicle's integration level, brings it
back.
Reproduces within a few order cycles on the bundled Demo-01 model with the
loopback driver, on both v7.5.0 and master.
Expected
An idle vehicle at a known position, integrated as TO_BE_UTILIZED, should be
assigned a dispatchable order that has a valid route to its destination.
Actual
Vehicle-02 state=IDLE procState=IDLE position=Point-0004
allocatedResources = ['Point-0004'] <- just its own point, nothing blocked
orders pinned to it: DISPATCHABLE, indefinitely
Vehicle-03 state=IDLE procState=IDLE position=Point-0027
allocatedResources = ['Point-0027']
Vehicle-04 state=EXECUTING procState=PROCESSING_ORDER
working normally at the same time
Every documented assignment precondition holds for Vehicle-02:
- order state
DISPATCHABLE
procState IDLE, state IDLEintegrationLevel TO_BE_UTILIZED- reported at a known position
- a route exists (checked with
routeComputationQuery)
acceptableOrderTypes is *, vehicle not paused, energy level 100
POST /v1/dispatcher/trigger changes nothing. Waiting does not help either; the
order was still DISPATCHABLE after several minutes.
Steps to reproduce
- Start the kernel with the bundled
Demo-01 plant model.
- Attach the loopback adapter to
Vehicle-02, Vehicle-03 and Vehicle-04,
set integrationLevel to TO_BE_UTILIZED, initialise their positions.
- Repeatedly: pick a vehicle that is
IDLE / TO_BE_UTILIZED at a known
position, create a single-destination order pinned to it via
intendedVehicle, wait for it to reach a final state.
- Within a few cycles one order stays
DISPATCHABLE for good.
The attached script does this over the web API and reports the cycle where it
stalls. Python 3, no dependencies:
python3 repro_dispatch_stall.py --cycles 12 --no-withdraw
[ 1] ok (Vehicle-02 -> Goods in north 02)
[ 2] ok (Vehicle-03 -> Goods in south 01)
[ 3] NOT ASSIGNED -> dispatcher stalled
order=REPRO-0003-... vehicle=Vehicle-02 pos=Point-0025
Reproducibility
Reproduced on v7.5.0 (tag) and on master @ 391c842e (7.6.0-SNAPSHOT),
multiple runs each, kernel restarted between runs.
What was ruled out
- Not withdrawal-related. A control run that never withdraws, letting every
order finish, stalls just as reliably.
- Not resource contention. With
kernelapp.vehicleResourceManagementType = LENGTH_IGNORED the affected vehicle
holds only its own point, nothing waits on it, and the stall still happens.
(With the default LENGTH_RESPECTED the 1000 mm loopback vehicle also keeps
the short path behind it, which looks similar but is separate and appears to be
working as designed.)
- Not a general dispatcher failure. Orders pinned to other vehicles are
assigned and executed while one order sits in DISPATCHABLE.
Workaround
Setting the affected vehicle's integrationLevel to TO_BE_IGNORED and back to
TO_BE_UTILIZED restores it without a kernel restart. That it is recoverable
per vehicle, without touching anything else, suggests some per-vehicle state in
the dispatcher rather than resource accounting.
One place worth a look: OrderReservationPool releases reservations in only two
spots (AssignReservedOrdersPhase), plus a full clear on dispatcher shutdown.
Not verified, since the pool is not observable through the web API.
Environment
- openTCS
v7.5.0, also reproduced on master @ 391c842e
- Plant model: bundled
Demo-01
- Driver: loopback (
LoopbackCommunicationAdapter)
- JDK: Temurin 21.0.12.1, Linux (Ubuntu 22.04)
- Web API v1 only, no GUI actions
Reproduction script
repro_dispatch_stall.py (Python 3, no dependencies)
#!/usr/bin/env python3
"""Reproduce: openTCS stops assigning transport orders to a vehicle.
Orders sit in DISPATCHABLE while the vehicle they are pinned to is IDLE /
TO_BE_UTILIZED at a known position with a route available, and an explicit
dispatcher trigger changes nothing. Only a kernel restart clears it.
Root cause observed: after finishing an order, a vehicle can keep an allocation
on the *path* it arrived on, not just its current point. Anything routed through
that path then waits indefinitely.
Withdrawals were the original suspicion and are not the cause: --no-withdraw,
which lets every order finish normally, stalls just as reliably. Both modes are
kept so the control run stays easy to repeat.
Every few cycles the script probes whether dispatching still works at all, and
reports the cycle where it breaks.
python3 repro_dispatch_stall.py --cycles 12 --no-withdraw # control run
python3 repro_dispatch_stall.py --cycles 20 # with withdrawals
"""
from __future__ import annotations
import argparse
import json
import sys
import time
import urllib.error
import urllib.request
DEFAULT_API = "http://localhost:55200/v1"
TERMINAL = {"FINISHED", "FAILED", "WITHDRAWN", "UNROUTABLE"}
def call(api, path, payload=None, method=None):
data = json.dumps(payload).encode() if payload is not None else None
req = urllib.request.Request(
f"{api}{path}", data=data,
method=method or ("POST" if data is not None else "GET"),
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=15) as resp:
body = resp.read().decode()
return json.loads(body) if body.strip() else {}
except urllib.error.HTTPError as exc:
return {"__error": f"HTTP {exc.code}: {exc.read().decode()[:120]}"}
except urllib.error.URLError as exc:
raise SystemExit(f"cannot reach kernel: {exc.reason}")
def vehicles(api):
"""Loopback vehicles only; the VDA5050 one waits for a real vehicle."""
return [v for v in call(api, "/vehicles") if "VDA5050" not in v["name"]]
def free_vehicle(api):
for v in vehicles(api):
if (
v.get("procState") == "IDLE"
and v.get("state") in ("IDLE", "CHARGING")
and v.get("integrationLevel") == "TO_BE_UTILIZED"
and v.get("currentPosition")
):
return v
return None
def wait_until(predicate, timeout, interval=1.0):
deadline = time.time() + timeout
while time.time() < deadline:
value = predicate()
if value:
return value
time.sleep(interval)
return None
def order_state(api, name):
o = call(api, f"/transportOrders/{name}")
return o.get("state") if "__error" not in o else None
def dispatch_works(api, jobs, timeout=25) -> bool:
"""Ground truth: can a fresh order still be assigned to a free vehicle?"""
v = free_vehicle(api)
if not v:
return True # No free vehicle to test with; not evidence of a stall.
name = f"PROBE-{int(time.time() * 1000) % 10**9}"
loc, op = jobs[0]
res = call(api, f"/transportOrders/{name}", {
"destinations": [{"locationName": loc, "operation": op}],
"intendedVehicle": v["name"],
})
if "__error" in res:
print(f" probe could not be created: {res['__error']}")
return False
assigned = wait_until(
lambda: order_state(api, name) not in (None, "RAW", "ACTIVE", "DISPATCHABLE"),
timeout,
)
call(api, f"/transportOrders/{name}/withdrawal", {})
return bool(assigned)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--api", default=DEFAULT_API)
ap.add_argument("--cycles", type=int, default=40)
ap.add_argument("--no-withdraw", action="store_true",
help="control run: never withdraw, let each order finish")
ap.add_argument("--drive-seconds", type=float, default=3.0,
help="seconds to let the vehicle drive before withdrawing")
args = ap.parse_args()
model = call(args.api, "/plantModel")
ops = {t["name"]: [o for o in t.get("allowedOperations", []) if o.upper() != "NOP"]
for t in model["locationTypes"]}
jobs = [(l["name"], ops[l["typeName"]][0])
for l in model["locations"] if ops.get(l["typeName"])]
if len(jobs) < 2:
raise SystemExit("model has too few usable locations")
mode = "without withdrawals" if args.no_withdraw else "with withdrawals"
print(f"model {model['name']}: {len(jobs)} jobs | {args.cycles} cycles, {mode}\n")
for cycle in range(1, args.cycles + 1):
v = wait_until(lambda: free_vehicle(args.api), 60)
if not v:
print(f"[{cycle:3d}] no free vehicle within 60s -> STALLED")
return 1
loc, op = jobs[cycle % len(jobs)]
name = f"REPRO-{cycle:04d}-{int(time.time()) % 100000}"
res = call(args.api, f"/transportOrders/{name}", {
"destinations": [{"locationName": loc, "operation": op}],
"intendedVehicle": v["name"],
})
if "__error" in res:
print(f"[{cycle:3d}] could not create: {res['__error']}")
continue
got = wait_until(
lambda: order_state(args.api, name) not in ("RAW", "ACTIVE", "DISPATCHABLE"),
30,
)
if not got:
print(f"[{cycle:3d}] NOT ASSIGNED -> dispatcher stalled")
print(f" order={name} vehicle={v['name']} pos={v['currentPosition']}")
return 1
if args.no_withdraw:
wait_until(lambda: order_state(args.api, name) in TERMINAL, 90)
else:
# The point of the test: cut in while the vehicle is driving.
time.sleep(args.drive_seconds)
call(args.api, f"/transportOrders/{name}/withdrawal", {})
if cycle % 5 == 0:
ok = dispatch_works(args.api, jobs)
print(f"[{cycle:3d}] dispatch still healthy: {'yes' if ok else 'NO'}")
if not ok:
print("\n>>> REPRODUCED: the dispatcher no longer assigns orders")
print(f">>> after {cycle} cycles, {mode}")
return 1
else:
print(f"[{cycle:3d}] ok ({v['name']} -> {loc})")
print(f"\n{args.cycles} cycles completed, no stall observed.")
return 0
if __name__ == "__main__":
sys.exit(main())
Summary
A vehicle can stop receiving transport orders entirely. Orders pinned to it stay
in
DISPATCHABLEforever while the vehicle sitsIDLEat a known position witha route available, holding nothing that blocks anyone. Orders pinned to other
vehicles are picked up immediately, so the dispatcher itself is alive.
Only a kernel restart, or toggling that vehicle's integration level, brings it
back.
Reproduces within a few order cycles on the bundled
Demo-01model with theloopback driver, on both
v7.5.0andmaster.Expected
An idle vehicle at a known position, integrated as
TO_BE_UTILIZED, should beassigned a dispatchable order that has a valid route to its destination.
Actual
Every documented assignment precondition holds for
Vehicle-02:DISPATCHABLEprocStateIDLE,stateIDLEintegrationLevelTO_BE_UTILIZEDrouteComputationQuery)acceptableOrderTypesis*, vehicle not paused, energy level 100POST /v1/dispatcher/triggerchanges nothing. Waiting does not help either; theorder was still
DISPATCHABLEafter several minutes.Steps to reproduce
Demo-01plant model.Vehicle-02,Vehicle-03andVehicle-04,set
integrationLeveltoTO_BE_UTILIZED, initialise their positions.IDLE/TO_BE_UTILIZEDat a knownposition, create a single-destination order pinned to it via
intendedVehicle, wait for it to reach a final state.DISPATCHABLEfor good.The attached script does this over the web API and reports the cycle where it
stalls. Python 3, no dependencies:
Reproducibility
Reproduced on
v7.5.0(tag) and onmaster@391c842e(7.6.0-SNAPSHOT),multiple runs each, kernel restarted between runs.
What was ruled out
order finish, stalls just as reliably.
kernelapp.vehicleResourceManagementType = LENGTH_IGNOREDthe affected vehicleholds only its own point, nothing waits on it, and the stall still happens.
(With the default
LENGTH_RESPECTEDthe 1000 mm loopback vehicle also keepsthe short path behind it, which looks similar but is separate and appears to be
working as designed.)
assigned and executed while one order sits in
DISPATCHABLE.Workaround
Setting the affected vehicle's
integrationLeveltoTO_BE_IGNOREDand back toTO_BE_UTILIZEDrestores it without a kernel restart. That it is recoverableper vehicle, without touching anything else, suggests some per-vehicle state in
the dispatcher rather than resource accounting.
One place worth a look:
OrderReservationPoolreleases reservations in only twospots (
AssignReservedOrdersPhase), plus a full clear on dispatcher shutdown.Not verified, since the pool is not observable through the web API.
Environment
v7.5.0, also reproduced onmaster@391c842eDemo-01LoopbackCommunicationAdapter)Reproduction script
repro_dispatch_stall.py (Python 3, no dependencies)