Queue stuck after a while #3
Description
Hi @shimaore,
Thanks for this awesome package 📦!
I encounter some really complex and hard to reproduce issue with ESL Lite with Freeswitch 1.10.11/12 version
My nodejs 24 application calls bgapi commands to play/stop audio, etc.
However after some days at best, hours at worse when running my application with multiple replicas k8s, the commands hangs in your internal fifo queue.
As much as I could understand, at some point after my app starts, and new bg_api (A) is played, set in the queue, received by Freeswitch and executed (visible is fs logs).
Then, the next bg_api (B) command added to the queue will never be sent/played (nothing in fs logs).
Logging inside the lib the sendNextCommand current value, it stays the same as the previous bg_api (A) forever
- play bg_api A
- FS play the command
- play bg_api B
- current queue still stuck on bg_api A ignoring other commands, FS receive absolutely nothing
- keeps receiving events howewer from FS
I have logs from lib send: await enqueue but no more following send: enqueue done or sent
here's a simplified version of my esl service (using nestjs) running once per instance (singleton)
export class EslService {
private readonly logger = new Logger(EslService.name)
private client: FreeSwitchClient
private connect() {
this.client = new FreeSwitchClient({
...config,
logger: pino(),
})
this.attachListeners()
}
// [...]
async playAudio(
callId: string,
resource: string,
domain: string,
leg: BroadcastLeg = BroadcastLeg.BOTH
) {
this.logger.verbose(
`play audio resource: ${resource} on call ${callId}@${domain} with leg: ${leg}`
)
// awaits and format response for users
return this.formatResponse(
this.client.bgapi(
`uuid_broadcast ${callId} ${resource} ${leg}`,
this.timeout // 2000-5000ms env
)
)
}
}when it the queue is stuck, my play audio function will not return anything for 5 minutes (exact time) and each new attempt will resolve identically until I reboot instance of app.
Using FS ESL on port 8021 just like it's suggested.
Here's some logs I have from trace level on FreeswitchClient before the queue gets stuck:
{
"level": "trace",
"time": 1763745014874,
"instanceId": "xxxx",
"pid": 1,
"module": "FreeSwitchResponse",
"ref": "01KAKKZ0PNTZV7V31CJK7XYTKE",
"ev": {
"name": "FreeSwitchInvalidBodyError",
"description": "JSON parsing error",
"exception": {},
"body": ""
},
"message": "event"
}
{
"level": "trace",
"time": 1763738418227,
"instanceId": "xxxx",
"pid": 1,
"module": "FreeSwitchResponse",
"ref": "01KAKF01MCXM3NZ4WHNZXY38TE",
"ev": {
"name": "FreeSwitchMissingContentTypeError",
"headers": {
"replyText": "+OK Job-UUID",
"headers": {}
},
"body": {
"type": "Buffer",
"data": []
}
},
"message": "event"
}
{
"level": "debug",
"instanceId": "xxxx",
"pid": 1,
"timestamp": 1763731250001,
"message": {
"missingContentType": "1",
"missingEventName": "0",
"authRequest": "1",
"commandReply": "102",
"events": "0",
"bodyParseErrors": "549",
"logData": "0",
"disconnect": "0",
"apiResponses": "0",
"rudeRejections": "0",
"unhandled": "0",
"unflushedWrites": "0",
"nonEmptyBufferAtEnd": "0"
},
"context": "EslService"
}after the queue is locked, those parse body or content type errors are gone...
when it works I am able to trace this kind of background jobs:
{
"level": "trace",
"time": 1763747425349,
"instanceId": "xxxx",
"pid": 1,
"module": "FreeSwitchResponse",
"ref": "01KAKKZDAZFVTRCWA2P1BDAN0E",
"ev": {
"event": "BACKGROUND_JOB",
"headers": {
"contentLength": 794,
"contentType": "text/event-json",
"headers": {}
},
"body": {
"eventName": "BACKGROUND_JOB",
"jobUUID": "01KAKRKV2060FGY9MVK96K543R",
"response": "+OK Message sent\n",
"data": {
"Event-Name": "BACKGROUND_JOB",
"Core-UUID": "f98ced4d-9bd1-47a4-98cd-081bbcdbd0ff",
"FreeSWITCH-Hostname": "<hostname>",
"FreeSWITCH-Switchname": "<hostname>",
"FreeSWITCH-IPv4": "10.0.140.5",
"FreeSWITCH-IPv6": "::1",
"Event-Date-Local": "2025-11-21 18:50:24",
"Event-Date-GMT": "Fri, 21 Nov 2025 17:50:24 GMT",
"Event-Date-Timestamp": "1763747424906986",
"Event-Calling-File": "mod_event_socket.c",
"Event-Calling-Function": "api_exec",
"Event-Calling-Line-Number": "1572",
"Event-Sequence": "177226",
"Job-UUID": "01KAKRKV2060FGY9MVK96K543R",
"Job-Command": "uuid_broadcast",
"Job-Command-Arg": "efad4a11-1f87-4a66-ad4f-397f99d42c4d /var/lib/freeswitch/recordings/<hostname>/suzume_no_tojimari.wav both",
"Content-Length": "17",
"_body": "+OK Message sent\n"
}
}
},
"message": "event"
}when failing I receive only this logs then nothing more about the command:
{
"level": "trace",
"time": 1763748616606,
"instanceId": "xxxx",
"pid": 1,
"module": "FreeSwitchResponse",
"ref": "01KAKKYZBVBEKPJN7N1QJAQ5BD",
"command": "bgapi uuid_broadcast ec601dc0-1067-4575-8f5f-26de28c7adbd /var/lib/freeswitch/recordings/<hostname>/suzume_no_tojimari.wav both",
"commandHeaders": {
"job-uuid": "01KAKSR6CYXRS6S8GHQXG0NR53"
},
"timeout": 2000,
"message": "send: await enqueue"
}I can provide more traces (tcp dump) per email if needed as it's tricky to reproduce with a single instance (fewer logs but longer to occur)
Thanks in advance looking into it :)