Skip to content

MBP-389: Monitoring crash - #52

Merged
federrg merged 19 commits into
masterfrom
MBP-389-monitoring-crash
Sep 24, 2026
Merged

federrg merged 19 commits into
masterfrom
MBP-389-monitoring-crash

Conversation

@szilard-ess

@szilard-ess szilard-ess commented May 18, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes the infinite loop inside monitoring base with an early return. This caused an issue, that would crash the IPC. Also logs an error message when no EC slaves found.

@szilard-ess szilard-ess self-assigned this May 18, 2026
@szilard-ess szilard-ess added the bug Something isn't working label May 18, 2026
Comment thread POUs/Monitoring/FB_ESSMonitoringBase.TcPOU

@federrg federrg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implement suggested changes proposed by @szilard-ess

@szilard-ess
szilard-ess requested a review from federrg July 7, 2026 10:54
@szilard-ess

Copy link
Copy Markdown
Collaborator Author

I added the message and in one of the commits also included my proposal to fix the cycle time spikes happening during EtherCAT monitoring. See the task MBP-389 and its subtask MBP-390 for more information.

Comment thread POUs/Monitoring/FB_ESSMonitoringBase.TcPOU Outdated
@szilard-ess
szilard-ess requested a review from mac-kan July 13, 2026 09:26
@szilard-ess

Copy link
Copy Markdown
Collaborator Author

I have added clear of aSlaveConfig if the current slave count is less than, the count in the previous cycle to not have invalid data stored inside aSlaveConfig from previous copy.

@federrg federrg left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chages requsted by me have been implemented. This is a bug fix, if the bug has been fixed then this can be approved.
I am the one who requeted the changes, so I guess I need to approve. so just waiting for the ocnfimration from @mac-kan @szilard-ess tha the bug has benn fixed.

@szilard-ess

Copy link
Copy Markdown
Collaborator Author

Chages requsted by me have been implemented. This is a bug fix, if the bug has been fixed then this can be approved. I am the one who requeted the changes, so I guess I need to approve. so just waiting for the ocnfimration from @mac-kan @szilard-ess tha the bug has benn fixed.

With the latest changes, there is no more MEMSET/MEMMOVE use which could of caused issues as @mac-kan mentioned it, instead we swap between the 2 buffers.

Note: The evaluation of the buffer will be running on empty buffer until one of the buffers is filled the first time, but this can be solved with a check.

I had this note previously, but in commit a1d4c76 I changed the delay time to be calculated based on the number of slaves, this makes sure that we have enough time to read the slaves atleast once. Another solution would be to have bFirstReadDone variable which is written after the first swap of the pointers.

@federrg

federrg commented Sep 11, 2026

Copy link
Copy Markdown
Member

@szilard-ess I do not have a strong perference. If the variable will facilitate debugging and trouble shooting by all means add it. if not at least a comment in the code explaining what you metioned would be enough. I leave it up to your criteria.

@szilard-ess

Copy link
Copy Markdown
Collaborator Author

@szilard-ess I do not have a strong perference. If the variable will facilitate debugging and trouble shooting by all means add it. if not at least a comment in the code explaining what you metioned would be enough. I leave it up to your criteria.

If I understand the old code fully, the timer fbEtherCATAlarmDelayTON is there to delay the alarms and wait for the read of fbGetEcSlaveStates. If there is no other reason, then I would prefer a variable named bFirstReadDone, instead of a timer. @mac-kan could you please confirm this for me?

@federrg

federrg commented Sep 11, 2026

Copy link
Copy Markdown
Member

Hi @szilard-ess , yes, that timer is there exactly for that reason. we were getting EtherCAT error every start up or activating configuration, so we included the delay.

@szilard-ess

szilard-ess commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

Hi @szilard-ess , yes, that timer is there exactly for that reason. we were getting EtherCAT error every start up or activating configuration, so we included the delay.

Thank you for the info, I changed it to wait until the first read is finished instead.

@mac-kan mac-kan left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job with changing the swap logic to pointers, and your suggestion of moving away from the time dependency when starting up! 👍

I think there is still some work required related to the guard logic though (see specific comment). Because of this, another bug seems to be masked, and I think it would make sense to introduce a startup timeout, so that we can trigger the '0 EtherCAT Slaves found' error after some seconds. As is now, that error should be triggered the cycle after the first failed read attempt during startup, then it will pop through. After the first issue has been fixed, the message will be prevented to pop through if happening during startup.

Comment thread POUs/Monitoring/FB_ESSMonitoringBase.TcPOU Outdated
@szilard-ess
szilard-ess requested a review from mac-kan September 14, 2026 12:48

@mac-kan mac-kan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like most things have been fixed, good job!

Comment thread POUs/Monitoring/FB_ESSMonitoringBase.TcPOU Outdated
Comment thread POUs/Monitoring/FB_ESSMonitoringBase.TcPOU Outdated
Updates and enabled error message of slave link error, when no EtherCAT
slave were found.
Reduces the amount of data being copied drung EtherCAT monitoring,
which caused cycle time spikes.
Adds clear to aSlaveConfig if the current slave count is less than,
the count in the previous cycle to not have invalid data stored inside
aSlaveConfig from previous copy.
Uses read/write buffers to remove the use of MEMCOPY/MEMMOVE/MEMSET,
which increased cycle time.
Fixes the timer of get slave config data to work as a delay after a
finished read out instead of triggering execute every 100 ms, eventough
the read function block was still busy.
Changes early return logic, checks when FB_EcGetConfSlaves is done with
the first readout, instead of relying on a timer.
Changes order of early returns to be able print error message even when
there was no sucessful read at all.
Adds clear of zero slave found error to be able to detect it more than
once.
Fixes the issue of possible buffer swap without sucessful read of
configured slave data, by introducing the check of returned slave count.
Changes execute logic of all diagnostic function blocks to start the
execution on startup and to have a consistent logic.
Adds init of buffer addresses, to fix initialization restriction of
variables inside VAR_INST.
@mac-kan
mac-kan force-pushed the MBP-389-monitoring-crash branch from b8717e3 to 482a15f Compare September 24, 2026 07:53

@federrg federrg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested by @mac-kan

@federrg
federrg merged commit 2723af7 into master Sep 24, 2026
2 checks passed
@federrg
federrg deleted the MBP-389-monitoring-crash branch September 24, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants