Intermittent initialization failure on some Kasli SoC banks

[kaolpr]

Summary

We have identified a recurring initialization failure when Fastino is connected to Kasli SoC EEM ports driven by 1.8V banks (2-7, 10, 11). The issue does not appear when using 2.5V banks (ports 0, 1, 8, 9).

This behavior is inconsistent across units; it was identified in the latest production batch, though it may be latent in others. All affected units pass stackup verification and operate correctly on non-1.8V ports.

Observations & Investigation

While troubleshooting, we noticed clock signals exposed via TP22 and TP23 CMOS test points.

Investigation of signals entering the Fastino FPGA with differential and active probes showed no significant distortion or Signal Integrity (SI) issues, even when the initialization failed.

However, disabling the drivers for these two test points consistently resolved the initialization issue in our test setup. With the test point drivers removed, the Fastino units initialize and operate correctly across all Kasli SoC EEM ports, including those on 1.8V banks.

Conclusion

While the differential probes did not reveal obvious SI issues, the fact that disabling these CMOS outputs resolves the failure suggests we may be on the tight margin of operation at some point. The question is whether the problem is on the gateware or layout side, and if removing TP drivers creates enough safety margin for reliable operation.

[gkasprow]

Can't it be explained by poor power sequencing where FPGA banks are back supplies via LVDS links from Fastino or the other way? That may also cause failure of the on die termination and other effects like latchup. The effect will strongly depend on component variation causing race condition and some combination of port voltage, turn on time and production batch may work and some may not.
ICE40 FPGA do not have termination at all so don't expect any differences in SI. The clock signals are likely to be present all the time and maybe those signals wake up first just after FPGA boots supplying Fastino FPGA before it gets its power