Attribute reporting checks in testing framework

[cecille]

The current set of certification tests mostly use read actions to verify device changes. This does not match the normal production flow for controllers, which mostly rely on subscriptions for change notifications rather than polling reads.

We want to retrofit the existing tests to verify subscription reporting as a side effect of the existing read tests in order to ensure that attributes being read are properly reported.

Proposed high level design

• at the start of each test, do a wildcard subscription to the device (setup_test)
• build an initial cache of attribute values at this point
• when reports are received, update the cache of reported values
• at each read instance (probably in read_single_attribute_check_success) validate that the read value matches the cached value, or the cached value is updated within the subscription ceiling.
• shutdown the subscription on test teardown

Potential problems

Tests that re-subscribe with keepSubscriptions=false

Some options:

• deal with the tests individually - better if there's only a few
• have some kind of reporting mechanism that disables these checks if such a subscription is detected?
• mint another fabric specifically for this (has implications for RR-1.1, and whatever other test it is that tries to fill the fabric table - SC something?) - we would want to make this a long-running fabric so it doesn't fill up admin_json. This would be a problem for fabric scoped attributes, so this option is less desirable if there are only a few tests with keepSubscriptions=false.

Changes omitted (C) and quieter report (Q) attributes

• need to find these in the DM files and omit them from checks
• for now you can use the most recent XML files, though there are other plans to have the DM files for the specific device available as part of the PICS effort - a problem for later

Real failures

• There are definitely some clusters that don't report properly and those will need to be identified and fixed (in separate PRs)

Reads that don't go through read_single_attribute_check_success

• maybe we just swap these tests to use this function? In a separate PR? Depends on how widespread this is.

Open questions, needs further thought:

Test plans

• we need a way to indicate that we're doing this in the test plans in a way that doesn't require modifying every test plan

Widespread support

• once there's a proof of concept and the bugs are fixed, it would be good to present this to the CSG (along with a list of anything that got fixed as a result of this effort)

YAML tests

• how do we do this in YAML? Do we need to? Or is most complex logic handled in python nowadays?

Backwards compatibility

• how do we deal with older devices that may not have the fixes you implement? We need to balance asking for improvement from devices vs. not unfairly punishing manufacturers

[j-ororke]

Providing list of Attributes from 1.6 DM cluster XML's currently being omitted from wildcard subscription method in test framework as mentioned in PR 43274:
Changes Omitted (C) - changeOmitted="true" - 72 attributes across 8 clusters:

• Power Source: WiredAssessedInputVoltage, WiredAssessedInputFrequency, WiredAssessedCurrent, BatVoltage, BatChargingCurrent
• General Diagnostics: UpTime, TotalOperationalHours, DeviceLoadStatus
• Thread Network Diagnostics: OverrunCount + all the Tx/Rx MAC counters (47 attributes total)
• Wi-Fi Network Diagnostics: RSSI, BeaconLostCount, BeaconRxCount, all packet counts, OverrunCount
• Ethernet Network Diagnostics: all packet/error/count attributes
• Time Synchronization: UTCTime, LocalTime
• Operational Credentials: NOCs, TrustedRootCertificates
• ICD Management: ICDCounter

Quieter Reporting (Q) - quieterReporting="true" - 46 attributes across 14 clusters:

• Identify: IdentifyTime
• On/Off: OnTime, OffWaitTime
• Level Control: CurrentLevel, RemainingTime, CurrentFrequency
• Power Source: BatPercentRemaining, BatTimeRemaining, BatTimeToFullCharge
• Operational State: CountdownTime
• Valve Configuration and Control: RemainingDuration
• Electrical Power Measurement: Voltage, ActiveCurrent, all power/current/harmonic attributes
• Electrical Energy Measurement: all 4 cumulative/periodic energy attributes
• Device Energy Management: PowerAdjustmentCapability, Forecast
• Energy EVSE: SessionDuration, SessionEnergyCharged, SessionEnergyDischarged
• Closure Control/Dimension: CountdownTime, CurrentState
• Service Area: EstimatedEndTime
• Color Control: CurrentHue, CurrentSaturation, RemainingTime, CurrentX, CurrentY, ColorTemperatureMireds, EnhancedCurrentHue

[j-ororke]

Updated TC_SC_3_6 python3 test module and TestReadSubscribeAceExistenceErrors.py to call super().setup_class() at the start of overridden setup_class method. The original implementation silently skipped the base class initialisation in MatterBaseTest, leaving subscription_controller that we are implementing uninitialised, which caused an error at runtime. Both tests now work correctly with the background wildcard subscription being introduced in PR #43274.

[j-ororke]

Additionally added a new script-arg --no-wildcard-subscription to PR #43274 in order to gate the background subscription and new controller initialization before tests start to exclude tests similar to ACL_2_4 and ACL_2_6 that experience issues when new ACL entries are added before tests start.