bug: Regression in 375 with modules not found

[marcelmamula]

@richm @tomjelinek Please take this with priority, because #375 made this role unusable.

Problem

All custom modules in ha_cluster are in /library and they are called with plain name. This is not working, because Ansible expects them to be collection wide modules or they need to have path properly setup.

This was not a problem before, because all these modules were conditioned. This one is not, so it showed this role wide problem in
#375.

Environment

Tested with same results on:

• Python 3.11 and ansible-core 2.16
• Python 3.13 and ansible-core 2.20

NOTE: This is not SLES specific problem, because role fails before it even gets to shell files.

Testing

1. Clone latest repository

2. Execute role:

     - name: Execute Ansible Role ha_cluster
      ansible.builtin.include_role:
        name: fedora.linux_system_roles.ha_cluster

NOTE: Tested also with using roles: with same result.

3. Observe error because module does not exist.

[ERROR]: couldn't resolve module/action 'sr_fingerprint'. This often indicates a misspelling, missing collection, or incorrect module path.
Origin: /ansible_collections/fedora/linux_system_roles/roles/ha_cluster/tasks/set_vars.yml:8:3

6     difference(ansible_facts.keys() | list) | length > 0
7
8 - name: Record role begin fingerprint
    ^ column 3

Additional

We use role specific filters in SLL roles in "{{ role_path }}/plugins/filters" and those can be called with __sap_swpm_hostnames_var | community.sap_install.validate_sap_abap_platform_hostname .

I tried moving sr_fingerprint.py to "{{ role_path }}/plugins/modules" and call it with fedora.linux_system_roles.ha_cluster.sr_fingerprint: but it did not work.

[spetrosi]

Seems like you build the fedora.linux_system_roles manually from the roles repos that we have upstream. This won't work with a simply copy-paste. To build the collection, we use the https://github.com/linux-system-roles/auto-maintenance/blob/main/lsr_role2collection.py converter script that converts roles to a collection format, part of which is moving modules from roles' plugins dir to the collection's root plugins dir.
Try copying sr_fingerprint.py to ~/.ansible/collections/ansible_collections/fedora/linux_system_roles/plugins/modules, Ansible should be able to find it there.

Or wait until we release a new version of the fedora.linux_system_roles collection.

[marcelmamula]

@richm @tomjelinek After few rounds of testing, I can confirm that it works only in 1 case:

• Git clone + name: ~/ansible_collections/fedora/linux_system_roles/roles/ha_cluster

• Galaxy install (when new release is published) + name: fedora.linux_system_roles.ha_cluster should work, but it cannot be tested now because module is not on collection level.

This means that we cannot properly test code that is not published.

When I am doing PRs to ha_cluster, they usually go with PR into https://github.com/sap-linuxlab/community.sap_install/tree/main/roles/sap_ha_pacemaker_cluster which uses:

  ansible.builtin.include_role:
    name: "{{ sap_ha_pacemaker_cluster_system_roles_collection }}.ha_cluster"

Using these internal modules before they are published would mean I have to hardcode local git cloned path for testing, which is not optimal at all.

[spetrosi]

Or use this ha_cluster role natively, but use it in the original legacy role format - copy it to ~/.ansible/roles/linux-system-roles.ha_cluster/ and call with linux-system-roles.ha_cluster. The legacy format is indeed legacy and might be dropped by Ansible.

[marcelmamula]

@spetrosi My setup that worked for past 2 years without any issue:

ANSIBLE_COLLECTIONS_PATH=~/ansible_collections

ls -la ~/ansible_collections/fedora/linux_system_roles/roles/ | grep ha_cluster
lrwxrwxrwx  1 mmamula mmamula   38 Apr  8 13:00 ha_cluster -> /home/mmamula/workspace/ha_cluster/wt1

This way I can repoint underlying cloned code (work trees) for rapid testing. But current problem exposed issue with LSR build process, that moves those modules outside of symlink.

And I only needed to refresh this from time to time since I do not use any other LSR roles.

[spetrosi]

I see, you want to test the latest changes this ha_cluster repo, earlier than they are released officially.
The easiest way would be to test it as a standalone legacy role as I described in #377 (comment).
I cannot really think of how we can fix it from this repo.
@richm is on PTO until next week, and he plans to do this change with sr_fingerprint module to all our roles, after which we can do a new release of the collection. I can do a release now but this won't really solve the original issue that you report.

[marcelmamula]

@spetrosi I dont need you to rush releasing new version.

I just wanted to figure out some safe way for testing new code, which entails work on our SLL repository and LSR ha_cluster at same time.
From our previous comments it seems I have only two choices, while keeping my current setup:

• Copy new module from ha_cluster to root of collection.
• Temporarily change our include_role to point towards hard filepath.

[spetrosi]

Temporarily change our include_role to point towards hard filepath.

This is a true working solution to call role from this repo, so maybe make this the default.
But ha_cluster uses other roles from fedora.linux_system_roles collection, so test will still need to install this collection and update it, alongside clone this ha_cluster repo to legacy roles path.

[richm]

I don't understand how you can test unreleased code now. Let's say that there was a PR that changed library/ha_cluster_info.py, and you wanted to test the new functionality in that module. How would you do that?

[marcelmamula]

@richm I am updating whole collection infrequently, while working always on latest cloned ha_cluster. I did not get to situation when I had to touch these modules so I did not face it.

This is reason why I had to raise this issue in order to find out best way to handle this.

[tomjelinek]

I've run into similar issues. After #375 got merged, CI scripts in tmt/ started failing. However, it turned out that there is nothing wrong with the role. The CI scripts needed fixing: #379

[marcelmamula]

@tomjelinek @richm I think we can close this issue, but it would make sense to add note for contributors, outlining that changes to library modules needs workaround for testing during development.

[richm]

@tomjelinek @richm I think we can close this issue, but it would make sense to add note for contributors, outlining that changes to library modules needs workaround for testing during development.

Ok - what do you suggest?

[marcelmamula]

@richm Is there README.md or some document for contributors? This would make sense to add as TIP or NOTE when dealing with plugins.

[richm]

@richm Is there README.md or some document for contributors? This would make sense to add as TIP or NOTE when dealing with plugins.

https://github.com/linux-system-roles/ha_cluster/blob/main/contributing.md

What do you suggest we add? I'm not sure what the problem is, and how it should be addressed in the doc.

All of our developer tooling uses tox-lsr which "just works" and handles this situation.

All of our github action CI either handles this situation, or otherwise uses tox-lsr.

[marcelmamula]

I would update it like this:

Running tests without tox

This Ansible Role can be tested locally without using tox, but there are limitations that have to be worked around.

1. Install collection fedora.linux_system_roles.
2. Delete ha_cluster role from installed collection and create symlink towards your folder with git clone.
3. Trigger role with using fedora.linux_system_roles.ha_cluster.

NOTE: Using this configuration can lead to issues when plugins were recently updated and they are not in collection root!

• Publishing of fedora.linux_system_roles will collate all modules from library/ inside of roles and merge them together.
• When you are testing locally, this is not covered and can lead to issues where role expects new module to be present.