Skip to content

Prevent processing of datasets belonging to unapproved reporting organisations#138

Merged
simon-20 merged 5 commits into
developfrom
sk-fix-showing-unapproved-orgs--add-test-flag
Apr 22, 2026
Merged

Prevent processing of datasets belonging to unapproved reporting organisations#138
simon-20 merged 5 commits into
developfrom
sk-fix-showing-unapproved-orgs--add-test-flag

Conversation

@simon-20
Copy link
Copy Markdown
Contributor

This PR:

  • Fixes the bug whereby datasets marked as Public were being processed even if they belonged to an unapproved reporting organisation
  • Adds a cli flag --run-for-single-reporting-org to aid in targeted debugging
  • Fixes a few flake8 warnings and runs some linting on tests

This will resolve #136

@simon-20 simon-20 requested a review from a team April 22, 2026 11:21
Copy link
Copy Markdown
Contributor

@chrisarridge chrisarridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of small non-breaking comments. Otherwise it looks good.

Comment thread src/config/bds_context.py
def RUN_FOR_N_DATASETS(self) -> int | None:
return self._RUN_FOR_N_DATASETS

@property
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not make it a bool? Similar remark on line 23?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes should be a string - it contains the short name of the reporting org.

suitecrm_reporting_org_records = [
o
for o in suitecrm_reporting_org_records
if o.get("attributes", {}).get("iati_short_name", "") == context.RUN_FOR_SINGLE_REPORTING_ORG
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type hint is for this to be an integer - should it be a string? It's a string in the command line arguments.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, thanks!

Comment thread CHANGELOG.md
## [1.4.4] - 2026-04-22

### Fixed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a change for the command line argument addition?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

This commit adds a feature to limit processing to a single
reporting org, which helps with more targeted debugging
when there are issues with real data that need investigating.
Housekeeping commit which fixes most of the outstanding
flake8 warnings.
This commit stops unapproved orgs from being downloaded, which in turn
stops datasets from those orgs from being included, because any dataset
which belongs to an unknown org is rejected. This resolves #136.
@simon-20 simon-20 force-pushed the sk-fix-showing-unapproved-orgs--add-test-flag branch from 7140447 to 5b0ed1b Compare April 22, 2026 11:55
@simon-20 simon-20 merged commit 227cdc1 into develop Apr 22, 2026
1 check passed
@simon-20 simon-20 deleted the sk-fix-showing-unapproved-orgs--add-test-flag branch April 27, 2026 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make sure datasets are only processed from organisations that are "approved to publish"

2 participants