Register global J2K and HTJ2K codec ids

[alemirone]

Summary

This PR registers two new global codec ids in the c-blosc2 codec registry:

• BLOSC_CODEC_J2K = 39
• BLOSC_CODEC_HTJ2K = 40

and associates them with the codec names:

• "j2k"
• "htj2k"

Motivation

The main goal is to support a new family of external multi-backend JPEG 2000 plugins for Blosc2.

These plugins are designed so that the same logical codec id can be used with different runtime backends, including both:

• open-source backends
• commercial backends

This gives an important interoperability advantage:

• users do not need to duplicate compressed files depending on which backend is locally available
• files can be exchanged between sites/users that may or may not have access to commercial codecs
• when commercial backends are not available, open-source backends can still be used
• when commercial backends are available, users can select them at runtime without changing the stored codec identity

In other words, the registry-level codec id identifies the compressed format family, while backend selection remains a runtime choice.

Where the new plugins live

This registry change is meant to support external plugin repositories that are already well advanced:

• blosc2_j2k
  • https://github.com/Blosc/blosc2_j2k
• blosc2_htj2k
  • https://github.com/Blosc/blosc2_htj2k

These plugins implement runtime backend selection for JPEG 2000 / HTJ2K codec paths and already support multiple backends.

This PR is therefore the registry-side step needed to make their codec ids official and stable.

What this PR changes

• adds BLOSC_CODEC_J2K = 39
• adds BLOSC_CODEC_HTJ2K = 40
• registers the corresponding codec names:
  • "j2k"
  • "htj2k"

Why this matters at the registry level

The plugin-side code needs stable global ids so that:

• blosc2_compname_to_compcode("j2k") == 39
• blosc2_compname_to_compcode("htj2k") == 40

and so that the same ids are consistently visible across:

• Blosc2 runtime registration
• plugin loading
• Python bindings
• HDF5 filter integration
• user-facing tools and workflows

Validation status

The plugin side is already in a fairly advanced state.

There are now:

• registry-aware tests
• real roundtrip compression/decompression tests
• small HDF5 end-to-end tests using numeric Blosc2 filter options

for both:

• blosc2_j2k
• blosc2_htj2k

Current transition status

The only remaining transitional detail is that these plugin tests currently install a slightly modified:

• c-blosc2
• python-blosc2

runtime in order to expose the new official ids during CI and validation.

That is exactly why this PR exists: to upstream the registry part and remove the need for that special setup over time.

Temporary compatibility workarounds

During this transition, a few temporary workarounds were introduced in the plugin repositories to keep standard builds and CI working before the new ids are fully available everywhere.

Examples include:

• fallback definitions for BLOSC_CODEC_J2K / BLOSC_CODEC_HTJ2K when building against older headers
• skipping registry-specific tests in standard CI when the runtime registry is not patched
• separate registry-aware CI workflows using patched python-blosc2
• some temporary portability/build fixes in plugin CI
• temporary Windows-side limitations for bundled OpenHTJ2K in blosc2_htj2k

These are transitional measures only.
Once the registry ids are upstream in c-blosc2 and propagated normally through python-blosc2, these workarounds can be simplified or removed.

Context

This PR is the upstream registry step requested while reviewing the plugin work around:

• Blosc/blosc2_grok#21
  • Add Runtime J2K/HTJ2K Backend Plugins for blosc2_grok blosc2_grok#21

Once this registry discussion is settled, the plugin-side workarounds can be progressively reduced.

Related plugin work

This registry change is meant to support the following external multi-backend plugin work:

• blosc2_j2k
  • currently at: https://github.com/alemirone/blosc2_j2k
• blosc2_htj2k
  • currently at: https://github.com/alemirone/blosc2_htj2k

[FrancescAlted]

I have just single comment.

[FrancescAlted]

This does not need to be updated. It is just for documenting how many regular codecs are registered. Plugins are not part in this count.

[FrancescAlted]

I think I'll go ahead merging this now, and will restore the BLOSC2_GLOBAL_REGISTERED_CODECS = 5 original value.

[FrancescAlted]

Restored original BLOSC2_GLOBAL_REGISTERED_CODECS value in 3350d95.

[FrancescAlted]

Thanks @alemirone !

[FrancescAlted]

BTW, C-Blosc2 3.1.0 is out now. It comes with the new global IDs for J2K and HTJ2K. Plus, it comes with new b2nd_deserialize_meta_inline() that should remove the need for plugins to link with libblosc2. More info: Blosc/blosc2_grok#22.

[amirone2]

Hi Francesc,

Thank you for merging the PR!

I have adapted the installation procedure in the README and the tests in both blosc2_htj2k and blosc2_j2k so that they now rely on upstream c-blosc2.

Everything is working now. By the way, for some reason unknown to me, a GitHub automated process has suspended my account. If needed, I can create the next MR from alemirone2 instead of alemirone.

Let me know what the next step should be.
Best

[FrancescAlted]

Great news. That means that you are using new b2nd_deserialize_meta_inline instead of b2nd_deserialize_meta? If so, can you confirm that the new plugins do not need to link against c-blosc2 for working correctly?

After you fix the remaining details, the best is that you transfer your repos under the GitHub Blosc umbrella.

[alemirone]

Hi Francesc,

I have updated both blosc2_j2k and blosc2_htj2k

The plugins now use the header-only b2nd_deserialize_meta_inline() helper instead of linking against the non-inline B2ND symbol. The main plugin libraries no longer link directly against libblosc2; on Linux I also verified that they do not have a needed dependency on libblosc2.

The registry-aware tests are now running successfully, including compression/decompression roundtrip checks. I also enabled them on Linux, macOS and Windows. For Windows, the workflow uses clang-cl/Ninja, as required by the current python-blosc2 build.

I have kept the HDF5-specific checks out of the registry workflow for now, because the currently available hdf5plugin wheel does not yet expose the updated c-blosc2 registry. The plugin-level roundtrip tests are passing.

Everything is working now in the tests.

Best,
Alessandro

[FrancescAlted]

That's great! I think @t20100 will be happy to hear about this.

Now, we can start moving your projects to the Blosc/ umbrella if you want. If so, tell me your desired username in github and I'll add it as a member of the Blosc organization and will grant you with at least the "Create repositories" permission. Then you can go to your repos → Settings → Danger Zone → Transfer ownership, selects Blosc as the target, and confirm.

Thank you for your contribution!

[alemirone]

Hi Francesc,

Great, thank you!

Please add my GitHub account alemirone to the Blosc organization. Once I have the permissions, I will transfer the repositories to the Blosc umbrella following the procedure you described.

Best,
Alessandro

[alemirone]

Hi Francesc,

Thank you for adding me to the Blosc organization.

It seems I still do not have sufficient permissions to transfer the repositories into the Blosc organization. Either option is fine for me, whichever you prefer:

1. You can temporarily give me permission to create repositories in the Blosc organization, so I can transfer the existing repositories.

2. Or, since these repositories do not have issues, pull requests, or relevant activity to preserve yet, you can create the two empty repositories directly under Blosc:

   • Blosc/blosc2_j2k
   • Blosc/blosc2_htj2k

Then I can push the current code there and update the remotes locally.

Thanks,
Alessandro

[FrancescAlted]

Good idea. So I went ahead and created Blosc/blosc2_j2k and Blosc/blosc2_htj2k and made alemirone admin for these projects. Tell me how it goes. Luck!

[alemirone]

Hi Francesc,

Thank you again for creating the repositories under the Blosc organization and for giving me admin access.

I have pushed both projects there:

• https://github.com/Blosc/blosc2_j2k
• https://github.com/Blosc/blosc2_htj2k

I also did some cleanup work on the README demo quickstarts so that they now clone the plugin repositories from the Blosc organization. These are just the copy-paste demo instructions in the README. The temporary python-blosc2 branch is still referenced there only because the python-blosc2 update exposing the new codec IDs is not merged yet.

For blosc2_htj2k, I also checked the OpenHTJ2K dependency. The previous build was still referring to OpenHTJ2K PR #190, but that PR has now been merged upstream and released as v0.4.0. I updated the build and documentation so that the bundled OpenHTJ2K backend now points to the official v0.4.0 release instead of the pull-request ref.

On my side, the local checks are OK: CMake build, wheel build/install, pytest with the registry checks enabled, the Python README quickstart variants, and the C++ HDF5 quickstart all pass. The only remaining consolidation point seems to be the small python-blosc2 update exposing the official codec IDs:

• blosc2.Codec.J2K = 39
• blosc2.Codec.HTJ2K = 40

Once that is in place, I think we should be able to simplify the demo quickstarts and CI workflows further, depending on how you prefer to proceed.

Thanks,
Alessandro

[FrancescAlted]

Cool! I exposed the new global IDs in commit 5a442377. There I assumed that you are going to distribute wheels in PyPI, so this is why I am mentioning blosc2-j2k and blosc2-htj2k packages; if that is not the intent, tell me and I'll remove the comments.

Good job!

[alemirone]

Hi Francesc,

Thank you! Yes, I can go ahead with the preparation of the wheels as soon as possible.

The Python enum entries are already available in python-blosc2 main:

blosc2.Codec.J2K = 39
blosc2.Codec.HTJ2K = 40

The remaining point is that the python-blosc2 wheels on PyPI should also bundle
a c-blosc2 version that includes the new runtime registry entries for J2K and
HTJ2K.

I checked the current PyPI wheel, blosc2 4.4.2, and it still seems to bundle an
older c-blosc2 runtime without those registry entries. So, for the moment, the
blosc2_j2k and blosc2_htj2k CI workflows still need the temporary git-based
python-blosc2 dependency.

Once a new python-blosc2 release including the updated c-blosc2 is available on
PyPI, I can remove that temporary dependency and publish the plugin wheels as
blosc2-j2k and blosc2-htj2k.

The package names you added in the comments are exactly the intended ones.

[FrancescAlted]

Python-Blosc2 4.4.3, with the new IDs in, has been released.

[alemirone]

Hi Francesc,

The two packages, blosc2-j2k and blosc2-htj2k, are now available on PyPI.

The remaining polishing step is the release of a new hdf5plugin wheel by @t20100, linked against a recent c-blosc2 version that includes the new J2K/HTJ2K codec IDs. Once that is available, the HDF5 quickstart can use the standard hdf5plugin wheel directly, without the current source-build workaround.

[t20100]

I'll prepare a release of hdf5plugin with updated c-blosc2 embedded library.

[alemirone]

Hi Thomas,

I checked the current hdf5plugin wheel from PyPI, and it still seems to embed an older c-blosc2 runtime:

hdf5plugin 6.0.0
embedded c-blosc2: 2.21.2
j2k -> -1
htj2k -> -1

So the HDF5 Blosc2 filter cannot yet resolve the new official codec IDs:

j2k -> 39
htj2k -> 40

Is there anything preventing a rebuild/release of hdf5plugin wheels against a recent c-blosc2 version containing these registry entries?

For now, blosc2-j2k and blosc2-htj2k are both available on PyPI and work for direct Blosc2/Python usage with python-blosc2 >= 4.4.3. The remaining missing piece for transparent HDF5 usage is a rebuilt hdf5plugin wheel.

Thanks!

Alessandro

[t20100]

Hi,

I just merged silx-kit/hdf5plugin#387 to update c-blsoc2 in the main branch. You can try it by installing from the main branch.

I've unpinned blosc2_grok and added j2k and htj2k blosc2 plugin to tests and CI, but there is some issues, see silx-kit/hdf5plugin#388

[alemirone]

Hi Thomas,

I checked the published wheel too.

The problem was on our side, in the OpenHTJ2K wrapper inside blosc2_htj2k: the decoder wrapper was preallocating component buffers before calling openhtj2k_decoder::invoke(), while OpenHTJ2K allocates them itself. That ownership mismatch could cause the double-free/corruption on image stacks.

I fixed it, added a regression test for chunked multi-frame stacks, and released blosc2-htj2k 0.4.1 on PyPI.

I also verified the published PyPI wheel in a fresh environment. With blosc2-htj2k==0.4.1 and an hdf5plugin rebuilt against the updated c-blosc2, the HDF5 stack roundtrip now works correctly on my side.

Thanks again for catching it.