Skip to content

Provide shared workaround for ipykernel TCP transport warning in notebook runners #633

Description

@redeboer

When executing notebooks in ComPWA projects through commands such as poe nb and poe docnb-force, recent ipykernel emits warnings like:

[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.

This is not specific to one project. In ComPWA/jpsi-nstar, poe nb executes notebooks through pytest --nbmake, while poe docnb-force executes through myst-nb / jupyter-cache / nbclient. Both paths eventually construct an nbclient.NotebookClient and a jupyter_client.KernelManager with the default transport="tcp".

This feels like policy/tooling territory rather than something that should live in individual analysis packages such as jpsi-nstar.

What I checked

With the current jpsi-nstar environment:

  • ipykernel == 7.3.0
  • jupyter_client == 8.9.1
  • nbclient == 0.11.0
  • jupyter_client.KernelManager.transport still defaults to tcp
  • setting KernelManager.transport = "ipc" works and suppresses the warning
  • setting KernelManager.transport_encryption = "required" generates CurveZMQ keys, but in this stack the client path failed because AsyncKernelClient.curve_publickey expected bytes and received a string
  • nbmake currently constructs NotebookClient(...) directly without exposing a config hook
  • myst-nb delegates execution through jupyter-cache.executors.utils.single_nb_execution(...), which also does not pass such config through by default

Verified hack

The following monkey patch was verified to make both notebook execution paths use IPC transport, suppressing the warning for:

  • direct nbclient.execute(...)
  • pytest --nbmake / poe nb
  • the jupyter-cache.single_nb_execution(...) path used by myst-nb
from __future__ import annotations

from typing import Any


def enable_ipc_kernel_transport() -> None:
    """Use IPC for nbclient-managed kernels."""
    from nbclient import NotebookClient
    from traitlets.config import Config

    marker = "_compwa_uses_ipc_transport"
    if getattr(NotebookClient, marker, False):
        return

    original_init = NotebookClient.__init__

    def init_with_ipc_transport(self: Any, *args: Any, **kwargs: Any) -> None:
        config = Config(kwargs.get("config") or {})
        config.KernelManager.transport = "ipc"
        kwargs["config"] = config
        original_init(self, *args, **kwargs)

    NotebookClient.__init__ = init_with_ipc_transport
    setattr(NotebookClient, marker, True)

For nbmake, this can be activated from a top-level conftest.py:

from compwa_policy_jupyter import enable_ipc_kernel_transport

enable_ipc_kernel_transport()

For Sphinx / myst-nb, this can be activated from docs/conf.py before notebook execution starts:

from compwa_policy_jupyter import enable_ipc_kernel_transport

enable_ipc_kernel_transport()

Caveat

This is intentionally a hack: it monkey-patches NotebookClient.__init__. It should not be copied into each project package. If we decide to use it, it should live in a shared ComPWA tooling/policy helper, or preferably be replaced by an upstream fix.

Possible upstream fixes

  • nbmake could expose/pass a traitlets.Config or a NotebookClient configuration option.
  • myst-nb / jupyter-cache could expose/pass NotebookClient kwargs for kernel manager config.
  • jupyter_client / nbclient could fix the transport_encryption="required" path so CurveZMQ keys work cleanly with AsyncKernelClient.

Metadata

Metadata

Assignees

Labels

🐛 BugSomething isn't working🖱️ DXImprovements to the Developer Experience

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions