Skip to content

refactor(docker): migrate single-node compose from host to bridge networking#2952

Open
bitflicker64 wants to merge 12 commits intoapache:masterfrom
bitflicker64:docker-fix-bridge-network
Open

refactor(docker): migrate single-node compose from host to bridge networking#2952
bitflicker64 wants to merge 12 commits intoapache:masterfrom
bitflicker64:docker-fix-bridge-network

Conversation

@bitflicker64
Copy link
Contributor

@bitflicker64 bitflicker64 commented Feb 15, 2026

Purpose of the PR

Fix Docker deployment failing on macOS due to Linux-only host networking. Originally scoped to single-node, expanded during review to cover the full 3-node distributed cluster as well.

close #2951

Main Changes

Docker Compose:

  • Remove network_mode: host from both single-node and 3-node compose files
  • Switch to Docker bridge network (hg-net) with container hostnames
  • Replace config file volume mounts with environment variable injection (HG_* prefix)
  • Add proper depends_on: condition: service_healthy, restart policies, and healthchecks

Entrypoint Scripts (complete rewrite):

  • hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.shSPRING_APPLICATION_JSON injection via HG_PD_* env vars, deprecated alias migration
  • hugegraph-store/hg-store-dist/docker/docker-entrypoint.shHG_STORE_* env vars
  • hugegraph-server/hugegraph-dist/docker/docker-entrypoint.shHG_SERVER_* env vars, auth support

Helper Scripts:

  • wait-storage.sh — configurable PD auth
  • wait-partition.sh — 120s timeout, configurable

Documentation:

  • Created docker/README.md — full setup guide, env var reference, port table, troubleshooting
  • Fixed 7 files pointing to dead example/ directory
  • Fixed hugegraph-store/docs/deployment-guide.md wrong env var names
  • Updated K8s manifest env var names
  • Added bridge network notes to PD configuration and README docs

Problem

The original Docker configuration uses network_mode: host which only works on native Linux. Docker Desktop on macOS does not implement host networking the same way. Containers start but HugeGraph services advertise incorrect addresses (127.0.0.1, 0.0.0.0).

Resulting failures:

  • Server stuck in loop waiting for storage backend
  • PD client UNAVAILABLE io exception errors
  • Store reports zero partitions
  • Cluster never becomes usable even though containers are running

Root Cause

  • network_mode: host is Linux-specific
  • Docker Desktop falls back to bridge networking silently
  • HugeGraph components advertise localhost-style addresses
  • Other containers cannot route to those addresses

Solution

Switch to bridge networking and advertise container-resolvable hostnames. Docker DNS resolves service names automatically. Configuration injected via SPRING_APPLICATION_JSON through HG_* env vars instead of mounted config files.

Verification

Tested on macOS (Apple M4, Docker Desktop):

Single-node cluster:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
hg-server   Up healthy   0.0.0.0:8080->8080
hg-store    Up healthy   0.0.0.0:8520->8520
hg-pd       Up healthy   0.0.0.0:8620->8620

3-node cluster:

curl -u store:admin -s http://localhost:8620/v1/stores | grep -o '"partitionCount":[0-9]*'
"partitionCount":12
"partitionCount":12
"partitionCount":12

All 3 stores show partitionCount:12. All 9 containers healthy.

Also verified on Ubuntu 24.04 (native Docker):

  • All 9 containers healthy
  • partitionCount:12 on all 3 stores
  • Cluster works regardless of which PD node wins leader election

Does this PR potentially affect the following parts?

  • Modify configurations
  • Dependencies
  • The public API
  • Other affects
  • Nope

Documentation Status

  • Doc - Done

Related fixes discovered during this work:

Bug Issue Fix
getLeaderGrpcAddress() NPE in bridge mode #2959 #2961
IpAuthHandler hostname vs IP mismatch #2960 #2962

Changes Checklist

Replace network_mode: host with explicit port mappings and add configuration
volumes for PD, Store, and Server services to support macOS/Windows Docker.

- Remove host network mode from all services
- Add explicit port mappings (8620, 8520, 8080)
- Add configuration directories with volume mounts
- Update healthcheck endpoints
- Add PD peers environment variable

Enables HugeGraph cluster to run on all Docker platforms.
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. pd PD module store Store module labels Feb 15, 2026
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Feb 15, 2026
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Feb 20, 2026
@bitflicker64
Copy link
Contributor Author

Bridge networking changes have been validated successfully across environments:

  • macOS (Docker Desktop)
  • Ubuntu 24.04.4 LTS

Observed behavior:

  • PD container starts and becomes healthy
  • Store container starts, registers, and receives partitions
  • Partitions are assigned and Raft leaders are elected
  • Server container initializes without errors
  • REST endpoints respond as expected

No regressions were observed in the single-node deployment. Service discovery and inter-container communication function correctly under bridge networking.


ARM64 Compatibility Fix — wait-storage.sh

Problem

The original wait-storage.sh relied on gremlin-console.sh for storage readiness detection:

On ARM64 (Apple Silicon), this fails due to a Jansi native library crash


Root Cause

  • gremlin-console.sh depends on Jansi, which is unstable on ARM64
  • The detection logic is triggered only when hugegraph.* environment variables are used
  • Volume-mounted configurations bypass this code path, masking the failure

Fix

Replaced Gremlin Console detection with a lightweight PD REST health check:

Cleanup

detect-storage.groovy is no longer required by the updated startup flow and can be removed

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the single-node Docker Compose configuration from Linux-specific host networking to cross-platform bridge networking. The change addresses a critical issue where Docker Desktop on macOS and Windows doesn't support host networking properly, causing services to advertise unreachable addresses and preventing cluster initialization.

Changes:

  • Replaced host networking with bridge networking and explicit port mappings
  • Added comprehensive environment-based configuration for PD, Store, and Server through new entrypoint scripts
  • Implemented health-aware startup with PD REST endpoint polling in wait-storage.sh
  • Added volume mounts for persistent data and deprecated variable migration guards

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
docker/docker-compose.yml Migrated from host to bridge networking, added environment variables, updated healthchecks, exposed required ports
hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh New comprehensive entrypoint with SPRING_APPLICATION_JSON configuration, deprecation guards, and required variable validation
hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh New comprehensive entrypoint with SPRING_APPLICATION_JSON configuration, deprecation guards, and required variable validation
hugegraph-server/hugegraph-dist/docker/docker-entrypoint.sh Refactored to use environment variables for backend and PD configuration with deprecation guards
hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh Replaced Gremlin-based storage detection with PD REST health endpoint polling, increased timeout to 300s

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@bitflicker64
Copy link
Contributor Author

Thank you for the review. I’ll take care of the suggested adjustments and will proceed with testing the 3 node cluster configuration next.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 33 out of 37 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@imbajin
Copy link
Member

imbajin commented Mar 9, 2026

Overall this PR looks good to me now.

The overall direction makes sense, and I really appreciate all the work you’ve put into this, especially the follow-up updates during review. Thanks a lot for the contribution and for continuing to improve it.

Once the last small issue is fixed and confirmed, I think it would be reasonable to merge it first and validate it in real-world use.

@imbajin imbajin changed the title fix(docker): migrate single-node compose from host to bridge networking refactor(docker): migrate single-node compose from host to bridge networking Mar 9, 2026
@bitflicker64
Copy link
Contributor Author

Overall this PR looks good to me now.

The overall direction makes sense, and I really appreciate all the work you’ve put into this, especially the follow-up updates during review. Thanks a lot for the contribution and for continuing to improve it.

Once the last small issue is fixed and confirmed, I think it would be reasonable to merge it first and validate it in real-world use.

@imbajin I'll get the remaining fix done shortly.

Also wanted to flag something I noticed while testing docker logs doesn't work properly for any of the HugeGraph containers right now. The startup scripts redirect JVM output to files inside the container (>> ${OUTPUT} 2>&1), so nothing reaches Docker's log capture. On top of that, the console appender is defined in the Log4j2 dist configs but never wired to the root logger. This means during debugging I had to manually exec into the container and tail the log files directly.

I also noticed the non-dist log4j2.xml configs already have the console appender wired correctly — the dist configs just never got updated to match. Should I open a separate PR for this?

@imbajin
Copy link
Member

imbajin commented Mar 10, 2026

Also wanted to flag something I noticed while testing docker logs doesn't work properly for any of the HugeGraph containers right now. The startup scripts redirect JVM output to files inside the container (>> ${OUTPUT} 2>&1), so nothing reaches Docker's log capture. On top of that, the console appender is defined in the Log4j2 dist configs but never wired to the root logger. This means during debugging I had to manually exec into the container and tail the log files directly.

I also noticed the non-dist log4j2.xml configs already have the console appender wired correctly — the dist configs just never got updated to match. Should I open a separate PR for this?

Thanks for catching that while testing — that does sound like a real issue.

I’d lean toward a separate PR for it, since it feels a bit beyond the main scope of this one. Keeping it separate would make this PR easier to land first, and the logging change can be reviewed on its own.

So I think it makes sense to finish the remaining fix here, and open a follow-up PR for the docker logs / dist logging config issue.

@bitflicker64
Copy link
Contributor Author

Thanks for catching that while testing — that does sound like a real issue.

I’d lean toward a separate PR for it, since it feels a bit beyond the main scope of this one. Keeping it separate would make this PR easier to land first, and the logging change can be reviewed on its own.

So I think it makes sense to finish the remaining fix here, and open a follow-up PR for the docker logs / dist logging config issue.

Makes sense, will do!

bitflicker64 added a commit to bitflicker64/incubator-hugegraph that referenced this pull request Mar 12, 2026
- Cache leader PeerId after waitingForLeader() and null-check to avoid
  NPE when leader election times out
- Remove incorrect fallback that derived leader gRPC address from local
  node's port, causing silent misroutes in multi-node clusters
- Wire config.getRpcTimeout() into RaftRpcClient's RpcOptions so Bolt
  transport timeout is consistent with future.get() caller timeout
- Replace hardcoded 10000ms in waitingForLeader() with config.getRpcTimeout()
- Remove unused RaftOptions variable and dead imports (ReplicatorGroup, ThreadId)

Fixes apache#2959
Related to apache#2952, apache#2962
@bitflicker64
Copy link
Contributor Author

Fixed in latest commit — wait-storage.sh now tries all PD peers in order instead of only the first one. Also fixed wait-partition.sh endpoint which was always returning partitionCount:0. Both verified on Ubuntu 24.04 with a fresh 3-node cluster.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 33 out of 37 changed files in this pull request and generated 9 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +85 to 86
./bin/start-hugegraph-pd.sh -j "${JAVA_OPTS:-}"
tail -f /dev/null
Comment on lines +80 to 81
./bin/start-hugegraph-store.sh -j "${JAVA_OPTS:-}"
tail -f /dev/null
Comment on lines +87 to 91
./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120

# Post-startup cluster stabilization check
./bin/wait-partition.sh || log "WARN: partitions not assigned yet"

@VGalaxies
Copy link
Contributor

VGalaxies commented Mar 18, 2026

I re-ran this locally with the Docker dev compose path and the single-node flow works fine for me now: PD, Store, and Server all came up healthy, and the basic endpoints responded as expected.

image

One small follow-up issue I noticed: the new PD/Store entrypoints seem to break the current standalone docker run examples in the module READMEs, because they now require the newer HG_PD_* / HG_STORE_* environment variables. This doesn’t seem blocking for the compose-based change in this PR, but it would be good to align the docs or add a small compatibility fallback in a follow-up.

@bitflicker64
Copy link
Contributor Author

I re-ran this locally with the Docker dev compose path and the single-node flow works fine for me now: PD, Store, and Server all came up healthy, and the basic endpoints responded as expected.
image

One small follow-up issue I noticed: the new PD/Store entrypoints seem to break the current standalone docker run examples in the module READMEs, because they now require the newer HG_PD_* / HG_STORE_* environment variables. This doesn’t seem blocking for the compose-based change in this PR, but it would be good to align the docs or add a small compatibility fallback in a follow-up.

The docker run examples in the module READMEs have already been updated to use the new HG_PD_* / HG_STORE_* variable names in the docs follow-up PR #2963. The entrypoints also include soft migration fallbacks so old variable names are automatically mapped to the new ones with a deprecation warning, meaning existing setups won't break immediately. Hope that covers the concern!

@bitflicker64
Copy link
Contributor Author

Note: testing the 3-node cluster locally currently requires building a local PD image (docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:local .) due to the temporary entrypoint volume mount workaround. This is because the current published images predate the getLeaderGrpcAddress() and IpAuthHandler fixes (#2961, #2962) — without them the cluster only works reliably when pd0 wins leader election. Once updated images are published this requirement will be cleaned up in a follow-up PR.

VGalaxies
VGalaxies previously approved these changes Mar 18, 2026
Copy link
Contributor

@VGalaxies VGalaxies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@imbajin Since #2962 is already merged and #2961 still seems relevant for the 3-node path, my suggestion would be to land #2961 first, then this PR, and follow with #2963.

@github-project-automation github-project-automation bot moved this from In progress to In review in HugeGraph PD-Store Tasks Mar 18, 2026
@bitflicker64
Copy link
Contributor Author

Two small cleanups: removed UTF-8 BOM from the 3-node compose (Copilot flag), and removed the image: tag from the dev compose so docker compose up actually builds locally instead of pulling from Docker Hub

@bitflicker64 bitflicker64 requested a review from VGalaxies March 18, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pd PD module size:L This PR changes 100-499 lines, ignoring generated files. store Store module

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

[Bug] Docker setup broken on macOS/Windows — network_mode: host replaced with bridge networking (single-node + 3-node cluster)

4 participants