v0.2.10-rc2

This is a non-urgent, pre-release update for hoodi operators containing improved monitoring (swapping promtail for alloy), alerting (add multiple Discord IDs to get tagged for issues), tracing (allowing us to better debug proposals), and updated client versions (all clients updated to their latest releases).

Please note the mandatory adjustment of .env vars required since v0.2.3 if updating to this release.

Warning

Breaking Change

This release adjusts how metrics are sent to Obol. After this update, you will no longer have to keep stashing and unstashing changes on prometheus/prometheus.yml every update. It will now be generated at runtime by ./prometheus/run.sh, and the required variables will be injected into the file from your .env. The steps you need to take to handle this change are described below.

1. If you currently have a locally modified prometheus/prometheus.yml, make a backup copy of it outside of the repository before upgrading (just in case). Ensure you record the value of PROM_REMOTE_WRITE_TOKEN from the authorization.credentials section of that file.

2. This token must now be provided via your .env file instead of being defined in prometheus.yml. The variable PROM_REMOTE_WRITE_TOKEN is present (commented out) in all .env.sample.* files. To configure it, copy the variable to your own .env, uncomment it and set your token, for example:

PROM_REMOTE_WRITE_TOKEN=obolH7d...

3. We are now adding a dedicated way to add discord IDs to your deployment, allowing you to be @'d directly by our Obol Agent if there is an issue with your node. Add (uncomment) ALERT_DISCORD_IDS= in your .env, and specify one or more (with comma separation) IDs. To get the ID that corresponds to your discord account; Enable developer mode on discord with User Settings > Advanced. Then right click on a user's profile picture or name and select Copy ID to get a unique 18-digit number that represents their account. If you previously changed your CHARON_NICKNAME to your discord ID, you can set that back to a human friendly name for your node.

4. If you have any other custom modifications to your original prometheus.yml, compare it against prometheus.yml.example, which is now used as the base template for generating the final configuration. Any additional fields or customizations should be added to prometheus.yml.example.

5. With your environment variables set, and any other modifications ported to the prometheus.yml.example file; run docker compose up -d as normal. Confirm that your metrics are being received on the Obol Grafana. Once you have verified that the new Prometheus setup is working as expected, you can safely delete your backup of the old prometheus.yml.

Important

To maximise compatibility of environment variable interpolation across operating systems, changes have been made between this version and v0.2.3.

Please set (uncomment) the following .env vars if you haven't already, or your charon, mev-boost, and charon-dv-exit-sidecar may not start properly.

(Copying the .env.sample.mainnet file afresh and making any modifications you may have made, might be the least disruptive way to pick up these changes.)

CHARON_BEACON_NODE_ENDPOINTS=http://${CL}:5052
CHARON_EXECUTION_CLIENT_RPC_ENDPOINT=http://${EL}:8545
VE_BEACON_NODE_URL=http://${CL}:5052
VE_EXECUTION_NODE_URL=http://${EL}:8545
LIDO_DV_EXIT_BEACON_NODE_URL=http://${CL}:5052
CLUSTER_NAME="Your cluster name here"
CLUSTER_PEER="Your peer name here"

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.10-rc2
# You should no longer need to apply any stashed changes if you followed the instructions above
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Update version to v1.8.0 by @github-actions[bot] in #227
• Bump stack versions by @KaloyanTanev in #228
• Update version to v1.8.1 by @github-actions[bot] in #232
• Add ALERT_DISCORD_IDS label and env variable by @DiogoSantoss in #218
• OTLP setup by @pinebit in #220
• Migrate to Grafana Alloy as the recommended replacement for log forwa… by @apham0001 in #201
• feat(dashboard): add Hoodi network support and update queries by @qwe638853 in #230
• Update prysm and nimbus vc by @OisinKyne in #233

New Contributors

• @qwe638853 made their first contribution in #230

Full Changelog: v0.2.9...v0.2.10-rc2

v0.2.9

This is a non-urgent, recommended update for mainnet operators containing improved block proposal monitoring, updated client versions and reducing storage requirements by execution layer client.

Please note the mandatory adjustment of .env vars required since v0.2.3 if updating to this release.

Important

To maximise compatibility of environment variable interpolation across operating systems, changes have been made between this version and v0.2.3.

Please set (uncomment) the following .env vars if you haven't already, or your charon, mev-boost, and charon-dv-exit-sidecar may not start properly.

(Copying the .env.sample.mainnet file afresh and making any modifications you may have made, might be the least disruptive way to pick up these changes.)

CHARON_BEACON_NODE_ENDPOINTS=http://${CL}:5052
CHARON_EXECUTION_CLIENT_RPC_ENDPOINT=http://${EL}:8545
VE_BEACON_NODE_URL=http://${CL}:5052
VE_EXECUTION_NODE_URL=http://${EL}:8545
LIDO_DV_EXIT_BEACON_NODE_URL=http://${CL}:5052

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.9
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Reduce disk usage by @DiogoSantoss in #217
• Remove nethermind flags by @DiogoSantoss in #219
• Bump NM, Prysm, Charon by @KaloyanTanev in #222
• chore(deps): update peter-evans/create-pull-request action to v8 by @renovate[bot] in #215
• chore(deps): update actions/checkout action to v6 by @renovate[bot] in #204
• Downgrade Nethermind to 1.35.2 by @KaloyanTanev in #224

Full Changelog: v0.2.8...v0.2.9

v0.2.8

This is a non-urgent, recommended update for mainnet operators containing improved block proposal monitoring.

Please note the mandatory adjustment of .env vars required since v0.2.3 if updating to this release.

Important

To maximise compatibility of environment variable interpolation across operating systems, changes have been made between this version and v0.2.3.

Please set (uncomment) the following .env vars if you haven't already, or your charon, mev-boost, and charon-dv-exit-sidecar may not start properly.

(Copying the .env.sample.mainnet file afresh and making any modifications you may have made, might be the least disruptive way to pick up these changes.)

CHARON_BEACON_NODE_ENDPOINTS=http://${CL}:5052
CHARON_EXECUTION_CLIENT_RPC_ENDPOINT=http://${EL}:8545
VE_BEACON_NODE_URL=http://${CL}:5052
VE_EXECUTION_NODE_URL=http://${EL}:8545
LIDO_DV_EXIT_BEACON_NODE_URL=http://${CL}:5052

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.8
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Bump grandine to 2.0.1 by @KaloyanTanev in #213
• Update version to v1.7.2 by @github-actions[bot] in #214

New Contributors

• @github-actions[bot] made their first contribution in #214

Full Changelog: v0.2.7...v0.2.8

v0.2.7

This is a required update for mainnet operators running Nimbus Validator Client (since pre-fusaka ready versions, see below), and a recommended update for operators running Lodestar Validator Client. This release is optional for other operators.

Please note the mandatory adjustment of .env vars required since v0.2.3 if updating to this release.

Operators must update to a Fusaka-ready release before the Mainnet Fusaka hard fork on December 3rd @ 10PM UTC. Please take a final review of your versions ahead of the fork.

Warning

For Nimbus Validator Client users. Please note this merged fix, there is a chance your validator client is still a pre-fusaka version, despite you updating to a 'fusaka-ready' version of this repo. Running this release should force your nimbus VC to notice and pull the newer version. Do consult the VCs startup logs to ensure its running v25.11.0 or v25.11.1.

Important

To maximise compatibility of environment variable interpolation across operating systems, changes have been made between this version and v0.2.3.

Please set (uncomment) the following .env vars if you haven't already, or your charon, mev-boost, and charon-dv-exit-sidecar may not start properly.

(Copying the .env.sample.mainnet file afresh and making any modifications you may have made, might be the least disruptive way to pick up these changes.)

CHARON_BEACON_NODE_ENDPOINTS=http://${CL}:5052
CHARON_EXECUTION_CLIENT_RPC_ENDPOINT=http://${EL}:8545
VE_BEACON_NODE_URL=http://${CL}:5052
VE_EXECUTION_NODE_URL=http://${EL}:8545
LIDO_DV_EXIT_BEACON_NODE_URL=http://${CL}:5052

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.7
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Fusaka ready client versions

Please ensure you are running these versions or newer before the hard fork.

Category	Client	Version
Charon	Charon	1.7.x
Consensus Layer / Validator Clients	Prysm	7.0.0
	Lighthouse	v8.0.1
	Lodestar	1.36.0
	Teku	25.11.1
	Nimbus	25.11.0
	Grandine	2.0.0
Execution Layer	Geth	1.16.7
	Nethermind	1.35.2
	Erigon	3.2.2
	Reth	1.9.1
	Besu	25.11.0

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Fix implicit image naming and tagging for vc-nimbus by @TobiWo in #209
• Update lodestar to v1.37.0 by @OisinKyne in #210
• Bump nimbus local image tag by @OisinKyne in #212

Full Changelog: v0.2.6...v0.2.7

v0.2.6

This is a recommended update for mainnet operators running Nimbus. This release is optional for other operators. Please note the mandatory adjustment to .env vars required since v0.2.3 if updating to this release.

Operators must update to a Fusaka-ready release before the Mainnet Fusaka hard fork on December 3rd. Many clients have released updates since their first Fusaka-ready client, please take a final review of your versions ahead of the fork. (Lodestar are expected to release v1.37.0 on December 1st, which will be a recommended update for those running it as a validator client).

Important

To maximise compatibility of environment variable interpolation across operating systems, changes have been made between this version and v0.2.3.

Please set (uncomment) the following .env vars if you haven't already, or your charon, mev-boost, and charon-dv-exit-sidecar may not start properly.

CHARON_BEACON_NODE_ENDPOINTS=http://${CL}:5052
CHARON_EXECUTION_CLIENT_RPC_ENDPOINT=http://${EL}:8545
VE_BEACON_NODE_URL=http://${CL}:5052
VE_EXECUTION_NODE_URL=http://${EL}:8545
LIDO_DV_EXIT_BEACON_NODE_URL=http://${CL}:5052

Copying the .env.sample.mainnet file afresh and making any modifications you may have made, might be the least disruptive way to pick up these changes.

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.6
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Fix quoting for cluster name and peer replacements in config by @Lorenz29 in #208
• Bump to latest nimbus patch by @OisinKyne in #211

New Contributors

• @Lorenz29 made their first contribution in #208

Full Changelog: v0.2.5...v0.2.6

v0.2.5

This is a required update for mainnet operators running Lighthouse, Teku, or Reth. This release is optional for other operators.

Operators must update to a fusaka-ready release before the Mainnet Fusaka hard fork on December 3rd.

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.5
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• fixed bugs & bumped LH,TEKU versions by @alansunny3 in #206
• Bump lh, reth, teku vc by @OisinKyne in #207

New Contributors

• @alansunny3 made their first contribution in #206

Full Changelog: v0.2.4...v0.2.5

v0.2.4

This is a required update for mainnet operators running Prysm VC or experiencing issues with .env variable interpolation in docker compose causing startup issues. This release is optional for other operators.

Operators must update to a fusaka-ready release before the Mainnet Fusaka hard fork on December 3rd.

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.4
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Remove double interpolation in docker composes, updated .envs by @KaloyanTanev in #203

Full Changelog: v0.2.3...v0.2.4

v0.2.3

This is a required update for all mainnet operators. Operators must update to this release before the Mainnet Fusaka hard fork on December 3rd.

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.3
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Update Mainnet Oracle Allowlist by @nameisremus in #202

New Contributors

• @nameisremus made their first contribution in #202

Full Changelog: v0.2.2...v0.2.3

v0.2.2

This is a required update for all mainnet operators running the included MEV-Boost client. Operators must update to this release before the Mainnet Fusaka hard fork on December 3rd.

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.2
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Update mev-boost by @OisinKyne in #200

Full Changelog: v0.2.1...v0.2.2

v0.2.1

This is the second stable Fusaka-ready release suitable for all mainnet operators. Operators must update to this release, or at least v0.2.0 depending on their client choices, before the Mainnet Fusaka hard fork on December 3rd.

Important

Users should no longer run docker compose -f docker-compose.yml -f logging.yml -f docker-compose.override.yml to run their cluster, they should instead only run docker compose up -d. Overrides to turn off certain containers (e.g. in the case where you use an external EL/CL rather than the one in this repo) should be done with .env file variables, by setting EL=el-none, CL=cl-none, and MEV=mev-none for example. Read more about client swapping in our docs.

Logging can be enabled by uncommenting the line #MONITORING=${MONITORING:-monitoring},monitoring-log-collector in their .env file, and setting CHARON_LOKI_ADDRESSES to a URI given to you by the Obol team.

To update to this version, please run the following commands:

# Stop the node
docker compose down
# Save any local changes
git stash
# Update your local copy of this repo
git pull
# Checkout this release
git checkout v0.2.1
# Re-apply the stashed local changes
git stash apply
# Restart the node
docker compose up -d

Note

lido-charon-distributed-validator-node is a repo intended as a deployment guide and is not intended to be the canonical way to deploy a distributed validator.

Operators are encouraged to use this repository to build and maintain their own configurations that work for their individual use case. Please work with your squad to have your cluster exposed to no single point of failure on three or more nodes.

What's Changed

• Update all latest releases except prysm version by @OisinKyne in #197
• Bump multiple clients, Nov 11th by @KaloyanTanev in #199

Full Changelog: v0.2.0...v0.2.1