Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6e4819f
test: add multi-SDK e2e command stress tests with dual-client mode
starrysky9959 May 22, 2026
1b7cddc
fix: update pre-push hook to use cmd_stress_test.sh
starrysky9959 May 22, 2026
85ffc0c
fix: use rocksdb storage provider (recognized by eloqctl)
starrysky9959 May 22, 2026
b1f3ad6
fix: strip seccomp/cap_add via sed instead of compose override
starrysky9959 May 22, 2026
e47776d
fix: restore full docker-compose with minio, control-node, and stress…
starrysky9959 May 22, 2026
39e4bc4
test: merge quality and e2e into single test workflow
starrysky9959 May 22, 2026
92af88e
fix: add ca-certificates and ssl certs symlink to Dockerfile
starrysky9959 May 22, 2026
fb93392
test: add pumba chaos (1ms delay, 0.1% loss); drop log_service from t…
starrysky9959 May 22, 2026
b2e7133
fix: disable WAL since log_service was removed
starrysky9959 May 22, 2026
2b16dc8
fix: skip ECHO failures caused by cluster routing (keyless command)
starrysky9959 May 22, 2026
c683cf4
fix: use cluster client for preload to handle MOVED redirects
starrysky9959 May 23, 2026
04de678
fix: add cluster stabilization wait; temporarily drop pumba
starrysky9959 May 23, 2026
99764b2
fix: use plain INFO instead of INFO stats across all SDKs
starrysky9959 May 23, 2026
2622c4d
fix: add ZADD cushions for Z* reads; tolerate <2% sporadic failures
starrysky9959 May 23, 2026
39354d3
fix: discover master dynamically via CLUSTER NODES; add host-side con…
starrysky9959 May 23, 2026
fae5c37
feat: add --repeat param to multiply per-command concurrency (default…
starrysky9959 May 23, 2026
bbeb3a4
fix: extract master discovery; run all SDKs even if one fails; report…
starrysky9959 May 23, 2026
031dbf8
fix: stabilize e2e ci workflow and keep pre-push lightweight
starrysky9959 May 23, 2026
63e15c5
ci: run e2e daily and increase mainline stress
starrysky9959 May 23, 2026
a503025
feat: add per-worker inflight concurrency to stress clients
starrysky9959 May 23, 2026
a4733cd
feat: streamline release updates and control-node e2e flow
starrysky9959 May 23, 2026
004c80a
fix: remove dead no-op upgrade step state
starrysky9959 May 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .githooks/pre-push
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ set -eo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
LOG_FILE="${REPO_ROOT}/.githooks/pre-push.log"

echo "[pre-push] Running full verification suite (output -> ${LOG_FILE})..." >&2
echo "[pre-push] Running fast verification suite (output -> ${LOG_FILE})..." >&2
if "${REPO_ROOT}/scripts/test-before-push.sh" > "${LOG_FILE}" 2>&1; then
echo "[pre-push] PASS" >&2
exit 0
Expand Down
63 changes: 0 additions & 63 deletions .github/workflows/quality.yml

This file was deleted.

120 changes: 120 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
name: Test

on:
pull_request:
push:
branches:
- main
schedule:
- cron: "0 2 * * *"
workflow_dispatch:
inputs:
version:
description: "EloqKV version"
required: false
default: "1.2.2"
workers:
description: "Stress test workers"
required: false
default: "32"
duration:
description: "Stress duration (seconds)"
required: false
default: "60"
repeat:
description: "Per-command repeat factor"
required: false
default: "10"
inflight:
description: "Concurrent execution slots per worker"
required: false
default: "4"

env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"

permissions:
contents: read

jobs:
test:
name: Test
runs-on: ubuntu-24.04
timeout-minutes: 30

steps:
- uses: actions/checkout@v4

- name: Install build dependencies
uses: awalsh128/cache-apt-pkgs-action@latest
with:
packages: protobuf-compiler pkg-config libssl-dev
version: test-ubuntu24-v1

- name: Set up Rust
uses: dtolnay/rust-toolchain@master
with:
toolchain: nightly-2026-04-21
components: rustfmt, clippy

- name: Cache Rust build artifacts
uses: swatinem/rust-cache@v2
Comment on lines +49 to +61

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Detect unpinned 'uses:' refs in workflow files (expects 40-char SHA pins).
rg -nP '^\s*uses:\s*[^@\s]+@(?!(?:[0-9a-f]{40})\b)' .github/workflows

Repository: monographdb/eloq_waiter

Length of output: 2144


Pin GitHub Actions in .github/workflows/e2e.yml to immutable commit SHAs (no tags/latest/branches).

Mutable uses: refs in this workflow include:

  • actions/checkout@v4
  • awalsh128/cache-apt-pkgs-action@latest
  • dtolnay/rust-toolchain@master
  • swatinem/rust-cache@v2
  • actions/cache@v4 (also applies at line 81)

Pin each uses: to a full 40+ char commit SHA.

🧰 Tools
🪛 zizmor (1.25.2)

[error] 44-44: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)


[error] 50-50: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)


[error] 55-55: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e.yml around lines 44 - 55, Replace mutable GitHub
Actions refs with immutable commit SHAs: update each `uses:` entry that
currently references tags/branches (e.g., actions/checkout@v4,
awalsh128/cache-apt-pkgs-action@latest, dtolnay/rust-toolchain@master,
swatinem/rust-cache@v2, actions/cache@v4) to the corresponding full 40+
character commit SHA for the specific release you want to lock; ensure you
replace every occurrence in the workflow and verify the SHA references point to
the intended commit on the action repositories.

with:
shared-key: test-ubuntu24
cache-on-failure: true

- name: Format check
run: cargo fmt --all -- --check

- name: Clippy
run: cargo clippy --all-targets --all-features -- -D warnings

- name: Unit tests
run: cargo test --tests -- --test-threads=1

- name: Build eloqctl
run: |
cargo build -p cluster_mgr
mkdir -p ~/.eloqctl/bin
cp target/debug/cluster_mgr ~/.eloqctl/bin/cluster_mgr
chmod 755 ~/.eloqctl/bin/cluster_mgr
rm -f ~/.eloqctl/config
ln -s "$PWD/src/cluster_mgr/config" ~/.eloqctl/config

- name: Run E2E test
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
workers="${{ github.event.inputs.workers || '32' }}"
duration="${{ github.event.inputs.duration || '60' }}"
repeat="${{ github.event.inputs.repeat || '10' }}"
inflight="${{ github.event.inputs.inflight || '4' }}"
elif [ "${{ github.event_name }}" = "pull_request" ]; then
workers="16"
duration="60"
repeat="10"
inflight="4"
else
workers="32"
duration="60"
repeat="20"
inflight="4"
fi

export DOCKER_E2E_DIR="$PWD/tests/docker_ha"
export ELOQCTL_HOME="$HOME/.eloqctl"
export ELOQCTL="$HOME/.eloqctl/bin/cluster_mgr"
export ELOQKV_VERSION="${{ github.event.inputs.version || '1.2.2' }}"
export DURATION_SECONDS="${duration}"
export WORKERS="${workers}"
export INFLIGHT="${inflight}"
export REPEAT="${repeat}"
export SKIP_DEPS=1
export STEPS="launch,py-stress,go-stress,ts-stress,remove"
echo "Running E2E with WORKERS=${WORKERS} INFLIGHT=${INFLIGHT} DURATION_SECONDS=${DURATION_SECONDS} REPEAT=${REPEAT}"
bash tests/e2e/cmd_stress_test.sh

- name: Cleanup Docker
if: always()
run: |
docker compose -f tests/docker_ha/docker-compose.yaml down -v --remove-orphans 2>/dev/null || true
docker system prune -af --volumes 2>/dev/null || true
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ src/log/
log/
.monograph_waiter/
.bin/
.tmp-eloqctl/
tests/ssh_keys/test_rsa
tests/**/.eloqctl/
tests/**/topology.generated.yaml
Expand All @@ -46,3 +47,6 @@ tests/**/id_ed25519
tests/**/id_ed25519.pub
tests/**/authorized_keys
tests/**/exported.yaml
tests/**/node_modules/
tests/**/__pycache__/
tests/**/go-cluster-tls-stress
41 changes: 38 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions doc/cluster_mgr.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ eloqctl remove eloqkv-cluster --force
| `connect <cluster>` | Print an EloqKV client command for the cluster. |
| `plan <topology.yaml>` | Preview supported changes without mutating local state or remote hosts. |
| `apply <topology.yaml>` | Execute the same plan shown by `plan`, gated by live critical-service health and verified afterward. |
| `update <cluster> <version>` | Rolling version update for an existing cluster. |
| `update <cluster> <version>` | Rolling version update for an existing cluster. Version lookup and tarball selection come from GitHub Releases for `eloqdata/eloqkv`. `--download-only` resolves and caches the required tarballs without touching remote hosts. `--monitor` updates only one monitor component. |
| `update-conf <cluster>` | Apply selected config fields and optionally restart tx nodes. |
| `scale <cluster>` | Add or remove EloqKV tx/standby nodes. Duplicate add/remove requests are no-ops. |
| `scalelog <cluster>` | Add or remove log service nodes. Duplicate add/remove requests are no-ops. |
Expand All @@ -59,7 +59,7 @@ eloqctl remove eloqkv-cluster --force
| `backup <cluster> ...` | Create, list, remove, restore, and dump backups. |
| `export <cluster> [--output file]` | Write the saved launch-compatible topology YAML. |
| `list` | List locally registered clusters. |
| `versions` | List available EloqKV versions. |
| `versions` | List available EloqKV versions by scanning GitHub Release assets for `eloqdata/eloqkv`. |
| `completion <shell>` | Generate shell completion scripts. |
| `upgrade` | Run local SQLite schema upgrades. |
| `remove <cluster> [--force]` | Remove local cluster metadata and perform best-effort remote cleanup. |
Expand Down
32 changes: 32 additions & 0 deletions doc/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,38 @@ Upgrade an existing EloqKV cluster to a specific version:
eloqctl update <cluster> <version>
```

`eloqctl` now resolves available versions and EloqKV tarballs from GitHub Releases for `eloqdata/eloqkv`. For example, asset `eloqkv-1.2.2-rocks_s3-ubuntu24-amd64.tar.gz` is treated as version `1.2.2` for store `rocks_s3`.

Download the required tarballs into the local cache without applying the update:

```sh
eloqctl update <cluster> <version> --download-only
```

`--download-only` still resolves the version against the saved cluster topology, so provide the existing cluster name and the target version exactly as you would for a real upgrade. It downloads the matching EloqKV release tarballs into `$ELOQCTL_HOME/download/...` and exits without changing remote files, processes, or cluster metadata.

Typical workflow:

```sh
eloqctl versions
eloqctl update <cluster> 1.2.2 --download-only
eloqctl update <cluster> 1.2.2
```

Update only one monitor component without touching the EloqKV cluster itself:

```sh
eloqctl update <cluster> --monitor grafana
```

Override the monitor tarball URL for a monitor-only update:

```sh
eloqctl update <cluster> --monitor grafana --monitor-url https://dl.grafana.com/oss/release/grafana-11.0.0.linux-amd64.tar.gz
```

GitHub-hosted EloqKV tarballs are now validated with the asset `sha256` digest when available. Cached files are reused only when the digest still matches, so a re-published asset under the same tag can be refreshed correctly.

Upgrade to the latest available version:

```sh
Expand Down
13 changes: 5 additions & 8 deletions scripts/test-before-push.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,15 @@ REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"

cd "${REPO_ROOT}"

echo "[1/5] Install dev eloqctl"
echo "[1/4] Install dev eloqctl"
"${REPO_ROOT}/scripts/install-dev.sh"

echo "[2/5] Check formatting"
echo "[2/4] Check formatting"
cargo fmt --all -- --check

echo "[3/5] Check and lint"
echo "[3/4] Check and lint"
cargo check -p cluster_mgr
cargo clippy --all-targets --all-features -- -D warnings

echo "[4/5] Run Docker E2E suite"
bash tests/e2e/test.sh

echo "[5/5] Done"
echo "PASS: pre-push test suite completed"
echo "[4/4] Done"
echo "PASS: pre-push checks completed (full E2E runs in GitHub Actions)"
Loading
Loading