Hardening and Refactoring

[chrismaddalena]

Summary

Significant refactoring to improve performance and logging.

CHANGELOG

[3.1.0] - 18 July 2026

Added

• Added an AOF-backed Redis retry queue so Ghostwriter tag failures do not block log entry ingestion and pending jobs survive mythic_sync restarts and service recreation, provided the Redis data volume is preserved
• Added Redis health checks, pending-job reporting, and automatic migration of legacy mappings and queued tag jobs
• Added tests for query retries, stale entry reconciliation, deleted entry recreation, tag retries, and source IP formatting
• Added GitHub Actions coverage for Python 3.10, Python 3.12, and production container builds, plus Dependabot configuration

Changed

• Changed Ghostwriter GraphQL retries to use exponential backoff with jitter, capped at five minutes
• Improved GraphQL error messages with the operation name and variables, including actionable context for ambiguous ModelDoesNotExist responses
• Changed source IP formatting from a JSON array to a sorted, comma-separated string
• Made Redis hostname, port, and database configurable and scoped Redis mappings by Mythic server and Ghostwriter oplog
• Enabled Redis append-only persistence for embedded Mythic deployments and for the standalone Compose Redis service with a named volume. Raw Mythic subscription events remain live-streamed and are not persisted locally
• Updated supported dependency pins and moved the production container to Python 3.11 on Debian Bookworm
• Made Mythic timestamp and IP parsing tolerant of timezone variants, IPv6, plain strings, CIDR notation, and malformed individual addresses
• Limited GraphQL error context to identifiers while redacting commands, comments, and other potentially sensitive values
• Made the Ghostwriter initialization entry idempotent and corrected Mythic API-key authentication so user credentials are not also required
• Made subscription and tag worker shutdown explicit so worker failures cancel and await the remaining tasks before the Ghostwriter client closes

Fixed

• Fixed stale Redis entry mappings by looking up the Ghostwriter entry by entryIdentifier and repairing the mapping or recreating a deleted entry
• Prevented repeated Mythic error notifications and notification delivery failures from interfering with GraphQL retries
• Fixed Redis startup checks reporting success without issuing a command
• Fixed conversion, creation, update, and Redis failures being swallowed after logging, which could allow processing to continue after an entry failed
• Fixed timezone-aware token expiration parsing for timestamps ending in Z

[Copilot]

Pull request overview

This PR delivers a substantial hardening/refactor of mythic_sync, centered on making Ghostwriter tag updates non-blocking and durable via Redis-backed queuing, improving GraphQL retry diagnostics/redaction, and strengthening operational behavior (startup checks, containerization, CI).

Changes:

• Added a Redis-backed pending-tag retry worker with namespaced keys, legacy migration, and dead-lettering for irrecoverable jobs.
• Refactored Ghostwriter GraphQL execution to use capped exponential backoff with jitter and safer diagnostic context (redaction).
• Updated container/runtime/CI scaffolding (Docker entrypoint, Compose health checks, GitHub Actions, Dependabot) and expanded unit test coverage.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
sync.py	Core refactor: Redis namespacing/migrations, tag retry queue + worker, improved GraphQL retry/backoff and diagnostic redaction, initialization idempotency, and shutdown handling.
test_sync.py	New async unit tests covering retries, reconciliation, tag-queue behavior, timestamp/IP normalization, migrations, and auth/Redis startup behavior.
requirements.txt	Dependency pin updates (but includes a gql pin that appears invalid).
Dockerfile	Moves to Bookworm base, venv install, non-root runtime, dedicated entrypoint, and /data volume.
docker-entrypoint.sh	New entrypoint to manage embedded Redis + graceful shutdown sequencing.
docker-compose.yml	Redis healthcheck + AOF settings + persistent volume; service restart policies.
settings.env	Adds REDIS_DB and documents optional REDIS_URL.
README.md	Expanded operational documentation, dead-letter inspection, and manual acceptance checklist/testing guidance.
CHANGELOG.md	Adds 3.1.0 release notes.
.github/workflows/test.yml	New CI workflow for unit tests + container build + Redis persistence check.
.github/dependabot.yml	New Dependabot configuration for pip/docker/actions.
.dockerignore	New ignore list to slim Docker build context (excludes tests).
.gitignore	Adds common Python/macOS ignores.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[augmentcode]

🤖 Augment PR Summary

Summary: Refactors Mythic→Ghostwriter syncing to be more resilient, especially around Redis durability and tag-update failures.

Changes:

• Adds an AOF-backed Redis pending-tag retry queue with migration of legacy keys, pending/dead-letter reporting, and operator-facing diagnostics.
• Namespaces Redis entry mappings and tag queues by Ghostwriter oplog and Mythic server, and hardens Redis startup checks (requires successful PING).
• Refactors Ghostwriter GraphQL execution with operation-aware error context, variable redaction, and exponential backoff with jitter (capped at 5 minutes).
• Makes the Ghostwriter initialization entry idempotent and improves stale/deleted entry reconciliation via entryIdentifier.
• Improves Mythic timestamp/IP parsing tolerance and changes source IP formatting to a sorted comma-separated string.
• Updates container/Compose setup for Redis AOF persistence and graceful shutdown via a custom entrypoint, plus adds CI (py310/py312 + container build) and a new unit test suite.

_{🤖 Was this summary useful? React with 👍 or 👎}

[augmentcode]

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.