Skip to content

Simplify Splunkbase → MinIO sync playbook with a more native approach #127

@JacobPEvans-personal

Description

@JacobPEvans-personal

Context

playbooks/sync-splunkbase.yml (added in #125) is a ~250-line pure-Ansible
playbook that authenticates to Splunkbase via an XML session-token flow,
queries existing MinIO tags per object, downloads drifted archives, uploads
them via mc cp, and tags via mc tag set. It works, it's idempotent, it
fails loud on real errors — but the implementation is long, fragile in a few
places, and relies on three separate command: mc … shell-outs per entry.

Goal

Replace the hand-rolled sync with something more native. Candidates:

  1. amazon.aws.s3_object (or community.aws.s3_object) with a custom
    endpoint_url pointing at MinIO. Supports tags: in a single module
    call, replacing both mc cp and mc tag set. Adds a boto3 dependency
    on the controller but removes the MC_HOST_homelab env-var trick, the
    mc tag list --json parsing, and the NoSuchKey stdout-filter
    workaround. Likely the biggest structural simplification available.

  2. Splunk's own docker-splunk app installer (SPLUNK_APPS_URL env
    var on the container). The official Splunk Docker image can accept a
    comma-separated list of Splunkbase URLs at startup and install them
    itself. If this works for all our apps, the entire sync layer could
    be deleted — Splunk pulls directly from Splunkbase on container start,
    no MinIO mirror needed for version rotation. Would require rethinking
    the role to pass SPLUNK_APPS_URL through docker-compose, and
    credentials would need to live in the container environment.

  3. Splunkbase JSON login endpoint instead of the XML
    api/account:login flow. If /api/v1/accounts/login returns JSON, the
    regex XML parse (regex_search('<id>([^<]+)</id>')) goes away.

  4. A dedicated community collection if one exists — check
    ansible-galaxy for anything like splunk.splunkbase or
    community.splunkbase. Unlikely, but worth a search.

Acceptance criteria

  • Resulting playbook is materially shorter AND cleaner (target: under 100
    lines, or deleted entirely if option 2 works)
  • Same functionality: pinned-version sync from Splunkbase to the install
    target, idempotent re-runs, fails loud on auth/network errors
  • No new shell scripts. Pure-Ansible modules where possible
  • Version metadata still visible per-app (either MinIO tags OR Splunkbase
    URL in the container env)
  • End-to-end verification against live cluster before merge

Non-goals

  • Scheduled automation (separate concern — needs self-hosted runner for
    home-LAN reach)
  • Latest-version auto-discovery (Splunkbase has no public API for this)
  • Re-adding any shell script

References

  • playbooks/sync-splunkbase.yml — current implementation
  • roles/splunk_docker/vars/addons.yml — registry consumed by the sync
  • splunk/docker-splunkSPLUNK_APPS_URL environment variable docs
  • amazon.aws.s3_object — Ansible module supporting custom S3 endpoints
  • feat(splunk): MinIO add-on registry + Splunkbase auto-sync #125 — PR that added the current sync

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:lowLow - Address when time permitssize:mM - Moderate effort, 1-2 daystype:refactorRefactor - Code change with no functional change

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions