KMS-649: PR adds Redis-backed response caching for hot read endpoints (/concepts*, /concept*, /tree*) with a scheduled cache-prime Lambda, plus deployment/runtime updates to support Redis in AWS and local development.

.gitignore

@@ -23,3 +23,24 @@ cmr
 cdk.context.json
 cdk.out
 infrastructure/rdfdb/cdk/cdk.context.json
+
+# Spike/generated artifacts
+*.log
+log.txt
+cdk/log.txt
+data/mongo-spike/
+data/mongo-spike/**/*.json
+data/mongo-spike/**/*.ndjson
+data/mongo-spike/**/*.xml
+data/mongo-spike/**/*.rdf
+scripts/spike-express/file.rdf
+loadtest/locust/.DS_Store
+loadtest/locust/**/__pycache__/
+loadtest/locust/**/*.pyc
+loadtest/locust/**/results*.csv
+loadtest/locust/**/locust_results.csv
+loadtest/locust/**/summary.csv
+loadtest/locust/**/summary.csv~
+loadtest/locust/
+scripts/load/hammer_endpoints_sequential.js
+data/scheme-size*

README.md

@@ -32,6 +32,75 @@ To start local server (including rdf4j database server, cdk synth and sam)
 npm run start-local
 ```
 
+To run local server with SAM watch mode enabled
+```
+npm run start-local:watch
+```
+
+### Optional: Enable Redis cache in local SAM/LocalStack
+
+By default, local `start-local` does not provision Redis in CDK. You can still test Redis caching by running Redis in Docker.
+Local defaults are centralized in `bin/env/local_env.sh`.
+
+1. Ensure the docker network exists:
+```
+npm run rdf4j:create-network
+```
+
+2. Start Redis on the same docker network used by SAM:
+```
+npm run redis:start
+```
+
+3. (Optional) override defaults in `bin/env/local_env.sh` or per-command, for example:
+```
+REDIS_ENABLED=true REDIS_HOST=kms-redis-local REDIS_PORT=6379 npm run start-local
+```
+
+4. Start local API:
+```
+npm run start-local
+```
+
+5. Verify cache behavior (published reads only):
+- `GET /concepts?version=published`
+- `GET /concepts/concept_scheme/{scheme}?version=published`
+
+6. Check Redis cache memory usage:
+```
+npm run redis:memory_used
+```
+
+### Redis node types and memory (ElastiCache)
+
+Common burstable node types for Redis/Valkey:
+
+| Node type | Memory (GiB) |
+| --- | --- |
+| `cache.t4g.micro` | `0.5` |
+| `cache.t4g.small` | `1.37` |
+| `cache.t4g.medium` | `3.09` |
+| `cache.t3.micro` | `0.5` |
+| `cache.t3.small` | `1.37` |
+| `cache.t3.medium` | `3.09` |
+
+For full and latest node-family capacities (m/r/t and region support), see:
+https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheNodes.SupportedTypes.html
+
+To disable local Redis cache again:
+```
+REDIS_ENABLED=false npm run start-local
+```
+
+### Invoke cache-prime cron locally
+
+Runs the same Lambda used by the scheduled EventBridge cache-prime target, locally via SAM.
+The script re-synthesizes `cdk/cdk.out/KmsStack.template.json` each run so local Redis env settings are baked into the template.
+
+```bash
+npm run prime-cache:invoke-local
+```
+
 ## Local Testing
 
 To run the test suite, run:
@@ -42,24 +111,25 @@ npm run test
 ## Setting up the RDF Database for local development
 In order to run KMS locally, you first need to setup a RDF database.
 ### Prerequisites
-#### Set the RDFDB user name and password
-```
-export RDF4J_USER_NAME=[user name]
-export RDF4J_PASSWORD=[password]
-```
+RDF4J local defaults are in `bin/env/local_env.sh`.
+If needed, override per command (for example: `RDF4J_USER_NAME=... RDF4J_PASSWORD=... npm run rdf4j:setup`).
 ### Building and Running the RDF Database
 #### Build the docker image
 ```
 npm run rdf4j:build
 ```
 #### Create a docker network
 ```
-npm run create-network
+npm run rdf4j:create-network
 ```
 #### Run the docker image
 ```
 npm run rdf4j:start
 ```
+#### Pull latest concepts RDF files from CMR
+```
+npm run rdf4j:pull
+```
 #### Setup and load data into the RDF database
 ```
 npm run rdf4j:setup
@@ -71,60 +141,15 @@ npm run rdf4j:stop
 ```
 
 # Deployments
-## Deploying RDF Database to AWS
-### Prerequisites
-#### Copy your AWS credentials and set these up as env variables
-```
-export AWS_ACCESS_KEY_ID=${your access key id}
-export AWS_SECRET_ACCESS_KEY=${your secret access key}
-export AWS_SESSION_TOKEN=${your session token}
-export VPC_ID={your vpc id}
-```
-#### Set the RDFDB user name and password
-```
-export RDF4J_USER_NAME=[your rdfdb user name]
-export RDF4J_PASSWORD=[your rdfdb password]
-export RDF4J_CONTAINER_MEMORY_LIMIT=[7168 for sit|uat, 14336 for prod]
-```
-
-#### Deploy Docker Container to Registry
-```
-cd cdk/rdfdb/bin
-./deploy_to_ecr.sh
-```
-
-#### Deploy ECS Service to AWS
-#### Deploy IAM, EBS, LB, and ECS stacks
-```
-export RDF4J_USER_NAME=[your rdfdb user name]
-export RDF4J_PASSWORD=[your rdfdb password]
-export RDF4J_CONTAINER_MEMORY_LIMIT=[7168 for sit|uat, 14336 for prod]
-export RDF4J_INSTANCE_TYPE=["M5.LARGE" for sit|uat, "R5.LARGE" for prod]
-
-cd cdk
-cdk deploy rdf4jIamStack
-cdk deploy rdf4jEbsStack
-cdk deploy rdf4jLbStack
-cdk deploy rdf4jEcsStack
-cdk deploy rdf4jSnapshotStack
-cdk deploy KmsStack
-```
-#### Alternatively, you can deploy all stacks at once
-```
-cd cdk
-cdk deploy --all
-```
-One thing to note is if you destroy the rdf4jEbsStack and redeploy, this will create a new EBS file system.  You will need to copy the data from the old EBS file system to the new one.  This can be done by mounting the old EBS file system to an EC2 instance and copying the data to the new EBS file system.
-
-## Deploying KMS to AWS
+## Deploying KMS Application to AWS
 ### Prerequisites
 #### Copy your AWS credentials and set these up as env variables
 ```
+export RELEASE_VERSION=[app release version]
 export bamboo_STAGE_NAME=[sit|uat|prod]
 export bamboo_AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
 export bamboo_AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
 export bamboo_AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}
-export bamboo_LAMBDA_TIMEOUT=30
 export bamboo_SUBNET_ID_A={subnet #1}
 export bamboo_SUBNET_ID_B={subnet #2}
 export bamboo_SUBNET_ID_C={subnet #3}
@@ -141,10 +166,13 @@ export bamboo_RDF4J_INSTANCE_TYPE=["M5.LARGE" for sit|uat, "R5.LARGE" for prod]
 export bamboo_RDF_BUCKET_NAME=[name of bucket for storing archived versions]
 export bamboo_EXISTING_API_ID=[api id if deploying this into an existing api gateway]
 export bamboo_ROOT_RESOURCE_ID=[see CDK_MIGRATION.md for how to determine]
-export bamboo_KMS_CACHE_TTL_SECONDS=[optional, default 3600]
-export bamboo_KMS_CACHE_CLUSTER_SIZE_GB=[optional, default 0.5]
-export bamboo_KMS_CACHE_CLUSTER_ENABLED=[optional, default true]
+export bamboo_LOG_LEVEL=[INFO|DEBUG|WARN|ERROR]
+export bamboo_KMS_REDIS_ENABLED=[true|false]
+export bamboo_KMS_REDIS_NODE_TYPE=[for example cache.t3.micro]
 ```
+Notes:
+- If you are not deploying into an existing API Gateway, set `bamboo_EXISTING_API_ID` and `bamboo_ROOT_RESOURCE_ID` to empty strings.
+
 #### Deploy KMS Application
 ```
 ./bin/deploy-bamboo.sh

bin/deploy-bamboo.sh

@@ -70,9 +70,8 @@ dockerRun() {
         --env "EXISTING_API_ID=$bamboo_EXISTING_API_ID" \
         --env "ROOT_RESOURCE_ID=$bamboo_ROOT_RESOURCE_ID" \
         --env "LOG_LEVEL=$bamboo_LOG_LEVEL" \
-        --env "KMS_CACHE_TTL_SECONDS=${bamboo_KMS_CACHE_TTL_SECONDS}" \
-        --env "KMS_CACHE_CLUSTER_SIZE_GB=${bamboo_KMS_CACHE_CLUSTER_SIZE_GB}" \
-        --env "KMS_CACHE_CLUSTER_ENABLED=${bamboo_KMS_CACHE_CLUSTER_ENABLED}" \
+        --env "KMS_REDIS_ENABLED=$bamboo_KMS_REDIS_ENABLED" \
+        --env "KMS_REDIS_NODE_TYPE=$bamboo_KMS_REDIS_NODE_TYPE" \
     $dockerTag "$@"
 }
 

bin/env/local_env.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+# Shared defaults for local SAM/CDK workflows.
+export RDF4J_SERVICE_URL="${RDF4J_SERVICE_URL:-http://rdf4j-server:8080}"
+export REDIS_ENABLED="${REDIS_ENABLED:-true}"
+export REDIS_HOST="${REDIS_HOST:-kms-redis-local}"
+export REDIS_PORT="${REDIS_PORT:-6379}"
+export KMS_DOCKER_NETWORK="${KMS_DOCKER_NETWORK:-kms-network}"
+export SAM_WARM_CONTAINERS="${SAM_WARM_CONTAINERS:-EAGER}"
+export SAM_LOCAL_WATCH="${SAM_LOCAL_WATCH:-false}"

bin/rdf4j/build.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+docker build -t rdf4j:latest ./cdk/rdfdb/docker

bin/rdf4j/create-network.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=bin/env/local_env.sh
+source "${SCRIPT_DIR}/../env/local_env.sh"
+
+NETWORK_NAME="${RDF4J_DOCKER_NETWORK:-${KMS_DOCKER_NETWORK:-kms-network}}"
+
+if docker network inspect "$NETWORK_NAME" >/dev/null 2>&1; then
+  echo "Docker network '${NETWORK_NAME}' already exists"
+  exit 0
+fi
+
+docker network create "$NETWORK_NAME"

bin/rdf4j/pull.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+node setup/scripts/pullRdfFromCmr.js

bin/rdf4j/setup.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=bin/env/local_env.sh
+source "${SCRIPT_DIR}/../env/local_env.sh"
+
+RDF4J_CONTAINER_NAME="${RDF4J_CONTAINER_NAME:-rdf4j-server}"
+RDF4J_USER_NAME="${RDF4J_USER_NAME:-rdf4j}"
+RDF4J_PASSWORD="${RDF4J_PASSWORD:-rdf4j}"
+RDF4J_REPOSITORY_ID="${RDF4J_REPOSITORY_ID:-kms}"
+# Setup runs from host machine against mapped port by default.
+export RDF4J_SERVICE_URL="${RDF4J_SETUP_SERVICE_URL:-http://127.0.0.1:8081}"
+export RDF4J_CONTAINER_NAME RDF4J_USER_NAME RDF4J_PASSWORD RDF4J_REPOSITORY_ID
+
+echo "Running RDF4J setup against ${RDF4J_SERVICE_URL}/rdf4j-server (repo=${RDF4J_REPOSITORY_ID})"
+
+node setup/setupRdf4j.js

bin/rdf4j/shell.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+CONTAINER_NAME="${RDF4J_CONTAINER_NAME:-rdf4j-server}"
+
+if ! docker ps -q --filter "name=^${CONTAINER_NAME}$" | grep -q .; then
+  echo "Container '${CONTAINER_NAME}' is not running." >&2
+  exit 1
+fi
+
+docker exec -it "$CONTAINER_NAME" /bin/bash

bin/rdf4j/start.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=bin/env/local_env.sh
+source "${SCRIPT_DIR}/../env/local_env.sh"
+
+NETWORK_NAME="${RDF4J_DOCKER_NETWORK:-${KMS_DOCKER_NETWORK:-kms-network}}"
+CONTAINER_NAME="${RDF4J_CONTAINER_NAME:-rdf4j-server}"
+RDF4J_USER_NAME="${RDF4J_USER_NAME:-rdf4j}"
+RDF4J_PASSWORD="${RDF4J_PASSWORD:-rdf4j}"
+RDF4J_CONTAINER_MEMORY_LIMIT="${RDF4J_CONTAINER_MEMORY_LIMIT:-4096}"
+
+if ! docker network inspect "$NETWORK_NAME" >/dev/null 2>&1; then
+  docker network create "$NETWORK_NAME" >/dev/null
+fi
+
+if docker ps -aq --filter "name=^${CONTAINER_NAME}$" | grep -q .; then
+  docker rm -f "$CONTAINER_NAME" >/dev/null
+fi
+
+docker run \
+  --name "$CONTAINER_NAME" \
+  --network "$NETWORK_NAME" \
+  -d \
+  -p 8081:8080 \
+  -e "RDF4J_USER_NAME=${RDF4J_USER_NAME}" \
+  -e "RDF4J_PASSWORD=${RDF4J_PASSWORD}" \
+  -e "RDF4J_CONTAINER_MEMORY_LIMIT=${RDF4J_CONTAINER_MEMORY_LIMIT}" \
+  -v logs:/usr/local/tomcat/logs \
+  rdf4j:latest

bin/rdf4j/stop.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+CONTAINER_NAME="${RDF4J_CONTAINER_NAME:-rdf4j-server}"
+
+if docker ps -q --filter "name=^${CONTAINER_NAME}$" | grep -q .; then
+  echo "Stopping container '${CONTAINER_NAME}'..."
+  docker kill "$CONTAINER_NAME" >/dev/null
+  echo "Stopped container '${CONTAINER_NAME}'."
+else
+  echo "Container '${CONTAINER_NAME}' is not running."
+fi
+
+if docker ps -aq --filter "name=^${CONTAINER_NAME}$" | grep -q .; then
+  echo "Removing container '${CONTAINER_NAME}'..."
+  docker rm "$CONTAINER_NAME" >/dev/null
+  echo "Removed container '${CONTAINER_NAME}'."
+else
+  echo "Container '${CONTAINER_NAME}' does not exist."
+fi

bin/redis/connect.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+CONTAINER_NAME="${REDIS_CONTAINER_NAME:-kms-redis-local}"
+HOST="${REDIS_CONNECT_HOST:-127.0.0.1}"
+PORT="${REDIS_HOST_PORT:-6380}"
+
+if command -v redis-cli >/dev/null 2>&1; then
+  echo "Connecting via host redis-cli to redis://${HOST}:${PORT}"
+  exec redis-cli -h "${HOST}" -p "${PORT}"
+fi
+
+if docker ps --format '{{.Names}}' | grep -qx "${CONTAINER_NAME}"; then
+  echo "Host redis-cli not found; connecting via docker exec to '${CONTAINER_NAME}'"
+  exec docker exec -it "${CONTAINER_NAME}" redis-cli
+fi
+
+echo "Could not connect to Redis."
+echo "Start Redis first with: npm run redis:start"
+echo "If using host redis-cli, install it or ensure '${CONTAINER_NAME}' is running for docker fallback."
+exit 1