Add recovery feature module

CONTRIBUTING.md

@@ -12,32 +12,35 @@ Thanks for contributing to TKeeper. It's a security-critical, distributed system
 ## Reporting a bug (Issues)
 
 Before opening an issue:
+
 - Search existing issues/discussions.
 - Test on the latest release (or `main` if you can).
 
 Include:
+
 - Expected vs actual behavior
 - TKeeper version / commit SHA
-- Minimal reproduction steps (copy-pasteable)
-- Logs/stack traces (don't forget to remove secrets if present)
+- Minimal reproduction steps
+- Logs/stack traces
 - Environment (OS, JDK)
 
 If you can provide a minimal failing test, even better.
 
 ## Feature requests (Discussions → Issues)
 
 Start in Discussions with:
+
 - The real problem you're solving
 - Proposed API/behavior
 - Compatibility notes (breaking vs non-breaking)
 - Any relevant standards/papers/RFCs
 
-If the direction is agreed, convert to an issue.
+If the direction is agreed, convert it to an issue.
 
 ## Testing expectations
 
-- If your change affects behavior, **add or update an integration test**. See [integration tests](integration-tests)
-- If you're not sure whether something is “behavioral”, assume it is and add the test.
+- If your change affects behavior, **add or update an integration test**. See [integration tests](integration-tests).
+- If you're not sure whether something is behavioral, assume it is and add the test.
 
 ## Code style
 
@@ -48,18 +51,21 @@ If the direction is agreed, convert to an issue.
 
 ## Commit messages
 
-Use clear messages. Conventional Commits are recommended:
-- `fix: ...`
-- `feat: ...`
-- `chore(deps): ...`
-- `test: ...`
+Use short, clear commit messages that describe the change. Prefixes such as `feat:`, `fix:`, or `chore:` are not required.
 
-If changes relate to Java SDK:
-- `fix(sdk): ...`
+Good examples:
+
+- `Fix Java SDK signature validation`
+- `Add integration test for expired keys`
+- `Update dependency versions`
+- `Document key rotation behavior`
+
+Keep each commit focused on one logical change.
 
 ## Pull requests
 
 A PR should include:
+
 - What changed (short summary)
 - Why it changed (rationale)
 - Tests added/updated (or why not)

README.md

@@ -104,7 +104,7 @@ Cryptographic implementations are selected at build time.
 
 A deployable artifact must include at least one platform.
 
-Build all production features and platforms:
+Build all default production features and platforms:
 
 ```bash
 ./gradlew :build -Pkeeper.features=all -Pkeeper.platforms=all
@@ -120,8 +120,9 @@ See [Build and Features](docs/deployment/build-and-features.md).
 
 ## Security references
 
-**Security assurance:** 346 automated scenarios, including 81 failure-injection
-scenarios, run for every pull request and commit to `main`. See
+**Security assurance:** 356 automated functional scenarios, including 90
+protocol and corruption failure-injection scenarios plus one 3-of-5 share-recovery scenario, run
+for every pull request and commit to `main`. See
 [Security Assurance](docs/security-model/security-assurance.md) for tested attack
 vectors and executable evidence.
 

build.gradle

@@ -8,7 +8,7 @@ plugins {
 def mainClassName = 'org.exploit.keeper.TKeeperKt'
 
 group = 'org.exploit'
-version = '2.3.2'
+version = '2.4.0'
 
 repositories {
     mavenCentral()
@@ -38,7 +38,8 @@ def featureModules = [
 ]
 
 def explicitFeatureModules = [
-        'features:auth-dev': 'keeper.feature.auth.dev'
+        'features:auth-dev'          : 'keeper.feature.auth.dev',
+        'features:recovery'          : 'keeper.feature.recovery',
 ]
 
 def testOnlyFeatureModules = ['integration-tests:failure-injection']
@@ -48,9 +49,15 @@ def platformModules = [
         'platform-pqc': 'keeper.platform.pqc'
 ]
 
+def recoveryPlatformModules = [
+        'platform-ecc': 'features:recovery:ecc',
+        'platform-pqc': 'features:recovery:pqc'
+]
+
 def shadowedRuntimeModules =
         featureModules.keySet() +
                 explicitFeatureModules.keySet() +
+                recoveryPlatformModules.values() +
                 testOnlyFeatureModules
 
 shadowedRuntimeModules.each { moduleName ->
@@ -132,7 +139,13 @@ def enabledPlatformModules = platformModules.findAll { moduleName, propertyName
             truthy(findProperty(propertyName))
 }
 
-enabledFeatureModules.keySet().each { moduleName ->
+def enabledRecoveryPlatformModules = enabledExplicitFeatureModules.containsKey('features:recovery')
+        ? recoveryPlatformModules.findAll { platformName, ignored ->
+            enabledPlatformModules.containsKey(platformName)
+        }.values()
+        : []
+
+(enabledFeatureModules.keySet() + enabledExplicitFeatureModules.keySet()).each { moduleName ->
     featurePlatformRequirements.get(moduleName, []).each { requiredPlatform ->
         if (!enabledPlatformModules.containsKey(requiredPlatform)) {
             def featureName = moduleName.tokenize(':').last()
@@ -218,6 +231,10 @@ dependencies {
         productionFeatureRuntime project(path: ":$moduleName", configuration: 'shadowRuntimeElements')
     }
 
+    enabledRecoveryPlatformModules.each { moduleName ->
+        productionFeatureRuntime project(path: ":$moduleName", configuration: 'shadowRuntimeElements')
+    }
+
     enabledPlatformModules.keySet().each { moduleName ->
         productionFeatureRuntime project(path: ":$moduleName", configuration: 'shadowRuntimeElements')
     }
@@ -230,6 +247,10 @@ dependencies {
         integrationFeatureRuntime project(path: ":$moduleName", configuration: 'shadowRuntimeElements')
     }
 
+    recoveryPlatformModules.values().each { moduleName ->
+        integrationFeatureRuntime project(path: ":$moduleName", configuration: 'shadowRuntimeElements')
+    }
+
     testOnlyFeatureModules.each { moduleName ->
         integrationFeatureRuntime project(path: ":$moduleName", configuration: 'shadowRuntimeElements')
     }
@@ -286,6 +307,7 @@ tasks.named('build') {
     dependsOn ':sdk:test'
     dependsOn enabledFeatureModules.keySet().collect { ":$it:test" }
     dependsOn enabledExplicitFeatureModules.keySet().collect { ":$it:test" }
+    dependsOn enabledRecoveryPlatformModules.collect { ":$it:test" }
     dependsOn enabledPlatformModules.keySet().collect { ":$it:test" }
 }
 
@@ -353,7 +375,10 @@ tasks.register('shadowJarProductionTest', com.github.jengelman.gradle.plugins.sh
     doLast {
         def forbidden = [
                 'auth-dev'         : 'org/exploit/keeper/auth/dev/provider/DevAuthProvider.class',
-                'failure-injection': 'org/exploit/keeper/module/failure/FailureInjectionController.class'
+                'failure-injection': 'org/exploit/keeper/module/failure/FailureInjectionController.class',
+                'recovery'         : 'org/exploit/keeper/controller/external/RecoveryController.class',
+                'recovery-ecc'     : 'org/exploit/keeper/module/recovery/ecc/service/ECCRecoveryService.class',
+                'recovery-pqc'     : 'org/exploit/keeper/module/recovery/pqc/service/PQCRecoveryService.class',
         ]
         def archive = archiveFile.get().asFile
         def zip = new java.util.zip.ZipFile(archive)
@@ -378,7 +403,10 @@ tasks.register('artifactIsolationTest') {
     doLast {
         def markers = [
                 'auth-dev'         : 'org/exploit/keeper/auth/dev/provider/DevAuthProvider.class',
-                'failure-injection': 'org/exploit/keeper/module/failure/FailureInjectionController.class'
+                'failure-injection': 'org/exploit/keeper/module/failure/FailureInjectionController.class',
+                'recovery'         : 'org/exploit/keeper/controller/external/RecoveryController.class',
+                'recovery-ecc'     : 'org/exploit/keeper/module/recovery/ecc/service/ECCRecoveryService.class',
+                'recovery-pqc'     : 'org/exploit/keeper/module/recovery/pqc/service/PQCRecoveryService.class',
         ]
 
         def productionJar = tasks.named('shadowJarProductionTest').get().archiveFile.get().asFile
@@ -472,6 +500,7 @@ def releaseChecksBeforeContainers = [
         ':integration-tests:test',
         featureModules.keySet().collect { ":$it:test" },
         explicitFeatureModules.keySet().collect { ":$it:test" },
+        recoveryPlatformModules.values().collect { ":$it:test" },
         testOnlyFeatureModules.collect { ":$it:test" },
         platformModules.keySet().collect { ":$it:test" },
         tasks.named('artifactIsolationTest')
@@ -500,7 +529,11 @@ tasks.register('securityFuzz') {
 
     dependsOn ':platform-ecc:fuzzSecurityParsers'
     dependsOn ':platform-ecc:fuzzProtocolStateMachines'
+    dependsOn ':platform-ecc:fuzzKeeperProtocolStateMachines'
     dependsOn ':platform-pqc:fuzzMLDSAStateMachine'
+    dependsOn ':platform-pqc:fuzzKeeperMLDSAProtocols'
+    dependsOn ':features:recovery:ecc:fuzzRecoveryProtocol'
+    dependsOn ':features:recovery:pqc:fuzzRecoveryPayloads'
 }
 
 test {

docs/api-reference/permissions.md

@@ -40,6 +40,7 @@ tkeeper.key.*.sign
 | `tkeeper.storage.write` | trusted-dealer import |
 | `tkeeper.quorum.promote` | promote mono to threshold |
 | `tkeeper.consistency.fix` | repair threshold state |
+| `tkeeper.recover` | run share-recovery protocols |
 | `tkeeper.expired.view` | read key-expiration indexes |
 | `tkeeper.compliance.inventory` | read asset inventory |
 | `tkeeper.integrity.rotate` | rotate audit integrity key |
@@ -52,7 +53,7 @@ tkeeper.key.*.sign
 - Give signing services only the key identities they need.
 - Keep lifecycle permissions out of normal signing clients.
 - Keep import permissions highly restricted.
-- Restrict consistency repair and quorum promotion to recovery operators.
+- Restrict consistency repair, share recovery, and quorum promotion to recovery operators.
 - Treat control-plane read permissions as sensitive operational metadata.
 - Keep destroy permissions separate from rotate and refresh.
 - Use negative permissions to carve high-risk identities out of broad grants.

docs/deployment/README.md

@@ -32,6 +32,7 @@ Mono and threshold use the same public API model. The difference is where key au
 For production:
 
 - build only required features
+- include recovery only in temporary maintenance artifacts and replace them after repair
 - include every required platform explicitly
 - use a production authenticator
 - keep the internal API private to the cluster

docs/deployment/backup-and-recovery.md

@@ -38,6 +38,153 @@ Restore a peer in an isolated environment before reconnecting it to the cluster.
 
 An older backup may represent a generation that the cluster has already rotated, refreshed, destroyed, or repaired. Do not run consistency fix as an automatic restore step; use it only when the operator can establish which quorum state is safe.
 
+## Share recovery mode
+
+Share recovery is an explicit build capability. Select `recovery` and the platforms used by the
+key histories that need repair. Gradle adds the corresponding platform recovery modules.
+
+ECC recovery artifact:
+
+```bash
+./gradlew :build -Pkeeper.features=recovery -Pkeeper.platforms=ecc
+```
+
+ML-DSA recovery artifact:
+
+```bash
+./gradlew :build -Pkeeper.features=recovery -Pkeeper.platforms=pqc
+```
+
+Artifact for both platforms:
+
+```bash
+./gradlew :build -Pkeeper.features=recovery -Pkeeper.platforms=ecc,pqc
+```
+
+The `all` feature selector does not include recovery. Build recovery as a separate maintenance
+artifact and run that artifact on every helper and target only for the repair window.
+
+### Preconditions
+
+- The configured quorum satisfies `1 < threshold < total`.
+- The operator has selected a sorted list of exactly `threshold` healthy helper peer ids.
+- The target peer id is not in the helper list.
+- The helpers agree on the complete key history and have no pending state for the key.
+- Public TLS, internal mTLS with client authentication, outbound client mTLS, and a distinct
+  `tls-spki-sha256` pin for every peer are configured.
+- Initialization, database identity, integrity, TLS, and seal state are intact on the target.
+
+Legacy unversioned generation zero is outside the recovery boundary.
+
+### Run recovery
+
+1. Stop normal traffic to the participating keepers.
+2. Deploy the recovery-capable artifact to the target and selected helpers.
+3. Restart each participant with recovery mode enabled:
+
+   ```bash
+   java -Dkeeper.recovery=true -jar build/libs/tkeeper-2.4.0.jar
+   ```
+
+4. Unseal the keepers if required and confirm health and status.
+5. Call the recovery endpoint on the damaged keeper with a principal that has
+   `tkeeper.recover`:
+
+   ```bash
+   curl --fail-with-body \
+     --cacert public-ca.crt \
+     -H 'Authorization: Bearer <token>' \
+     -H 'Content-Type: application/json' \
+     -d '{"keyId":"signing-key","helperIds":[1,2,3]}' \
+     https://keeper-4:8080/v1/keeper/recovery/recover
+   ```
+
+6. Compare the returned algorithm, current generation, recovered generations, and helpers with
+   the expected state. Read every generation's public key while recovery mode is still enabled.
+7. Repeat for each damaged logical key.
+8. Rebuild the normal production artifact with the deployment's regular feature and platform
+   selectors, without `recovery`:
+
+   ```bash
+   ./gradlew :build -Pkeeper.features=<production-features> -Pkeeper.platforms=<production-platforms>
+   ```
+
+9. Replace the maintenance artifact on every participant and restart with
+   `keeper.recovery=false`. Do not return the recovery-capable artifact to normal service.
+10. Verify historical inventory, current generation, public keys, owner and expiration indexes,
+   destroyed generations, and a normal signing operation.
+
+The endpoint requires the operator-provided helper list. It does not discover or replace helpers.
+If a selected helper is missing or has conflicting key state, the request fails before target writes.
+
+### Rebuilt state
+
+The target first obtains an identical manifest from every selected helper. Only after validation
+does it replace all local state for that logical key in one database transaction. The rebuild covers:
+
+- active key material for every non-destroyed generation;
+- signed versioned metadata, policies, authorities, owner, and expiration data;
+- the current-generation pointer and historical inventory;
+- destroy markers;
+- ECC commitments or ML-DSA public keys and recovery commitments; and
+- derived owner and expiration indexes.
+
+Target-local pending, missing, stale, or conflicting records for the key are discarded. A failure
+inside the replacement transaction rolls back the reset and install together. Prefix-adjacent key
+ids are not part of the reset.
+
+If the helpers mark a generation destroyed, recovery restores its metadata and destroy marker but
+does not install secret material. Recovery can rebuild a peer with no local records for the logical
+key, provided the target database identity and other node-scoped state still exist.
+
+### Platform behavior
+
+ECC recovery uses exactly `threshold` remote helpers. Helpers exchange fresh pairwise masks and
+return masked Lagrange contributions. The target validates each reconstructed share against the
+generation's polynomial commitments. It works for any configured quorum with `1 < threshold < total`;
+`threshold = total` cannot recover a missing share because only `total - 1` shares survive.
+
+ML-DSA helpers return only replicated RSS components that belong to the target. The target validates
+each component against the signed per-subset commitments stored during DKG or dealer split. Legacy
+ML-DSA generations without these commitments remain usable but return `RECOVERY_NOT_POSSIBLE`.
+ML-DSA parameter changes between generations are supported. A history that crosses between ECC and
+PQC returns `RECOVERY_NOT_POSSIBLE` without installing partial state.
+
+Recovery is limited to 64 historical generations per logical key.
+
+### Recovery mode boundary
+
+Recovery mode blocks normal key operations. The public API allows health, status, seal, unseal,
+public-key reads, and recovery routes. The internal API allows health, integrity-public-key discovery,
+and the selected platform's recovery routes. Helpers reject recovery calls while running in normal
+mode. Startup fails if the artifact lacks the recovery feature or the required TLS and peer-pin
+configuration.
+
+Recovery repairs keeper state. It does not determine whether the lost or modified share was exposed.
+Rotate or reshare after recovery when required by the incident response decision.
+
+Disabling the runtime flag is not the final deployment state. The normal artifact must exclude the
+recovery base module and both platform recovery implementations.
+
+### ECC mask boundary
+
+The mask construction hides an honest helper contribution from the target colluding with up to
+`t - 2` helpers. A malicious helper can cause the final commitment check to fail; this version does
+not include a zero-knowledge proof that attributes that denial of service. Secret scalar arithmetic
+uses random quotient blinding, pair masks, and one secure modular reduction, but the bigint backend
+is not a formally verified fixed-width constant-time field implementation. Co-resident timing and
+microarchitectural attackers remain outside the protocol's threat model.
+
+Within one recovery session each helper caches its contribution, so retries return the same value
+and do not create a second equation. Every new session uses independent CSPRNG pair masks. Reusing a
+pair mask in a session with a different helper set (and therefore a different Lagrange coefficient)
+would reveal that helper's share; the signed transcript binds the session id, target, canonical
+helper set, algorithm, generation, commitments, metadata, and expiry, and rejects such reuse.
+
+The complete graph contains `t(t - 1) / 2` mask deliveries. The implementation bounds setup to 64
+concurrent requests; validate recovery latency at the deployment's largest supported `t` before
+relying on the default session lifetime.
+
 ## Recovery objectives
 
 Define and test separately: