Aurora Identity Risk Engine

The server half of Aurora Identity. It receives the events that the browser extension sends, stores them in ClickHouse, scores them, and serves a dashboard that turns them into a picture of which outside services an organisation's people are actually using, and which of those have seen personal data.

The other repository in the project is website, which holds the source for auroraidentity.com.

It is a single Go binary and a ClickHouse instance. There is no queue and no second service, and the only caches are small in-memory LRUs inside the process itself: the tenant lookup, the duplicate-request check and the domain classifications. The extension posts one JSON object per request it saw, the server writes one row, and every screen in the dashboard is a SQL query over that one table. This has been sufficiently fast so far, and scaling it further is a future direction.

What arrives

Everything comes in on POST /ingest. The body is capped at one megabyte and holds a single event:

Field	What it holds
api_key	The tenant's public API key, which is what authenticates the event
pseudonym	An opaque string standing in for one person, computed in the browser
signature	Sent by the extension, but see the warning further down
request_id	A fresh UUID, used to reject an event that arrives twice
timestamp	Milliseconds since the epoch
http_method	POST, PUT, and so on
domain	The scheme, host and path the browser was about to send to
metadata	Free-form JSON: requestSize, fileTypeFromAnalysis, piiDetected

The server never sees a request body, an email address or a Google account. The extension does its analysis on the machine and sends only this description of what it found. The pseudonym is derived in the browser from the person's Chrome account identifier using a key that never leaves the browser, so the server can tell one person's events from another's without ever holding the account itself.

What happens to an event

Four checks, in order, and any of them drops the event.

The api_key is looked up in the tenants table, and the row must have status active. Successful lookups are held in a 100-entry cache, so a busy tenant costs one query rather than one per event. An unknown or suspended key gets a 401.

The domain and http_method must be present and must not be the strings "null" or "undefined", which is what a browser sends when something upstream went wrong. The pseudonym must not be empty, because an event that cannot be attributed to a person is of no use to anyone looking at the dashboard.

The timestamp must be within five minutes of now in either direction. That window is what stops an old event being replayed later.

The request_id must not have been seen before. The server keeps the last ten thousand in memory and rejects a repeat, which is what makes the extension's retry-three-times behaviour safe: a request that actually arrived but whose response was lost will not be counted twice.

An event that survives all four is inserted into telemetry_events and the server answers 204 No Content.

How the data is stored

Two tables, both in database/001-create-tables.sql.

telemetry_events is a ReplicatedMergeTree partitioned by tenant and month, and sorted by tenant, then pseudonym, then time. That sort order is the reason the dashboard is fast: every screen asks about one tenant, usually about one person or one span of days, and that is exactly the prefix of the sort key. Retention is set on the table itself. Data moves to the warm storage volume after seven days and is deleted after ninety.

tenants holds one row per customer: the numeric tenant_id, a display name, the email domain used to match people at sign-in, the public_api_key that authenticates ingest, and a status of active, suspended or offboarded. There is no sign-up flow. You add a tenant by inserting a row.

Who can see the dashboard

Sign-in is Google OIDC. The server takes the verified email out of the ID token, splits the domain off it, and looks that domain up in the tenants table. Whatever comes back becomes the tenant for the rest of the session, and every dashboard query is filtered by it.

Somebody whose email domain matches no tenant still signs in successfully, but lands on a holding page and can reach nothing else. That is deliberate: the person is told their organisation is not set up rather than being shown a login error they cannot act on.

Sessions live in a cookie for thirty days, marked HttpOnly and SameSite=Lax, and Secure when APP_ENV is production. The session token is rotated at sign-in.

What the dashboard shows

Tool registry is the full list of outside services seen, with how many people used each, how many events, when it was first and last seen, whether personal data was detected going to it, and whether it is an AI service. It exports to CSV.

Shadow discovery is the same data framed as the question an IT team actually asks, which is what is in use here that nobody approved.

Risk ranks person-and-service pairs and attaches a recommended action, so that sustained heavy use of a high-risk service reads differently from a one-day spike.

Signals groups by the kind of personal data detected, so you can see which people sent what category of data where. Only the last four characters of each pseudonym are shown.

Department exposure rolls services up into Engineering, Marketing, Sales, Finance, HR and Security. This is inferred from the category of the service, not from any HR system: if somebody uses a CRM, that use counts towards Sales. There is no org chart involved.

Leavers is the offboarding view. For one person it lists every outside service they touched, so that somebody leaving can be checked against the accounts that will still exist after they have gone.

Configuration edits the list of permitted domains. Anything on that list is excluded from the risk views, because a company that has approved a service does not want to be told about it every day.

How risk is scored

Each service starts from a base score set by its classification: high is 80, medium 50, unknown 40, low 20, and anything on the permitted list is 10. Signals are then added. Personal data detected adds 30, being an AI service adds 10, and recency, event volume and number of users add up to 15, 10 and 5 respectively. The total is divided by the theoretical maximum of 150 and expressed out of 100.

The weights are all in web/server/risk-scoring.go and the comment at the top of that file is the whole scheme in one place. They are a starting point that produced sensible orderings on real data, not a calibrated model, and they are meant to be edited.

The piece that is not in this repository

Classification comes from a separate program. web/server/api/domains.go runs /aurora-db --domain <domain> and reads JSON back:

[{"serviceName": "...", "domain": "...", "summary": "...",
  "risk": "high", "permitted": false, "categories": ["Sales / CRM"]}]

That binary is copied into the image by web/Dockerfile from a bin/ directory that this repository does not contain, so the Docker build will fail on that line until you supply one. You need one either way to run the service.

Aurora's own binary is part of Aurora's IP and is not published here, so get in touch if you want to run that one. Otherwise write your own service that speaks the contract above; anything that answers it will do.

Without a working lookup the server still ingests and stores events, and the raw data screens work, but every service comes back unclassified and scores 40.

Repository layout

web/server/          The Go server
  main.go            Routing, OIDC setup, health checks
  ingest.go          The /ingest endpoint: tenant lookup, validation, insert
  auth.go            Google OIDC sign-in and the auth middleware
  risk-scoring.go    The scoring weights, all in one file
  api/domains.go     Calls out to the aurora-db classification binary
  database/          ClickHouse connection and tenant lookup
  *.go               One file per dashboard screen, each a page and a data handler

web/app/             Templates, CSS and per-page JavaScript. No build step.
database/            ClickHouse image and the schema
aws/                 CloudFormation, Kubernetes manifests and deployment notes
tests/               A shell script that posts to /ingest

Running it locally

You need Docker and a Google OAuth client. Copy .env.example to .env and fill in GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET; the server refuses to start without both. The redirect URI on that OAuth client has to match AUTH_REDIRECT_URL exactly, which defaults to http://localhost:8080/auth/google/callback.

docker compose up --build

That brings up ClickHouse on 8123 and 9000 and the server on 8080. ClickHouse data lives in a Docker named volume, so it survives a restart and never lands in your working copy.

Before you can sign in you need a tenant row whose domain matches your email domain, and before an event will be accepted you need its public_api_key:

docker exec -it clickhouse clickhouse-client -q "
  INSERT INTO tenants (tenant_id, display_name, domain, public_api_key)
  VALUES (1, 'Example', 'example.com', 'pick-a-long-random-string')"

Then put that key in AURORA_PUBLIC_API_KEY in .env and run tests/test-ingest-api.sh, which posts one event that should be accepted and one that should be rejected. database/README.md covers querying the data, including the SQL console ClickHouse serves at http://localhost:8123/play.

Deploying it

aws/ holds the real deployment: CloudFormation for the EKS cluster and networking, Kubernetes manifests for the web server and for ClickHouse as two replicas backed by three Keeper nodes across three availability zones, and aws/deploy.md as the step-by-step. The third-party manifests it depends on, cert-manager and the AWS load balancer controller, are downloaded during the walkthrough rather than kept here, so you get versions that match what you are installing.

Every account-specific value is a placeholder. AWS_ACCOUNT_ID appears in the image references, app.example.com is the hostname, and aws/.env.example lists the rest.

Known gaps and future directions

The signature is not checked. The extension derives a signing key and signs every payload, and this server reads the signature field into a struct and then never looks at it. Authentication is the API key alone, which makes it a bearer token: anyone who obtains one can post whatever events they like for that tenant. Treat those keys accordingly until this is closed.

The extension and this server do not agree on the wire format. The extension sends tenant_id; this server expects api_key. One of the two has to move before they will talk to each other.

Both of these are work in progress and should be merged along shortly.

The schema says ON CLUSTER aurora_cluster. That is right for the Kubernetes deployment and wrong for the single-node local stack, which has no such cluster defined, so the init script will not run cleanly under docker-compose. Strip the ON CLUSTER clauses for local work.

Sessions are held in memory. Restarting the server signs everybody out, and the Kubernetes deployment runs two replicas with no shared session store and no sticky sessions, so a person will be bounced back to sign-in whenever they land on the other pod. This needs a shared store before it is comfortable in production.

The permitted-services list is one global file. It is not per tenant, and it is written to the container's own filesystem, so an edit does not survive a restart and does not reach the other replica. In a single-tenant deployment it behaves fine.

Health checks sit behind authentication. /livez and /readyz are wrapped in the same auth middleware as everything else, so an unauthenticated probe gets a redirect rather than a 200. The load balancer accepts it only because its success codes are set to 200-399. Move both routes outside the middleware if you want honest probes.

Ingest logs are chatty at Info level. Every event logs its domain and request ID at info level, which is useful while you are getting the extension talking to the server and is more than you want to keep once it works.

Replay protection is per process and best-effort. The ten-thousand-entry duplicate cache lives in one server's memory, so a duplicate that reaches the other replica, or one that arrives after ten thousand others, will be stored twice.

Extending it

A new dashboard screen is one Go file with a page handler and a data handler, one template, one JavaScript file, and a route in main.go. There is no framework and no build step to learn; the existing screens are all the same shape and any of them is a fair template.

Changing what counts as risky is editing the numbers at the top of risk-scoring.go. Changing what counts as approved is the permitted list. Changing what is classified how is the aurora-db contract, which is yours to implement.

Never edit an existing migration. Add a new SQL file. database/README.md has worked examples of altering a live ClickHouse table, including the awkward case of changing a column that is part of the primary key.

Getting in touch

Questions, bugs and patches are welcome as issues on this repository.

If you would like help deploying this, detectors or classifications built for your own data, or consulting work around it, we are happy to help: https://github.com/georgismitev.

Licence

The Aurora Identity Risk Engine uses the Business Source License 1.1 (BSL 1.1).

• It is free for individuals, hobbyists, researchers and open-source projects.
• It is free for organisations with fewer than 50 employees and under $1,000,000 in annual gross revenue.
• It is free for all internal use.
• Offering it to third parties as a hosted, SaaS or managed service requires a commercial licence from Aurora Identity.
• It becomes Apache 2.0 automatically on September 9, 2030.

Full terms are in LICENSE.