Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# The canonical scanner is shared byte-for-byte with the sibling repositories.
# Without this rule, a Windows checkout (core.autocrlf=true) would rewrite it to
# CRLF on disk and break the published SHA-256. Scoped to that one file on purpose.
scripts/check_no_client_leak.py text eol=lf
118 changes: 0 additions & 118 deletions .github/scripts/check_client_leak.py

This file was deleted.

8 changes: 6 additions & 2 deletions .github/workflows/no-client-leak.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Scan tracked files for customer names / hard-coded identity GUIDs
run: python .github/scripts/check_client_leak.py
- name: Scan tracked files for leaked identifiers / customer names
# The denylist lives in a secret, never in the repo. If the secret is not
# configured the name rule is skipped with a warning — never a failure.
env:
CLIENT_DENYLIST: ${{ secrets.CLIENT_DENYLIST }}
run: python scripts/check_repo_leaks.py
- name: Ensure no PowerPoint deck is tracked
run: |
if git ls-files '*.pptx' '*.ppt' | grep .; then
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,9 @@ src/config.yaml
src/state.json
.vscode/mcp.json

# Local client denylist for scripts/check_repo_leaks.py — one entry per line.
# Names are detected, never published: in CI the same list comes from the
# ${{ secrets.CLIENT_DENYLIST }} secret. This file must stay untracked.
.clientdeny

# (The *.example files ARE committed: src/config.example.yaml, src/state.example.json, .vscode/mcp.example.json)
23 changes: 20 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,9 @@
## Project layout

```
.github/workflows/ — CI: client-leak guard + pytest
.github/scripts/ — check_client_leak.py (run it locally before pushing)
.github/workflows/ — CI: leak guard + pytest
scripts/ — check_no_client_leak.py (canonical, shared verbatim)
+ check_repo_leaks.py (entry point; run before pushing)
docs/ARCHITECTURE.md — full design + data model + demo storyboard
src/config.yaml — workspace, item names, topology + telemetry config
src/generate_data.py — synthetic topology + telemetry (with the scripted PFC storm)
Expand Down Expand Up @@ -176,10 +177,26 @@ This is a **synthetic** demo: sites are generic French metro areas, and every cu
service and hardware vendor is fictional. Two guards enforce it, in CI and locally:

```bash
python .github/scripts/check_client_leak.py # customer names + hard-coded identity GUIDs
python scripts/check_repo_leaks.py # real GUIDs, Fabric endpoints, personal paths
python -m pytest tests/ -q
```

The scan reads `git ls-files` only — never the working tree, because `__pycache__/*.pyc`
and `node_modules` embed absolute build paths and produce false positives.

**Detection is by shape, never by name.** `scripts/check_no_client_leak.py` is a
byte-identical copy of the canonical scanner shared across the sibling repos; do not
edit it, configure it from `scripts/check_repo_leaks.py` instead.

An optional customer-name rule is driven by a list that is **never committed**:

- in CI, the `CLIENT_DENYLIST` repository secret (one entry per line);
- locally, a `.clientdeny` file at the repo root (git-ignored, same format, `#` comments).

Without it the rule is skipped with a warning — it never fails the build. Matches are
reported as `file:line` plus the entry rank only, so the terms are not echoed back into
public CI logs.

---

## Key facts
Expand Down
130 changes: 130 additions & 0 deletions scripts/check_no_client_leak.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
#!/usr/bin/env python3
"""Fail if real customer / tenant identifiers leak into Git-tracked files.

This repo is public. It must never contain real Microsoft Fabric identifiers,
real workspace SQL endpoints, or personal filesystem paths.

Only files returned by ``git ls-files`` are scanned. The working tree is
deliberately NOT walked: ``__pycache__/*.pyc`` and ``node_modules`` embed
absolute build paths and would produce false positives.

Usage:
python scripts/check_no_client_leak.py

Exit code 0 = clean, 1 = leak detected.
"""

from __future__ import annotations

import os
import re
import subprocess
import sys
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parent.parent

# Placeholder GUIDs that are allowed to appear verbatim.
ALLOWED_GUIDS = {
"00000000-0000-0000-0000-000000000000",
}

BINARY_SUFFIXES = {
".png", ".jpg", ".jpeg", ".gif", ".ico", ".pdf", ".pptx", ".xlsx",
".docx", ".zip", ".gz", ".woff", ".woff2", ".ttf", ".eot",
}

GUID_RE = re.compile(
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-"
r"[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
)

# A real Fabric SQL/warehouse endpoint host: a long opaque token followed by
# ".datawarehouse.fabric.microsoft.com" (or the Synapse / Power BI equivalents).
FABRIC_ENDPOINT_RE = re.compile(
r"\b[a-z0-9]{20,}(?:-[a-z0-9]{20,})?\."
r"(?:datawarehouse|dev\.azuresynapse|pbidedicated)\b",
re.IGNORECASE,
)

# Personal home directories. Placeholders such as C:\Users\<you> are fine.
PERSONAL_PATH_RE = re.compile(
r"(?:[A-Za-z]:\\Users\\|/Users/|/home/)(?!<|\{|\$|%|USERNAME\b|username\b|you\b|user\b)"
r"[A-Za-z0-9._-]+",
)

# Extra literals to forbid, as a comma-separated list. Deliberately empty by
# default: hardcoding an author's username here would re-publish the very thing
# this checker exists to remove.
# NO_LEAK_EXTRA_LITERALS="acme,contoso" python scripts/check_no_client_leak.py
FORBIDDEN_LITERALS = [
literal.strip().lower()
for literal in os.environ.get("NO_LEAK_EXTRA_LITERALS", "").split(",")
if literal.strip()
]

CHECKS = [
("real GUID", GUID_RE, lambda m: m.group(0).lower() not in ALLOWED_GUIDS),
("Fabric SQL endpoint", FABRIC_ENDPOINT_RE, lambda m: True),
("personal filesystem path", PERSONAL_PATH_RE, lambda m: True),
]


def tracked_files() -> list[str]:
out = subprocess.run(
["git", "ls-files", "-z"],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=True,
).stdout
return [p for p in out.split("\0") if p]


def scan_file(rel_path: str) -> list[str]:
path = REPO_ROOT / rel_path
if path.suffix.lower() in BINARY_SUFFIXES or not path.is_file():
return []
try:
text = path.read_text(encoding="utf-8", errors="ignore")
except OSError:
return []

findings = []
for lineno, line in enumerate(text.splitlines(), start=1):
for label, pattern, keep in CHECKS:
for match in pattern.finditer(line):
if keep(match):
findings.append(
f"{rel_path}:{lineno}: {label}: {match.group(0)}"
)
lowered = line.lower()
for literal in FORBIDDEN_LITERALS:
if literal in lowered:
findings.append(
f"{rel_path}:{lineno}: author-specific marker: {literal}"
)
return findings


def main() -> int:
findings: list[str] = []
for rel_path in tracked_files():
findings.extend(scan_file(rel_path))

if findings:
print("Client / tenant data leak detected in Git-tracked files:\n")
for finding in findings:
print(f" {finding}")
print(
"\nReplace real values with visibly fake placeholders "
"(e.g. <YOUR_WORKSPACE_ID> or 00000000-0000-0000-0000-000000000000)."
)
return 1

print("No client data leak detected in Git-tracked files.")
return 0


if __name__ == "__main__":
sys.exit(main())
Loading
Loading