Add profile registry and URL-based profile selection

CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## [0.3.0] - 2026-05-19
+
+### Added
+- Explicit per-path S3 profile selection via the URL userinfo slot:
+  `s3://<profile>@bucket/key` ([#8](https://github.com/Positronic-Robotics/pos3/issues/8)).
+  A profile in the URL takes precedence over the `profile=` argument and works
+  for every pos3-powered CLI without code or env-var changes.
+- Name-keyed local profile registry auto-loaded from
+  `~/.config/pos3/profiles.toml` (override with `POS3_PROFILES_FILE`).
+  Non-secret config (`endpoint`/`region`/`local_name`/`public`) is kept
+  separate from a secret `credentials_file`.
+- Profiles with explicit credentials build their own isolated `boto3.Session`,
+  never reading or mutating the user's ambient AWS configuration.
+- Unknown profile names (in URL or argument) are a hard error with no silent
+  fallback to the default credential chain.
+
+### Changed
+- **Python 3.11+ is now required** (`requires-python = ">=3.11"`). The TOML
+  profile registry uses the standard-library `tomllib`; the previously
+  declared 3.9/3.10 support was never exercised by CI.
+- Profile logic (the `Profile` dataclass, registry loading, resolution, and
+  client creation) moved into the internal `pos3.profiles` module. The public
+  API (`pos3.Profile`, `pos3.register_profile`) is unchanged.
+
 ## [0.2.2] - 2026-02-06
 
 ### Fixed

README.md

@@ -173,3 +173,52 @@ with pos3.mirror(default_profile='nebius-public'):
 ```
 
 Each profile has a `local_name` used in the cache path to keep files from different endpoints separate. When registering profiles, `local_name` defaults to the profile name. The default AWS profile uses `_` as its local name.
+
+#### Explicit profile selection in the URL
+
+For CLI tools where the S3 path is the only thing the user controls, a profile
+can be selected directly in the URL using the userinfo slot:
+
+```bash
+some-pos3-cli --dataset.path=s3://acme@bucket/dataset/
+```
+
+- The scheme stays `s3://`, so the URL still parses with standard tooling.
+- pos3 extracts the profile name, resolves it, and strips it before talking to
+  boto3 — no changes are needed in consuming tools.
+- A profile in the URL **takes precedence** over an explicit `profile=`
+  argument.
+- An unknown profile is a **hard error** — there is no silent fallback to the
+  default credential chain.
+- No userinfo means the behavior is unchanged (boto3 default chain or the
+  context default profile).
+
+#### Local profile registry
+
+pos3 auto-loads named profiles from `~/.config/pos3/profiles.toml` (override
+the location with the `POS3_PROFILES_FILE` environment variable, or use
+`XDG_CONFIG_HOME`). This lets a recipient run any pos3-powered CLI against a
+custom S3-compatible endpoint with isolated credentials — no env vars, no code,
+no collision with their own AWS setup:
+
+```toml
+[profiles.acme]
+endpoint = "https://s3.example-provider.com"
+region   = "us-east-1"
+# Optional: secret kept in a separate file (AWS-style INI).
+credentials_file = "~/.config/pos3/acme.creds"
+```
+
+```ini
+# ~/.config/pos3/acme.creds
+[acme]
+aws_access_key_id = AKIA...
+aws_secret_access_key = ...
+# aws_session_token = ...   # optional
+```
+
+The non-secret config (endpoint/region) is safe to share or commit; the
+credentials file is the only secret and lives outside code. Each profile builds
+its **own isolated `boto3.Session`**, so it never reads or mutates the user's
+ambient AWS configuration in either direction. Programmatically registered
+profiles take precedence over registry entries of the same name.

pos3/__init__.py

@@ -16,97 +16,19 @@
 from typing import Any
 from urllib.parse import urlparse
 
-import boto3
-from botocore import UNSIGNED
-from botocore.config import Config
 from botocore.exceptions import ClientError
 from tqdm import tqdm
 
-logger = logging.getLogger(__name__)
-
-
-@dataclass(frozen=True)
-class Profile:
-    """Configuration for an S3-compatible endpoint.
-
-    Attributes:
-        local_name: Identifier used in cache path (e.g., 'nebius'). Cannot be '_' (reserved).
-        endpoint: S3 endpoint URL (e.g., 'https://storage.eu-north1.nebius.cloud').
-        public: If True, use anonymous access (no credentials required).
-        region: Optional AWS region name.
-    """
-
-    local_name: str
-    endpoint: str
-    public: bool = False
-    region: str | None = None
-
-    def __post_init__(self):
-        if self.local_name == "_":
-            raise ValueError("Profile local_name cannot be '_' (reserved for default)")
-        if not self.local_name or not all(c.isalnum() or c in "-_" for c in self.local_name):
-            raise ValueError(f"Invalid local_name '{self.local_name}': use only alphanumeric, dash, underscore")
-
-
-_PROFILES: dict[str, Profile] = {}
-
-
-def register_profile(
-    name: str,
-    endpoint: str,
-    public: bool = False,
-    region: str | None = None,
-    local_name: str | None = None,
-) -> None:
-    """Register a named profile for S3 access.
-
-    Creates a Profile with the given parameters. See Profile class for field details.
-    The `local_name` defaults to the profile `name` if not specified.
-    """
-    config = Profile(local_name=local_name or name, endpoint=endpoint, public=public, region=region)
-    existing = _PROFILES.get(name)
-    if existing is not None and existing != config:
-        raise ValueError(f"Profile '{name}' already registered with different config")
-    _PROFILES[name] = config
-
-
-def _resolve_profile(profile: str | Profile | None) -> Profile | None:
-    """Resolve a profile name to a Profile object.
-
-    Args:
-        profile: None, registered profile name (string), or Profile object.
-
-    Returns:
-        Profile object or None.
-
-    Raises:
-        ValueError: If profile is a string that is not registered.
-    """
-    if profile is None or isinstance(profile, Profile):
-        return profile
-    if profile not in _PROFILES:
-        raise ValueError(f"Unknown profile: '{profile}'. Register with pos3.register_profile() first.")
-    return _PROFILES[profile]
-
-
-def _create_s3_client(profile: Profile | None = None):
-    """Create boto3 S3 client, optionally using a profile.
-
-    Args:
-        profile: None (use boto3 defaults) or Profile config.
-    """
-    if profile is None:
-        return boto3.client("s3")
-
-    kwargs: dict[str, Any] = {"endpoint_url": profile.endpoint}
-
-    if profile.region:
-        kwargs["region_name"] = profile.region
+from .profiles import _PROFILES as _PROFILES  # re-exported for tests/back-compat
+from .profiles import (
+    Profile,
+    _create_s3_client,
+    _resolve_profile,
+    _url_profile,
+    register_profile,
+)
 
-    if profile.public:
-        kwargs["config"] = Config(signature_version=UNSIGNED)
-
-    return boto3.client("s3", **kwargs)
+logger = logging.getLogger(__name__)
 
 
 class _NullTqdm(nullcontext):
@@ -121,7 +43,12 @@ def _parse_s3_url(s3_url: str) -> tuple[str, str]:
     parsed = urlparse(s3_url)
     if parsed.scheme != "s3":
         raise ValueError(f"Not an S3 URL: {s3_url}")
-    return parsed.netloc, parsed.path.lstrip("/")
+    netloc = parsed.netloc
+    if "@" in netloc:
+        # Drop the explicit profile selector from the userinfo slot; it is a
+        # routing hint, not part of the bucket identity.
+        netloc = netloc.rsplit("@", 1)[1]
+    return netloc, parsed.path.lstrip("/")
 
 
 def _normalize_s3_url(s3_url: str) -> str:
@@ -327,8 +254,17 @@ def __init__(self, options: _Options):
         self._stop_event: threading.Event | None = None
         self._sync_thread: threading.Thread | None = None
 
-    def _effective_profile(self, profile: str | Profile | None) -> Profile | None:
-        """Resolve profile name and substitute default if None."""
+    def _effective_profile(self, profile: str | Profile | None, remote: str | None = None) -> Profile | None:
+        """Resolve the effective profile for an operation.
+
+        An explicit profile in the S3 URL userinfo slot
+        (`s3://<profile>@bucket/key`) takes precedence over the `profile=`
+        argument. Falls back to the context default when nothing is specified.
+        """
+        if remote is not None and _is_s3_path(remote):
+            url_profile = _url_profile(remote)
+            if url_profile is not None:
+                profile = url_profile
         resolved = _resolve_profile(profile)
         return resolved if resolved is not None else self._default_profile
 
@@ -383,7 +319,7 @@ def download(
             FileNotFoundError: If remote is a local path that does not exist.
             ValueError: If download registration conflicts with an existing download or upload or parameters differ.
         """
-        effective_profile = self._effective_profile(profile)
+        effective_profile = self._effective_profile(profile, remote)
 
         if not _is_s3_path(remote):
             path = Path(remote).expanduser().resolve()
@@ -459,7 +395,7 @@ def upload(
         Raises:
             ValueError: If upload registration conflicts with an existing download or upload or parameters differ.
         """
-        effective_profile = self._effective_profile(profile)
+        effective_profile = self._effective_profile(profile, remote)
 
         if not _is_s3_path(remote):
             path = Path(remote).expanduser().resolve()
@@ -516,14 +452,14 @@ def sync(
             return local_path
 
         normalized = _normalize_s3_url(remote)
-        effective_profile = self._effective_profile(profile)
+        effective_profile = self._effective_profile(profile, remote)
         # Unregister the download to allow upload registration for the same remote
         self._downloads.pop((normalized, effective_profile), None)
         return self.upload(remote, local_path, interval, delete_remote, sync_on_error, exclude, profile)
 
     def ls(self, prefix: str, recursive: bool = False, profile: str | Profile | None = None) -> list[str]:
         """Lists objects under the given prefix, working for both local directories and S3 prefixes."""
-        effective_profile = self._effective_profile(profile)
+        effective_profile = self._effective_profile(profile, prefix)
 
         if _is_s3_path(prefix):
             normalized = _normalize_s3_url(prefix)