Enhance social media support and improve code quality

.env

@@ -0,0 +1,4 @@
+# Environment variables for profile-checker
+# Available levels: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET
+LOG_LEVEL=DEBUG
+LOG_FILE=finder.log

.github/workflows/pylint.yml

@@ -17,6 +17,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
+        pip install -r requirements.txt
         pip install pylint
     - name: Analysing the code with pylint
       run: |

README.md

@@ -1,78 +1,52 @@
-# scan
-scan is an OSINT microservice that searches the [WhatsMyName](https://github.com/WebBreacher/WhatsMyName) dataset for username hits from various websites.
-The service uses a database to cache search results because validating usernames is costly and time-consuming. Cached results are reused on repeat lookups, and a refresh option exists to rebuild the results.
-A status endpoint is exposed so clients can poll long-running searches instead of relying on a single request timeout.
+# profile-checker
+
+Profile validation microservice called `profile-checker` that checks whether usernames exist on social media platforms. The service validates profiles by performing HTTP requests to platform-specific endpoints and confirms existence through response validation.
 
 ## Requirements
 - Python 3.9 or higher
 
 ### Tech Stack
 - The REST API framework being used is [FastAPI](https://fastapi.tiangolo.com/)
-- The database engine is [SQLite](https://www.sqlite.org/index.html)
 - The REST client being used is [httpx](https://www.python-httpx.org/)
 
 ### Setup
-1. Create Python virtual environment. `python -m venv venv`
-2. Activate virtual environment. `source venv/bin/activate`
+1. Create Python virtual environment. `python -m venv .venv`
+2. Activate virtual environment. `source .venv/bin/activate`
 3. Install dependencies. `pip install -r requirements.txt`
-4. Start server. `fastapi dev main.py`
+4. Create a `.env` file with `LOG_LEVEL` and `LOG_FILE` as needed.
+5. Start server. `fastapi dev main.py`
+
+### Configuration
+- `config.py` initializes logging and reads environment variables.
+- `.env` supports:
+  - `LOG_LEVEL` — log verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`)
+  - `LOG_FILE` — path to the log output file
+- Site verification templates are stored in `sites.yaml`.
 
 ### Endpoints
 - GET `/scan/{username}`
-    * Starts a new search if the username has not been cached yet.
-    * Returns `202 Accepted` when a new background search is started.
-    * Returns `102 Processing` when a search for the username is already pending or in progress.
-    * Returns cached results with `200 OK` if the username has already been searched and no refresh is requested.
-    * Query Parameters:
-        * `refresh` (`true` or `false`, default `false`) — when set to `true`, the cache is invalidated and the username search is rebuilt.
+    * Validates whether the given username exists on supported platforms.
+    * Returns `200 OK` with results containing found profiles.
+    * Performs HEAD requests to platform validation endpoints and follows up with GET requests to confirm profile existence.
 
-- GET `/scan/status/{username}`
-    * Returns the current status of a background search.
-    * Use this endpoint to poll progress until the search reaches `completed`.
-
-### Example response for `/scan/{username}` when cached
+### Example response for `/scan/{username}`
 ```json
 [
   {
-    "title": "GitHub",
-    "uri": "https://github.com/osint-services",
-    "profile_url": "https://github.com/osint-services",
-    "category": "coding",
-    "search_timestamp": "2024-11-02 17:01:27",
-    "found_timestamp": "2024-11-02 17:01:28"
-  },
-  {
-    "title": "Facebook",
-    "uri": "https://facebook.com/osint-services",
-    "profile_url": "https://facebook.com/osint-services",
-    "category": "social",
-    "search_timestamp": "2024-11-02 17:01:27",
-    "found_timestamp": "2024-11-02 17:01:28"
+    "title": "X",
+    "profile_uri": "https://x.com/osint-services",
+    "validation_uri": "https://api.x.com/i/users/username_available.json?username=osint-services",
+    "is_valid_profile": true
   }
 ]
 ```
 
-### Example response for `/scan/status/{username}`
-```json
-{
-  "status": "in_progress",
-  "found_sites": [
-    {
-      "title": "GitHub",
-      "uri": "https://github.com/osint-services",
-      "profile_url": "https://github.com/osint-services",
-      "category": "coding",
-      "search_timestamp": "2024-11-02 17:01:27",
-      "found_timestamp": "2024-11-02 17:01:28"
-    }
-  ],
-  "total_sites": 200,
-  "checked_sites": 50
-}
-```
-
 ### Notes
-- This service uses background processing and polling rather than long-lived request timeouts.
-- The result validation step checks page content and common profile signals to reduce false positives from login, not-found, or generic landing pages.
-- If a search fails, `GET /scan/status/{username}` will return a `failed` status and an `error` field may be included in the response.
+- Profile validation checks page responses and validation endpoints to confirm actual profile existence and reduce false positives.
+- The service currently supports X (Twitter) profile validation.
+- Logging is configured in `config.py` and writes to both console and the file specified by `LOG_FILE`.
+- Supported site templates are defined in `sites.yaml`, making it easy to add or update platform endpoints without changing application logic.
+- `LOG_LEVEL` and `LOG_FILE` can be configured via `.env`.
+- The project root already includes `__init__.py`, so the package can be imported if needed, but script execution does not require additional `__init__.py` files.
+- CORS is enabled for localhost:3000 frontend connections.
 

config.py

@@ -0,0 +1,30 @@
+"""Application configuration and logging setup."""
+
+import logging
+import os
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+LOG_LEVEL = os.getenv("LOG_LEVEL", "DEBUG").upper()
+if LOG_LEVEL not in {"CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET"}:
+    LOG_LEVEL = "DEBUG"
+
+LOG_FILE = os.getenv("LOG_FILE", "finder.log")
+
+logger = logging.getLogger("profile-checker")
+logger.setLevel(LOG_LEVEL)
+
+ch = logging.StreamHandler()
+ch.setLevel(LOG_LEVEL)
+
+fh = logging.FileHandler(LOG_FILE)
+fh.setLevel(LOG_LEVEL)
+
+formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+ch.setFormatter(formatter)
+fh.setFormatter(formatter)
+
+logger.addHandler(ch)
+logger.addHandler(fh)