INFound Influencer Platform (Backend + Data Collection + Desktop Client)

This workspace combines FastAPI backend services, Playwright- and RabbitMQ-powered data collectors, and a desktop Electron client to showcase the sample ingestion, creator outreach, seller-side RPA, and chatbot automation flows that support the INFound influencer pipeline. The materials are curated for an UCSD MS committee presentation, so every sensitive string has been sanitized.

Architecture snapshot

• infound-backend-services houses the internal ingestion API, creator portals, request filters, scheduler, and MQ producers that run inside FastAPI processes.
• infound-data-collection implements RabbitMQ consumers, Playwright crawlers, and chatbot dispatchers (now covering partner, creator, and seller/shop portals) that normalize TikTok data and hand it to the backend.
• infound-desktop-client is a pnpm monorepo built with Electron + Vue 3 + TypeScript; it contains the operator desktop app plus an RPA simulation app for seller login, outreach, sample-management, chatbot, and creator-detail flows.
• Shared helpers in common/ modules centralize multi-environment config, structured logging, SQLAlchemy/Redis sessions, RabbitMQ connections, and model definitions so business logic can stay focused.

Workspace layout

.
├── infound-backend-services/   # FastAPI APIs, schedulers, MQ producers
├── infound-data-collection/    # Playwright crawlers, RabbitMQ consumers, chatbots
└── infound-desktop-client/     # Electron desktop app + seller RPA simulation app

Backend services

• Internal ingestion APIs under apps/portal_inner_open_api rely on SampleIngestionService, RequestFilterMiddleware, and Pydantic models to validate rows, deduplicate samples/sample_contents, persist *_crawl_logs, and maintain sample_chatbot_schedules.
• ChatbotSchedulePublisher periodically polls due schedules, builds follow-up content via chatbot_message_builder, and publishes tasks to chatbot.topic through the shared RabbitMQProducer; /chatbot/messages accepts prebuilt payloads as well.
• Creator portals under apps/portal_creator_open_api provide login plus sample/chatbot endpoints, while the inner API exposes /creators/ingest, /outreach_tasks/ingest, and /chatbot/messages so crawlers can ingest creator metadata, sync outreach task state, and request message dispatch.
• Common infrastructure (common/core/config.py, common/core/logger.py, common/core/database.py, common/services/redis_service.py, etc.) keeps configuration, logging, database pooling, and Redis helpers consistent across services.

Data collection services

• common/mq contains a resilient RabbitMQConnection and ConsumerBase that every consumer inherits; they auto-handle reconnection, error classification (MessageProcessingError, ValueError), and JSON parsing with contextual logs.
• apps/portal_tiktok_sample_crawler_html rebuilds the legacy sample_all.py/sample_process.py workflow with CrawlerRunnerService, Playwright browser pools, Gmail code handlers, account/region selection, tab navigation, view-content/logistics extraction, and data normalization/export routines that still align with samples, sample_contents, and sample_chatbot_schedules through SampleIngestionClient.
• apps/portal_tiktok_campaign_crawler logs into TikTok Shop Partner Center, crawls campaign/product tables, optionally exports Excel snapshots, and posts normalized payloads to /campaigns/ingest and /products/ingest via CampaignIngestionClient.
• apps/portal_tiktok_creator_crawler gathers creator metadata, contact info, and outreach-relevant stats, then posts them to the inner API endpoints for creator ingestion, outreach task syncing, and chatbot scheduling.
• apps/portal_tiktok_shop_creator_crawler targets the seller/shop portal to pull creator-connection lists, persist them through /creators/ingest, and (optionally) publish shop-outreach chatbot tasks to chatbot.topic.
• apps/sample_chatbot, apps/outreach_chatbot, and apps/unified_chatbot consume the chatbot.topic exchange, respect DLQs, and use Playwright instances to send sample and outreach messages, while providing manual overrides for accounts/regions and retry limits.
• apps/portal_tiktok_shop_chatbot consumes shop-outreach messages (topic-based) and drives seller-portal IM to creators with the same Playwright/Gmail fallback stack as other chatbots.
• RabbitMQ tasks are routed through crawler.direct, chatbot.topic, and their DLQs; each consumer can also be run with AT_MOST_ONCE semantics or batched delivery and can export normalized rows to data/manage_sample/ or data/manage_campaign/ when exportExcel is enabled.

Desktop client

• apps/frontend.desktop is the operator-facing Electron app. It splits logic across Electron main/preload/renderer layers, persists non-sensitive state via electron-store, and stores tokens in the OS credential store.
• apps/frontend.rpa.simulation is a lightweight Electron shell for seller-side Playwright execution. It can start a reusable browser session and dispatch independent tasks for seller login, outreach, sample management, chatbot messaging, and creator-detail extraction.
• packages/frontend.desktop.shared holds shared RPA abstractions, task executors, logger helpers, time helpers, and cross-app TypeScript types.
• The desktop client complements the backend/data-collection services: it is primarily for interactive operator workflows and RPA debugging, while the Python services remain the production ingestion and MQ automation path.

Committee showcase: sample crawler, campaign crawler, creator crawler, and chatbots

• Sample crawler story: the Playwright-backed portal_tiktok_sample_crawler_html service drives [region, tab, campaign] exploration, extracts content/logistics/promotions, enforces the same normalization rules as sample_process.py, exports verification files, then hands every batch to /samples/ingest via SampleIngestionClient. This chain demonstrates the playbook for how TikTok sample requests become structured data.
• Campaign crawler story: portal_tiktok_campaign_crawler logs into Partner Center, scans campaigns or specific IDs, opens campaign detail tables, exports optional spreadsheets, and ships campaign/product rows to /campaigns/ingest and /products/ingest for catalog seeding and monitoring.
• Creator crawler story: portal_tiktok_creator_crawler logs into the partner creator portal, scrolls results, captures creator stats/contact info, and calls /creators/ingest//outreach_tasks/ingest to sync records with the backend. It showcases how outreach datasets and creator contact pipelines are seeded before any chatbot outreach is scheduled.
• Seller/shop outreach story: portal_tiktok_shop_creator_crawler logs into the seller portal creator-connection page, applies keyword/brand filters, sends normalized creator rows to /creators/ingest, and emits chatbot.shop_outreach.* MQ messages so the shop chatbot can DM those creators without human intervention.
• Chatbot story: sample_chatbot, outreach_chatbot, and unified_chatbot consume chatbot.topic, honor DLQs, and use Playwright to send the prepared sequences from the inner API. The scheduler + RabbitMQ flow demonstrates how the platform closes the loop, moving from discovery (sample & creator crawlers) to automated engagement without human-in-the-loop tweaks once the MQ payloads arrive.

Data pipeline overview

1. Operators publish RabbitMQ jobs (typically function: "sample" or outreach tasks) to crawler.direct.
2. Playwright crawlers log into TikTok partner portals, read tab states, parse actions/logistics/promotions, and normalize each row into the schema the backend expects.
3. Normalized rows or creator/outreach payloads are POSTed to the inner API (/samples/ingest, /creators/ingest, /outreach_tasks/ingest), with X-INFound-Inner-Service-Token headers validated by RequestFilterMiddleware.
4. The inner API upserts MySQL tables, writes sample_chatbot_schedules, and publishes chatbot jobs via RabbitMQProducer to chatbot.topic with routing keys like chatbot.sample.batch or chatbot.outreach.batch.
5. sample_chatbot/outreach_chatbot/unified_chatbot consume those jobs, launch Playwright sessions, and send the prepared message sequences; failures go to DLQs for manual or scripted re-processing.

Getting started

Backend services

cd infound-backend-services
poetry install
export SERVICE_NAME=portal_inner_open_api
export ENV=dev
poetry run python main.py

• Switch SERVICE_NAME to portal_creator_open_api or other services to run additional FastAPI apps.
• Override configs/base.yaml placeholders by creating environment-specific YAML files or by setting environment variables (e.g., MYSQL_HOST, RABBITMQ__PASSWORD, AUTH__VALID_TOKENS).

Data collection services

cd infound-data-collection
poetry install --no-root
poetry run playwright install
export SERVICE_NAME=portal_tiktok_sample_crawler_html
export ENV=dev
poetry run python main.py --consumer portal_tiktok_sample_crawler_html --env "$ENV"

• Swap SERVICE_NAME/--consumer to run portal_tiktok_campaign_crawler, portal_tiktok_creator_crawler, sample_chatbot, outreach_chatbot, or unified_chatbot.
• Playwright credentials, Gmail app passwords, and RabbitMQ/inner API hosts are loaded through environment variables or local YAML overrides (see common/core/config.py for aliasing rules).
• Replace every <...> placeholder in the YAML configs with real values before deploying (RabbitMQ hosts, MySQL credentials, inner API tokens, etc.).

Desktop client

cd infound-desktop-client
pnpm install

# Operator desktop app
pnpm --filter frontend.desktop dev

# Seller RPA simulation app
pnpm --filter frontend.rpa.simulation dev

• Production/staging endpoints in apps/frontend.desktop/.env.* have been replaced with placeholders; define local values before running against a real environment.
• The desktop app uses Electron main-process services for auth and state persistence, while the RPA simulation app expects local Playwright runtime data under ignored data/ paths.
• See infound-desktop-client/README.md and the infound-desktop-client/docs/ folder for desktop architecture, IPC contracts, and RPA payload examples.

Startup checklist (committee demo)

Sample crawler (Playwright HTML version)

cd infound-data-collection
export SERVICE_NAME=portal_tiktok_sample_crawler_html
export ENV=dev
export PLAYWRIGHT_HEADLESS=false
poetry run python main.py --consumer portal_tiktok_sample_crawler_html --env "$ENV"

Campaign crawler

cd infound-data-collection
export SERVICE_NAME=portal_tiktok_campaign_crawler
export ENV=dev
export PLAYWRIGHT_HEADLESS=false
poetry run python main.py --consumer portal_tiktok_campaign_crawler --env "$ENV"

Creator crawler

cd infound-data-collection
export SERVICE_NAME=portal_tiktok_creator_crawler
export ENV=dev
export PLAYWRIGHT_HEADLESS=false
poetry run python main.py --consumer portal_tiktok_creator_crawler --env "$ENV"

Seller portal creator crawler (shop side)

cd infound-data-collection
export SERVICE_NAME=portal_tiktok_shop_creator_crawler
export ENV=dev
export PLAYWRIGHT_HEADLESS=false          # set true for headless
# Optional overrides:
# export SHOP_OUTREACH_DEFAULT_REGION=MX
# export SHOP_OUTREACH_ACCOUNT_NAME=MX1   # match configs/accounts.json
poetry run python main.py --consumer portal_tiktok_shop_creator_crawler --env "$ENV"

Sample chatbot

cd infound-data-collection
export SERVICE_NAME=sample_chatbot
export ENV=dev
poetry run python main.py --consumer sample_chatbot --env "$ENV"

Outreach chatbot

cd infound-data-collection
export SERVICE_NAME=outreach_chatbot
export ENV=dev
poetry run python main.py --consumer outreach_chatbot --env "$ENV"

Unified chatbot (sample + outreach)

cd infound-data-collection
export SERVICE_NAME=unified_chatbot
export ENV=dev
poetry run python main.py --consumer unified_chatbot --env "$ENV"

Shop chatbot (seller IM to creators)

cd infound-data-collection
export SERVICE_NAME=portal_tiktok_shop_chatbot
export ENV=dev
export PLAYWRIGHT_HEADLESS=false          # false to watch browser
# Optional: export SHOP_CHATBOT_ACCOUNT_NAME=MX1  # pick account from configs/accounts.json
poetry run python main.py --consumer portal_tiktok_shop_chatbot --env "$ENV"

Security & sanitization

This presentation copy removes all secrets: IP addresses, hostnames, tokens, Gmail credentials, passwords, demo account identifiers, and embedded git remotes are shown as <PLACEHOLDER> values or example.com domains. Treat the placeholders in every YAML or .env file (infound-backend-services/configs/base.yaml, infound-data-collection/configs/base.yaml, infound-desktop-client/apps/frontend.desktop/.env.*, etc.) as prompts to load production values from a secure store before running.

Additional pointers

• Refer to infound-backend-services/readme.md for detailed sample ingestion contracts, chatbot scheduling, creator/outreach APIs, and SQL model generation guidance.
• Refer to infound-data-collection/readme.md for consumer configuration, RabbitMQ routing keys, Playwright behavior, data exports, and chatbot/creator crawler operations.