Set up SNAP cluster automation server for uutils services

[brando90]

Summary

Plan and document how to set up a persistent automation server on the Stanford SNAP cluster to run uutils automation services (email, Google Drive sync, social media posting, Slack/Zulip notifications).

Context & Purpose

This SNAP cluster setup is exclusively for research and educational purposes. The automation services support:

• Sharing research results (plots, papers) via email, Slack, Zulip
• Syncing experiment data/images via Google Drive
• Posting research updates to social media
• Monitoring research group communications

This is part of Brando's ML/AI and programming languages research at Stanford.

Proposed Architecture

Option 1: Persistent tmux/screen session on a login node

• Simplest: SSH into SNAP, start a tmux session, run cron jobs
• Downside: login nodes may have usage policies against long-running processes
• Best for: lightweight periodic tasks (cron every N minutes)

Option 2: SLURM job with long wall-time

• Submit a lightweight job that runs automation scripts periodically
• Can request minimal resources (1 CPU, minimal RAM)
• Downside: job may get preempted or hit wall-time limits

Option 3: Dedicated screen/tmux + systemd user service

• Create a systemd user service (if allowed) for persistent background tasks
• Most robust for always-on automation

Recommended approach

Option 1 for simplicity. Use cron on a SNAP login node for periodic tasks:

# Example crontab
# Sync Drive images every 30 minutes
*/30 * * * * cd ~/ultimate-utils && python -c "from uutils.gdrive_uu import sync_drive_folder; sync_drive_folder('FOLDER_ID', '~/drive_images')" >> ~/logs/gdrive_sync.log 2>&1

# Send daily research summary email
0 18 * * * cd ~/ultimate-utils && python scripts/daily_summary.py >> ~/logs/daily_summary.log 2>&1

Setup Steps

1. Environment setup on SNAP

ssh brando9@snap-gpu.stanford.edu
cd ~
git clone https://github.com/brando90/ultimate-utils.git  # or git pull
pip install -e ~/ultimate-utils

2. Credentials setup

mkdir -p ~/keys && chmod 700 ~/keys

# Google Drive service account
# (copy from local machine via scp)
scp ~/keys/gdrive_service_account.json brando9@snap-gpu:~/keys/
chmod 600 ~/keys/gdrive_service_account.json

# Gmail app password
scp ~/keys/gmail_app_password.txt brando9@snap-gpu:~/keys/
chmod 600 ~/keys/gmail_app_password.txt

# Slack bot token
scp ~/keys/slack_bot_token.txt brando9@snap-gpu:~/keys/
chmod 600 ~/keys/slack_bot_token.txt

# Zulip config
scp ~/keys/zuliprc brando9@snap-gpu:~/keys/
chmod 600 ~/keys/zuliprc

3. Cron setup

crontab -e
# Add jobs as needed (see examples above)

4. Logging

mkdir -p ~/logs
# All cron jobs should redirect output to ~/logs/
# Rotate logs periodically or use logrotate

🔐 Secrets & Security

• All credentials in ~/keys/ on SNAP with chmod 600 permissions
• SNAP home directories are private to each user
• NEVER commit credentials to the repo
• NEVER store credentials in shared SNAP directories
• Consider: credentials are only as safe as your SNAP account — use strong SSH keys

🧑 Human Interaction Required

• One-time: SSH into SNAP and set up environment (~30 min)
• One-time: Transfer credential files from local machine
• One-time: Set up cron jobs
• Periodic: Check logs, renew tokens that expire (e.g., Facebook, Instagram)
• Periodic: Update uutils (git pull && pip install -e .)

💻 Machine Uptime

• SNAP login nodes: generally always on, but may be rebooted for maintenance
• Cron jobs survive reboots (crontab is persistent)
• For critical always-on tasks, consider a SLURM job as backup
• Monitor with: crontab -l and check ~/logs/

⚠️ Research & Education Use Only

This server setup on the SNAP cluster is exclusively for Stanford research and educational purposes — automating research communications, sharing results, and managing experiment data. No commercial use.

Tasks

• Document SNAP-specific setup instructions (which login node, Python version, etc.)
• Write a setup script (scripts/setup_snap_automation.sh)
• Create example crontab entries for each service
• Test each service individually on SNAP
• Set up log rotation
• Document how to monitor and debug automation failures