imgurdex

A lightweight asynchronous crawler that fetches resources from Imgur, processes them through a pluggable architecture, and stores results locally or in Google Cloud Storage.

This project follows a layered architecture:

• domain/ → Core models and interfaces
• services/ → Orchestration logic (download, consumption, iteration)
• infrastructure/ → Implementations of external systems (storage, HTTP, etc.)
• scripts/ → CLI entrypoints

Overview

The system is built around four main abstractions:

Resource

The core entity in the system.

Any Imgur image (the main resource) can be retrieved by a link, such as https://i.imgur.com/cGV1iV5.jpeg.

That 7 character string, composed of letters (uppercase and lowercase) and digits (from 0 to 9), is the ID of the resource.

Note that a resource can be a JPEG (as seen above), a PNG, a GIF or even a MP4. For the sake of simplicity, only JPEGs, PNGs and GIFs will be considered as resource in this project.

Even if a image is a JPEG or a GIF, Imgur makes it accessible through a .png extension. If the final .jpeg or .gif is replaced with .png, Imgur will redirect to the same image. We take advantage of that feature (or bug) such that a image can be found just by its ID. So .png here is called the raw extension of the resource.

To validate its true extension, file signatures are checked: the first four bytes of any image can tell if it's a JPEG, a PNG or a GIF. That would be called the real extension of the resource, or simply the resource extension.

Finally, after a resource is fetched, its bytes are downloaded, which are the contents of the resource.

Downloader

An abstract service responsible for fetching a resource from Imgur by its ID.

It returns a resource if found or None if the resource does not exist.

A resource does not exist if trying to access its URL returns a 302 Found to https://i.imgur.com/removed.png.

Implementations may include HTTP-based or API-based downloaders.

Consumer

An abstract service responsible for handling the result of a download.

It's a callback that receives a resource ID if the resource does not exist or the resource itself otherwise.

Implementations may include storing to disk or cloud storage, logging results or forwarding to another pipeline.

ID iterator

An abstract service responsible for providing a stream of IDs to crawl.

Implementations may include generating random IDs, reading from a file or from a in-memory iterator.

sequenceDiagram
    participant It as IdIterator
    participant D as Downloader
    participant C as Consumer

    loop Until iterator exhausted
        It->>D: yield id
        D->>D: download(id)
        alt Resource exists
            D->>C: consume_hit(resource)
        else Resource missing
            D->>C: consume_miss(id)
        end
    end

    It->>It: close()

Loading

Installation

It's recommended to have Poetry in your machine.

1. Clone the repository:

git clone https://github.com/ezgrs/imgurdex
cd imgurdex

2. Install the dependencies:

poetry install

Usage

CLI

The crawler is configured via CLI arguments:

poetry run python -m imgurbc.scripts.crawler

Flag	Description
-d, --delay	Delay between requests in seconds (default: 1)
--gcloud-storage-bucket-name	GCS bucket name (optional)
--gcloud-storage-bucket-location	GCS bucket location (default: us-east1)
-i, --input	Input IDs (optional, strings)
--no-stdout	Disable stdout logging
-o, --output	Local directory to store the images found

Note that

• If --delay is 0, Imgur may rate-limit the connection
• If --gcloud-storage-bucket-name is not given, no Cloud access will be made
• --gcloud-storage-bucket-location is only applied if --gcloud-storage-bucket-name is given and if its bucket does not already exist inside the Google Cloud project's Storage
• If --input is not given, a infinite random stream of IDs will be tried instead
• If --output is not given, images found will be discarded
• You can use both --gcloud-storage-bucket-name and --output, saving images both locally and to the cloud

Examples

If you want to download a known set of images and save it to the output directory:

poetry run python -m imgurbc.scripts.crawler -i dzgsGOW ogHxQ5Y cGV1iV5 -o ./output

If you want to download a random set of images and save it to Google Cloud Storage:

poetry run python -m imgurbc.scripts.crawler --gcloud-storage-bucket-name my-bucket

Google Cloud

Setup

1. Check if you have the Google Cloud SDK installed:

gcloud --version

Google Cloud SDK 565.0.0
beta 2026.04.10
bq 2.1.31
core 2026.04.10
gcloud-crc32c 1.0.0
gsutil 5.3

2. Log in into your Google Cloud account:

gcloud auth login

This command will first open your browser to the sign-in page where you complete authentication.

Then it'll show your current list of projects: choose which one you would like to use its Storage.

Running this will allow you to run gcloud commands from your terminal, finding your credentials automatically.

3. Create your Application Default Credentials (ADC) file:

gcloud auth application-default login

Running this will allow your SDK library to run the SDK code, finding your credentials automatically.

4. If your project is not already set for some reason, you can do so by running:

gcloud config set project PROJECT_ID

Cloud Run service

This section walks through setting up and deploying a function on Google Cloud Run so the crawler can be executed on a schedule using Google Cloud Scheduler.

Instead of running the crawler in a continuous loop on a dedicated server, this approach treats each run as a short-lived, stateless job. Cloud Scheduler triggers the function at regular intervals, and Cloud Run spins up just enough compute to handle that execution before scaling back down.

1. Create a service account to manage the Cloud Run service:

gcloud iam service-accounts create crawler

Created service account [crawler].

Add the following roles:

• storage.admin (gives full control over Cloud Storage buckets)

For instance,

gcloud projects add-iam-policy-binding PROJECT_ID
  --member="serviceAccount:crawler@PROJECT_ID.iam.gserviceaccount.com"
  --role="roles/storage.admin"

Updated IAM policy for project [PROJECT_ID].

2. Deploy the API to Google Cloud Run:

gcloud run deploy CLOUD_FUNCTION_NAME
  --source .
  --region us-east1
  --allow-unauthenticated
  --service-account=crawler@PROJECT_ID.iam.gserviceaccount.com

This will deploy a FastAPI application with two endpoints:

• POST /imgur/{imgur_id}, which returns 200 OK if the provided image ID exists in Imgur otherwise 404 Not Found. It also saves the Imgur image to Google Cloud Storage if the former.
• POST /imgur/random, which generates a random Imgur ID and acts like calling the previous endpoint.

Deploying automatically via GitHub

Considering you have a GitHub repository https://github.com/YOUR_GITHUB_USER/YOUR_REPO, you can set up a service account to deploy the Cloud Run service whenever you do a push.

Check out the workflow file at .github/workflows/cloud-run-deploy.yml.

1. Enable the IAM Service Account Credentials API:

gcloud services enable iamcredentials.googleapis.com

Operation "operations/XXXX.X9-9999999999999-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" finished successfully.

2. Create a Workload Identity pool:

gcloud iam workload-identity-pools create github-pool
  --location=global
  --display-name="GitHub Pool"

Created workload identity pool [github-pool].

3. Create a GitHub OIDC provider:

gcloud iam workload-identity-pools providers create-oidc github-provider 
  --location=global 
  --workload-identity-pool=github-pool
  --display-name="GitHub Provider"
  --issuer-uri="https://token.actions.githubusercontent.com"
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository"
  --attribute-condition="assertion.repository=='YOUR_GITHUB_USER/YOUR_REPO'"

Created workload identity pool provider [github-provider].

4. Query the provider resource name:

gcloud iam workload-identity-pools providers describe github-provider
  --location=global
  --workload-identity-pool=github-pool
  --format="value(name)"

projects/9999999999999/locations/global/workloadIdentityPools/github-pool/providers/github-provider

The 13-digit string will be refered as the PROJECT_NUMBER.

5. Create a service account to execute the deploy:

gcloud iam service-accounts create github-deployer

Created service account [github-deployer].

Add the following roles (as explained in the previous section):

• run.admin (gives full control over Cloud Run services)
• iam.serviceAccountUser (allows GitHub to use a service account when deploying)
• artifactregistry.writer (allows pushing build artifacts)
• cloudbuild.builds.editor (allows Cloud Build to run builds)
• storage.objectAdmin (gives full control over objects inside Cloud Storage buckets)
• storage.bucketViewer (allows reading Cloud Storage bucket metadata)

Also link the service account to the GitHub provider:

gcloud iam service-accounts add-iam-policy-binding
  github-deployer@PROJECT_ID.iam.gserviceaccount.com
  --role="roles/iam.serviceAccountTokenCreator"
  --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github-pool/attribute.repository/YOUR_GITHUB_USER/YOUR_REPO"

Updated IAM policy for serviceAccount [github-deployer@PROJECT_ID.iam.gserviceaccount.com].

Integrating with Secret Manager

A .env file for os.environ must be uploaded to Cloud Run server to make the code run correctly. This section will integrate with Secret Manager so that the Cloud Run has no access to the secret variables.

1. Enable the IAM Service Account Credentials API:

gcloud services enable secretmanager.googleapis.com

Operation "operations/XXXX.X9-9999999999999-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" finished successfully.

2. Save all the sensitive variables:

echo|set /p="SECRET_VALUE" > secret.txt
gcloud secrets create SECRET_KEY --replication-policy="automatic" --data-file=secret.txt

Created version [1] of the secret [SECRET_KEY].

To update it later:

echo|set /p="NEW_SECRET_VALUE" > secret.txt
gcloud secrets versions add SECRET_KEY --data-file=secret.txt

Created version [2] of the secret [SECRET_KEY].

3. Grant the Cloud Run service account manager acess to the variables (as defined below, crawler):

gcloud secrets add-iam-policy-binding SECRET_KEY
    --member="serviceAccount:crawler@PROJECT_ID.iam.gserviceaccount.com"
    --role="roles/secretmanager.secretAccessor"

Updated IAM policy for secret [SECRET_KEY].

When the gcloud run deploy command is now executed, the --set-env-vars and --set-secrets parameters need to be passed. No code change is required, since it'll be automatically injected into os.environ:

gcloud run deploy CLOUD_FUNCTION_NAME
  --source .
  --region us-east1
  --allow-unauthenticated
  --service-account=crawler@PROJECT_ID.iam.gserviceaccount.com
  --set-env-vars EMAIL_HOST=smtp.gmail.com,EMAIL_PORT=587,...
  --set-secrets EMAIL_USERNAME=EMAIL_USERNAME:latest,EMAIL_PASSWORD=EMAIL_PASSWORD:latest

Enabling authentication

Right now, if the Cloud Run function is called by anyone in the internet, that person will be able to access it because of the --allow-unauthenticated parameter. To revoke it, it's possible to restrict access to a dedicated service account.

1. Create a service account to run the scheduler:

gcloud iam service-accounts create cron-scheduler

Created service account [cron-scheduler].

2. Grant it permission to run that specific Cloud Run function:

gcloud run services add-iam-policy-binding CLOUD_FUNCTION_NAME
  --region us-east1
  --member="serviceAccount:cron-scheduler@PROJECT_ID.iam.gserviceaccount.com"
  --role="roles/run.invoker"

Updated IAM policy for service [CLOUD_FUNCTION_NAME].

3. Query the Cloud Run canonical URL:

gcloud run services describe CLOUD_FUNCTION_NAME
  --region us-east1
  --format="value(status.url)"

The output text will be refered as the CLOUD_FUNCTION_URL.

4. Create a Cloud Scheduler job:

gcloud scheduler jobs create http croncrawl
  --location us-east1
  --schedule="* * * * *"
  --http-method=POST
  --uri="CLOUD_FUNCTION_URL/imgur/random"
  --oidc-service-account-email="cron-scheduler@PROJECT_ID.iam.gserviceaccount.com"
  --oidc-token-audience="CLOUD_FUNCTION_URL"
  --headers="User-Agent=Google-Cloud-Scheduler"
  --time-zone="XXXXX/XXXXX"

5. Remove the public access:

gcloud run services get-iam-policy CLOUD_FUNCTION_NAME --region us-east1

bindings:
- members:
  - allUsers
  - serviceAccount:cron-scheduler@PROJECT_ID.iam.gserviceaccount.com
  role: roles/run.invoker

gcloud run services remove-iam-policy-binding CLOUD_FUNCTION_NAME
  --region us-east1
  --member="allUsers"
  --role="roles/run.invoker"

Updated IAM policy for service [CLOUD_FUNCTION_NAME].
bindings:
- members:
  - serviceAccount:cron-scheduler@PROJECT_ID.iam.gserviceaccount.com
  role: roles/run.invoker

When the next gcloud run deploy runs, the --allow-unauthenticated flag now must be omitted.