magpie

A curation of my own watch later videos, powered by machine learning

Table of contents

1. Motivation
2. Architecture
3. Built with
4. Project structure
5. Development setup
6. Deployment
7. Known issues
8. Quick reference
9. Hosting locally
10. Enhancements

Motivation

I have been using YouTube for about nine years at this point. Over time, my viewing preferences have changed a lot. It can be hard to keep track of those changing preferences, and I would have liked a native way inside YouTube to categorise videos so that I know what I am/was really into at any point.

One might say that the aforementioned categorisation already exists, and they are right - it is the Category attribute assigned to every video via YouTube's own machine-learning algorithm. But it's bad - really bad, as it very frequently mis-categorises videos. YouTube as a platform doesn't really care about it, as their entire brand is based on a randomised selection of videos curated for you via a blackbox algorithm. I think that's fine, but I would really want to know what the current "map" of my interests looks like, especially on a website like YouTube, which I use a lot. They do provide you with a "map" at the end of the year - which is the personalised YouTube rewind for every user. It's more accurate and reflects one's interests faithfully. "What if there was something like that, but available at all times?" - that was my initial thought before starting this project.

Architecture

We use a Sentence-Transformers model to represent video metadata as vectors in a high-dimensional vector space. We then project that space to low dimension for visualisation and clustering, using another model (UMAP). We finally use a clustering algorithm (HDBSCAN) to classify semantically similar neighbors into a single category. The last step, labeling of the genres, was done manually by myself.

flowchart TD
    WL([Watch Later Playlist]) --> PREPROCESS[Clean & Preprocess<br/>Video Metadata]

    subgraph Embed["Embedding"]
        PREPROCESS --> MINILM["Sentence-Transformers<br/>(all-MiniLM-L6-v2)<br/> 384-dim vector"]
    end

    subgraph Pipeline["Pipeline"]
        subgraph Reduce["Dimensionality Reduction"]
            MINILM --> UMAP["UMAP<br/> 2D projection"]
        end
    
        subgraph Cluster["Clustering"]
            UMAP --> HDBSCAN["HDBSCAN<br/>(density-based)"]
        end
    end

    HDBSCAN --> YESORNO{Cluster found?}
    YESORNO -->|Yes| MANUAL["Manual labeling<br/>e.g. 'Horror', 'Programming'"]
    YESORNO -->|No| MIXED[Mixed Bag]

    MANUAL --> FINAL[(Final Genre)]
    MIXED --> FINAL

Loading

We use the YouTube API v3 to ingest the data from a playlist. The actual watch-later playlist on YouTube does not have a public URL one can use to call the API, so I had to create a separate playlist with the same videos on which the API could be called. The UMAP and HDBSCAN steps are wrapped together using sklearn.pipeline.make_pipeline, and persisted as genre_pipeline.pkl which is the pickle file present inside backend/.

A note on performance measure: HDBSCAN's relative_validity_ (DBCV) was initially chosen to be the performance measure for the model, but I quickly realised that it was not a good measure as human evaluation was the only metric that actually mattered for my clusters.

Built with

I used sklearn (Scikit-Learn) to train the machine learning model (using a pipeline). The vector embeddings were generated using all-miniLM-L6-v2 from Sentence-Transformers. Other algorithms used were UMAP for dimensionality reduction and HDBSCAN for clustering. I used pandas and numpy for dataframe manipulation, as well as plotly for generating a visualisation. I also used joblib to download the trained model. Most of the machine learning steps were done on marimo's molab platform.

The web application was created using Svelte (vanilla, not SvelteKit) + TypeScript using Vite as the build tool/bundler for the user interface; and FastAPI as the backend server, with SQLModel serving as the ORM. The database used was SQLite.

Project structure

The project is structured around two main folders, for the backend and the user interface respectively. The frontend is simple - we have an src/ folder inside of which the entire application is written. For the backend, we have a slightly more complex setup with multiple folders; but the main FastAPI app is still served through the backend root, using main.py. A snapshot of the project structure is given below.

.
├── backend
│   ├── app # utilities for interacting with the database
│   ├── data # contains the .parquet and .csv files
│   ├── database # contains database.db
│   ├── docker-entrypoint.sh # entrypoint for Docker container
│   ├── Dockerfile # for backend container
│   ├── genre_pipeline.pkl # pickle file for the model
│   ├── main.py # main FastAPI app
│   ├── pipeline # machine learning pipeline utilities
│   ├── pyproject.toml
│   ├── README.md # unused for now/stub
│   ├── scripts # scripts for populating the database
│   ├── test.py # minor dataframe test, irrelevant
│   └── uv.lock
├── compose.yaml # main compose file for the project
├── compose.production.yaml # alternate compose file for hosting locally 
├── frontend
│   ├── Dockerfile # for frontend container
│   ├── index.html
│   ├── package.json
│   ├── package-lock.json
│   ├── public
│   ├── .env.example # example .env file
│   ├── README.md # autogenerated by vite
│   ├── src # main user interface logic
│   ├── svelte.config.js
│   ├── tsconfig.app.json
│   ├── tsconfig.json
│   ├── tsconfig.node.json
│   └── vite.config.ts # vite configuration, for svelte and tailwind
└── .github # GitHub Actions workflows and scripts for CI
├── LICENSE # project license
└── README.md # you are here

Development setup

For developing magpie locally, you can either use Docker or follow the steps manually. Both are supported. If you are installing the dependencies manually, I will be making two assumptions: first, that you have uv installed; and second, that you have npm installed. That's it. For Docker I will just assume that you have docker compose installed. On some systems you also need the docker-buildx package for latest support.

Part A: Using Docker

1. First, clone the repository:

   git clone https://github.com/cosmognaut/magpie.git
   cd magpie

2. Then, let's create a .env file so that we have some environment variables in place.

   # this should be done at the project root
   touch .env
   vim .env

   Inside the .env, you need to add a YouTube Data API v3 key (which is available via GCP), a playlist ID, (available via the playlist URL), and the URL for the backend server. An example:

   YOUTUBE_API_KEY=AI..
   PLAYLIST_ID=PL..
   VITE_API_URL="http://127.0.0.1:8000/videos/"

3. Now you will find three things: a compose.yaml folder in the project root, a Dockerfile in frontend/, and another in backend/. To orchestrate your containers using docker compose, use the below command:

   docker compose up --build --watch

   This might take a while to install project dependencies. When it's done, you should be able to access the user interface at http://localhost:5173/, while the server is available via http://127.0.0.1:8000/videos/. To check the server health you can use http://127.0.0.1:8000/api/health/ - you should see a status: ok message.

   The --watch flag makes you absolutely ready for development. Thus, you can go ahead and modify anything (like /frontend/src/) and you should have the container sync things automatically.

   Note: On initial startup it might take a minute for the SQL transactions to go through, so you will see an error at http://localhost:5173/. This resolves automatically after about a minute, when fastapi finally runs the server.

4. The database is mounted as a volume, so it will persist between teardowns:

   docker compose down
   docker compose up --watch

   To delete the persisted volume, use docker compose down --volumes.

5. If you prefer to orchestrate your containers manually (i.e. without using docker compose), you can follow the below steps:

   cd frontend
   docker build --tag cosmognaut/magpie-frontend . # you can use any tag you like, use docker image ls to check it later
   docker run --env-file ../.env --detach -p 127.0.0.1:5173:90 cosmognaut/magpie-frontend:latest
   cd ../backend/
   docker build --tag cosmognaut/magpie-backend .
   docker run --env-file ../.env --detach -p 127.0.0.1:8000:80 cosmognaut/magpie-backend:latest

   If you are experiencing an error because of space related issues, it might be useful to prune the build cache. Use docker system prune to do that.

   That's it for Docker! I have also published an image to Docker Hub. Fore more information on that, see hosting.

Part B: Manual setup

1. Again, clone the repository

   git clone https://github.com/cosmognaut/magpie.git
   cd magpie

2. Inside the backend directory, you need to first sync all dependencies. uv is supported here.

   cd backend
   uv venv # this creates a virtual environment
   source .venv/bin/activate # this activates the virtual environment
   uv sync # actually sync deps

3. Now for the environment variables - grab your YouTube API key from GCP, making sure that YouTube Data v3 is enabled. Also grab your playlist's ID from its URL. Create a .env inside the backend root folder.

   touch .env
   vim .env

   Inside the file, it should have something like this:

   YOUTUBE_API_KEY=AI...
   PLAYLIST_ID=PL...

4. You will find a /scripts directory here. We will use this to populate the database. You may also find a genre_pipeline.pkl pickle file which is the trained machine learning model. You may not use this for now, as the relevant .parquet files are already present inside the /data folder. But if you still want to edit the genre names and label the clusters yourself, you may edit the genre_assignment.py file.

   vim scripts/genre_assignment.py # to name the clusters
   uv run -m scripts.genre_assignment # to generate the assigned_genres.parquet file - this should already be present inside the data/ folder

5. Now we will actually populate the database. We are using a simple sqlite database here, powered by the SQLModel ORM. Use the following commands:

   uv run -m scripts.main # creates and populates the database; also assigns the genres as per assigned_genres.parquet
   uv run -m app.test_db # to test if the database is populated and how many genres are there

   Any videos that don't have a genre are currently being assigned the 'Mixed Bag' genre by default. This is intended, but is not optimal. A more pragmatic fix would be to train a classifier based on the existing labels and have that categorise the videos.

6. You're ready to serve the endpoint now. Just use FastAPI here, the entrypoint has already been set to main:app.

   fastapi dev

7. Now visit http://127.0.0.1:8000/api/health to see if the endpoint is running. You should see a status: ok message. You are now serving the videos via the endpoint at http://127.0.0.1:8000/videos/.

8. Let's now shift our attention to the user interface. First, go into the frontend directory and install dependencies:

   cd .. # you should now be at the root
   cd frontend
   npm ci

9. Add a .env file for the frontend, using the backend API URL. This has already been done for you in .env.example.

   touch .env
   vim .env

   Inside the file, it should have something like this:

   VITE_API_URL="http://127.0.0.1:8000/videos/"

10. For local development setups, you can use Vite.

    npx vite

    The user interface should now be available at http://localhost:5173/ - it depends on your port.

Deploying the application

The project has been deployed using FastAPI Cloud. This makes it really easy to deploy the app via the command line. I am considering adding GitHub Actions CI later that deploys this on every push. To deploy the project locally for now, you can follow the below steps:

1. Make sure that you're in the frontend directory.

   cd frontend

2. Build the user interface and move the folder to the backend folder.

    rm -rf ../backend/dist/  # only if a previous deploy's dist/ still exists
    npm run build
    mv dist/ ../backend/

   FastAPI Cloud requires you to have the dist/ folder in the same directory as the main FastAPI app. That's the reason for moving the dist/ folder inside backend/.
3. Because the .fastapicloudignore is already present, you can simply run the below commands.

   cd ../backend # you should now be inside the backend folder
   mv .dockerignore .. # needed, see below
   fastapi deploy
   mv ../.dockerignore .

   The .dockerignore file was found to supersede the .fastapicloudignore file. Thus, we need to remove it from the folder either by deleting it altogether before deployment, or moving it up a folder so that .fastapicloudignore takes precedence. This was done in the CI pipeline too. That's it! fastapi deploy makes it really easy for you to deploy your applications.

Known issues

This can still be considered as a v0 for the project. As such, there are some known issues that may arise while developing or interacting with the application.

• If the database has more videos than the number used to train the model (2339), videos are automatically assigned the 'Mixed Bag' genre. While this is intended, this is not optimal and it would be better for new videos to get a proper genre assignment. The main source of the problem is the clustered_watch_later.parquet file inside data/. This file was generated after the model was trained on the videos and was downloaded from molab. A fix would be to train a classifier on the existing videos and labels and get it to categorise new videos after the database is populated.

Quick reference

I am assuming you are present in the root folder (i.e. ., neither frontend/ nor backend/). This is a quick reference for some common tasks you may need to perform:

• Populate the database

  cd backend
  uv run -m scripts.main # assuming you already have a .env here
  uv run -m app.test_db # test DB

• Redeploy the application

  cd frontend
  npm run build
  rm -rf ../backend/dist/ # if there is a dist/ from a previous deploy
  mv dist/ ../backend/
  cd ..
  cd backend
  mv .dockerignore ..
  fastapi deploy
  mv ../.dockerignore .

Hosting locally

If you want to host magpie locally, you can use the compose.production.yaml file present inside the project root. It pulls images from the Docker Hub registry. First get set up with a .env in the project root. Refer to the instructions given above.

Now, use the following command:

docker compose --file compose.production.yaml up

Enhancements

There is already a "magpie enhancement proposals" file inside frontend/src/, but this one supersedes that. This list may also include some TODOs. Some enhancements I could do to make this project better include, in no particular order:

• A classifier trained on existing video data that fixes the current issue of newer videos not being assigned a genre.
• Add a .env for frontend API_URL.
• A unified database script to make things cleaner to develop. (Priority)
• Use some other model to generate the embeddings just to test if it results in more accurate vectors (in terms of semantic similarity).
• Use GitHub actions for continuous integration; i.e. on a commit the project gets deployed.
• Write a Dockerfile to make the project easier to develop.
• Fuzzy search using Fuse.js. I had this idea after discussing genre search with Claude. I am implementing this on my own right now, but later in the future I would want to add fuzzy search for genres, as well as the video titles using that library.
• Semantic search across the entire database. This means that I would need to revamp by sqlite database to also store the "final string" column of my pandas dataframe. Or I could use a vector database, but I really don't want to do that. I could take a query from the user and run a fuzzy search across the final strings field of my database and display the relevant results. It'd also teach me much about optimising for quick data lookup.
• In-app video player using the YouTube iframe API.