Feature request: make an API out of the website

[aymanbagabas]

It would be nice to expose the database behind a simple API to use it in cli tools and with curl

[lazamar]

This seems like a duplicate of #34
The machine powering this service is very weak so if scripts start hammering it, the main use case of people going to the website and finding stuff might be affected.

What kind of application did you have in mind that needed an API?

[aymanbagabas]

I just need a quick way to access the website data from the command line. I ended up writing a Go scraper using go-colly. I'm pretty sure this can be more lightweight using standard shell scripting with AWK, grep, and sed.

package main

import (
	"fmt"
	"os"
	"sort"
	"time"

	"github.com/gocolly/colly"
)

func usage() {
	fmt.Fprintf(os.Stderr, "Usage: %s <package> [channel=nixpkgs-unstable]\n", os.Args[0])
}

func main() {
	if len(os.Args) < 2 {
		usage()
		os.Exit(1)
	}

	pkg := os.Args[1]
	channel := "nixpkgs-unstable"
	if len(os.Args) > 2 {
		channel = os.Args[2]
	}

	var pkgs []Package
	c := colly.NewCollector()

	// Find all table rows
	c.OnHTML("tr", func(e *colly.HTMLElement) {
		var pkg Package
		e.ForEach("td", func(i int, e *colly.HTMLElement) {
			switch i {
			case 0:
				pkg.Name = e.Text
			case 1:
				pkg.Version = e.Text
			case 2:
				pkg.Revision = e.Text
			case 3:
				t, err := time.Parse("2006-01-02", e.Text)
				if err != nil {
					fmt.Fprintf(os.Stderr, "Error parsing date for %q: %v\n", e.Text, err)
					return
				}

				pkg.Date = t
			}
		})

		pkgs = append(pkgs, pkg)
	})

	url := fmt.Sprintf("https://lazamar.co.uk/nix-versions/?channel=%s&package=%s", channel, pkg)
	if err := c.Visit(url); err != nil {
		fmt.Fprintf(os.Stderr, "Error visiting %q: %v\n", url, err)
		os.Exit(1)
	}

	sort.Slice(pkgs, func(i, j int) bool {
		return pkgs[i].Date.After(pkgs[j].Date)
	})

	for _, p := range pkgs {
		if p.Name == pkg {
			fmt.Printf("%-20s\t%s\t%s\n", p.Version, p.Revision, p.Date.Format("2006-01-02"))
		}
	}
}

type Package struct {
	Name     string
	Version  string
	Revision string
	Date     time.Time
}

[aymanbagabas]

Ok, with some help from Claude, I was able to write a shell script that scrapes the website to get the revisions and dates. It uses Gum, bash, grep, and sed. You can find it here.

Usage:

Usage: ./nix-versions [-c channel] [-f format] package

Options:
  -h            Show this help message
  -c <channel>  Specify the Nix channel (default: nixpkgs-unstable)
  -f <format>   Output format (env|shell|profile|in-shell)

Examples:
  ./nix-versions python                         # Search python in nixpkgs-unstable
  ./nix-versions -c nixos-23.05 python         # Search python in nixos-23.05
  ./nix-versions -f shell python               # Get nix shell command for python

[lazamar]

I just need a quick way to access the website data from the command line.

Yeah, so why do you need that? What are you trying to achieve that you need historical version info in a script?
If you are going to scrape the website to collect a bunch of versions for packages you might as well run the project locally and build the whole database so you can query it directly?

[aymanbagabas]

Yeah, so why do you need that? What are you trying to achieve that you need historical version info in a script? If you are going to scrape the website to collect a bunch of versions for packages you might as well run the project locally and build the whole database so you can query it directly?

I ran into situations where I need to test a specific version of some package, and I find Nix package versions very useful for this sort of thing. With this cli script, I can start a nix-shell with the specified package using eval "$(nix-versions -f shell python3)" for example

[dfandrich]

I'd be interested in an API as well, which I would use to determine the most recent package version available.