Load PyPI dependency info into a database

[chadwhitacre]

Resolving dependencies by running pip install -r requirements.txt and then sniffing the results is a reeeaalllyy inefficient way to go about it. Better is going to be loading each package index into a database. Let's do this once, starting with PyPI.

First step is to basically download PyPI.

[chadwhitacre]

https://twitter.com/whit537/status/788359742271791104

[chadwhitacre]

https://packaging.python.org/mirrors/
https://pypi.org/
https://pypi.python.org/mirrors
https://www.pypi-mirrors.org/
https://pypi.python.org/pypi/bandersnatch
https://bitbucket.org/pypa/bandersnatch/
https://github.com/openstack-infra/pypi-mirror
https://github.com/pypa/warehouse/
http://doc.devpi.net/latest/

[chadwhitacre]

Ultimately what we want is to pass in a requirements.txt and/or a setup.py and get back a structure representing the dependencies.

Hmm ... https://graphcommons.com/

[chadwhitacre]

Jackpot!

https://github.com/anvaka/allpypi

[chadwhitacre]

https://github.com/koder-ua/python_deps

[chadwhitacre]

Well, jackpot: https://github.com/anvaka/pm#individual-visualizations. Indexers for 13 ecosystems!

[chadwhitacre]

As of https://mail.python.org/pipermail/distutils-sig/2015-January/025683.html we don't have to worry about mutability in PyPI. That means we never need to update info once we have it. It's possible to delete packages but I think we don't want to do that. We want to keep old info around.

[chadwhitacre]

And we only need one-week granularity. If we update once a day we'll be well inside our loop.

[chadwhitacre]

Where the mutability comes in is that dependencies are subject to a range. If I depend on Foo >= 1.0, then when Foo 1.1 comes out my dependency chain will need to be updated.

[chadwhitacre]

What's the data structure we want?

[chadwhitacre]

We want to support taking in a list of files of type text/x-python (for setup.py) and/or text/plain (for requirements.txt), and returning a single flattened list of dependencies with this info:

• package_manager—PyPI, but also GitHub, Bitbucket, ... probably Git? SCMs in general?
• name—name for PyPI, repo name for GitHub, Bitbucket, ... something for SCMs?
• url—on package manager
• license—a string or url for the actual license
• osi_license—a link under https://opensource.org/licenses; null indicates non OSI-approved
• version—the single latest version satisfying version_range; null means dep hell
• version_range—a single range computed from folding together required_by; empty set indicates dependency hell
• required_by—a list of (location, version range) tuples
  • location for text/plain—filename or URL; and line number
  • location for text/x-python—filename, URL, or package; parameter (*_require{s}); and index in the argument

Take care to handle different files with the same name in the upload.

[chadwhitacre]

Two more hops:

• database schema to support that query
• ETL flow to populate the database from PyPI

[ewdurbin]

Briefly spoke on the phone with @whit537 and summarized the basic guidance for PyPI data analytics that require traversing dependency trees.

Note that right now metadata available via PyPI is limited and may be for some time as PEP 426 is indefinitely deferred. Information is available in the deferment section of the PEP describing other PEPs addressing these topics.

In order to crawl PyPI for dependency links, you'll need general metadata for "indexing" as well as the package files themselves to obtain dependency information via setuptools/distutils.

Recommended approach:

• Setup and maintain a bandersnatch based mirror of PyPI's simple index and package data (wheels, tarballs, zips, exes... all of it) (Around 450GB right now)
• Use the simple index or XMLRPC list_packages() call to retrieve a list of packages registered with PyPI.
• Use individual packages JSON API https://pypi.org/pypi/<package_name>/json to retrieve metadata and releases for individual packages (https://pypi.org/pypi/requests/json for example)
• For specific releases, use the JSON API https://pypi.org/pypi/<package_name>/<release_identifier>/json (https://pypi.org/pypi/requests/2.11.1/json for example)
• In order to obtain dependency information for a given package and release, you'll need to read in from the artifact itself (obtained via bandersnatch)

All of the above endpoints and tools with the exception of the XMLRPC are designed to minimize impact on the PyPI backend infrastructure, as they are easily cached in our CDN.

[chadwhitacre]

A table for releases, unique on (package_manager_id, package_id, version), also has path, license, and osi_license columns.

A releases.deps column stores upstream dependencies, which are self-references to specific releases, along with version_range and required_by for each. Resolving a set of dependencies is then a matter of merging the deps chains for the input set. The process used to precompute deps should be usable on-the-fly for queries.

PyPI gives us a changelog that includes new release events. With that, we should be able to recompute deps for the subset of affected packages; we'll need a reverse mapping (release_id, required_by) for that. Basically, when a new release comes out we want to:

1. compute deps for that release, and
2. go through everything that depends on any other version of the newly released package, and update deps.

We'll need to keep a table of packages, and do the reverse mapping based on that. Something like:

for each package in might_depend_on(new_release):
    for release in package.releases:
        release.update_deps(new_release)