[WIP] Add "index" subcommand

[vpavlin]

@kbsingh's idea of index and cccp was always very interesting for me. This PR adds new subcommand index which has then 3 verbs:

• list - to list Nulecule apps/components in the Index
• info - to print details about the app/component
• generate - to generate the Index based on the local path or a git URL containing unpacked Nulecule apps/components

It operates on top of https://github.com/projectatomic/nulecule-library where I have sent another pull request which is needed for this to work properly - projectatomic/nulecule-library#17

Workflow

1. generate
   • based on the content of nulecule-library generate gen_index.yaml (this is just a POC and WIP;) )
   • This step is normally not used by a user! This is used in a Dockerfile for Automated Build in nulecule-library to build an image containing index.yaml which is pulled and unpacked by index list and index info if the index file is missing locally - works already as I set up the Automated Build for my other PR branch (image: nulecule/index)
2. list - parse gen_index.yaml and list all available apps/components
   
3. info - print info for a specific component and suggest a way how to add a component to your app
   

I haven't tried to map it to atomic command anyhow yet..:(

Reasoning

This index functionality can later be used by other tools to query available component and information about them (searching, filtering, automated addition of components...). It will also help developers to find components for their apps easily using Atomic App directly.

I'd also like to see filtering being added for the list verb to make it easy to search for f.e.

• Apps with a specific version of the Nulecule spec (--filter=specversion=0.02)
• Apps supporting specific provider (--filter=provider=docker)
• Apps with a some name (--filter=id=mysql*)

Comments are very welcome:-)

[landscape-bot]

Repository health decreased by 0.19% when pulling 4a478d8 on vpavlin:feature/index into edc6fc5 on projectatomic:master.

• 3 new problems were found (including 0 errors and 3 code smells).
• No problems were fixed.

[cdrage]

This looks awesome.

The only thing I'm concerned about is it limits people outside of the project from being included in the nulecule-library and list may be much longer as we progress with more examples.

Would the epic/end-goal be our own hub (hub.projectatomic.io) similar to Docker's with a hub of atomicapp's people can download, test, etcetera? Of course we'd add a query / regex that Nulecule needs to be included in the upload, etcetera.

Or is the goal of this to provide a good set of examples for user's to use, etc?

What about a search command too?

[vpavlin]

@charliedrage How does that limit people from being included in nulecule-library? It's just about sending a PR:-) I've also proposed an "external entries" for index in projectatomic/nulecule-library#17

I don't know about hub.projectatomic.io, but this index thing would certainly help with stuff like Fedora modularization - Fedora could easily create their own index based on existing Atomic Apps and add it to projects like Software Center

Search would be definitely valuable part of index implementation in Atomic App (this proposal already mentions --filter which is kinda similar..)

The initial goal is to make it easy for developers to find well managed Atomic Apps which they can use or incorporate in their own apps.

[cdrage]

@vpavlin
Maybe as an atlernative we can add a atomicapp index push $IMAGENAME ? Similar to how homebrew for OS X takes in PR requests as has a bot automatically test && push any updates / requests to the repo.

Otherwise, really good work! I like it 👍 👍

[cdrage]

I believe should keep gitpython within requirements.txt in order to keep this consistent across all Dockerfile's.
gitpython==1.0.1 and prevent compatibility issues (Debian may have a different package) as seen with our previous lockfile issue.

[dustymabe]

just to be clear let's leave this on the burner until beta3 has passed (and maybe even GA)

[bexelbie]

@cdrage's comment about a registry made me wonder. If we could get a label based search on docker registry then we could just leverage that (or any private registries). That seems more extensible, although it would require a big patch upstream. However I see label searches providing even more value down the road outside of this function.

[vpavlin]

@bexelbie This idea is not new:) Me and @shaded-enmity had that one a long time ago (basically when we started to use labels like vendor in the base images.

Anyway, I talked to Distribution (v2 registry) maints last week at DockerCon and they said there is no plan for this yet. I am pretty sure there is/was some issue for that, I'll try to find it (or if @shaded-enmity knows about any?).

I also talked to quay.io devs and they said this feature should land in their registry in a couple of months:)

Anyway, yes, this would be the best solution for "index"

Also we shouldn't forget about the moratorium for Remote API which might influence addition of this feature https://github.com/docker/docker/blob/master/ROADMAP.md#23-remote-registry-operations

[shaded-enmity]

@vpavlin The original issue can be found here: distribution/#600, the later discussion happened via emails so let me search the archives.

The proof of concept code works by scanning the Distribution/Registry file system storage for a magic header found in all manifests. Labels and some additional bits of information are then collated on a per-repo basis and signed with distribution key. There's also an index.html generated for all the content.
You can give it a quick test drive using this Vagrant box, which does the following:

• Builds Distribution from source and starts it
• Creates configurable number of random repositories with random images and metadata
• Pushes those images to the local Distribution

Then you also need to install the jq utility from Fedora 23 and git clone the proof of concept repo and run it like:

mkdir metadata/ keys/
python crypto_setup.py -g keys/ md-key.json
python service.py -g md-key.json /opt/data/cache metadata/

I should probably include these in the provisioning script and update it to use a better base image like this one if this gets any attention :)

[cdrage]

Hey @vpavlin So i've been working on refactoring this and making it viable for a merge into Atomic App :) One question, in regards to the fetch functionality you added (and from the looks if it isn't used). Would it be viable for the user to automically download the gen_index.yml into their home directory if it doesn't exist? Ex. ~/.atomicapp/gen_index.yml

[centos-ci]

Can one of the admins verify this patch?

[centos-ci]

Can one of the admins verify this patch?

[centos-ci]

Can one of the admins verify this patch?

[centos-ci]

Can one of the admins verify this patch?

[vpavlin]

Yeah, I was trying to find the usage of fetch, but with no luck:D I guess I just lost the idea in the middle of coding:-) But yes, generally, if you try to list, you might either get it pull automatically, or be informed there is not index locally and if you want to pull it. I didn't follow development lately, so I have not idea what fits better to overall UX of AtomicApp now. Location sounds ok

[cdrage]

@vpavlin thanks man :)

Yeah, I was looking at how cccp-index does it, if we can keep an index.yml within nulecule-library that would be great (or like what you and dusty said, keep it in a container and rely on docker hub for hosting, which may be better).

I'll refactor some of this code and ping you :)

[cdrage]

Hey @vpavlin I'm going to close this in favour of #750 :) Thank you so much for contributing this code!