Vendoring Instructions Using Conda in Addition to Pip

[patanijo]

It would be good to have instructions on how to do "vendoring" of dependencies when using conda as a package manager instead of pip.

I believe the instructions would be similar to the pip instructions
For example

1. Create a virtual environment containing all of your dependencies

conda create -n app_vendor pandas numpy scikit-learn -- copy```
2) Copy contents of <env_dir>/lib/site-packages to `vendor` subdirectory under your main app directory.
3) Push your app to cloud foundry using `cf push`. The build-pack will install dependencies directly from the `vendor` directory

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/151927173

The labels on this github issue will be updated when the story is started.

[jdamato3]

Lol @patanijo I was just trying to do this and log an issue...looks like you beat me to it 👍

[ianrcooke]

Is there an undocumented way to do this? What's listed above doesn't appear to work.

I'm using PCF Dev behind a corporate proxy with ssl interception, so I need either a way to tell conda about the self-signed cert the interceptor uses or, better, a way to vendor my conda environment. It doesn't appear that there are hooks to modify .condarc for the installing user in the installation environment, or a way to run conda config --set ssl_verify /path/to/cert. Vendoring seems the only path forward.

[denverquane]

In a similar situation here, hoping there is a way to vendor dependencies using conda. Didn't see any instructions or indications of whether or not this is possible, and my attempts have been unsuccessful. If anyone can shed light on this issue, or provide any direction as to how to vendor dependencies in Conda instead of Pip, that would be greatly appreciated.

[patanijo]

See Ian Huston's blog post I haven't tried this yet, but reading through this the only thing that could be a gotcha is that I would follow the instructions on a Linux machine if you have any dependencies that have non-python components, as dependencies in a conda package for something like numpy would have C++ or Fortran dependencies that are compiled per platform.

[robdimsdale]

Hi folks. Sorry for the delay.

As this issue is fairly old and the docs have been updated a lot since then, I'd like to start by asking if any of you still find this an issue? Are there documentation changes you'd still like to see?

[tadufre]

as far as I can tell, the docs still do not explain how to vendor using conda.

[arjun024]

I tried to have a go at this to get some documentation, but couldn't get a vendored app to work. I'm not 100% sure this case was ever supported.

Experiment

Here's what I tried (using this app):

1. On an ubuntu machine , use the docker image continuumio/miniconda3 to create a container with app mounted.
2. Install conda build tools: conda install conda-build
3. cd <conda-app>
4. CONDA_PKGS_DIRS=vendor/noarch conda env create -f environment.yml -n <env_name>
5. conda index vendor
   (vendor directory is generated)
6. Add the following channel to the environment.yml:

channels:
  - /tmp/app/vendor

(/tmp/app is the location of the app during staging)

Now, when I push the app to a CF with public internet blocked using a cached python buildpack,
I get the following error:

Preparing transaction: ...working... done
Executing transaction: ...working... done
installation finished.
Installing Dependencies
Installing conda environment from environment.yml
Collecting package metadata (repodata.json): ...working... failed
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/linux-64/repodata.json>
       Elapsed: -

       An HTTP error occurred when trying to retrieve this URL.
       HTTP errors are often intermittent, and a simple retry will get you on your way.

       If your current network has https://www.anaconda.com blocked, please file
       a support request with your network engineering team.

       'https://repo.anaconda.com/pkgs/main/linux-64'


       **ERROR** Could not update conda env: Could not run conda env update: exit status 1
Failed to compile droplet: Failed to run all supply scripts: exit status 14

Observation

• Maybe this was never supported? To support vendoring, perhaps the logic needs to fork out here and run create with --offline flag when vendor directory is present. I couldn't investigate this further due to some platform issues.
• The repodata.json file (which staging is trying to download) is present in my vendor. Not sure if I'm missing something obvious while vendoring. I'd be curious to hear from someone who has successfully vendored the app, or an expert in the area before I venture further
• @thitch97 @robdimsdale - any thoughts?