Streaming dvc files

A dvc pull will download dvc files to your local repository. But this may not be the best way to proceed! In particular, dvc offers an api which permits one to “stream” or cache files, leaving your storage local to the working repository free of big data files.

To illustrate,

import dvc.api
import pandas as pd

with dvc.api.open('BigRemoteFile.dta',mode='rb') as dta:
    df = pd.read_stata(dta)

This will result in a pandas.DataFrame in RAM, but will use no additional disk (except that, depending on what’s being used as the dvc store, the file may actually be stored in .dvc/cache; this cache can be cleared with dvc gc).

Pulling dvc files

If you need the actual file instead of a “stream” you can instead “pull” the dvc files, using

dvc pull

and files should be added from the remote dvc data store to your working repository.

Adding New Data

Additional S3 Credentials

Write access to the remote s3 repository requires additional credentials; contact ligon@berkeley.edu to obtain these.

Procedure to Add Data

To add a new LSMS-style survey to the repo, you’ll follow the following steps. Here we give the example of adding a 2015–16 survey from Uganda, obtained from https://microdata.worldbank.org/index.php/catalog/3460. The same steps should work for you mutatis mutandis:

1. Create a directory corresponding to the country or area; e.g.,

   mkdir Uganda
       

2. Create a sub-directory indicating the time period for the survey; e.g.,

   mkdir Uganda/2015-16
       

3. Create a Documentation sub-directory for each survey; e.g.,

   mkdir Uganda/2015-16/Documentation
       

   In this directory include the following files:

   SOURCE

   A text file giving both a url (if available) and citation information for the dataset.

   LICENSE

   A text file containing a description of the license or other terms under which you’ve obtained the data.

4. Add other documentation useful for understanding the data to the Documentation sub-directory.
5. Add all the contents of the Documentation folder to the git repo; e.g.,

   cd ./Uganda/2015-16/Documentation
   git add .
   git commit -m"Add Uganda 2015-16 documentation to repo."
   git push
       

6. Create a Data sub-directory for each survey; e.g.,

   mkdir Uganda/2015-16/Data
       

7. Obtain a copy of the data you’re interested in, perhaps as a zip file or other archive. Store this in some temporary place, and unzip (or whatever) the files into the relevant Country/Year/Data directory, taking care to preserve any useful directory structure in the archive. E.g.,

   cd Uganda/2015-16 && unzip -j /tmp/UGA_2015_UNPS_v01_M_STATA8.zip
       

8. Add the data you’ve unarchived to dvc, then add the pointers (i.e., files with a .dvc extension to git). For the Uganda case we assume that all the relevant data comes in the form of stata *.dta files, since this is what we downloaded from the World Bank. For example,

   cd ../Data
   dvc add *.dta
   git commit -m"Add Uganda/2015-16/Data/*.dta files to dvc store."
   git pull && git push
       

9. Push the data files to the dvc store. Make sure you have good internet connection! Then a simple

   dvc push
       

   will copy the data to the remote data store. NB: If this is the first time you’ve done this for this repository, then you’ll first need to jump through some simple hoops to authenticate with gdrive.

10. With the files pushed to the dvc store, you won’t need them locally anymore, so you can do something like

    cd ../Data && rm *.dta
        

    or (if you have a more complex directory structure) perhaps

    find ../Data -name \*.dta -exec rm \{\} \;