LSMS_Library

A Python library providing a uniform interface to Living Standards Measurement Study (LSMS) household surveys from multiple countries and years, without the data loss typical of traditional harmonization approaches.

The Problem

LSMS datasets are invaluable for studying poverty, consumption, and household welfare across developing countries. However, each country’s survey uses different:

• Variable names and encodings
• Food classification systems
• Questionnaire structures
• File formats and organization

Researchers typically spend weeks learning each new dataset’s idiosyncrasies or use pre-harmonized datasets that sacrifice detail and comparability. Cross-country or longitudinal analyses become prohibitively time-consuming.

The Solution

LSMS_Library provides an abstraction layer that gives you a consistent interface to work with any supported LSMS dataset. Instead of harmonizing the data itself (which loses information), we harmonize the way you access the data.

Installation

From PyPI (when available):

pip install LSMS_Library

From github (current release):

pip install git+https://github.com/ligon/LSMS_Library.git@v0.7.0

From a source checkout (for contributors):

git clone https://github.com/ligon/LSMS_Library.git
cd LSMS_Library
poetry install

Quick Start

import lsms_library as ll

# Single-country access
uga = ll.Country('Uganda')
uga.waves          # ['2005-06', '2009-10', ..., '2019-20']
uga.data_scheme    # ['food_acquired', 'household_roster', ...]
food = uga.food_expenditures()   # Standardized DataFrame, all waves

# Cross-country analysis
roster = ll.Feature('household_roster')
roster.countries   # ['Burkina_Faso', 'Ethiopia', 'Mali', 'Uganda', ...]
df = roster()      # Harmonized DataFrame across all countries

Data Access

This library abstracts over Living Standards Measurement Study (LSMS) survey data. The underlying microdata belongs to the respective national statistics offices and the World Bank; users must accept the World Bank Microdata Library’s terms of use before accessing it.

Authentication: the WB Microdata API key

1. Register at https://microdata.worldbank.org/ (free).
2. Accept the terms of use for the LSMS collections you want to access.
3. Get your API key from your account dashboard.
4. Create ~/.config/lsms_library/config.yml with:

   microdata_api_key: your_key_here
   # data_dir: /path/to/override   # same as LSMS_DATA_DIR env var
       

   or set MICRODATA_API_KEY as an environment variable.

5. On import lsms_library, the library validates your key against the WB catalog and automatically unlocks access to the S3 read cache. No further setup is required.

The S3 cache

Once your WB API key is validated, the library unlocks a read-only S3 cache that mirrors the WB Microdata downloads. This is a convenience – the S3 cache is faster than the WB NADA API and reduces load on the WB service – but it is not a separate access layer. The WB terms of use are the authoritative gate; the S3 cache just provides the same data faster.

Decrypted plaintext credentials are written to ~/.config/lsms_library/s3_creds (or the path in the LSMS_S3_CREDS environment variable), not into the package tree – so the library is safe to install from a wheel into a read-only site-packages directory.

Non-interactive environments

In CI, Docker builds, or other non-interactive contexts, set LSMS_SKIP_AUTH=1 to suppress the import-time authentication flow. In that mode you are responsible for ensuring ~/.config/lsms_library/s3_creds exists (e.g. via a CI secret mount) before the first data access.

Data cache location

Parquet caches materialize under the platform-appropriate user data directory:

• Linux: ~/.local/share/lsms_library/ by default
• Override with LSMS_DATA_DIR env var or data_dir in config.yml

See the caching guide for details on assume_cache_fresh, the Country and Feature classes, and per-country build methods.

Documentation

Full documentation is available at ligon.github.io/LSMS_Library, including:

• **Getting Started** – installation and first steps
• **Country Guide** – single-country workflows, harmonization pipeline, derived tables
• **Feature Guide** – cross-country analysis with ll.Feature
• **Caching** – performance tuning, build backends, cache management
• **Panel Data** – longitudinal analysis and ID harmonization
• **API Reference** – complete class documentation (auto-generated from source)

Contributing

See CONTRIBUTING.org for detailed guidelines on adding new datasets using DVC.

Citation

If you use LSMS_Library in your research, please cite:

@software{ligon25:lsms_library,
  author =    {Ethan Ligon},
  title =     {{\tt LSMS_Library}: Abstraction layer for working with Living Standards Measurement Surveys},
  year =      2025,
  doi = {10.5281/zenodo.17258079},
  url = {https://pypi.org/project/lsms_library/}
}

License

See the LICENSE file in the repository for details.

Acknowledgments

This project builds on data collection efforts by:

• The World Bank’s Living Standards Measurement Study (LSMS) team
• National statistical offices in participating countries
• The LSMS-ISA initiative