-
Notifications
You must be signed in to change notification settings - Fork 22
Data sources
Below is an overview of some data providers, data sources and data sets that are used in the field and it's made available to help navigating the matter. Data sources actually used in ExpectedReturns for reproducibility purposes will be reported in the usual package documentation (repo docs, vignettes). Likewise, if possible, functionality that allows to conveniently import such resources in the R environment will be referred to.
Of course, there are databases which make meticulous work in this sense at least unnecessary. Well known examples are WRDS, CRSP and Morningstar Direct. Unfortunately, such products and services are not open to everyone and data they allow to access to is copyrighted with sharing restrictions.
The tables below are a summary of most of Dr. Ilmanen's book Appendix B, "Data sources and data-series construction".
Appendix sections B1 -- B3 are included, whereas sections B4 -- B6 are excluded as they are outside the scope of our project for now. However, from B6 are of interest specifications on illiquidity estimates and eventually notes on CPI and real GDP series.
While reading the tables, please note the following:
- To avoid confusion, subsections below are reported with the same names Ilmanen gave them.
- The syntax "Author* (year, *)" is a citation from the book bibliography.
- "???" signals that the reference wasn't found at a first glance and needs to be searched again.
- "self-collected data" refer to data collected by Dr. Ilmanen himself. Clearly the data is used in his book, but to our knowledge is not open sourced.
- Often the same or very similar data series are shared by several providers. In such cases consider the Link/Reference to be an example.
| Provider | Link/Reference | Notes | |
|---|---|---|---|
| Equity | |||
| Stocks | Dimson–Marsh–Staunton (since 1900) | Dimson–Marsh–Staunton (2002, 2010) | |
| MSCI Barra (since 1970 for developed markets, since 1988 for emerging markets) | https://www.msci.com/index-solutions | ||
| Fama-French's data library | http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html | ||
| Shiller's online data | http://www.econ.yale.edu/~shiller/data.htm | ||
| Arnott–Bernstein (since 1802), extended with S&P500 data (since 2000) | Arnott-Bernstein (2002) | ||
| Private Equity | CA indexes (since 1980s) | https://www.cambridgeassociates.com/private-investment-benchmarks/ | |
| Venture Capital | LPX50 index (since 1994) | https://www.lpx-group.com/lpx_indexing/lpx-nav-indices/lpx-nav-index-series/ | |
| Fixed-income | |||
| Government bonds (global) | Dimson–Marsh–Staunton (since 1900, currency-unhedged) | Dimson–Marsh–Staunton (2002, 2010) | |
| WGBI (since 1985, currency-hedged) | https://www.yieldbook.com/m/indices/single.shtml?ticker=WGBI | ||
| BarCap indexes (since 1970s/80s) | https://indices.barclays/IM/21/en/indices/list.app | ||
| US Treasury | Fama-French's data library | http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html | |
| CRSP (since 1952) | http://www.crsp.org/products/research-products/crsp-us-treasury-database | Integrates with WRDS | |
| ML Treasury index subsector (since 1978) | ??? | ||
| Arnott–Bernstein (since 1802) | Arnott–Bernstein (2002) | ||
| Homer–Sylla (before 1926) | Homer–Sylla (1991) | ||
| Ibbotson Associates (since 1926) | https://indexes.morningstar.com/yearbooks | ||
| Bloomberg (recent data) | https://www.bloomberg.com/markets/rates-bonds/government-bonds/us | ||
| Corporate bonds | Moody's (since 1919) and Ibbotson Associates (since 1926) | https://fred.stlouisfed.org/series/AAA and https://indexes.morningstar.com/yearbooks | |
| Convertible bonds | ML US convertible bond index returns (since late 1980s) | ML US treasury index subsector??? | |
| Emerging market debt | EMBI Global index (since 1994) extended with Citi's Brady bond index data (from 1990 to 1993) | https://www.jpmorgan.com/country/US/EN/jpmorgan/investbk/research/indexresearch/vendor/packages extended with Citi's Brady bond index data??? | |
| Alternative | |||
| Real estate | MIT/CRE transactions-based index (since 1984, direct) | https://mitcre.mit.edu/research-publications/cred/transaction-based-index | |
| NAREIT composite index (since 1972, listed) | https://www.reit.com/data-research | ||
| GPR global property stocks index (since 1984, listed) | https://www.globalpropertyresearch.com/indices | ||
| Shiller (since 1890) | http://www.econ.yale.edu/~shiller/data.htm | ||
| Davis–Lehnert–Martin (since 1960) | Davis–Lehnert–Martin (2008) | ||
| LILP | https://www.lincolninst.edu/research-data | ||
| Dimson (2004) | Dimson (2004) | ||
| Infrastructure | UBS Global Infrastructure and Utilities index (since 1990) | UBS Global Infrastructure and Utilities index??? | |
| Timber | NCREIF (since 1987) | https://www.ncreif.org/research/ | |
| Art | AMR (since 1976) | https://www.artmarketresearch.com/art-market-overview/ | |
| artprice.com (since 1990) | https://www.artprice.com/search | ||
| Hedge fund | HFR (since 1990) | https://www.hedgefundresearch.com/hfr-database | Must be an 'accredited investor' as per SEC Rule 501(a) to access. Integrates with WRDS and Morningstar Direct. |
| Dow Jones Credit Suisse AllHedge index (since 1994) | https://lab.credit-suisse.com/#/en/index/SECT/SECT/overview | ||
| Derivatives | |||
| Commodity futures | GSCI (since 1970s) | https://tradingeconomics.com/commodity/gsci | |
| Oil price data from Knuth Kjaer (since 1900) | ??? | ||
| Gold price data from Bloomberg (since 1920) | Bloomberg terminal | ||
| CRB index data (since 1957) | https://tradingeconomics.com/commodity/crb | ||
| Self-collected data | ??? | ||
| Sovereign CDS spreads | Bloomberg | Bloomberg terminal |
| Style | Provider | Link/Reference | Notes |
|---|---|---|---|
| Value | Fama-French's data library | http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html | |
| Giuliano De Rossi (UBS) | ??? | ||
| Momentum/Trend | MLM Commodity Index (before 1990) | ??? | |
| Self-collected data (since 1990, mainly from Bloomberg) | Bloomberg terminal | ||
| Carry | Self-collected data (since 1970s, mainly from Bloomberg) | Bloomberg terminal | |
| Volatility selling | ML Equity Volatility Arbitrage Index (since 1989) | ??? | |
| CBOE S&P500 Index Options | http://www.cboe.com/products/stock-index-options-spx-rut-msci-ftse/s-p-500-index-options | ||
| Andrew Ang (Columbia University, BlackRock) | ??? | ||
| David Blitz (Robeco) | ??? | ||
| Giuliano De Rossi (UBS) | ??? |
| Factor | Provider | Link/Reference | Notes |
|---|---|---|---|
| Growth | Consensus Economics US real GDP growth | https://www.consensuseconomics.com/special-data/ | |
| Inflation | Consensus Economics US CPI | https://www.consensuseconomics.com/special-data/ | |
| Liquidity | BoE Financial Market Liquidity Index extended with Citi liquidity index (1990–1991, 2009) | Bank of England (2007), ??? | |
| Volatility | CBOE S&P500 VIX | http://www.cboe.com/products/vix-index-volatility/vix-options-and-futures/vix-index/vix-historical-data |
Some researchers (and companies), whose works we are interested in reproducing, make data sets used in their studies publicly available. Indeed, this happens precisely as an essential element to allow reproduction studies. Although this alone is insufficient to make the study fully reproducible in a strict sense, it certainly constitute a necessary (and much appreciated) step toward that ideal. In accordance with copyrights and intellectual rights, every work involved and the corresponding authors will always be cited and receive proper credit where they are being introduced, discussed and "reproduced". Below we gather the main sources to provide an overview:
- AQR data sets: https://www.aqr.com/Insights/Datasets
- Prof. Pastor's liquidity factor data: https://faculty.chicagobooth.edu/-/media/faculty/lubos-pastor/data/liq_data_1962_2019.txt
- Global Q-factors: http://global-q.org/index.html