deduplication of data in the full dataset

[ginacuomo]

Ensuring there is no duplication of data in the various sources

[ginacuomo]

assigning Cecile because you are leading on this effort -- feel free to link with branches where you're working on this so that when we merge it will close the issue. available to help on this as much as you need!!