Analysis and code of HiC data for 16p samples
Table of Contents
File structure of all input data, reference files and results
All filepaths below are relative to the project root on our server: /data/talkowski/Samples/16p_HiC/
Files for reproducibility
$ tree -L 2 dependencies.files
└── dependencies.files/
├── conda.envs
│ ├── cooltools.yml
│ ├── distiller.yml
│ ├── hicrep.yml
│ ├── multiqc.yml
│ ├── TADs.yml
│ └── r.yml
├── packages
│ ├── functionsdchic_1.0.tar.gz
│ └── hashmap_0.2.2.tar.gz
└── README.md
mamba environments can be installed via:
$ find ./dependencies.files/ -type f -name '*.yml' -exec mamba env create -f {} \;Install r packages as follows:
$ Rscript dependencies.files/install.R.packages.RThe files used as input or that were generated by the distiller-nf are listed below
NOTE: ... indicates that these files exist for all samples/pairs, truncated for brevity
$ tree -L 1 /data/talkowski/Samples/16p_HiC/results/ -P "*_library|mapped_*|*qc"
└── results/
├── sample.QC/
│ ├── multiqc.reports/
│ │ ├── fastp.multiqc.html
│ │ ├── fastqc.multiqc.html
│ │ ├── pairtools.multiqc.html
│ │ └── qc3C.multiqc.html
├── coolers_library/
│ ├── 16p.NSC.DEL.A3.TR1/
│ │ ├── 16p.NSC.DEL.A3.TR1.hg38.mapq_30.1000.cool
│ │ ├── 16p.NSC.DEL.A3.TR1.hg38.mapq_30.1000.mcool
│ │ ├── 16p.NSC.DEL.A3.TR1.hg38.no_filter.1000.cool
│ │ └── 16p.NSC.DEL.A3.TR1.hg38.no_filter.1000.mcool
│ └── .../
├── fastqc/
│ ├── 16p.NSC.DEL.A3.TR1/
│ │ ├── 16p.NSC.DEL.A3.TR1.lane1.0.2_fastqc.html
│ │ ├── 16p.NSC.DEL.A3.TR1.lane1.0.2_fastqc.zip
│ │ ├── 16p.NSC.DEL.A3.TR1.lane1.0.1_fastqc.html
│ │ └── 16p.NSC.DEL.A3.TR1.lane1.0.1_fastqc.zip
│ └── .../
├── mapped_parsed_sorted_chunks
│ ├── 16p.NSC.DEL.A3.TR1/
│ │ ├── 16p.NSC.DEL.A3.TR1.lane1.hg38.0.fastp.json
│ │ ├── 16p.NSC.DEL.A3.TR1.lane1.hg38.0.fastp.html
│ │ ├── 16p.NSC.DEL.A3.TR1.lane1.hg38.0.pairsam.gz
│ │ └── 16p.NSC.DEL.A3.TR1.lane1.hg38.0.bam
│ └── .../
└── pairs_library
├── 16p.NSC.DEL.A3.TR1/
│ ├── 16p.NSC.DEL.A3.TR1.hg38.dedup.stats
│ ├── 16p.NSC.DEL.A3.TR1.hg38.dups.bam
│ ├── 16p.NSC.DEL.A3.TR1.hg38.dups.pairs.gz
│ ├── 16p.NSC.DEL.A3.TR1.hg38.nodups.bam
│ ├── 16p.NSC.DEL.A3.TR1.hg38.nodups.pairs.gz
│ ├── 16p.NSC.DEL.A3.TR1.hg38.unmapped.bam
│ ├── 16p.NSC.DEL.A3.TR1.hg38.unmapped.pairs.gz
│ └── 16p.NSC.DEL.A3.TR1.hg38.nodups.pairs.gz.px2
└── .../
List of coallated results across annotation types + analyses
$ tree -L 1 /data/talkowski/Samples/16p_HiC/results/ -I "*_library|mapped_*|*qc"
results/
├── RNASeq
│ ├── all.DESeq2.results.tsv
│ ├── all.expression.data.tsv
│ ├── all.TRADE.results.tsv
│ ├── DESeq2/
│ └── expression/
├── sample.QC
│ ├── coverage/
│ ├── expected.coverage/
│ ├── minimum.viable.resolutions.tsv
│ ├── multiqc.reports/
│ └── resolution.coverage.summaries.tsv
├── hicrep
│ ├── all.hicrep.cmds.txt
│ ├── all.hicrep.scores.tsv
│ └── results
├── weiner.replication
│ └── plots/
├── TAD
├── loops
├── compartments
│ ├── all.cooltools.compartments.tsv
│ └── results_compartments/
├── multiHiCCompare
│ ├── all.multiHiCCompare.n.results.tsv
│ ├── all.multiHiCCompare.results.tsv
│ └── results/
└── gghic.plots
└── plots/
Each sample as a .yml file (in ./sample.configs) specifiying the params for distiller-nf to run the sample with. We separte each sample into its own file so we can run them in parallel, but the only difference between files are the input fastq files, all pipeline parameters are the same.
We use 64 cores total, with 128 maxCPUs set in the nextflow config.
This fully processes a sample (.fastq -> .mcool) with ~400M reads in ~9h.
Notice that all files contain a SampleID specifying which library the results are.
Some files contain a pair of SampleIDs since they are comparing 2 samples e.g. HiCRep.
All SampleIDs are follow the same format format Project.CellType.Genotype.BioRepID.TechRepID e.g. 16p.NSC.DEL.A3.TR1
16p: project this sample is a part ofNSC: Celltype {NSC, iN}DEL: Genotype of the sample for the region/gene of interest {WT,DEL,DUP}A3: ID string specifiying which biological replicate the sample isTR1: ID string specifiyng which technical replicate the sample is
$ tree /data/talkowski/Samples/16p_HiC/
./
├── GRCh38.reference
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.chrom.sizes
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.amb
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.ann
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.bwt
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.fai
│ ├── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.pac
│ └── GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.sa
├── HiC.16p.sample.metadata.tsv # metadata for all HiC samples
├── fastq/ # Raw reads for HiC samples
│ ├── 22LCC2LT4_3_2148261314_16pDELA3NSCHiC_S1_L003_R1_001.fastq.gz
│ ├── 22LCC2LT4_3_2148261314_16pDELA3NSCHiC_S1_L003_R2_001.fastq.gz
│ └── *.fastq.gz
└── sample.configs/ # Config files for distiller
├── template.distiller.yml # defining distiler-nf params
├── 16p.NSC.DEL.A3.TR1.distiller.yml # Template with specific fastq files
└── *.distiller.yml
Annotations of genomic features, used for association analyses
$ tree -L 1 /data/talkowski/Samples/16p_HiC/reference.files/
reference.files/
├── raw.FGE.data/ # downloaded functional genomic element annotations used for FGE analysis
├── genome.bins/ # coordinates of genomic bins at various resolutions
└── genome.tracks/ # binwise metrics computed across the genome by cooltools
└── track.type_genecov/
Generate .mcool files from .fastq files using the distiller-nf pipeline
# Generate individual yml files each sample for the distiller-nf pipeline using a template
# Variables in the template are defined in locations.R or in make.distiller.configs.R
Rscript ./scripts/distiller/make.distiller.configs.R ./sample.configs/template.distiller.yml
# Run the distiller-nf pipeline by submitting each sample config as an individual SLURM job
module load wget; module unload java; module load singularity/3.7.0; conda activate distiller
./scripts/distiller/run.distiller.sh -w "./work_${USER}" -a ~/miniforge3 ./sample.configs/16p.*.ymldistiller-nf outputs multiple QC reports/files per sample than can each be aggregated into a single multiqc report docs. In the two previous steps we have also generated stats files which can be aggretaed via multiqc into their own reports. Ultimately we can generate 3 multiqc reports
fastqc: Quality statistics for readsfastp: Trimming+MAPQ statistics for filtered reads that were aligned
pairtools stats: Summary statistics of processed HiC pairs
Generate various QC results from distiller-nf output files
# Generate qc3C report for each bam file
# ./scripts/matrix.utils.sh qc3C ./results/sample.QC/qc3C/ DpnII HinfI results/mapped_parsed_sorted_chunks/**/*.bam
# Generate multiQC reprots from fastqc and pairtools resutls
./scripts/matrix.utils.sh multiqcs ./results/sample.QC/multiqc.reports/ ./results/Two things we want to check to QC matrix samples
- pair frequency by distance
- Cis/Trans pair frequency
- minimum resolution as defined in Rao et al. 2014.
Metrics 1 and 2 are calcualted by distiller-nf and found in the *.dedup.stats files.
For metric 3 and subsequent plots we need to calculate the per-bin coverage using cooltools coverage .
For some subsequent steps we will analyze matrices formed by merging all biological/technical replicates for a given Edit+Genotype+CellType (e.g. 16p+WT+NSC). Merging her means summing all the total number of contacts over all samples for each bin-bin pair (matrix entry) and is handled by cooler merge.
# Generated merged matrices for each condition
./scripts/matrix.utils.sh merge_16p ./results/coolers_libraryWe create merged matrices for MAPQ filtered for each which produces the following files
results
└── coolers_library
├── 16p.NSC.WT.Merged.Merged/
│ ├── 16p.NSC.WT.Merged.Merged.hg38.mapq_30.1000.cool
│ └── 16p.NSC.WT.Merged.Merged.hg38.mapq_30.1000.mcool
├── 16p.NSC.DEL.Merged.Merged/
│ └── ...
└── ...We use HiCRep to calculate the "reproducibility score" for all pairs of sample matrices, under several parameter combinations. The command below actually runs the HiCRep and produces 1 file per sample pair + parameter combination, each file contains scores for each chromosome separately (chr{1..22,X,Y}).
After this there is a .Rmd notebook that coallates these files into a single neat dataframe that is used for plotting.
Generate HiCRep results
# Generate list of commands to run for all pairs of matrices + all hyper-param combos
conda activate hicrep
Rscript ./scripts/hicrep/run.hicrep.R && parallel -j $(nproc) --bar --eta :::: ./results/hicrep/all.hicrep.cmds.txtGenerate Weiner et al. 2022 replication analysis figures
Rscript ./scripts/weiner.replication/make.replication.figures.RMake annotated HiC heatmaps w/gghic
Rscript ./scripts/gghic.plots/plot.annotated.contact.heatmaps.RHere is a convenient function to compile rmarkdown notebooks in bash
Put this in your .bashrc to use it
# can put this in your .bashrc or in a script
knit() {
input_rmd="$(readlink -e "${1}")"
if [[ -z "${2}" ]]; then
self_contained="FALSE"
else
self_contained="TRUE"
fi
echo "self_contained: ${self_contained}"
output_html="${input_rmd%.Rmd}.html"
R -q -e "
library(rmarkdown)
rmarkdown::render(
input='${input_rmd}',
output_file='${output_html}',
output_dir=NULL,
output_format=
rmdformats::html_clean(
code_folding='hide',
df_print='paged',
self_contained=${self_contained},
lightbox=TRUE,
gallery=TRUE,
toc_depth=5,
thumbnails=FALSE
)
)"
}
# run function to compile notebook
$ knit notebook.Rmd List all rmarkdown notebooks withsets of results
$ tree -L 1 ./notebooks -P '*.html'
/data/talkowski/Samples/16p_HiC/notebooks/
├── matrix.QC.html
├── matrix.coverage.html
├── hicrep.html
├── weiner.replication.html
├── TADs.html
├── loops.html
├── loop.reproducibility.html
└── multiHiCCompare.html