Home

Unix scripts:

Transfer directories/files from one server to another one using rsync:

rsync -avp --progress <SOURCE PATH> <YOUR_USER@YOUR_SERVER.XX.EDU>:TARGET PATH

[Need to add documentation to this useful command]

---

Define a new command/variable in PATH:

vim ~/.bash_profile, and then:

Inside the opened file, .bash_profile: add the path of your program or whatever to the PATH variable by adding :, then the path.

Then run, source ~/.bash_profile to apply the changes.

---

Append a header using sed:

sed -i $'1 i\id\tfilename\tmd5\tsize\tstate' segment_a[b-z]

• $: permits adding \t, \n to the expression.
• segment_a[b-z]: this will append to the segment_a files that its name ends with [b-z] letters.

Split large file into file with smaller chunks using split:

split -l 50 manifest.txt segment_

• -l option to specifies the number of lines (chunks) that we would like to be in each of the smaller files.
• manifest.txt, is the large filename to be chunked.
• segment_, is a prefix that will be used in front of each of the smaller files' names.

Run a command iteratively on each of the files using a for loop:

for file in ./*  
do  
    gdc-client download -m "$file"  
done

Search a set of directories and execute a command using find:

find * -maxdepth 0 -type d -not -name "data_batch_1" -exec mv {} ./data_batch_1/ \;

• -maxdepth: specifies the depth of the search.
• -type: the type of object to be search d or f, namely directory or file.
• -not and -name: find any directory or file that does not have the specified name.
• -exec: used to specify a command to be executed on the found directory(s) or file(s).
• \;: these symbols must be used at the end of find command after using -exec option.

Draw a hierarchical structure of the directories using tree:

tree -d -C -L 1

• -d: specifies directories.
• -C: color the the directories.
• -L: the depth of the tree.

Listing all the directory under current directory using ls:

ls -d */

Listing all .fastq files inside each of the children directories using ls [NEED TO DOUBLE CHECK THIS CMD]

ls */ *.fastq

Count the number of pattern matching in vim using gn flag:

:%s/pattern//gn

Configure Git command-line colors in the server

git config --global color.ui true

Python core tricks:

Suppress warning message while executing python program:

python -W ignore file.py

Pandas scripts:

Know the maximum width of columns:

pd.options.display.max_colwidth

Adjust the the width, by simply assigning a value:

pd.options.display.max_colwidth = 500

Extract string from column values:

pd.Series.str.extract('(TCGA-\w{2}\-)', expand=False)

• I am using ( and ) to signify the capturing group i.e. the group of regex that I want to capture.
• Note: Make sure the series is essentially str object using the dtype method. In case the type is vague, use .apply(str) method on the series to change its type to str object.

Sort DataFrame column using .sort_values() method:

df_expre.sort_values('case_ids')

Count Number of `NaN in a dataframe:

df.isnull().T.any().T.sum()

Find the rows that haveNaN values in a dataframe:

df[df.isnull().T.any().T]

Jupyter/iPython tricks:

Run python script file inside ipython:

%run "C:\code\python_code.py"

Access each file in each of the sub-directories (children) using glob method:

import glob
files = glob.glob('/parent_directory/**/*.htseq_count.txt')

• ** matched 0 or more sub-directories (children directories).
• the * in ``*.htseq_count.txt is expanded to every file ending with `.htseq_count.txt.