Unix capstone exercise

Ths capstone exercise covers essentially all of the Unix commands you need to develop your bioinformatics piepliens. If you can do this exercise without too much difficulty, you are ready to process raw RNA-seq data!

But don’t be discouraged if you find this difficutl - unless you have quite a lot of previous programming experience, this exerise is likely to be challenging.

General hints:

• Use pwd liberally to check that you are in the right directory

• Each command should execute almost instantly - if not, something is problay wrong with your command

• Commands in Jupyter can appear to “hang” when there is an error - try the command in a terminal to see what’s wrong

• Remember to configure so that you get a warning with unbound variabels

• Use variables liberally - if a name is used more than once, it should probably be assinged to a variable

• There is no restriction on using any search engine or reviewing previous notebooks

1. Make two directories data and results. Check that the directories were created.

Hint:

• You can list just directories with ls -d */

pwd

/home/jovyan/work/Duke_HTS2019/cliburn

if [ -d "data" ]
then
    echo "Directory data exists."
else
    echo "Create folder data."
    echo mkdir data
fi

Directory data exists.

if [ -d "results" ]
then
    echo "Directory data exists."
else
    echo "Create folder results."
    echo mkdir results
fi

Directory data exists.

ls -d */

data/  figs/  results/  seqs/

2. In data, create the files expt-1, expt-2, epxt-3, each with 5 rows of 3 random numbers between 0 and 9 (inclusive) separated by tabs. When done, display the contents of each file.

Hint:

• Create 3 for loops

  • the first to create a file,

  • the next to append a row of numbers, and add a newline

  • and the third to generate the numbers in the row.

• You can generate random integers in Unix by repeatedly capturing the value of the special variable $RANDOM.

• Initialize the RANDOM variable with the seed 42.

• The arithmetic modulo opertor is % (gives remainder after diviison)

• The tab character is represented by the string \t

• The argument -ne to echo means do not emit a newline, evaluate specail characaters like \t

• You can generate sequences of numbers using brace expansion or seq

• Arithmetic operatinons are evaluated like this $((1+2))

cd data

RANDOM=42
for FILE in expt-{1..3}
do
    rm $FILE
    touch $FILE
    for row in {1..5}
    do
        for col in {1..3}
        do
            echo -ne $(($RANDOM % 10))'\t' >> $FILE
        done
        echo >> $FILE
    done
done

rm: cannot remove 'expt-1': No such file or directory
rm: cannot remove 'expt-2': No such file or directory
rm: cannot remove 'expt-3': No such file or directory

for FILE in expt-{1..3}
do
    echo $FILE
    cat $FILE
done

expt-1
6       1       1
3       8       7
9       0       6
3       4       9
2       5       2
expt-2
0       4       6
8       4       3
6       2       7
3       2       9
1       5       9
expt-3
0       2       5
7       8       5
4       5       3
5       8       7
8       3       1

3. Create an MD5SUM file containing the checksums of the contents within expt-1, expt-2, and epxt-3. Show the contents of the MD5SUM file.

Hint:

• md5sum needs a list of files as arguments

ls

expt-1  expt-2  expt-3  gene_counts_raw.txt  gene_counts.txt

md5sum * > MD5SUM

cat MD5SUM

72e21e4726baa4296d0722161c92d19b  expt-1
c98f878743247e78e2dc5dacfb0e3b7f  expt-2
0bd25d1865615e48cc55d8f7ad88c440  expt-3
a61102c86253625061a43f7a56e3131c  gene_counts_raw.txt
a71c7c2c2b84cc10d9c1519d5cac5bc5  gene_counts.txt

4. Create a data.tar.gz file containing all the contents of the data directory

cd ..

tar -czvf data.tar.gz data

data/
data/expt-1
data/expt-2
data/gene_counts.txt
data/expt-3
data/gene_counts_raw.txt
data/MD5SUM

5. Move data.tar.gz to the results directory and recreate the original files

mv data.tar.gz results

cd results

tar -xzvf data.tar.gz

data/
data/expt-1
data/expt-2
data/gene_counts.txt
data/expt-3
data/gene_counts_raw.txt
data/MD5SUM

ls

data  data.tar.gz

6. Create a new file expt-sum.txt in results with 3 lines showing the sum of the numbers in the 1st, 2nd and 3rd columns over all 3 expt-x files. That is, the first row of expt-sum.txt contains the sum of the 1st column of expt-1, expt-2, and expt-3 and so on. Use globbing to get the files expt-1, expt-2, and expt-3.

Hint:

• Remember the cut command

• Use a double for loop

  • The outer loop selects a file

  • The inner loop sums up the numbers in the appropriate column of the file

DEST_FILE='expt-sum.txt'
touch $DEST_FILE
COL=1
for FILE in $(ls data/expt*)
do
    SUM=0
    NS=$(cat $FILE | cut -f$COL)
    for N in $NS
    do
        SUM=$((SUM + N))
    done
    COL=$((COL + 1))
    echo $SUM >> $DEST_FILE
done

cat $DEST_FILE

23
17
21

7. Delete the data and reults directory recursively

pwd

/home/jovyan/work/Duke_HTS2019/cliburn/results

cd ..

ls -d */

data/  figs/  results/  seqs/

rm -r data results

ls -d */

figs/  seqs/