{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unix capstone exercise\n",
    "\n",
    "Ths capstone exercise covers essentially all of the Unix commands you need to develop your bioinformatics piepliens. If you can do this exercise without too much difficulty, you are ready to process raw RNA-seq data!\n",
    "\n",
    "But don't be discouraged if you find this difficutl - unless you have quite a lot of previous programming experience, this exerise is likely to be challenging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "General hints:\n",
    "\n",
    "- Use `pwd` liberally to check that you are in the right directory\n",
    "- Each command should execute almost instantly - if not, something is problay wrong with your command\n",
    "- Commands in Jupyter can appear to \"hang\" when there is an error - try the command in a terminal to see what's wrong\n",
    "- Remember to configure so that you get a warning with unbound variabels\n",
    "- Use variables liberally - if a name is used more than once, it should probably be assinged to a variable\n",
    "- There is no restriction on using any search engine or reviewing previous notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Make two directories `data` and `results`. Check that the directories were created.\n",
    "\n",
    "Hint:\n",
    "\n",
    "- You can list just directories with `ls -d */`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/work/Duke_HTS2019/cliburn\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory data exists.\n"
     ]
    }
   ],
   "source": [
    "if [ -d \"data\" ] \n",
    "then\n",
    "    echo \"Directory data exists.\" \n",
    "else\n",
    "    echo \"Create folder data.\" \n",
    "    echo mkdir data\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory data exists.\n"
     ]
    }
   ],
   "source": [
    "if [ -d \"results\" ] \n",
    "then\n",
    "    echo \"Directory data exists.\" \n",
    "else\n",
    "    echo \"Create folder results.\" \n",
    "    echo mkdir results\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34mdata/\u001b[0m  \u001b[01;34mfigs/\u001b[0m  \u001b[01;34mresults/\u001b[0m  \u001b[01;34mseqs/\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -d */"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. In `data`, create the files `expt-1`, `expt-2`, `epxt-3`, each with 5 rows of 3 random numbers between 0 and 9 (inclusive) separated by tabs. When done, display the contents of each file.\n",
    "\n",
    "  \n",
    "Hint: \n",
    "\n",
    "- Create 3 for loops \n",
    "    - the first to create a file, \n",
    "    - the next to append a row of numbers, and add a newline\n",
    "    - and the third to generate the numbers in the row.\n",
    "- You can generate random integers in Unix by repeatedly capturing the value of the special variable `$RANDOM`.\n",
    "- Initialize the `RANDOM` variable with the seed `42`.\n",
    "- The arithmetic modulo opertor is `%` (gives remainder after diviison)\n",
    "- The tab character is represented by the string `\\t`\n",
    "- The argument `-ne` to `echo` means do not emit a newline, evaluate specail characaters like `\\t`\n",
    "- You can generate sequences of numbers using brace expansion or `seq`\n",
    "- Arithmetic operatinons are evaluated like this $((1+2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: cannot remove 'expt-1': No such file or directory\n",
      "rm: cannot remove 'expt-2': No such file or directory\n",
      "rm: cannot remove 'expt-3': No such file or directory\n"
     ]
    }
   ],
   "source": [
    "RANDOM=42\n",
    "for FILE in expt-{1..3}\n",
    "do\n",
    "    rm $FILE\n",
    "    touch $FILE\n",
    "    for row in {1..5}\n",
    "    do\n",
    "        for col in {1..3}\n",
    "        do\n",
    "            echo -ne $(($RANDOM % 10))'\\t' >> $FILE\n",
    "        done\n",
    "        echo >> $FILE\n",
    "    done\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "expt-1\n",
      "6\t1\t1\t\n",
      "3\t8\t7\t\n",
      "9\t0\t6\t\n",
      "3\t4\t9\t\n",
      "2\t5\t2\t\n",
      "expt-2\n",
      "0\t4\t6\t\n",
      "8\t4\t3\t\n",
      "6\t2\t7\t\n",
      "3\t2\t9\t\n",
      "1\t5\t9\t\n",
      "expt-3\n",
      "0\t2\t5\t\n",
      "7\t8\t5\t\n",
      "4\t5\t3\t\n",
      "5\t8\t7\t\n",
      "8\t3\t1\t\n"
     ]
    }
   ],
   "source": [
    "for FILE in expt-{1..3}\n",
    "do\n",
    "    echo $FILE\n",
    "    cat $FILE\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Create an `MD5SUM` file containing the checksums of the contents within `expt-1`, `expt-2`, and `epxt-3`. Show the contents of the `MD5SUM` file.\n",
    "\n",
    "Hint:\n",
    "\n",
    "- `md5sum` needs a list of files as arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "expt-1  expt-2  expt-3  gene_counts_raw.txt  gene_counts.txt\n"
     ]
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "md5sum * > MD5SUM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72e21e4726baa4296d0722161c92d19b  expt-1\n",
      "c98f878743247e78e2dc5dacfb0e3b7f  expt-2\n",
      "0bd25d1865615e48cc55d8f7ad88c440  expt-3\n",
      "a61102c86253625061a43f7a56e3131c  gene_counts_raw.txt\n",
      "a71c7c2c2b84cc10d9c1519d5cac5bc5  gene_counts.txt\n"
     ]
    }
   ],
   "source": [
    "cat MD5SUM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Create a `data.tar.gz` file containing all the contents of the `data` directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/\n",
      "data/expt-1\n",
      "data/expt-2\n",
      "data/gene_counts.txt\n",
      "data/expt-3\n",
      "data/gene_counts_raw.txt\n",
      "data/MD5SUM\n"
     ]
    }
   ],
   "source": [
    "tar -czvf data.tar.gz data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Move `data.tar.gz` to the `results` directory and recreate the original files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "mv data.tar.gz results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/\n",
      "data/expt-1\n",
      "data/expt-2\n",
      "data/gene_counts.txt\n",
      "data/expt-3\n",
      "data/gene_counts_raw.txt\n",
      "data/MD5SUM\n"
     ]
    }
   ],
   "source": [
    "tar -xzvf data.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34mdata\u001b[0m  \u001b[01;31mdata.tar.gz\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Create a new file `expt-sum.txt` in `results` with 3 lines showing the sum of the numbers in the 1st, 2nd and 3rd columns over all 3 `expt-x` files. That is, the first row of `expt-sum.txt` contains the sum of the 1st column of `expt-1`, `expt-2`, and `expt-3` and so on. Use globbing to get the files `expt-1`, `expt-2`, and `expt-3`.\n",
    "\n",
    "Hint:\n",
    "\n",
    "- Remember the `cut` command\n",
    "- Use a double for loop\n",
    "    - The outer loop selects a file\n",
    "    - The inner loop sums up the numbers in the appropriate column of the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "DEST_FILE='expt-sum.txt'\n",
    "touch $DEST_FILE\n",
    "COL=1\n",
    "for FILE in $(ls data/expt*)\n",
    "do\n",
    "    SUM=0\n",
    "    NS=$(cat $FILE | cut -f$COL)\n",
    "    for N in $NS\n",
    "    do\n",
    "        SUM=$((SUM + N))\n",
    "    done\n",
    "    COL=$((COL + 1))\n",
    "    echo $SUM >> $DEST_FILE\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23\n",
      "17\n",
      "21\n"
     ]
    }
   ],
   "source": [
    "cat $DEST_FILE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Delete the `data` and `reults` directory recursively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/work/Duke_HTS2019/cliburn/results\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34mdata/\u001b[0m  \u001b[01;34mfigs/\u001b[0m  \u001b[01;34mresults/\u001b[0m  \u001b[01;34mseqs/\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -d */"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm -r data results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34mfigs/\u001b[0m  \u001b[01;34mseqs/\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -d */"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}