{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Overview\n", "1. [Personal Notebooks](#Maintaining-Personal-Files): Downloading notebooks that you modified\n", "2. [HTS Course Material](#Accessing-Course-Material): Downloading official course material: notebooks, lecture slides, etc\n", "3. [HTS Computing Environment](#Docker-Containers): Downloading and running the HTS Docker Image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# High Priority\n", "\n", "If you have **created or modified files** in your Jupyter container that you would like to preserve, we recommend that you follow the instructions for [Maintaining Personal Files](#Maintaining-Personal-Files).\n", "- We recommend you do this before *as soon as possible*, certainly before the beginning of August, regardless of whether you plan to renew your Docker container or not\n", "- Keep in mind that you only need to worry about this for files that you have **created or modified**. The course material that we created and shared with you will continue to be publicly available.\n", "\n", "## Important Notes\n", "\n", "See details below, but please keep the following in mind:\n", "1. The course material will remain in the [hts2019-notebooks repo](https://gitlab.oit.duke.edu/HTS2019/hts2019-notebooks) and will be publicly available, in perpetuity (or as long as continues to exist), regardless of your affiliation with Duke (or lack thereof). See below for details.\n", "2. The configuration and build information for the course Docker container will remain in the [jupyter-hts-2019 repo](https://gitlab.oit.duke.edu/HTS2019/jupyter-HTS-2019) and will be publicly available, in perpetuity (or as long as continues to exist), regardless of your affiliation with Duke (or lack thereof).\n", "3. The HTS 2019 Docker image will remain at and will be publicly available, in perpetuity (or as long as continues to exist), regardless of your affiliation with Duke (or lack thereof).\n", "\n", "# Maintaining Personal Files\n", "\n", "## Tar and Download\n", "\n", "### Tarring\n", "#### Tarring: Get Everything (simplest)\n", "The simplest thing to do is to make a tarball of the whole directory that contains your **personal files**. Assuming that you have just been copying and renaming files in the hts2019-notebooks directory." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tar -cvzf ~/work/hts2019-notebooks.tar.gz ~/work/hts2019-notebooks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Tarring: Only Notebooks\n", "If you *only want to get notebooks*, you could use the following to only grab the notebooks from hts2019-notebooks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "find ~/work/hts2019-notebooks \\\n", " -name \"*.ipynb\" \\\n", " -not -path \"*/.ipynb_checkpoints/*\" \\\n", " | tar -cvzf ~/work/hts2019-notebooks.tar.gz -T -" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Tarring: Only Notebooks with a common name\n", "\n", "And if you want only modified notebooks, and saved them with a standard naming scheme, e.g. leaving `-Copy1` in the name, for example, renaming `demultiplex.ipynb` to `demultiplex-Copy1.ipynb`, you could use the following to only grab the modified files from hts2019-notebooks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "find ~/work/hts2019-notebooks \\\n", " -name \"*-Copy1*\" \\\n", " -not -path \"*/.ipynb_checkpoints/*\" \\\n", " | tar -cvzf ~/work/hts2019-notebooks.tar.gz -T -" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tar -tvf ~/work/hts2019-notebooks.tar.gz" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Downloadng the tarball\n", "\n", "Now you can do one of the following to download the tarball to your laptop:\n", "\n", "1. Click on the \"Jupyter\" logo above to open the Jupyter file browser\n", "2. Naviagte your way to the directory where you saved the tarball. The tarball should be in the toplevel directory, which is usually what opens when you click the Jupyter logo, but if not, you can click on the folder icon near the top of the file browser\n", "3. Click the checkbox next to `hts2019-notebooks.tar.gz`\n", "4. Click the *Download* button near the top of the Jupyter window to download it.\n", "\n", "### Unpacking the tarball\n", "On a Mac you can \"untar\" by double clicking on the file in finder, or at the terminal with the command `tar -zxf hts2019-notebooks.tar.gz`.\n", "\n", "On Windows, you can download software that will do it, such as [7-Zip](http://www.7-zip.org/)\n", "\n", "\n", "# Accessing Course Material\n", "\n", "You can access the course material in three different ways:\n", "1. You can browse and download the materials from the [hts2019-notebooks repo](https://gitlab.oit.duke.edu/HTS2018/hts2018-notebooks)\n", "2. You can **clone** the repo using git: `git clone `\n", "3. You can browse the material at: \n", "\n", "# Docker Containers\n", "\n", "## Course Instances\n", "\n", "### Duke Affiliates\n", "\n", "#### Container Renewal\n", "Before the begining of the fall semester you **should** receive an email telling you that your HTS2019 container will expire soon, unless you renew it. If you want to maintain your container, we strongly recommend that you keep an eye out for this email, or just be proactive and regularly check [VM Manage](https://vm-manage.oit.duke.edu/containers) for a \"renewal\" link. In the past course participants have lost their containers and contents because they failed to renew in time. \n", "\n", "#### Reduced Capabilities\n", "Soon after the course ends, the containers will be shifted to a server with drastically reduced the resources. That means that running analyses on those containers may not work and may crash the container.\n", "\n", "### Non-affiliates\n", "\n", "For those not affiliated with Duke, the containers will be deactivated before the fall semester. If you want to save any files from the course that you have modified, you must follow the instructions for [Maintaining Personal Files](#Maintaining-Personal-Files) as soon as possible.\n", "\n", "## Running the Docker Image Elsewhere\n", "\n", "### Running on your local machine\n", "#### Quick Start\n", "The script that was demonstrated in class for installing the course Jupyter image and downloading the hts2019-notebooks repo is at . It should work in any environment that meets the requirements listed below. Run it like this (on your local computer or server):\n", "\n", "```\n", "bash reproducible_demo.sh\n", "```\n", "Once you start the Jupyter container it will continue running, even if you are not actively using Jupyter.\n", "\n", "For detailed instructions see [Install docker](#Install-docker) and [Run docker](#Run-docker) below. If this \n", "\n", "##### Requirements\n", "\n", "- git\n", "- docker\n", "- bash\n", "\n", "###### Install docker\n", "\n", "To run a container on your local machine or laptop, download the docker program from . There is a tab at the top of the page that says 'Get Docker'. You can run it on OS X, Windows, and Linux:\n", "- [Docker for Mac](https://docs.docker.com/docker-for-mac/) \n", "- [Docker for Windows](https://docs.docker.com/docker-for-windows/)\n", "- [Linux: Docker Community Edition](https://store.docker.com/search?type=edition&offering=community&operating_system=linux)\n", "\n", "\n", "\n", "#### Run docker\n", "If you successfully ran `reproducible_demo.sh`, as described above, you can skip this step and go on to [Open the notebook in your browser\n", "](#Open-the-notebook-in-your-browser)\n", "\n", "Once you have the docker program installed, open the program (you should get a terminal screen with command line). Enter the command:\n", "```\n", "docker pull dukehtscourse/jupyter-hts-2019\n", "```\n", "\n", "This will pull down the course docker image from dockerhub. It may take a few minutes. Next, run the command to start a container:\n", "```\n", "docker run --name hts-course -v YOUR_DIRECTORY_WITH_COURSE_MATERIAL:/home/jovyan/work \\\n", "-d -p 127.0.0.1\\:9999\\:8888 \\\n", "-e PASSWORD=\"YOUR_CHOSEN_NOTEBOOK_PASSWORD\" \\\n", "-e NB_UID=1000 \\\n", "-t dukehtscourse/jupyter-hts-2019\n", "```\n", "The most important parts of this verbiage are the `YOUR_DIRECTORY_WITH_COURSE_MATERIALS` and `YOUR_CHOSEN_NOTEBOOK_PASSWORD`. \n", "- `YOUR_DIRECTORY_WITH_COURSE_MATERIALS` (Bind mounting): The directory name is the one you extracted your course materials into. So, if you put them in your home directory, it might look something like: `-v /home/janice/HTS2019-notebooks:/home/jovyan/work`\n", "- `YOUR_CHOSEN_NOTEBOOK_PASSWORD`: The password is whatever you want to use to password protect your notebook. Now, this command is running the notebook so that it is only 'seen' by your local computer - no one else on the internet can access it, and you cannot access it remotely, so the password is a bit of overkill. Use it anyway. An example might be: `-e PASSWORD=\"Pssst_this_is_Secret\"` except that this is a terrible password and you should follow standard rules of not using words, include a mix of capital and lowercase and special symbols. etc.\n", "- `-d -p 127.0.0.1\\:9999\\:8888` part of the command is telling docker to run the notebook so that it is only visible to the local machine. It is absolutely possible to run it as a server to be accessed across the web - but there are some security risks associated, so if you want to do this proceed with great caution and get help.\n", "\n", "#### Open the notebook in your browser\n", "\n", "Open a browser and point it to http://127.0.0.1:9999\n", "You should get to a Jupyter screen asking for a password. This is the password you created in the docker run command.\n", "Now, you should be able to run anything you like from the course. Depending on your laptop's resources (RAM, cores), this might be slow, so be aware and start by testing only one file (vs the entire course data set).\n", "\n", "#### Stopping Docker\n", "The container will continue running, even if you do not have Jupyter open in a web browser. If you don't plan to use it for a while, you might want to shut it down so it isn't using resources on your computer. Here are two ways to do that:\n", "##### Kitematic\n", "Included in the [Docker for Mac](https://docs.docker.com/docker-for-mac/) and the [Docker for Windows](https://docs.docker.com/docker-for-windows/) installations.\n", " \n", "##### Commandline\n", "You may want to familiarize yourself with the following Docker commands.\n", "- `docker stop`\n", "- `docker rm`\n", "- `docker ps -a`\n", "- `docker images`\n", "- `docker rmi`\n", "\n", "#### Windows Note\n", "These instructions have not been tested in a Windows environment. If you have problems with them, please give us feedback\n", "\n", "### Running on a server\n", "To run on a remote server you will want to use a slightly different command from above, because you *will need to connect remotely*:\n", "\n", "```\n", "docker run --name hts-course \\\n", "-v YOUR_DIRECTORY_WITH_COURSE_MATERIAL:/home/jovyan/work \\\n", "-d -p 8888:8888 \\\n", "-e USE_HTTPS=\"yes\" \\\n", "-e PASSWORD=\"YOUR_CHOSEN_NOTEBOOK_PASSWORD\" \\\n", "-e NB_UID=1000 \\\n", "-t dukehtscourse/jupyter-hts-2019\n", "```\n", "\n", "#### Running on a Duke server (Duke Affiliates)\n", "\n", "If you want to use the HTS Docker container for research, contact Mark Delong and the research computing people. Andy Ingham (one of Mark DeLong's guys) can help with setup of a research VM with the HTS Docker container. If there is enough interest we could also look at having an option for an HTS image that could be automatically provisioned as part of the Research Toolkits/RAPID service for Duke researchers in general.\n", "\n", "#### Running on a non-Duke server\n", "\n", "Sys admins at other institution may be able to help set you up with compute resources if you let them know that you want to use a dockerhub image. Mark McCahill (OIT Wizard, class 50, and the person who keeps our course containers humming along) has generously offered to consult with IT people at other institutions to get you up and running. You can also run Docker images in AWS or Azure cloud environments. Mark McCahill can probably give you some pointers here too.\n", "\n", "### Running the Course Image with Singularity\n", "Docker requires root permissions to run, so you are unlikely to be able to run Docker on a computer that you are not fully in control of. As an alternative you can run the course image with [Singularity](https://sylabs.io/singularity/), another container system. Singularity is similar to Docker, and can run Docker images, but you do not need special permissions to run Singularity images *or* Docker images with Singularity (as long as Singularity is actually installed on the computer).\n", "\n", "The following command uses Singularity to start up a container from the course Jupyter image.\n", "```\n", "export USE_HTTPS=yes; \\\n", " singularity exec docker://dukehtscourse/jupyter-hts-2019 \\\n", " /usr/local/bin/start-notebook.sh \n", "```\n", "\n", "#### Install Singularity\n", "Here are instructions for installing:\n", "\n", "- [Singularity version 2.6](https://sylabs.io/guides/2.6/user-guide/quick_start.html#quick-installation-steps)\n", "- [Singularity version 3.2](https://sylabs.io/guides/3.2/user-guide/quick_start.html#quick-installation-steps)\n", "- [Singularity Desktop for macOS (Alpha Preview)](https://sylabs.io/singularity-desktop-macos/)\n", "\n", "\n", "# Select Duke Computing Resources\n", "\n", "- [Duke Research Computing Website](https://rc.duke.edu)\n", "- [Research Computing Links on OIT Website](https://oit.duke.edu/what-we-do/services/research-computing)\n", "\n", "## Individual Virtual Machines\n", "\n", "- [Virtual Computing Manager](https://oit.duke.edu/help/articles/vcm-how-use-virtual-computing-manager): Duke offers free small VMs (2 core, 2GB RAM) all affiliates through a system called VCM. For running the Jupyter containers you will want some version of Linux, such as \"Ubuntu 16.04\"\n", "- [Research Toolkits](https://oit.duke.edu/what-we-do/applications/research-toolkits): Duke faculty researchers have access to moderate size VMs (4 core, 32GB RAM)\n", "\n", "## Clusters\n", "\n", "- [Duke Compute Cluster](https://rc.duke.edu/the-duke-compute-cluster/)\n", "- [HARDAC](https://genome.duke.edu/cores-and-services/computational-solutions/compute-environments-genomics)\n", "\n", "# Assorted Notes on Approaches to Reproducible Analysis\n", "\n", "## Scripting Options\n", "\n", "I have used shell scripts, makefiles, Jupyter notebooks, and Rmarkdown notebooks as “master scripts” for running complex analyses. My current preference is for Rmarkdown notebooks. Jupyter and Rmarkdown let you combine code from different languages in a single notebook, so you can have a bash cell followed by an R cell followed by a python cell. You can’t put other languages in shell scripts or makefiles, so if you want to combine multiple languages with these, you need to put that code in its own script and run that script similar to how you would run a binary executable.\n", "\n", "One thing I like about Rmarkdown is how easy it is to generate good looking reports/manuscripts.\n", "\n", "## Singularity\n", "\n", "I switched from Docker to Singularity for several reasons that are mostly summed up by the fact that Singularity was designed for use in a research environment. As far as you are concerned the most important issue is that you have to have root to run Docker on a machine, so no cluster sysadmin will ever let you run your own Docker container. Singularity runs fine in user space and is installed on both of the Duke clusters I have access to. If you need to convince the cluster admin to install it, see this: .\n", "\n", "I put together these notes about [getting started with Singularity](https://gist.github.com/granek/6f492ed1593bcc0c1c6cafe242a2e5d4). They were intended to run on a Duke VCM or RAPID VM, but should work on any machine where you have root. If you don’t have root on any linux machine, you can try to run Singularity on your local machine: or \n", "\n", "## Etc\n", "\n", "I almost never run anything locally on my computer, I do everything through web interfaces (RStudio Server) and command line session over ssh, both running on remote servers. I end up running most of my research projects on the Duke VCM or RAPID machines, which I can get away with because microbial datasets tend to be relatively modest. \n", "\n", "These days when I start a new project, I prepare a Singularity Recipe with the software I expect to need and build a Singularity image from it. I usually include RStudio Server in these images so I can use that interface to develop Rmarkdown notebooks and run the analysis. For the occasional project where I need a bigger machine I still build the image on my VCM or RAPID machine (building a singularity image requires sudo, and these are the only machines where I have root), and copy the image over to a bigger server. I have access to one biggish server where I can run RStudio Server from a Singularity image, but I don’t think the clusters on campus allow this, so I am stuck working over an ssh session.\n", "\n", "All my scripts, config files, metadata, and Singularity Recipes/Dockerfiles live in a git repo that is synced to Github/Bitbucket/Gitlab. This makes it easy to move this stuff between servers if I need to upgrade. I usually end up having to rsync data and Singularity images between servers, because only the VCM and RAPID servers can share storage. If I do this, I will tend to move everything over to the bigger server, instead of trying to work in two places, because it gets confusing. The one exception is that if I need to update the singularity image that has to be done on one of the VMs I control.\n", "\n", "As much as possible, I avoid running things on clusters because they make things more complicated: I can’t use RStudio and I have to fiddle with SLURM. When I do need to work on a machine that won’t allow RStudio, I will often edit files with Emacs running locally (because it is set up how I like and has a nice mac os interface) using tramp-mode. Several other text editors allow you to edit remote files over ssh." ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 2 }