HTS2019 Source Materials

Source materials can be cloned from the HTS2019 GitLab Repository

The working environment can be built using the dockerfile from Dockerhub: /dukehtscourse/jupyter-hts-2019

Slides

• Slide sets
  • Background knowledge of HTS2019
  • Wet Lab
  • Bioinformatics
  • Biostatistics

Quick Intro of Analysis Tools

• Duke HTS Summer Course 2019
  • Purpose of this component
  • How this is all setup
  • The Notebook
• Introduction to Unix and bash
  • What is an operating system?
• Introduction to R
  • R as simple calculator
  • Basic Programming Concepts
• R libraries and Bioconductor
  • Packages and Libraries
  • BioConductor
• Hands-On Tidyverse - with count data
  • Setup
  • Reading in count data
• Simulations and Statistical Inference
  • Set random number seed for reproducibility
  • Generating numbers
  • Using rep
  • Using sample
  • Random number generators
  • Looping
  • Using replicate
  • Making a data.frame or tibble of simulated data
• Estimation and Hypothesis Testing
  • Set random number seed for reproucibility
  • Functions around probability distributions
  • Point estimates
  • Interval estimates
  • Hypothesis testing
  • Two-sample model

Unix Shell

• Working with Unix
  • File and directory navigation
  • Exercises
• File I/O
  • Creating a file
  • View file contents
  • More redirection
  • View file contents
  • More redirection
  • Redirecting command output
  • Exercises
• File archival and recovery
  • File compression and archival
  • Checksums
  • Exercises
• Working with text
  • Create a FASTA file to play with
  • Using regular expresssions
  • Transliteration
  • Sorting
  • Downloading files
  • File compression/uncompression
  • Inspecting the GTF file
  • Remove comment lines
  • Spliting columns
  • Exercises
• Unix Variables
  • Safety - Disable default variables
  • Assigning variables
  • Common mistakes
  • Using a variable
  • Assigning command outputs to variables
  • Assigning results of an arithmetic expression (integers only)
  • Using variables in loops
  • Fibonacci series
  • Single and double quotes
  • Environment variables
  • Brace expansion
  • Shell scripts
  • Exercises
• Using bash in bioinformatics
  • 0.1. Files, directories and working with text
  • 0.2 Getting files
  • 0.3 Compressing ard archiving files
  • 0.4 Checksums
  • 1. Working with FASTA files
  • 2. Working with FASTQ files
  • 3.Working wtih GTF files
  • 4. Looping
• Unix capstone exercise
• Unix capstone exercise
• The Unix Shell: Regular Expressions
  • Regular Expressions
  • Character set modifiers
• Unix Review of Basics
  • Navigation
  • File manipulation
  • Text
  • Regular expressions
  • I/O and streams
  • Pipe
  • Utilities
  • Scripting

R Graphics and Data Manipulation

• Basics of R (Review)
  • Assignment
  • Custom function
  • Vectors
  • Lists
  • Data Frames
• Basics of R (Review)
  • Assignment
  • Custom function
  • Vectors
  • Lists
  • Data Frames
• Data Manipulation in the tidyverse
  • Load library
  • What data sets are available?
  • Bssic information
  • Piping
  • Select
  • Arrange
  • Filter
  • Mutate
  • Summarize
  • Group by
  • Exercises
• Tidy Data
  • Data set
  • Gather
  • Separate
  • Clean-up and type coercion
  • Joins
  • Exercise
• File I/O
  • Strucrure of count data from STAR
  • Get list of files
  • Read in first 2 files
  • Combining files
  • Custom function to read files
• Unsupervised Learning
  • Review of tidyverse
  • Pairwise scatter plot
  • Calculating distances
  • Agglomerative hierarchical clustering
  • K-means clustering
  • Dimension reduction and ordination
  • Heatmaps
• Unsupervised Learning
  • Golub data set
  • Distances
  • Ordination
  • Preserving the distances
  • Clustering
  • Semi-supervised learning (Noise discovery)
• R Graphics
  • R Graphics Systems
• R Graphics
  • Base Graphics
• R Graphics
  • Simulate some data for plotting
  • Grammar of Graphics
  • Saving plots
• R Graphics Exercise
• R Graphics Exercise (Solutions)

Bioinformatics (FASTQ, FASTQC, STAR, etc)

• Introduction to FASTQ Files
  • Shell Variables
  • Looking at a FASTQ
• What do quality scores mean?
• What do quality scores mean?
  • Phred Quality Scores
  • Ascii Codes
  • Phred Encodings
  • Why +33?
• Quality Control
  • Shell Variables
  • Making New Directories
  • Fastqc
  • MultiQC
• Trimming and Filtering a FASTQ
  • Shell Variables
  • Making New Directories
• Trimming and Filtering
  • Making an adapter file
  • fastq-mcf
• Download and Index Genome
  • Shell Variables
  • STAR
  • Reference Genome and Annotation
• Counting Reads
  • Shell Variables
  • Counting Reads
  • Running htseq-count
• Mapping Reads to a Reference Genome
  • Shell Variables
  • Mapping with STAR
• Making Generic Commands
  • Shell Variables
  • Using a FASTQ variable
  • Now for the real thing …
• Making a Pipeline
  • Shell Variables
  • Combining the Steps
• Working with Loops
  • Shell Variables
  • A Brief journey into for loops
  • Now for the real thing …
• Working with Loops
  • Shell Variables
  • Now let’s run both samples throught the pipeline:
• Looping with Globs
  • Shell Variables
  • Globs?
  • More Complex Globs
  • Looping over a glob
  • String Manipulation
  • A globy pipeline
  • Glob References
• Initialize Variables and Make Directories
• Make adapter file
• Download and Index Genome
• Prepare Data
• Downloading Everything
  • Downloading files through Jupyter
  • Download IGV
• Using IGV
  • Run IGV
  • Load Files
  • Configurations to explore
  • Look around
• Run with shorter intron limit
  • Link Directory
  • Tarring

Analysis of Pilot Data (Gene Count, DESeq2 & DE Analysis)

• Get Star Count of the Pilot 2019 Data
  • Load paths and libraries
  • Gather and spread the first four rows to nmisc
  • Gather and spread the gene rows
  • merge in the 4 misc counts and add summaries
• DESeq2: Create count matrix and metadata data frame
  • Objective: create a count matrix and metadata frame for the 2019 pilot data
• DESeq2: Create Data Objects
  • Objective: Create a DESeqDataSet object
• DESeq2: Basic Differential Expression (DE) analysis
  • Objective: Carry out a basic set of DE analyses using DESeq2 and visualize the results
  • Load the 2019 pilot dds object from image file
  • Estimate Size Factors and Dispersion Parameters
  • Differential Expression Analysis
  • Clustering
• DESeq2: Basic Differential Expression (DE) analysis
  • Objective: Carry out a basic set of DE interaction analysis using DESeq2 and visualize the results
  • Load the 2019 pilot dds object from image file
  • review the steps of DE analysis
  • Exercise: Compare the size factors and dispersion estimates between ddsmult and ddsad
  • Estimate DE effect due to condition given genotype
• Generate figures from DESEq2 analysis
  • Load data and generate DE results
  • Dot Plots
  • Volcano plots
  • Get “expression” data from counts
  • Principal Components Analysis
  • Clustering

Reproducible Analysis (Git)

• Data handling Demo
  • Switch to Jupyter
  • Checksums
  • Checksums File
  • Catching Changes
  • READ-ONLY
  • Preventing Modification
  • Preventing Deletion
  • Clean up
• Git Lesson
  • Fork a Repo in Gitlab
• Git in class
  • Automated Version Control
  • Create a Repository
  • Setting Up Git
  • Tracking Changes
  • Exploring History

After the course... (Docker & Singularity)

• Overview
• High Priority
  • Important Notes
• Maintaining Personal Files
  • Tar and Download
• Accessing Course Material
• Docker Containers
  • Course Instances
  • Running the Docker Image Elsewhere
• Select Duke Computing Resources
  • Individual Virtual Machines
  • Clusters
• Assorted Notes on Approaches to Reproducible Analysis
  • Scripting Options
  • Singularity
  • Etc

Indices and tables

• Index

• Module Index

• Search Page