Working with text¶
Create a FASTA file to play with¶
[1]:
cat > bgp.fasta << EOF
>HSBGPG Human gene for bone gla protein (BGP)
GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
ATAAACAGTGCTGGAGGCTGGCGGGGCAGGCCAGCTGAGTCCTGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
GAGAGGAGGGAAGAGCAAGCTGCCCGAGACGCAGGGGAAGGAGGATGAGGGCCCTGGGGATGAGCTGGGGTGAAC
CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
GCCATCAGGAAGGCCAGCCTGCTCCCCACCTGATCCTCCCAAACCCAGAGCCACCTGATGCCTGCCCCTCTGCTC
CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
GGCCTATCGGCGCTTCTACGGCCCGGTCTAGGGTGTCGCTCTGCTGGCCTGGCCGGCAACCCCAGTTCTGCTCCT
CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
EOF
[2]:
wc bgp.fasta
18 25 1294 bgp.fasta
Using regular expresssions¶
[3]:
cat bgp.fasta |
grep "CCCCC"
GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
[4]:
cat bgp.fasta |
grep -n "CCCCC"
2:GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
9:TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
[5]:
cat bgp.fasta |
grep -nv "CCCCC"
1:>HSBGPG Human gene for bone gla protein (BGP)
3:ATAAACAGTGCTGGAGGCTGGCGGGGCAGGCCAGCTGAGTCCTGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
4:ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
5:CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
6:GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
7:GAGAGGAGGGAAGAGCAAGCTGCCCGAGACGCAGGGGAAGGAGGATGAGGGCCCTGGGGATGAGCTGGGGTGAAC
8:CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
10:GCCATCAGGAAGGCCAGCCTGCTCCCCACCTGATCCTCCCAAACCCAGAGCCACCTGATGCCTGCCCCTCTGCTC
11:CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
12:GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
13:GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
14:AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
15:CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
16:GGCCTATCGGCGCTTCTACGGCCCGGTCTAGGGTGTCGCTCTGCTGGCCTGGCCGGCAACCCCAGTTCTGCTCCT
17:CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
18:ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
[6]:
cat bgp.fasta |
grep "GA*TT.*CA"
GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
[7]:
cat bgp.fasta |
grep -E "^C"
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
[8]:
cat bgp.fasta |
grep -E "G$"
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
[9]:
cat bgp.fasta |
grep -E "^C.*G$"
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
[10]:
cat bgp.fasta |
grep -o "GA*TT.*CA"
GATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCA
GTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCA
GTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCA
GTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCA
GTTGGCTGACCACATCGGCTTTCA
[11]:
cat bgp.fasta |
grep -E "(GCAT)+"
GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
[12]:
cat bgp.fasta |
grep -Eon "(GCA){2,}"
3:GCAGCA
12:GCAGCA
Transliteration¶
[13]:
cat bgp.fasta |
grep -E "^C.*G$"
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
Complement¶
[14]:
cat bgp.fasta |
grep -E "^C.*G$" |
tr ACTG TAGC
ATAAGAAAAGATCCAACATGGCATCGCCCCCAGCTCTCCTCCTTCATAATGCCAAATAAGAGGAGATAAAAGGGC
ATATCAAGGGCGCGAATTCATCCTCCCATCACTCCGTCGCTTCTCTAAATCCACAGTAAGCGTGATTGCCAGCCC
Reverse complement¶
[15]:
cat bgp.fasta |
grep -E "^C.*G$" |
tr ACTG TAGC |
rev
CGGGAAAATAGAGGAGAATAAACCGTAATACTTCCTCCTCTCGACCCCCGCTACGGTACAACCTAGAAAAGAATA
CCCGACCGTTAGTGCGAATGACACCTAAATCTCTTCGCTGCCTCACTACCCTCCTACTTAAGCGCGGGAACTATA
Sorting¶
[16]:
cat bgp.fasta |
grep -v "^>" |
sort
AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
ATAAACAGTGCTGGAGGCTGGCGGGGCAGGCCAGCTGAGTCCTGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
GAGAGGAGGGAAGAGCAAGCTGCCCGAGACGCAGGGGAAGGAGGATGAGGGCCCTGGGGATGAGCTGGGGTGAAC
GCCATCAGGAAGGCCAGCCTGCTCCCCACCTGATCCTCCCAAACCCAGAGCCACCTGATGCCTGCCCCTCTGCTC
GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
GGCCTATCGGCGCTTCTACGGCCCGGTCTAGGGTGTCGCTCTGCTGGCCTGGCCGGCAACCCCAGTTCTGCTCCT
GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
Sort by default uses lexicographic order¶
[17]:
cat bgp.fasta |
grep -nv "^>" |
sort
10:GCCATCAGGAAGGCCAGCCTGCTCCCCACCTGATCCTCCCAAACCCAGAGCCACCTGATGCCTGCCCCTCTGCTC
11:CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
12:GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
13:GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
14:AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
15:CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
16:GGCCTATCGGCGCTTCTACGGCCCGGTCTAGGGTGTCGCTCTGCTGGCCTGGCCGGCAACCCCAGTTCTGCTCCT
17:CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
18:ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
2:GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
3:ATAAACAGTGCTGGAGGCTGGCGGGGCAGGCCAGCTGAGTCCTGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
4:ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
5:CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
6:GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
7:GAGAGGAGGGAAGAGCAAGCTGCCCGAGACGCAGGGGAAGGAGGATGAGGGCCCTGGGGATGAGCTGGGGTGAAC
8:CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
9:TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
use -n flag for numeric order¶
[18]:
cat bgp.fasta |
grep -nv "^>" |
sort -n
2:GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
3:ATAAACAGTGCTGGAGGCTGGCGGGGCAGGCCAGCTGAGTCCTGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
4:ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
5:CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
6:GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
7:GAGAGGAGGGAAGAGCAAGCTGCCCGAGACGCAGGGGAAGGAGGATGAGGGCCCTGGGGATGAGCTGGGGTGAAC
8:CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
9:TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
10:GCCATCAGGAAGGCCAGCCTGCTCCCCACCTGATCCTCCCAAACCCAGAGCCACCTGATGCCTGCCCCTCTGCTC
11:CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
12:GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
13:GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
14:AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
15:CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
16:GGCCTATCGGCGCTTCTACGGCCCGGTCTAGGGTGTCGCTCTGCTGGCCTGGCCGGCAACCCCAGTTCTGCTCCT
17:CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
18:ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
Sort descending¶
[19]:
cat bgp.fasta |
grep -nv "^>" |
sort -rn
18:ATCCCAGCTGCTCCCAAATAAACTCCAGAAG
17:CTCCAGGCACCCTTCTTTCCTCTTCCCCTTGCCCTTGCCCTGACCTCCCAGCCCTATGGATGTGGGGTCCCCATC
16:GGCCTATCGGCGCTTCTACGGCCCGGTCTAGGGTGTCGCTCTGCTGGCCTGGCCGGCAACCCCAGTTCTGCTCCT
15:CCTGGAGCCCAGGAGGGAGGTGTGTGAGCTCAATCCGGACTGTGACGAGTTGGCTGACCACATCGGCTTTCAGGA
14:AGGGATGGGCATTTTGCACGGGGGCTGATGCCACCACGTCGGGTGTCTCAGAGCCCCAGTCCCCTACCCGGATCC
13:GCCTCTCTGGGTTGTGGTGGGGGTACAGGCAGCCTGCCCTGGTGGGCACCCTGGAGCCCCATGTGTAGGGAGAGG
12:GTGAGAGAAAAGGCAGAGCTGGGCCAAGGCCCTGCCTCTCCGGGATGGTCTGTGGGGGAGCTGCAGCAGGGAGTG
11:CACAGCCTTTGTGTCCAAGCAGGAGGGCAGCGAGGTAGTGAAGAGACCCAGGCGCTACCTGTATCAATGGCTGGG
10:GCCATCAGGAAGGCCAGCCTGCTCCCCACCTGATCCTCCCAAACCCAGAGCCACCTGATGCCTGCCCCTCTGCTC
9:TGATGGGTTCCTGGACCCTCCCCTCTCACCCTGGTCCCTCAGTCTCATTCCCCCACTCCTGCCACCTCCTGTCTG
8:CAGGCTCCCTTTCCTTTGCAGGTGCGAAGCCCAGCGGTGCAGAGTCCAGCAAAGGTGCAGGTATGAGGATGGACC
7:GAGAGGAGGGAAGAGCAAGCTGCCCGAGACGCAGGGGAAGGAGGATGAGGGCCCTGGGGATGAGCTGGGGTGAAC
6:GCTGGCAGTCCCTTTGCAGTCTAACCACCTTGTTGCAGGCTCAATCCATTTGCCCCAGCTCTGCCCTTGCAGAGG
5:CACCTCCCCTCAGGCCGCATTGCAGTGGGGGCTGAGAGGAGGAAGCACCATGGCCCACCTCTTCTCACCCCTTTG
4:ATGAGAGCCCTCACACTCCTCGCCCTATTGGCCCTGGCCGCACTTTGCATCGCTGGCCAGGCAGGTGAGTGCCCC
3:ATAAACAGTGCTGGAGGCTGGCGGGGCAGGCCAGCTGAGTCCTGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
2:GGCAGATTCCCCCTAGACCCGCCCGCACCATGGTCAGGCATGCCCCTCCTCATCGCTGGGCACAGCCCAGAGGGT
Downloading files¶
[20]:
wget ftp://ftp.ensemblgenomes.org/pub/release-39/fungi/gtf/fungi_basidiomycota1_collection/cryptococcus_neoformans_var_grubii_h99/Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz
--2019-06-26 09:17:00-- ftp://ftp.ensemblgenomes.org/pub/release-39/fungi/gtf/fungi_basidiomycota1_collection/cryptococcus_neoformans_var_grubii_h99/Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz
=> ‘Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz’
Resolving ftp.ensemblgenomes.org (ftp.ensemblgenomes.org)... 193.62.197.94
Connecting to ftp.ensemblgenomes.org (ftp.ensemblgenomes.org)|193.62.197.94|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /pub/release-39/fungi/gtf/fungi_basidiomycota1_collection/cryptococcus_neoformans_var_grubii_h99 ... done.
==> SIZE Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz ... 1796344
==> PASV ... done. ==> RETR Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz ... done.
Length: 1796344 (1.7M) (unauthoritative)
Cryptococcus_neofor 100%[===================>] 1.71M 2.44MB/s in 0.7s
2019-06-26 09:17:02 (2.44 MB/s) - ‘Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz’ saved [1796344]
[21]:
ls
a.txt
bgp.fasta
b.txt
Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz
c.txt
data
figs
hello.md5
hello.txt
lsd1.txt
lsd2.txt
lsd3.txt
MD5_CHECKSUM
MD5SUM
notebooks.tar.gz
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
File compression/uncompression¶
[22]:
ls -lh Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz
-rw-r--r-- 1 jovyan users 1.8M Jun 26 09:17 Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz
[23]:
gunzip Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf.gz
[24]:
ls -lh
total 39M
-rw-r--r-- 1 jovyan users 6 Jun 26 09:16 a.txt
-rw-r--r-- 1 jovyan users 1.3K Jun 26 09:16 bgp.fasta
-rw-r--r-- 1 jovyan users 6 Jun 26 09:16 b.txt
-rw-r--r-- 1 jovyan users 32M Jun 26 09:17 Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf
-rw-r--r-- 1 jovyan users 6 Jun 26 09:16 c.txt
drwxr-xr-x 2 jovyan users 4.0K Jun 25 15:34 data
drwxr-xr-x 2 jovyan users 4.0K Jun 26 08:52 figs
-rw-r--r-- 1 jovyan users 44 Jun 26 09:16 hello.md5
-rw-r--r-- 1 jovyan users 45 Jun 26 09:16 hello.txt
-rw-r--r-- 1 jovyan users 107 Jun 26 09:15 lsd1.txt
-rw-r--r-- 1 jovyan users 107 Jun 26 09:15 lsd2.txt
-rw-r--r-- 1 jovyan users 107 Jun 26 09:15 lsd3.txt
-rw-r--r-- 1 jovyan users 2.2K Jun 26 09:16 MD5_CHECKSUM
-rw-r--r-- 1 jovyan users 120 Jun 26 09:16 MD5SUM
-rw-r--r-- 1 jovyan users 2.8M Jun 26 09:16 notebooks.tar.gz
-rw-r--r-- 1 jovyan users 20K Jun 26 08:42 R00_Review_Basics.ipynb
-rw-r--r-- 1 jovyan users 2.9K Jun 25 15:34 R00_Review_Basics_Scratch.ipynb
-rw-r--r-- 1 jovyan users 11K Jun 25 15:34 R01_Data_Manipulation.ipynb
-rw-r--r-- 1 jovyan users 7.1K Jun 25 15:34 R01_Data_Manipulation_Scratch.ipynb
-rw-r--r-- 1 jovyan users 117K Jun 26 08:43 R01_Data_Manipulation_Solutions.ipynb
-rw-r--r-- 1 jovyan users 3.5K Jun 25 15:34 R01_Manipulating_Data_In_R.ipynb
-rw-r--r-- 1 jovyan users 40K Jun 25 15:34 R02_Tidying_Data_In_R.ipynb
-rw-r--r-- 1 jovyan users 4.8K Jun 25 15:34 R02_Tidying_Data.ipynb
-rw-r--r-- 1 jovyan users 64K Jun 26 08:45 R02_Tidying_Data_Solutions.ipynb
-rw-r--r-- 1 jovyan users 23K Jun 26 08:46 R03_FileIO.ipynb
-rw-r--r-- 1 jovyan users 1.2M Jun 26 08:55 R04_Unsupervised_Learning.ipynb
-rw-r--r-- 1 jovyan users 12K Jun 25 15:34 R04_Unsupervised_Learning_Scratch.ipynb
-rw-r--r-- 1 jovyan users 1.1M Jun 26 08:55 R05_Unsupervised_Learning_More_Examples.ipynb
-rw-r--r-- 1 jovyan users 154K Jun 26 08:55 R06_Graphics_Overview.ipynb
-rw-r--r-- 1 jovyan users 210K Jun 26 08:55 R07_Graphics_Base.ipynb
-rw-r--r-- 1 jovyan users 1.4M Jun 26 08:55 R08_Graphics_ggplot2.ipynb
-rw-r--r-- 1 jovyan users 4.9K Jun 26 08:55 R09_Graphics_Exercise.ipynb
-rw-r--r-- 1 jovyan users 102K Jun 26 08:55 R09_Graphics_Exercise_Solutions.ipynb
drwxr-xr-x 2 jovyan users 4.0K Jun 25 15:34 seqs
-rw-r--r-- 1 jovyan users 11K Jun 25 15:34 Unix01_File_And_Directory.ipynb
-rw-r--r-- 1 jovyan users 39K Jun 26 09:14 Unix01_File_And_Directory_Solutions.ipynb
-rw-r--r-- 1 jovyan users 12K Jun 25 15:34 Unix02_FileIO.ipynb
-rw-r--r-- 1 jovyan users 32K Jun 26 09:15 Unix02_FileIO_Solutions.ipynb
-rw-r--r-- 1 jovyan users 7.9K Jun 25 15:34 Unix03_File_Storage.ipynb
-rw-r--r-- 1 jovyan users 27K Jun 26 09:16 Unix03_File_Storage_Solutions.ipynb
-rw-r--r-- 1 jovyan users 11K Jun 25 15:34 Unix04_Text_Manipulation.ipynb
-rw-r--r-- 1 jovyan users 12K Jun 25 15:34 Unix04_Text_Manipulation_Solutinos.ipynb
-rw-r--r-- 1 jovyan users 16K Jun 25 15:34 Unix05_Variables.ipynb
-rw-r--r-- 1 jovyan users 16K Jun 25 15:34 Unix05_Variables_Solutions.ipynb
-rw-r--r-- 1 jovyan users 37K Jun 25 15:34 Unix06_Bash_Bioinformatics.ipynb
-rw-r--r-- 1 jovyan users 5.3K Jun 25 15:34 Unix07_Capstone_Exercise.ipynb
-rw-r--r-- 1 jovyan users 8.1K Jun 25 15:34 Unix07_Capstone_Exercise_Solutions.ipynb
-rw-r--r-- 1 jovyan users 11K Jun 25 15:34 Unix_Appendix01_Regular_Expressions.ipynb
-rw-r--r-- 1 jovyan users 5.9K Jun 25 15:34 Unix_Appendix02_Review.ipynb
Inspecting the GTF file¶
A GTF file has some header lines, followed by tabular data in 9 columns:
chromosome name > chr{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,X,Y,M}
annotation source > {ENSEMBL,HAVANA}
feature-type > {gene,transcript,exon,CDS,UTR,start_codon,stop_codon,Selenocysteine}
genomic start location > integer-value (1-based)
genomic end location > integer-value
score (not used) > .
genomic strand > {+,-}
genomic phase (for CDS features) > {0,1,2,.}
additional information as key-value pairs > (format: key “value”;)
[25]:
head Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf
#!genome-build CNA3
#!genome-version CNA3
#!genome-date 2015-11
#!genome-build-accession GCA_000149245.3
#!genebuild-last-updated 2015-11
1 ena gene 100 5645 . - . gene_id "CNAG_04548"; gene_source "ena"; gene_biotype "protein_coding";
1 ena transcript 100 5645 . - . gene_id "CNAG_04548"; transcript_id "AFR92135"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
1 ena exon 5494 5645 . - . gene_id "CNAG_04548"; transcript_id "AFR92135"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding"; exon_id "AFR92135-1";
1 ena CDS 5494 5645 . - 0 gene_id "CNAG_04548"; transcript_id "AFR92135"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding"; protein_id "AFR92135"; protein_version "1";
1 ena start_codon 5643 5645 . - 0 gene_id "CNAG_04548"; transcript_id "AFR92135"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
[26]:
tail Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf
Mt ena stop_codon 23840 23842 . + 0 gene_id "CNAG_09011"; transcript_id "AFR99113"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
Mt Ensembl_Fungi gene 23909 23980 . + . gene_id "ENSRNA049545749"; gene_name "tRNA-Val"; gene_source "Ensembl_Fungi"; gene_biotype "tRNA";
Mt Ensembl_Fungi transcript 23909 23980 . + . gene_id "ENSRNA049545749"; transcript_id "ENSRNA049545749-T1"; gene_name "tRNA-Val"; gene_source "Ensembl_Fungi"; gene_biotype "tRNA"; transcript_source "Ensembl_Fungi"; transcript_biotype "tRNA";
Mt Ensembl_Fungi exon 23909 23980 . + . gene_id "ENSRNA049545749"; transcript_id "ENSRNA049545749-T1"; exon_number "1"; gene_name "tRNA-Val"; gene_source "Ensembl_Fungi"; gene_biotype "tRNA"; transcript_source "Ensembl_Fungi"; transcript_biotype "tRNA"; exon_id "ENSRNA049545749-E1";
Mt ena gene 24096 24851 . + . gene_id "CNAG_09012"; gene_source "ena"; gene_biotype "protein_coding";
Mt ena transcript 24096 24851 . + . gene_id "CNAG_09012"; transcript_id "AFR99114"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
Mt ena exon 24096 24851 . + . gene_id "CNAG_09012"; transcript_id "AFR99114"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding"; exon_id "AFR99114-1";
Mt ena CDS 24096 24848 . + 0 gene_id "CNAG_09012"; transcript_id "AFR99114"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding"; protein_id "AFR99114"; protein_version "1";
Mt ena start_codon 24096 24098 . + 0 gene_id "CNAG_09012"; transcript_id "AFR99114"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
Mt ena stop_codon 24849 24851 . + 0 gene_id "CNAG_09012"; transcript_id "AFR99114"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
Remove comment lines¶
[27]:
tail +6 Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf | head -3
1 ena gene 100 5645 . - . gene_id "CNAG_04548"; gene_source "ena"; gene_biotype "protein_coding";
1 ena transcript 100 5645 . - . gene_id "CNAG_04548"; transcript_id "AFR92135"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
1 ena exon 5494 5645 . - . gene_id "CNAG_04548"; transcript_id "AFR92135"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding"; exon_id "AFR92135-1";
tail: error writing 'standard output': Broken pipe
[28]:
cat Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf |
grep -v '^#' |
head -3
1 ena gene 100 5645 . - . gene_id "CNAG_04548"; gene_source "ena"; gene_biotype "protein_coding";
1 ena transcript 100 5645 . - . gene_id "CNAG_04548"; transcript_id "AFR92135"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding";
1 ena exon 5494 5645 . - . gene_id "CNAG_04548"; transcript_id "AFR92135"; exon_number "1"; gene_source "ena"; gene_biotype "protein_coding"; transcript_source "ena"; transcript_biotype "protein_coding"; exon_id "AFR92135-1";
grep: write error: Broken pipe
cat: write error: Broken pipe
Spliting columns¶
[29]:
File compression and archivalcat Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf |
grep -v '^#' |
cut -f3 |
head -3
bash: File: command not found
[30]:
cat Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf |
grep -v '^#' |
cut -f4-5 |
head -3
100 5645
100 5645
5494 5645
cut: write error: Broken pipe
[31]:
cat Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf |
grep -v '^#' |
cut -f2,4-5 |
head -3
ena 100 5645
ena 100 5645
ena 5494 5645
cut: write error: Broken pipe
Exercises¶
Waht is the mRNA version of bgp.fasta?
[32]:
cat bgp.fasta |
grep -v '^>' |
tr T U
GGCAGAUUCCCCCUAGACCCGCCCGCACCAUGGUCAGGCAUGCCCCUCCUCAUCGCUGGGCACAGCCCAGAGGGU
AUAAACAGUGCUGGAGGCUGGCGGGGCAGGCCAGCUGAGUCCUGAGCAGCAGCCCAGCGCAGCCACCGAGACACC
AUGAGAGCCCUCACACUCCUCGCCCUAUUGGCCCUGGCCGCACUUUGCAUCGCUGGCCAGGCAGGUGAGUGCCCC
CACCUCCCCUCAGGCCGCAUUGCAGUGGGGGCUGAGAGGAGGAAGCACCAUGGCCCACCUCUUCUCACCCCUUUG
GCUGGCAGUCCCUUUGCAGUCUAACCACCUUGUUGCAGGCUCAAUCCAUUUGCCCCAGCUCUGCCCUUGCAGAGG
GAGAGGAGGGAAGAGCAAGCUGCCCGAGACGCAGGGGAAGGAGGAUGAGGGCCCUGGGGAUGAGCUGGGGUGAAC
CAGGCUCCCUUUCCUUUGCAGGUGCGAAGCCCAGCGGUGCAGAGUCCAGCAAAGGUGCAGGUAUGAGGAUGGACC
UGAUGGGUUCCUGGACCCUCCCCUCUCACCCUGGUCCCUCAGUCUCAUUCCCCCACUCCUGCCACCUCCUGUCUG
GCCAUCAGGAAGGCCAGCCUGCUCCCCACCUGAUCCUCCCAAACCCAGAGCCACCUGAUGCCUGCCCCUCUGCUC
CACAGCCUUUGUGUCCAAGCAGGAGGGCAGCGAGGUAGUGAAGAGACCCAGGCGCUACCUGUAUCAAUGGCUGGG
GUGAGAGAAAAGGCAGAGCUGGGCCAAGGCCCUGCCUCUCCGGGAUGGUCUGUGGGGGAGCUGCAGCAGGGAGUG
GCCUCUCUGGGUUGUGGUGGGGGUACAGGCAGCCUGCCCUGGUGGGCACCCUGGAGCCCCAUGUGUAGGGAGAGG
AGGGAUGGGCAUUUUGCACGGGGGCUGAUGCCACCACGUCGGGUGUCUCAGAGCCCCAGUCCCCUACCCGGAUCC
CCUGGAGCCCAGGAGGGAGGUGUGUGAGCUCAAUCCGGACUGUGACGAGUUGGCUGACCACAUCGGCUUUCAGGA
GGCCUAUCGGCGCUUCUACGGCCCGGUCUAGGGUGUCGCUCUGCUGGCCUGGCCGGCAACCCCAGUUCUGCUCCU
CUCCAGGCACCCUUCUUUCCUCUUCCCCUUGCCCUUGCCCUGACCUCCCAGCCCUAUGGAUGUGGGGUCCCCAUC
AUCCCAGCUGCUCCCAAAUAAACUCCAGAAG
Extract the nucleotides in positions 5,10 and 15 of each line of bgp.fasta.
[33]:
cat bgp.fasta |
grep -v '^>' |
cut -c5,10,15
GCA
AGA
GCA
TTC
GCT
GGG
CTT
GCA
TAC
GTC
GAA
CGT
ACT
GCG
TGT
ACC
CGC
Find the number of mitochondrial exons in the GTF file.
[34]:
cat Cryptococcus_neoformans_var_grubii_h99.CNA3.39.gtf |
grep '^M' |
cut -f3 |
grep "exon" |
wc -l
45
[ ]: