File I/O¶
Creating a file¶
Using an editor¶
[ ]:
Using redirection¶
[1]:
ls
data
figs
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
[2]:
echo "Hello Unix"
Hello Unix
[3]:
echo "Hello Unix" > foo.txt
Using heredocs¶
[4]:
cat > bar.txt << EOF
This old man he played one
He played knick knack on my thumb
EOF
View file contents¶
[5]:
cat foo.txt
Hello Unix
[6]:
cat bar.txt
This old man he played one
He played knick knack on my thumb
More redirection¶
[7]:
echo "I love bash" >> foo.txt
[8]:
cat >> bar.txt << EOF
This old man he played two
He played knick knack on my shoe
EOF
View file contents¶
[9]:
cat foo.txt
Hello Unix
I love bash
[10]:
cat bar.txt
This old man he played one
He played knick knack on my thumb
This old man he played two
He played knick knack on my shoe
More redirection¶
[11]:
echo "Many people hate bash" > foo.txt
Oops, you’ve over-written the original contents of foo.txt and they are lost forever.
[12]:
cat foo.txt
Many people hate bash
Safety¶
If you file contents are precious and should not be changed (e..g raw data), you can change permisions to read-only.
[13]:
ls -l foo.txt
-rw-r--r-- 1 jovyan users 22 Jun 26 09:15 foo.txt
[14]:
chmod -w foo.txt
[15]:
ls -l foo.txt
-r--r--r-- 1 jovyan users 22 Jun 26 09:15 foo.txt
[16]:
echo "Try to overwrite" > foo.txt
bash: foo.txt: Permission denied
Redirecting command output¶
[17]:
ls
bar.txt
data
figs
foo.txt
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
[18]:
ls > contents.txt
[19]:
cat contents.txt
bar.txt
contents.txt
data
figs
foo.txt
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
Redirecting error messages¶
[20]:
cat foobar.txt
cat: foobar.txt: No such file or directory
[21]:
cat foobar.txt > error.txt
cat: foobar.txt: No such file or directory
[22]:
ls e*txt
error.txt
[23]:
cat error.txt
Why is error.txt it empty?¶
The standard > is short for 1> which means re-direct the standard output stream.
[24]:
cat foobar.txt 1> stdout.txt
cat: foobar.txt: No such file or directory
[25]:
cat stdout.txt
To redirect the standard error stream, you need `2>``.
[26]:
cat foobar.txt 2> stderr.txt
[27]:
cat stderr.txt
cat: foobar.txt: No such file or directory
You can redirect to both.
[28]:
cat foobar.txt > results.txt 2> errors.txt
[29]:
cat results.txt
[30]:
cat errors.txt
cat: foobar.txt: No such file or directory
You can also redirect both streams to the same file with &>.
[31]:
cat foobar.txt &> combined.txt
[32]:
cat combined.txt
cat: foobar.txt: No such file or directory
Simple text operations¶
[33]:
cat contents.txt
bar.txt
contents.txt
data
figs
foo.txt
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
[34]:
head -3 contents.txt
bar.txt
contents.txt
data
[35]:
tail -3 contents.txt
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
[36]:
wc contents.txt
39 39 1089 contents.txt
[37]:
wc -l contents.txt
39 contents.txt
[38]:
wc -w contents.txt
39 contents.txt
[39]:
wc -c contents.txt
1089 contents.txt
If you don’t want the filename, force it to read from standard input.
[40]:
wc -c < contents.txt
1089
Piping¶
[41]:
head -3 contents.txt | wc -c
26
[42]:
head -3 contents.txt | wc -c > word_count.txt
[43]:
cat word_count.txt
26
Clean up¶
[44]:
ls
bar.txt
combined.txt
contents.txt
data
errors.txt
error.txt
figs
foo.txt
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
results.txt
seqs
stderr.txt
stdout.txt
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
word_count.txt
Fancy globbing¶
[45]:
shopt -s extglob
Specifying what NOT to list.
[46]:
ls !(*.ipynb)
bar.txt contents.txt error.txt results.txt stdout.txt
combined.txt errors.txt foo.txt stderr.txt word_count.txt
data:
gene_counts_raw.txt gene_counts.txt
figs:
box.png fig1.png fig2.png fig3.png fig4.png fig5.png g3.pdf lm.png
seqs:
CM000046.1.fa CM000047.1.fa
[47]:
ls !(*.txt)
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
data:
gene_counts_raw.txt gene_counts.txt
figs:
box.png fig1.png fig2.png fig3.png fig4.png fig5.png g3.pdf lm.png
seqs:
CM000046.1.fa CM000047.1.fa
Specifying alternatives.
[48]:
ls *{.txt,.ipynb}
bar.txt
combined.txt
contents.txt
errors.txt
error.txt
foo.txt
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
results.txt
stderr.txt
stdout.txt
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
word_count.txt
Clean up¶
[49]:
rm -f !(*.ipynb)
rm: cannot remove 'data': Is a directory
rm: cannot remove 'figs': Is a directory
rm: cannot remove 'seqs': Is a directory
[50]:
ls
data
figs
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
Exercises¶
Create 3 files with the same contents (say
lsd1.txt,lsd2.txt,lsd3.txt)
Two of the most famous products of Berkeley are LSD and Unix.
I don’t think that this is a coincidence.
using
A text editor
Echo
A HereDoc
[51]:
cat > lsd1.txt << EOF
Two of the most famous products of Berkeley are LSD and Unix.
I don’t think that this is a coincidence.
EOF
[52]:
cp lsd1.txt lsd2.txt
cp lsd1.txt lsd3.txt
[53]:
ls
data
figs
lsd1.txt
lsd2.txt
lsd3.txt
R00_Review_Basics.ipynb
R00_Review_Basics_Scratch.ipynb
R01_Data_Manipulation.ipynb
R01_Data_Manipulation_Scratch.ipynb
R01_Data_Manipulation_Solutions.ipynb
R01_Manipulating_Data_In_R.ipynb
R02_Tidying_Data_In_R.ipynb
R02_Tidying_Data.ipynb
R02_Tidying_Data_Solutions.ipynb
R03_FileIO.ipynb
R04_Unsupervised_Learning.ipynb
R04_Unsupervised_Learning_Scratch.ipynb
R05_Unsupervised_Learning_More_Examples.ipynb
R06_Graphics_Overview.ipynb
R07_Graphics_Base.ipynb
R08_Graphics_ggplot2.ipynb
R09_Graphics_Exercise.ipynb
R09_Graphics_Exercise_Solutions.ipynb
seqs
Unix01_File_And_Directory.ipynb
Unix01_File_And_Directory_Solutions.ipynb
Unix02_FileIO.ipynb
Unix02_FileIO_Solutions.ipynb
Unix03_File_Storage.ipynb
Unix03_File_Storage_Solutions.ipynb
Unix04_Text_Manipulation.ipynb
Unix04_Text_Manipulation_Solutinos.ipynb
Unix05_Variables.ipynb
Unix05_Variables_Solutions.ipynb
Unix06_Bash_Bioinformatics.ipynb
Unix07_Capstone_Exercise.ipynb
Unix07_Capstone_Exercise_Solutions.ipynb
Unix_Appendix01_Regular_Expressions.ipynb
Unix_Appendix02_Review.ipynb
Count the number of words in the second sentence only of lsd1.txt
[54]:
tail -1 lsd1.txt |
wc -w
8