Data handling Demo

Switch to Jupyter

Setup Files and Directories

RAWDATA_DIR=/tmp/rr_demo
DATA1="${RAWDATA_DIR}/data1.csv"
DATA2="${RAWDATA_DIR}/data2.csv"
mkdir -p $RAWDATA_DIR

printf "1\n2\n3\n4" > $DATA1
printf "5\n6\n7\n8" > $DATA2

Checksums

cd $RAWDATA_DIR
head *.csv

==> data1.csv <==
1
2
3
4
==> data2.csv <==
5
6
7
8

md5sum *.csv

257cde944bf50d4fe05001bc33dd0ca4  data1.csv
347f3f3465f592fd05200c622045948f  data2.csv

Checksums File

md5sum *.csv > mydata_md5.txt
md5sum -c mydata_md5.txt

data1.csv: OK
data2.csv: OK

Catching Changes

sed -i s/7/3/ $DATA2
cat $DATA2

5
6
3
8

md5sum -c mydata_md5.txt

data1.csv: OK
data2.csv: FAILED
md5sum: WARNING: 1 computed checksum did NOT match

READ-ONLY

ls -ld $RAWDATA_DIR
ls -ltr $RAWDATA_DIR

drwxr-xr-x 2 jovyan users 4096 Jun 25 17:10 /tmp/rr_demo
total 12
-rw-r--r-- 1 jovyan users  7 Jun 25 17:10 data1.csv
-rw-r--r-- 1 jovyan users 88 Jun 25 17:10 mydata_md5.txt
-rw-r--r-- 1 jovyan users  7 Jun 25 17:10 data2.csv

chmod -R a-w $RAWDATA_DIR
ls -ld $RAWDATA_DIR
ls -ltr $RAWDATA_DIR

dr-xr-xr-x 2 jovyan users 4096 Jun 25 17:10 /tmp/rr_demo
total 12
-r--r--r-- 1 jovyan users  7 Jun 25 17:10 data1.csv
-r--r--r-- 1 jovyan users 88 Jun 25 17:10 mydata_md5.txt
-r--r--r-- 1 jovyan users  7 Jun 25 17:10 data2.csv

Preventing Modification

echo "This is not the data you are looking for" > $DATA2

bash: /tmp/rr_demo/data2.csv: Permission denied

sed s/7/3/ -i $DATA2

sed: couldn't open temporary file /tmp/rr_demo/sedzRnyBp: Permission denied

Preventing Deletion

rm -rf $RAWDATA_DIR

rm: cannot remove '/tmp/rr_demo/data1.csv': Permission denied
rm: cannot remove '/tmp/rr_demo/data2.csv': Permission denied
rm: cannot remove '/tmp/rr_demo/mydata_md5.txt': Permission denied

Clean up

Let’s clean up the directory we made. To do this we need to reset permissions to allow write, otherwise we will get an error again

chmod -R a+w $RAWDATA_DIR
rm -rf $RAWDATA_DIR