File Test Operators

Testing files in scripts is easy and straight forward. This is where shell scripting starts to show its glory! In Bash you can do file testing for permissions, size, date, filetype or existence.

Integer Comparison Operators

How to compare integers or arithmetic expressions in shell scripts.

Single Comparison Operators

How to compare strings in Bash.

wget

The below wget command will download all HTML pages for a given website and all of the local assets (CSS/JS/etc) needed to correctly display the pages.

wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=unix http://example.com --no-check-certificate

grep

Searches all files in a directory

grep -rnw '/path/to/somewhere/' -e 'pattern'

• -r or -R is recursive,
• -n is line number, and
• -w stands for match the whole word.
• -l (lower-case L) can be added to just give the file name of matching files.
• -i stands for ignore case
• -e regex pattern to match

Search only files with .c or .h extensions

grep --include=\*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"

Exclude files with a .o extension

grep --exclude=*.o -rnw '/path/to/somewhere/' -e "pattern"

Exclude dirs dir1/, dir2/ and all of them matching *.dst/:

grep --exclude-dir={dir1,dir2,*.dst} -rnw '/path/to/somewhere/' -e "pattern"

find

delete all directories older than 10 days

find /path/to/base/dir/* -type d -ctime +10 -exec rm -rf {} \;

• find: the unix command for finding files / directories / links etc.
• /path/to/base/dir: the directory to start your search in.
• -type d: only find directories
• -ctime +10: only consider the ones with modification time older than 10 days
• -exec ... \;: for each such result found, do the following command in ...
• rm -rf {}: recursively force remove the directory; the {} part is where the find result gets substituted into from the previous part.

replace all spaces in filenames with an underscore

for file in *; do mv "$file" `echo $file | tr ' ' '_'` ; done

replace all strings in filename with other string

find . -name '*pdf' -exec bash -c ' mv "$0" "${0/string-to-find/string-to-replace}"' {} \;

swap filenames in a dir from FIRST LAST to LAST, FIRST

find * -maxdepth 1 -type d -print0 | sed -ze "p;s/^\(.*\) \([^ ]*\)/\2, \1/" | xargs -0 -n2 mv

sed

replace all instances of text in a file

sed -i 's/SEARCH_REGEX/REPLACEMENT/g' INPUTFILE

• -i - By default, sed writes its output to the standard output. This option tells sed to edit files in place. If an extension is supplied (ex -i.bak), a backup of the original file is created.
• s - The substitute command, probably the most used command in sed.
• / / / - Delimiter character. It can be any character but usually the slash (/) character is used.
• SEARCH_REGEX - Normal string or a regular expression to search for.
• REPLACEMENT - The replacement string.
• g - Global replacement flag. By default, sed reads the file line by line and changes only the first occurrence of the SEARCH_REGEX on a line. When the replacement flag is provided, all occurrences are replaced.
• INPUTFILE - The name of the file on which you want to run the command.

tar

basic options

• x instructs tar to extract the files from the zipped file
• v means verbose, or to list out the files it’s extracting
• z instructs tar to decompress the files – without this, you’d have a folder full of compressed files
• f tells tar the filename you want it to work on\

examples

To list the contents of a .tar file before you extract it, enter:

tar –tzf documents.tar.gz

To instruct tar to put the extracted unzipped files into a specific directory, enter:

tar –xvzf documents.tar.gz –C /home/user/destination

To extract the files from a .tar file, enter:

tar –xvf documents.tar

for loop

Rename Files and Directories Add Prefix

for f in * ; do mv -- "$f" "PRE_$f" ; done

converting all mp4 to mp3 w/ ffmpeg

#!/bin/bash
MP4FILE=$(ls ~/Music/ |grep .mp4)
for filename in $MP4FILE
do 
 name=`echo "$filename" | sed -e "s/.mp4$//g"`
 ffmpeg -i ~/Music/$filename -b:a 192K -vn ~/Music/$name.mp3
done

ocrmypdf

• Documentation

pdfjoin

• Man page

References

• How to Extract or Unzip tar.gz Files from Linux Command Line
• Shell Builtins
• How to use sed to find and replace strings in files
• sed - how to rename multiple files by replacing string in file name?…
• linux - Rename Files and Directories (Add Prefix)
• Linux Basics: How To Convert Mp4 To MP3