Chapter 9 Filtering and Editing Files

A filter is a program that reads standard input, performs an operation upon it and writes the results to standard output. It can therefore be used to process information in powerful ways such as restructuring output to generate reports, modifying text in files or other system administration tasks. The word filter here is defined in a broader context that what you might intuitively think, you actually came across a few such filters already, just think of the commands cat, head or wc.

In this chapter, we will look at a few additional command line tools that can be used to filter and edit files and we will also have a short look at regular expressions. One particular command, awk, is quite powerful and it is actually its own language, we will therefore dedicate an entire chapter to it later in the tutorial.

Note: Some examples in this section require the Banthracis proteome BanthracisProteome.txt. If you haven’t already, you may get the file as follows:

$ # download the zipped file using wget
$ wget --compression=auto https://bitbucket.org/wegmannlabteaching/bash_lecture/raw/master/Files/BanthracisProteome.txt.gz
$ # unzip the file
$ gunzip BanthracisProteome.txt.gz