If my last post Shell 101 was the shell elementary school then we are graduating to the shell high school now.

The commands and concepts that I explain in this post are a bit more advanced and have a more general purpose. They can also be very useful in shell scripts, about which I will write in a later post.

Globbing

This weird name describes a concept that allows you to save a lot of manual writing. It can be used in most commands that I’ve introduced in my previous post and also in the commands of this post.

Lets assume we have a folder full of files that we want to move to another directory. Let the filenames be:

top_secret.txt
top_mega_hyper_secret.txt
cool_stuff.txt
top_secret_passwords.csv
public_passwords.csv

If we would want to move all these files to the directory above without typing mv [filename] ../ each time we could write instead:

mv * ../

The magic in here is the “*” character. The shell will handle * as a wildcard to match as many files (or directories) as possible. Since we are not giving any further restrictions it will match all files in the current directory and move them to the new directory (in this case ../ which is the directory above).

But what if we only want to move files that end with .txt? In that case we could write

mv *.txt ../

Again, the shell will match as many files as possible while treating the * as a wildcard. This means that only top_secret.txt and top_mega_hyper_secret.txt will be moved.

We can advance this even more. If we want to move only files that start with “top” and end with “.txt” we could write.

mv top*.txt ../

We can even use multiple globs like this:

mv *secret* ../

This will move all files that have “secret” anywhere in their name.

There are even more options for globbing but for now this should be enough. If you want to read more on this you can type man 7 glob to get the manual for globbing.

cat (print file content)

Sometimes you want to get the content of a file without opening a text editor for that. In that case you can use cat which will print the content of all the specified files to the output.

cat [file1] [file2] [...]

echo (print a specified string)

Often you want to print a string or a variable to the output. This could look like this:

echo "Hello world!"

or for a variable:

echo $cool_varible_name

grep (print lines that match pattern)

This command is useful for a lot of things. The general syntax looks like this:

grep [expression] [file(s)]

Let’s assume we have a file full of passwords that’s called passwords.txt. The file could have content like this:

bank account: 1234
netflix.com: breakingbad
cat-videos.cute: cutekittens
reddit.com: dankmemes

Now we want to look up the password for cat-videos.cute to get our daily dose of internet cuteness.

For that we could write

grep cat-videos passwords.txt

This would print only the line on the file which contains the “cat-videos” string and we could copy our password into the browser.

Disclaimer: Please don’t save your passwords in an unencrypted text file. Use a password manager instead.

cut (cutting strings)

I absolutely love this program because it follows the unix philosophy (Make each program do one thing well.) much better than other programs like awk. awk is often misused on occasions where cut would do the job perfectly fine and faster… But I digress.

The syntax of cut is a bit more complex so I recommend reading the man-page (man cut). I’ll show an example of how I use it most of the time.

In the previous example we had a text file that contained websites and their corresponding passwords. But what if we only want to get the passwords without the website names? For this we could write something like this:

cut -d ':' -f 2 passwords.txt

Let’s look at the components of this command. -d ':' this tells the program that we want to split each line into fields that are separated by the delimiter ‘:’. -f 2 This specifies that we only want to see the second field, which in our case would be the passwords.

Often times we want to separate the lines into different words. In that case we would write -d ' ' to split at each space.

wc (word count)

Sometimes we want to know the number of words or lines in a file. This can be useful when we are saving one piece of information per line and want to know how many entries we have. Just like in our password.txt file where the number of passwords could be interesting.

So if we run

wc passwords.txt

we get 4 9 95 as output. The numbers here are representing the lines, words and bytes counts of the file respectively. We can also add the -l option to just get the number of lines, which we want most of the time.

sed and awk

These two commands are some of the most powerful built-in commands in Linux and they can do most of the things that the previous commands do (not as efficient though). They can be used for all kind of stream manipulation and filtering.

I have to admit that I’m not very familiar with neither sed nor awk and I therefore don’t feel comfortable writing about it before I learned to use them myself. Especially awk deserves a separate blog post since it’s a full programming language.