Table of Contents

Text Manipulation/Analysis

grep

Description

Search for text in files or in piped output from other commands. Grep does line-by-line matching. That is to say, given a pattern, it will display only lines from the input that match that pattern.

Grep patterns are in regular expression format. This is not to say that you have to know regex to use grep, but it can be more powerful if you do.

Usage

grep <pattern> <file>

sed

Description

Filters and transforms text given input or output. Text can be in regular expression format or simple text.

Example

echo 'I have a dog named cat' | sed 's/cat/Buddy/'

awk

Description

Very powerful text parsing tool. Mainly used to manipulate text in a programatic way.

Example

Let's say we were given a csv file and it looks like this:

Name,Age,ID  
Bob,35,1  
Jack,45,2  
Jill,24,3  
Nancy,56,4  
Tommy,32,5

and we wanted to grab a list of everyone's ages. We can use the awk command to split each line by the ',' delimiter using the -F flag and output to us the second item with "{print $2}"

cat /tmp/employees.csv | awk -F',' '{print $2}'

cut

We can do that same operation with the cut command:

cat /tmp/employees.csv | cut -d ',' -f 2

head

The head command will show us only the top N lines of the given file or output. We could use this to get the headings of our csv file:

cat /tmp/employees.csv | head -n 1

tail

The tail command does the opposite of the head command. It will show you the last N lines of the given file or output. We can use this to get the last entry in our list:

cat /tmp/employees.csv | tail -n 1

wc

The wc command will give us a word count of our file. Typically, however, it is used with the much more useful "-l" flag, which will instead give us a line count. Let's use this to get the total number of entries in our CSV.

cat /tmp/employees.csv | wc -l

File/Directory Management & Navigation

pwd

To print our current working directory, we can use the pwd command:

pwd

cd

To change our current working directory, we can use the cd command:

cd /home
pwd

mv

To move a file to a new directory OR to rename the file, we can use the mv command:

mv /path/to/somefile.txt /new/path/to/somefile.txt # Moving
mv somefile.txt somenewfile.txt # Renaming

cp

To copy a file from one directory to another, we can use the cp command:

cp /path/to/somefile.txt /new/path/to/somefile.txt # copying

We can also copy directories with the "-r" flag:

cp -r /some/dir /some/new/dir

rmdir

We can remove empty directories with rmdir (does not work if the directory is not empty):

rmdir /some/dir

rm

We can remove files with the rm command:

rm somefile.txt

Similar to the cp command, we can remove directories with the "-r" flag (use with caution):

rm -r /some/dir

find

We can use the find command to search our file system for a specific file. This supports wildcards (*) as well. The below command will search our entire file system (starting at /) for any files that end in ".csv"

find / -name *.csv

diff

We can use the diff command to get the differences between two files. For example, let's same we had the following two files:

/tmp/filev1

3+1=5
2+3=5
4+1=5

/tmp/filev2

3+1=4
2+3=5
4+1=5

diff /tmp/file_v1 /tmp/file_v2

tar

The main archiving utility in linux is tar. This utility is equivelant to 7zip or the zip utility in Windows. Tar uses gzip for compression.

To "tar" up a directory, use the following command:

tar -czf archive.tgz some_directory/ 

Here we are using three flags:

  • -c - create an archive
  • -z - compress with gzip
  • -f - specifies the output file, so it has to be last, with the output file directory after

You can also extract using the -x flag:

tar -xf archive.tgz

This will extract the archive into your current directory.

zip

The zip utility is another zip utility. However, it is usually only used when tar is not an accepted format, such as when someone sends you a zip file from windows or if you have to package a zip file for use in a program that doesn't accept tar.

The zip command is very simple to use:

zip -r archive.zip some_dir

It comes with a companion command to unzip archives:

unzip archive.zip

Similaryly, this command will extract the archive to your current directory.

du

The ls -l command does not give us real byte-size of a the contents of a directory. It doesn't do this because this has to be calculated, and can be quite slow. You may notice this issue in Windows explorer. The properties page is where you would find this information for a directory.

Luckily, if this information is important to you, you can use the "du" command, which will recursively calculate the size of a given directory.

du -sh ~/

Here we use two flags:

  • -s - calculate only the total size of the directory, not each of its descendants
  • -h - use human readable sizes

You will almost always want to use both of these flags.

Disk Management

df

If you are after information about the size of each partition/file system on the drive, the "df" command can be used:

df -h

We use the -h flag to make the file sizes human-readable

lsblk

lsblk can be used when we need to find information about the disks attached to the system as well as their partitions.

lsblk

Users and Permissions

chmod

chmod can be used to change the permissions of a file.

Adding Single Permissions

chmod +<perm> file.txt

where <perm> is one of:

  • r - read
  • w - write
  • x - execute

Removing Permissions

chmod -<perm> file.txt

Setting Permissions

Permissions can be set using a set of three numbers, which correspond to a different permission set:

0: — 1: –x. 2: -w- 3: -wx. 4: r- 5: r-x. 6: rw- 7: rwx.

To fully set the permissions of a file, you will have to use three of these numbers: first for the owner, then for the group and lastly for everyone else:

chmod 740 file.txt

In this example, we give the following permissions:

  • Owners: rwx permissions (7 - All Permissions)
  • Group: r– permissions (4 - Read Only)
  • Everyone else: — permissions (0 - No Permissions)

Directories and Recursion

You can set the permissions of a directory recursively (i.e. it and all of its descendants) using the -R flag:

chmod 400 -R ~/.ssh

This will set the permissions of everything under ~/.ssh to 400.

chown

Similarly, you may need to change the owner or group of a file or directory. You can do this with chown. The syntax for this is as follows:

chown <user>:<group> file.txt

You also may want to do this recursively. This works the same as chmod:

chown <user>:<group> -R some_dir/

useradd

You can add users to the system with the useradd command:

useradd --create-home -s /bin/bash <username>

We use the following flags:

  • –create-home - create a home directory in /home for the user
  • -s - default shell for the user

usermod

You can make changes to a user with the "usermod" command. For example, we can add a user to a group with the following command:

usermod -aG group username

  • -a - Add to group
  • -G - The group to add to

    You can also change the users default shell with:

usermod -s /usr/bin/fish username

  • -s - The path to the new default shell

groupadd

You can add groups with the groupadd command as follows:

groupadd group

userdel

You can delete users with the userdel command as follows:

userdel user

Networking

curl

The curl command is used to make web requests. It has various functionalities to control how those requests are made, but here are a few use cases you might use frequently.

Download a file

curl https://somedownloadsite.com/myfile.txt -o ~/myfile.txt

  • -o - The output file. This is where the file will be downloaded to

Send a POST request to an API

curl -X POST https://somesite.com/api/dostuff -d '<payload>'

  • -X - The request method (default is GET)
  • -d - The body of the request.

Other useful flags

  • -k - Ignore certificate warnings
  • -s - Hide curl metadata output
  • -F - Send form data
  • -d @somefile.json - to send payload from a file

ping

You can use the ping command to check if a particular device is active on the network or to verify if you have connectivity to a particular portion of the network.

This is also useful to determine if you have internet. In this case, we can ping google DNS servers (8.8.8.8):

ping 8.8.8.8 -c 3

  • -c - Ping three times (default is infinite)

ip

The ip command can be used to give us information about the networking configuration on our system.

ip addr

ip addr will tell us our layer 3 configuration for our network interfaces

ip addr  

ip link

The ip link command can be used to give us layer 2 information about our network interfaces

ip link

Process & Resource Management

top

The top command will give us an interactive view of our currently running processes as well as how many resources they are consuming:

top

You can exit top with the "q" key.

top_example_output.png

htop

An easier and much more useful alternative to top is htop. This tool is more advanced and is much preferred to top. However, it is usually not installed by default.

htop

htop_example_output.png

kill

You can kill processes with the kill command. This does, however require you to have the process PID

kill <pid> 

killall

Another alternative might be to kill all processes with a specific name. You can do this with the killall command:

killall <name>

Other

uname

You can get information about your system using the uname command. By default, this will just say "Linux" if you are using a linux system. However, a more useful way to use it might be:

uname -a

  • -a - Give all system information

You will notice this will give you the current kernel version that you are using.

history

Bash will keep a certain number of commands ran in its history. You can access this data using the history command:

history

This will output your command history.

You can also use "!!" to access the last command that was ran:

sudo !!

This is a great way to run your last command as sudo when you get a permission denied on the first try.