| Home | Configs | Guides | About |
Table of Contents
Text Manipulation/Analysis
grep
Description
Search for text in files or in piped output from other commands. Grep does line-by-line matching. That is to say, given a pattern, it will display only lines from the input that match that pattern.
Grep patterns are in regular expression format. This is not to say that you have to know regex to use grep, but it can be more powerful if you do.
Usage
grep <pattern> <file>
See also: Commands In-Depth: grep
sed
Description
Filters and transforms text given input or output. Text can be in regular expression format or simple text.
Example
echo 'I have a dog named cat' | sed 's/cat/Buddy/'
awk
Description
Very powerful text parsing tool. Mainly used to manipulate text in a programatic way.
Example
Let's say we were given a csv file and it looks like this:
Name,Age,ID Bob,35,1 Jack,45,2 Jill,24,3 Nancy,56,4 Tommy,32,5
and we wanted to grab a list of everyone's ages. We can use the awk command to split each line by the ',' delimiter using the -F flag and output to us the second item with "{print $2}"
cat /tmp/employees.csv | awk -F',' '{print $2}'
cut
We can do that same operation with the cut command:
cat /tmp/employees.csv | cut -d ',' -f 2
head
The head command will show us only the top N lines of the given file or output. We could use this to get the headings of our csv file:
cat /tmp/employees.csv | head -n 1
tail
The tail command does the opposite of the head command. It will show you the last N lines of the given file or output. We can use this to get the last entry in our list:
cat /tmp/employees.csv | tail -n 1
wc
The wc command will give us a word count of our file. Typically, however, it is used with the much more useful "-l" flag, which will instead give us a line count. Let's use this to get the total number of entries in our CSV.
cat /tmp/employees.csv | wc -l
File/Directory Management & Navigation
pwd
To print our current working directory, we can use the pwd command:
pwd
cd
To change our current working directory, we can use the cd command:
cd /home pwd
mv
To move a file to a new directory OR to rename the file, we can use the mv command:
mv /path/to/somefile.txt /new/path/to/somefile.txt # Moving mv somefile.txt somenewfile.txt # Renaming
cp
To copy a file from one directory to another, we can use the cp command:
cp /path/to/somefile.txt /new/path/to/somefile.txt # copying
We can also copy directories with the "-r" flag:
cp -r /some/dir /some/new/dir
rmdir
We can remove empty directories with rmdir (does not work if the directory is not empty):
rmdir /some/dir
rm
We can remove files with the rm command:
rm somefile.txt
Similar to the cp command, we can remove directories with the "-r" flag (use with caution):
rm -r /some/dir
find
We can use the find command to search our file system for a specific file. This supports wildcards (*) as well. The below command will search our entire file system (starting at /) for any files that end in ".csv"
find / -name *.csv
diff
We can use the diff command to get the differences between two files. For example, let's same we had the following two files:
/tmp/filev1
3+1=5 2+3=5 4+1=5
/tmp/filev2
3+1=4 2+3=5 4+1=5
diff /tmp/file_v1 /tmp/file_v2
tar
The main archiving utility in linux is tar. This utility is equivelant to 7zip or the zip utility in Windows. Tar uses gzip for compression.
To "tar" up a directory, use the following command:
tar -czf archive.tgz some_directory/
Here we are using three flags:
- -c - create an archive
- -z - compress with gzip
- -f - specifies the output file, so it has to be last, with the output file directory after
You can also extract using the -x flag:
tar -xf archive.tgz
This will extract the archive into your current directory.
zip
The zip utility is another zip utility. However, it is usually only used when tar is not an accepted format, such as when someone sends you a zip file from windows or if you have to package a zip file for use in a program that doesn't accept tar.
The zip command is very simple to use:
zip -r archive.zip some_dir
It comes with a companion command to unzip archives:
unzip archive.zip
Similaryly, this command will extract the archive to your current directory.
du
The ls -l command does not give us real byte-size of a the contents of a directory. It doesn't do this because this has to be calculated, and can be quite slow. You may notice this issue in Windows explorer. The properties page is where you would find this information for a directory.
Luckily, if this information is important to you, you can use the "du" command, which will recursively calculate the size of a given directory.
du -sh ~/
Here we use two flags:
- -s - calculate only the total size of the directory, not each of its descendants
- -h - use human readable sizes
You will almost always want to use both of these flags.
Disk Management
df
If you are after information about the size of each partition/file system on the drive, the "df" command can be used:
df -h
We use the -h flag to make the file sizes human-readable
lsblk
lsblk can be used when we need to find information about the disks attached to the system as well as their partitions.
lsblk
Users and Permissions
chmod
chmod can be used to change the permissions of a file.
Adding Single Permissions
chmod +<perm> file.txt
where <perm> is one of:
- r - read
- w - write
- x - execute
Removing Permissions
chmod -<perm> file.txt
Setting Permissions
Permissions can be set using a set of three numbers, which correspond to a different permission set:
0: — 1: –x. 2: -w- 3: -wx. 4: r- 5: r-x. 6: rw- 7: rwx.
To fully set the permissions of a file, you will have to use three of these numbers: first for the owner, then for the group and lastly for everyone else:
chmod 740 file.txt
In this example, we give the following permissions:
- Owners: rwx permissions (7 - All Permissions)
- Group: r– permissions (4 - Read Only)
- Everyone else: — permissions (0 - No Permissions)
Directories and Recursion
You can set the permissions of a directory recursively (i.e. it and all of its descendants) using the -R flag:
chmod 400 -R ~/.ssh
This will set the permissions of everything under ~/.ssh to 400.
chown
Similarly, you may need to change the owner or group of a file or directory. You can do this with chown. The syntax for this is as follows:
chown <user>:<group> file.txt
You also may want to do this recursively. This works the same as chmod:
chown <user>:<group> -R some_dir/
useradd
You can add users to the system with the useradd command:
useradd --create-home -s /bin/bash <username>
We use the following flags:
- –create-home - create a home directory in /home for the user
- -s - default shell for the user
usermod
You can make changes to a user with the "usermod" command. For example, we can add a user to a group with the following command:
usermod -aG group username
- -a - Add to group
-G - The group to add to
You can also change the users default shell with:
usermod -s /usr/bin/fish username
- -s - The path to the new default shell
groupadd
You can add groups with the groupadd command as follows:
groupadd group
userdel
You can delete users with the userdel command as follows:
userdel user
Networking
curl
The curl command is used to make web requests. It has various functionalities to control how those requests are made, but here are a few use cases you might use frequently.
Download a file
curl https://somedownloadsite.com/myfile.txt -o ~/myfile.txt
- -o - The output file. This is where the file will be downloaded to
Send a POST request to an API
curl -X POST https://somesite.com/api/dostuff -d '<payload>'
- -X - The request method (default is GET)
- -d - The body of the request.
Other useful flags
- -k - Ignore certificate warnings
- -s - Hide curl metadata output
- -F - Send form data
- -d @somefile.json - to send payload from a file
ping
You can use the ping command to check if a particular device is active on the network or to verify if you have connectivity to a particular portion of the network.
This is also useful to determine if you have internet. In this case, we can ping google DNS servers (8.8.8.8):
ping 8.8.8.8 -c 3
- -c - Ping three times (default is infinite)
ip
The ip command can be used to give us information about the networking configuration on our system.
ip addr
ip addr will tell us our layer 3 configuration for our network interfaces
ip addr
ip link
The ip link command can be used to give us layer 2 information about our network interfaces
ip link
Process & Resource Management
top
The top command will give us an interactive view of our currently running processes as well as how many resources they are consuming:
top
You can exit top with the "q" key.
htop
An easier and much more useful alternative to top is htop. This tool is more advanced and is much preferred to top. However, it is usually not installed by default.
htop
kill
You can kill processes with the kill command. This does, however require you to have the process PID
kill <pid>
killall
Another alternative might be to kill all processes with a specific name. You can do this with the killall command:
killall <name>
Other
uname
You can get information about your system using the uname command. By default, this will just say "Linux" if you are using a linux system. However, a more useful way to use it might be:
uname -a
- -a - Give all system information
You will notice this will give you the current kernel version that you are using.
history
Bash will keep a certain number of commands ran in its history. You can access this data using the history command:
history
This will output your command history.
You can also use "!!" to access the last command that was ran:
sudo !!
This is a great way to run your last command as sudo when you get a permission denied on the first try.