logo

Shell cheatsheet - Commands

Last Updated: 2024-02-06

Getting help

--help

Most commands have built-in help info, usually with the --help or -h option, e.g. $ ls --help will give you more info about the command ls.

man

# Display manual of the command.
$ man $COMMAND

# Specify section number.
$ man 2 mount # the system call
$ man 8 mount # the admin command

# Display manual of man itself. Section info are included.
$ man man

# Search the man pages
$ man -k mount

# List all sections that have the term.
$ man -f mount
mount (8)            - mount a filesystem
mount (2)            - mount filesystem
...
  • Section 1: user commands
  • Section 2: system calls
  • Section 3: library functions
  • Section 4: special files
  • Section 5: file formats
  • Section 6: games
  • Section 7: conventions and miscellany
  • Section 8: administration and privileged commands
  • Section L: math library functions
  • Section N: tcl functions

Current Working Directory

ls lists CWD if given no other parameters.

Check CWD, these 2 are equivalent:

$ pwd # Print Working Directory
$ echo $PWD

Nohup

Run command in background and redirect output to file.

$ nohup command > output &

nohup is a POSIX command to ignore the HUP (hangup) signal, enabling the command to keep running after the user who issues the command has logged out. The HUP (hangup) signal is by convention the way a terminal warns depending processes of logout. Then use

$ tail -f output

to print the last few lines of the output.

Rename extensions of multiple files

for file in *.mdx; do mv -- "$file" "${file%.mdx}.md"; done

Jobs / Foreground / Background

List all jobs:

$ jobs

Run something in background

$ <command> &

Foreground to Background:

[Ctrl + z]

$ bg

Background to Foreground:

# something is running in background
$ fg

Ctrl-C vs Ctrl-V

  • Ctrl-C: to (politely) kill a process with signal SIGINT; cannot be resumed.
  • Ctrl-Z: to suspend a process with the signal SIGTSTP, like a sleep signal, can be undone and the process can be resumed:
    • fg: resume in foreground
    • bg: resume in background

More or Less

  • more: can only go forward, bash autocompletion uses more
  • less: can go backward, search, and more; almost everything else uses less

dd

The dd Unix utility program reads octet streams from a source to a destination, possibly performing data conversions in the process.

Create File With Zeroes

Creating a 1 MiB file, called foobar, filled with zeroes:

dd if=/dev/zero of=foobar count=1024 bs=1024

Note: The block size value can be given in SI (decimal) values, e.g. in GB, MB, etc. To create a 1 GB file one would simply type:

Test IO Performance

$ dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 3.94954 s, 272 MB/s

Read operations from /dev/zero return as many null characters (0x00) as requested in the read operation.

Unlike /dev/null, /dev/zero may be used as a source, not only as a sink for data.

dd if=/dev/zero of=/dev/<destination partition>

dd if=/dev/zero of=foobar count=1 bs=1GB

diff

diff
sdiff
vimdiff
colordiff

env

Filter:

env | grep HOME
env | egrep 'HOME|USER|VERSION|SHELL|PWD'

Set PATH for sudo in Ubuntu

In Ubuntu, these commands are showing different results:

$ env
$ sudo env

This may cause a problem when you are setting some bin folder in your path while you need root permission to execute, for example, if you install node in your own folder while you need $ sudo node bin/www to serve it, you will get an error saying node cannot be found.

Solution:

$ sudo env PATH=$PATH [COMMAND]

this will use your own PATH when executing the COMMAND, e.g.

$ sudo env PATH=$PATH node bin/www

or add this to ~/.bashrc

alias sudo='sudo env PATH=$PATH'

ldd

ldd is a powerful command-line tool that allows users to view an executable file's shared object dependencies.

ldd = List Dynamic Dependencies

$ ldd ./my-program
not a dynamic executable

$ ldd /usr/bin/gzip
linux-vdso.so.1 =>  (0x00007fff39fff000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb842afa000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb842ea0000)

$ ldd /usr/bin/ssh
linux-vdso.so.1 =>  (0x00007fffd0164000)
libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007fb62de63000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb62dc43000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fb62d8a9000)
...

LD_LIBRARY_PATH to include this path for the application to successfully run.

tar

  • x: extract
  • z: compress archive using gzip program
  • j: bzip2
  • c: create archive
  • v: verbose
  • f: archive file name

.tar

No compression.

# Create
$ tar -cvf archive_name.tar directory_to_compress

# Extract the archive
$ tar -xvf archive_name.tar.gz

# Extract the files to a different directory
$ tar -xvf archive_name.tar -C /tmp/extract_here/

.tar.gz

# Create and compress
$ tar -cvzf archive_name.tar.gz directory_to_compress

# Decompress and extract
$ tar -xvzf archive_name.tar.gz

# Extract the files to a different directory:
$ tar -xvzf archive_name.tar.gz -C /tmp/extract_here/

.tar.bz2

# Create
$ tar -jcvf archive_name.tar.bz2 directory_to_compress

# Extract
$ tar -jxvf archive_name.tar.bz2 -C /tmp/extract_here/

pigz

pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation.

$ pigz -dc archive.tar.gz | tar xf -

# or
$ tar -I pigz -xvf archive.tar.gz

-I is equivalent to --use-compress-program.

type

Shell Builtin

Linux built in commands:

$ type pwd cd
pwd is a shell builtin
cd is a shell builtin

And type itself is a shell builtin

$ type type
type is a shell builtin

Shell Keyword

Shell keywords are used in shell scripts

$ type if fi
if is a shell keyword
fi is a shell keyword

Alias

If you have alias defined in .bash_profile or .bashrc, for example

alias ls="ls -G"

where -G is set to enable colorized output. Then ls becomes an alias:

$ type ls
ls is aliased to `ls -G'

Executable

If you have docker installed, you will see something like this:

$ type docker
docker is /usr/local/bin/docker

Show all python executables

$ type -a python

xargs

Similar to map, apply functions on each item in the list

e.g. kill all ssh connections

$ ps -ax | grep ssh | cut -d ' ' -f 1 | xargs kill -9

tee

Reads the standard input and writes it to both the standard output and one or more files.

$ echo "test" | tee foo.txt
test
$ cat foo.txt
test

zip

Compress

$ zip -r archive_name.zip directory_to_compress

Extract:

$ unzip archive_name.zip

Trouble Shooting

Error

$ unzip *.zip
error: caution: filename not matched:

Solution

$ unzip \*.zip

Locate a command

Use type or command to locate exact path for the env command:

$ type env
env is hashed (/usr/bin/env)

$ command -V env
env is hashed (/usr/bin/env)

which vs whereis

which: based on PATH

$ which hadoop
/Users/myname/lib/hadoop

whereis: based on standard binary directories

$ whereis hadoop
hadoop: /usr/bin/hadoop /etc/hadoop /usr/lib/hadoop /usr/share/man/man1/hadoop.1.gz

Check If Command Exists

$ command -v foo >/dev/null 2>&1
$ type foo >/dev/null 2>&1
$ hash foo 2>/dev/null

Example: add HADOOP_CLASSPATH if hadoop exists

command -v hadoop > /dev/null && {
    HADOOP_CLASSPATH=`hadoop classpath`
    CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
}

Run at a specific time

$ at 2:00 PM
cowsay 'hello'
(CTRL + D)

the command will run at 2 PM

List all jobs

$ at -l
1	Tue Apr 12 14:00:00 2016

Remove a job

$ at -r 1

TTY

TTY: teletype, now refers to any device that opens a physical or virtual terminal session.

Serial Port Terminals

Each serial port is considered to be a "device". e.g. /dev/ttys0

Pseudo Terminals

Pairs of devices such as /dev/ptyp3 and /dev/ttyp3; no physical device directly associated with either.

Controlling Terminal

/dev/tty

SSH to a Linux server (Ubuntu)

$ tty
/dev/pts/1

On a Mac

$ tty
/dev/ttys001

Shortcut

Change to different tty terminals: Ctrl + Alt + F1-F6

cd

cd to symlinked folder

cd -P /bin will go to /usr/bin

tree

Print directories tree

$ tree -dlL 2
  • -d : show only the directories
  • -l : follow symbolic links
  • L 2 : show only level one and level two directories

Combine multiple commands

Change file extension (e.g. from .md to .mdx):

$ find . -type f -name "*.md" -exec rename 's/\.md$/.mdx/' '{}' \;

Text Processing

Useful tools:

  • grep, egrep and fgrep: match patterns.
  • sed(stream editor) is for programmatically editing files based on lines.
  • awk is for text processing, especially useful for table-like text files like csv.

Replace characters

$ cat foo.txt | tr "," "_" > bar.txt

for unprintable character, e.g. \u0007, press ctrl-V ctrl-G

Delete character

Use tr -d, e.g. to remove all the o:

$ echo "Hello World" | tr -d o
Hell Wrld

Change Everything to Uppercase

$ cat foo.txt | tr "[a-z]" "[A-Z]"

Count Rows

$ cat foo.txt | wc -l

Extract a Column

$ echo 'a b c' | cut -d ' ' -f1
a

$ echo 'a b c' | cut -d ' ' -f2
b

Add - to list everything to the right

$ echo 'a b c' | cut -d ' ' -f2-
b c

split

Split data into chunks

Split by number of lines: split myfile, each chunk has 500 lines, prefixed by segment_, i.e. segment_aa, segment_ab, segment_ac...

$ split -l 500 myfile segment_

Split by size: split myfile, each chunk is 40k

$ split -b 40k myfile segment_

sort

  • -k: (key) column number
  • -t: delimiter
$ cat file | sort -nr -t \| -k 2 | head

Pretty print JSON

$ cat data.json | python -m json.tool

Append Multiple Lines To File

Use Here Document syntax, << "EOF" means the multi-line text ends at string EOF, anything in between will be appended to file.

cat >> path/to/file/to/append-to.txt << "EOF"
Some text here
And some text here
EOF

How to show history with timestamp?

history with timestamp: export HISTTIMEFORMAT="%F %T "

  • %F: show the date in YYYY-MM-DD format.
  • %T: show time in HH:MM:SS format.

How to check glibc version?

# Check ldd version.
$ ldd --version
ldd (Debian GLIBC 2.37-6+gl0) 2.37

# Check a built-in command, like ls.
$(ldd `which /bin/ls` | grep libc | awk '{print $3}')
GNU C Library (Debian GLIBC 2.37-6+gl0) stable release version 2.37.

Troubleshooting

df not responding

Try df -l, -l for local only.

chroot

chroot: changes the root file system directory as seen by a job, so that one program cannot access files outside of its directory tree. (for isolation)

chroot is both a system call and a wrapper program.

Cowsay

$ brew install cowsay
$ cowsay hello
 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Also try:

$ fortune | cowsay
$ fortune | cowsay -f tux
$ cowsay -l

Shortcuts

grep | wc -> grep -c
sort | uniq | wc -> sort -u | wc

List the largest files/directories

If you want to find and print the top 10 largest files names (not directories) in a particular directory and its sub directories

$ find . -printf '%s %p\n'|sort -nr|head

To restrict the search to the present directory use -maxdepth 1 with find.

$ find . -maxdepth 1 -printf '%s %p\n'|sort -nr|head

print the top 10 largest "files and directories":

$ du -a . | sort -nr | head

e.g. check which log file/directory takes most usage

$ du -sh /var/log/* | sort -rh |less

Global Search And Replacement

$ find / -name game
$ find . -type f -name '*.txt' -exec sed -i '' s/this/that/g {} +

Count lines in markdown files

for file in `find ../src -name '*.md'`; do
  wc -l ${file}
done | sort -n > ../_tmp/result.txt

Change -l to -w to count words.

Start Daemon

if kill -0 $pid > /dev/null 2>&1; then
    echo Database is already running ...
else
    echo Starting Database ...
    bin/mongod &
fi

Step by Step:

  • kill -0 $pid: -0 for testing if the pid is available to kill. It will return 0 if you can kill <pid>, 1 if <pid> is invalid or you do not have access.
  • ... > /dev/null 2>&1: redirect the output to /dev/null, a blackhole. 2 stands for STDERR, 1 for STDOUT. This command means redirect STDOUT to /dev/null, and redirect STDERR to STDOUT(which is /dev/null), so both STDOUT and STDERR will be quiet. Remember the & before destination.
  • bin/mongod &: if it is not running, start the daemon. Remember the & for running in the background.

Wait before it is up:

while true; do
    sleep 1
    if pgrep mongod; then
        break
    fi
done

Stop Daemon

Similar to the previous section.

if kill -0 $pid_mongod > /dev/null 2>&1; then
    echo Stopping mongod Server ...
    kill -9 $pid_mongod
else
    echo No mongod Server to stop ...
fi

ps

ps accepts different types of options:

  • UNIX options: with a preceding dash. (e.g. ps -e)
  • BSD options: without a preceding dash. (e.g. ps aux)
  • GNU long options: with two preceding dashes.

Note that ps -aux is distinct from ps aux.

logo
PRIVACY POLICYABOUT中文