Don’t forget this week’s reading.

In this lecture, we discuss the Unix shell and its commands. The ‘shell’ is a command-line interpreter and invokes kernel-level commands. It also can be used as a programming language to design your own commands. We’ll come to shell programming in a future lecture.

We do not recommend that you buy a book about Unix or the shell; there are some very good references and free-access online books – see the resources page – and we have selected some interesting and useful readings.

If you need help on the meaning or syntax of any Unix shell command you can use the manual (man) pages on a Unix system (see below) or the web Unix commands. Just keep in mind that some commands’ syntax varies a bit across Unix flavors, so when in doubt, check the man page on the system you’re using.

Historical note - Unix

Unix was developed at Bell Labs in the 1970s by a group led by Doug McIlroy, now an Adjunct Professor at Dartmouth!

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. — Doug McIlroy

Goals

We plan to cover the following topics in today’s lecture:

  • The shell
  • The file system

We’ll do this activity in today’s class:

  • Get to know your group mates.
  • Get to know your Learning Fellow.
  • Choose a group name.

The shell

Commands, switches, arguments

The shell is the Unix command-line interpreter. It provides an interface between the user and the kernel and executes programs called ‘commands’. For example, if a user enters ls then the shell executes the ls command, which actually executes a program stored in the file /bin/ls. The shell can also execute other programs including scripts (text files interpreted by a program like python or bash) and compiled programs (e.g., written in C). Even your own programs – once marked ‘executable’ – become commands you can run from the shell!

You will get by in the course by becoming familiar with a subset of the Unix commands; don’t let yourself be overwhelmed by the presence of hundreds of commands. You will probably be regularly using 2-3 dozen of them by the end of the term.

Unix has often been criticized for being very terse (it’s rumored that its designers were bad typists). Many commands have short, cryptic names and vowels are a rarity:

awk, cat, cp, cd, chmod, echo, find, grep, ls, mv, rm, tr, sed, comm

We will learn to use all of these commands and more.

Unix command output is also very terse - the default action on success is silence. Only errors are reported, and error messages are often terse. Unix commands are often termed ‘tools’ or ‘utilities’, because they are meant to be simple tools that you can combine in novel ways.

Instructions entered in response to the shell prompt are interpreted first by the shell - expanding any variable references, filename wildcards, or special syntax. Thus, the shell can rewrite the command line; then, it expects the command line to have the following syntax:

    command [arg1]...

The brackets [ ] indicate that the arguments are optional, and the notation above means that there are zero or more arguments. Arguments are separated by white space. Many commands can be executed with or without arguments. Others require arguments, or a certain number of arguments, (e.g., cp sort.c anothersort.c) to work correctly. If none are supplied, they will provide some error message in return. Another part of the Unix philosophy is to avoid an explosion in the number of commands by having most commands support various options (sometimes called flags or switches), which modify the actions of the commands.

For example, let’s use the ls command and the -l option switch to list in long format the file filename.c.

    ls -l filename.c

Switches are often single characters preceded by a hyphen (e.g., -l). Most commands accept switches in any order, though they generally must appear before all the real arguments (usually filenames). In the case of the ls example below, the command arguments represent file or directory names. The options modify the operation of the command and are usually operated on by the program invoked by the shell rather than the shell itself.

Unix programs always receive a list of arguments, containing at least one argument, which is always the command name itself. So, for ls that first argument would be “ls”. The first argument is referred to as argument 0, “the zero-th argument”. In our ls example, argument 1 is -l and argument 2 is filename.c. Some commands also accept their switches grouped together. For example, the following switches to ls are identical:

    ls -tla foople*
    ...
    ls -t -l -a foople*

The shell parses the words or tokens (command name and arguments) you type on the command line, and asks the kernel to execute the program corresponding to that command; the interpretation of the arguments (as switches, filenames, or something else) is determined by that program.

Typically, the shell processes the complete line after a carriage return is entered and then goes off to find the program that the command line specified. If the command is a path name, whether relative (e.g., ./mycommand) or absolute (e.g., /bin/ls), the shell simply executes the program in that file. If the command is not a path name, the shell searches through a list of directories in your “path”, which is defined by the shell variable called PATH.

Shell’s ‘Path’

Take a look at your PATH by asking the shell to substitute its value ($PATH) and pass it as an argument to the echo command:

[plank ~]$ echo $PATH
/dartfs-hpc/admin/opt/el7/intel/compilers_and_libraries_2019.3.199/linux/bin/intel64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/dell/srvadmin/iSM/bin:/thayerfs/apps/other:/thayerfs/apps/abaqus/bin:/thayerfs/apps/ansys/current/Framework/bin/Linux64:/thayerfs/apps/ansys/current/fluent/bin:/thayerfs/apps/cadence/bin:/thayerfs/apps/comsol/bin:/thayerfs/apps/eclipse:/thayerfs/apps/fiji:/thayerfs/apps/gams:/thayerfs/apps/idl/bin:/thayerfs/apps/julia/bin:/thayerfs/apps/maple/bin:/thayerfs/apps/mathematica/bin:/thayerfs/apps/matlab/bin:/thayerfs/apps/maya/bin:/thayerfs/apps/netgen/bin:/thayerfs/apps/paraview/bin:/thayerfs/apps/sagemath:/thayerfs/apps/synopsys/pcmstudio/bin:/thayerfs/apps/synopsys/sentaurus/bin:/thayerfs/apps/tecplot/bin:/thayerfs/apps/totalview/bin:/thayerfs/apps/turbovnc/opt/TurboVNC/bin:/thayerfs/apps/visit/bin:/thayerfs/apps/xilinx/bin:/thayerfs/home/f001cxb/bin:/bin:/usr/java/jdk:/usr/bin:/sbin:/usr/sbin:/usr/ucb:/usr/bsd:usr/java/jdk:/usr/java/j2sdk:/usr/bin/X11:/usr/X11R6/bin:/usr/local/bin:/usr/local/bin/X11:/usr/contrib/bin:
[plank ~]$

So where does the ls command executed above reside in the Unix directory hierarchy? Let’s use another command to find out.

[plank ~]$ which ls
ls is aliased to `ls -F --color=auto`
ls is /bin/ls
ls is /bin/ls
[plank ~]$

The first line of response says that ls is “aliased”. This is a shell feature; the shell allows us to define “aliases”, which act just like commands but are actually just a textual substitution of a command name (the alias) to some other string (in this case, ls -F --color=auto). Thus, any time I type ls blah blah, it treats it as if I had typed ls -F --color=auto blah blah. The -F option tells ls to add a trailing symbol to some names in its output; it adds a / to the names of directories, a @ to the names of symbolic links (um, that’s another conversation), and some other even specialized cases.

Of course, the shell then still needs to resolve ls. It then searches the PATH to find an executable file with that name; in this case, it appears that ls exists in /bin. The shell will execute it. Below you can see the effect of running ls (the alias) and /bin/ls (the raw command, without the -F).

[plank ~]$ ls
 Archive   class/   mail/          Sent
 backup/   cs50/    public_html/  'Sent Messages'
[plank ~]$ /bin/ls
 Archive   class   mail      Sent
 backup    cs50    public_html  'Sent Messages'
[plank ~]$

Viewing files

You can see the contents of any file with the cat command, so named because it concatenates all the files listed as arguments, printing one after the other. For very long files, though, the output will quickly scroll off your terminal.

Less Is More: The less and more commands are handy for quickly looking at files. The syntax is less filename and more filename. Take a look at the man pages to get the details of each. Similarly, head and tail display a number of lines (selectable via switches, of course) at the beginning and end of a file, respectively.

See what these do: cat /etc/passwd, head /etc/passwd, tail /etc/passwd, more /etc/passwd, and less /etc/passwd. The file /etc/passwd lists all the accounts on the system, and information about each account.

Editing files

Long before there were windows and graphical displays, or even screens, there were text editors. Two are in common use on Unix system today: emacs and vi. Actually, there is an expanded/improved version of vi called vim, which is quite popular.

You should try both and become comfortable with at least one. Yes, it’s tempting to use an external graphical editor (like Sublime), but there are times when you must use a text-only editor and thus you should get used to it.

Unix file system

The Unix file system is a hierarchical file system. The file system consists of a very small number of different file types. The two most common types are files and directories.

A directory (akin to a folder on a MacOS or Windows computer) contains the names and locations of all files and directories below it. A directory always contains two special files . (dot) and .. (dot dot); . represents the directory itself, and .. represents the directory’s parent. In the following, I make a new directory with mkdir, change my current working directory to be that new directory with cd, create a new file in that directory, and use ls to explore the contents of the new directory and its parent.

[plank]$ mkdir test
[plank]$ cd test
[plank]$ echo hello > somefile
[plank]$ ls -a
./  ../  somefile
[plank]$ ls
somefile
[plank]$ ls .
somefile
[plank]$ ls ..
 Archive   class/   mail/          Sent             test/
 backup/   cs50/    public_html/  'Sent Messages'

Directory names are separated by a forward slash /, forming path names. A path name is a filename that includes some or all of the directories leading to the file; an absolute path name is relative to the root (/) directory and begins with a /, in the first example below, whereas a relative path name is relative to the current working directory, as in the second example below. Notice that a relative path name can also use . or .., as in the example below.

[plank]$ cd /thayerfs/home/f001cxb/cs50
[plank]$ cd 
[plank]$ cd cs50
[plank]$ ls ../backup
cs50/  cs60/  web/

The “change directory” command (cd) allows us to move around the Unix directory hierarchy, that is, to change our “current working directory” from which all relative filenames and path names will be resolved.

Listing files

Here are a popular set of switches you can use with ls:

-l list in long format (as we have been doing)
-a list all entries (including `dot` files, which are normally hidden)
-t sort by modification time (latest first)
-r list in reverse order (alphabetical or time)
-R list the directory and its sub-directories recursively

The shell also interprets certain special characters like *, ?, and []; * matches zero or more characters, ? matches one character, and [] matches one character from the set (or range) of characters listed within the brackets:

[plank ~]$ cd public_html/Resources/
[plank ~/public_html/Resources]$ ls
DougMcIlroy.pdf  Homebrew.html		     RC13972-C-Programming.pdf
Homebrew0.png	 index.html		     StartingSublime.pdf
Homebrew1.png	 RC13972-C-Programming.docx  toomey-unix.pdf
[plank ~/public_html/Resources]$ ls *.pdf
DougMcIlroy.pdf		   StartingSublime.pdf
RC13972-C-Programming.pdf  toomey-unix.pdf
[plank ~/public_html/Resources]$ ls H*
Homebrew0.png  Homebrew1.png  Homebrew.html
[plank ~/public_html/Resources]$ ls *-*
RC13972-C-Programming.docx  RC13972-C-Programming.pdf  toomey-unix.pdf
[plank ~/public_html/Resources]$ ls Homebrew*.*
Homebrew0.png  Homebrew1.png  Homebrew.html
[plank ~/public_html/Resources]$ ls Homebrew?.*
Homebrew0.png  Homebrew1.png
[plank ~/public_html/Resources]$ ls Homebrew[0-9].png
Homebrew0.png  Homebrew1.png
[plank ~/public_html/Resources]$

Hidden files

The ls program normally does not list any files whose filename begins with . There is nothing special about these files, except . and .., as far as Unix is concerned. It’s simply a convention - files whose names begin with . are to be considered ‘hidden’, and thus not listed by ls or matched with by the shell’s * character. Home directories, in particular, include many ‘hidden’ (but important!) files. The -a switch tells ls to list “all” files, including those that begin with a dot (aka, the hidden files).

[plank]$ ls
 Archive   class/   mail/          Sent
 backup/   cs50/    public_html/  'Sent Messages'
[plank]$ ls -a
 ./              .config/            .lesshst         .spamassassin/
 ../             cs50/               .local/          .ssh/
 Archive         .cshrc              .login           .subversion/
 backup/         .dbus/              .logout          .vim/
 .bash_history   .emacs.d/           mail/            .viminfo
 .bash_logout    .environset         .mailboxlist     .vimrc
 .bash_profile   .forward            .notfsquota      .vscode-server/
 .bashrc         .fvwm2rc            .pki/            .wget-hsts
 .cache/         .gitconfig          public_html/     .Xauthority
 .ccache/        .gitignore_global   Sent
 class/          .gnupg/            'Sent Messages'

to see just the dot files, let’s get clever with the shell’s glob characters:

[plank ~]$ ls -ad .??*
.bash_history  .cshrc       .gitignore_global  .notfsquota     .vimrc
.bash_logout   .dbus/       .gnupg/            .pki/           .vscode-server/
.bash_profile  .emacs.d/    .lesshst           .spamassassin/  .wget-hsts
.bashrc        .environset  .local/            .ssh/           .Xauthority
.cache/        .forward     .login             .subversion/
.ccache/       .fvwm2rc     .logout            .vim/
.config/       .gitconfig   .mailboxlist       .viminfo
[plank ~]$

Here the .??* pattern ensures that the name of the hidden file/directory has more than two characters to skip special files . (current directory) and .. (parent directory). The switch -d avoids showing the content of each hidden directory, instead shows only the directory name.

All of these “dot files” (or “dot directories”) are important to one program or another. Here are some examples:

  • .bash_history - used by bash to record a history of the commands you’ve typed
  • .bash_logout - executed by bash when you log out
  • .bash_profile - executed by bash when you log in
  • .bashrc - executed by bash whenever you start a new shell
  • .emacs.d/ - a directory used by emacs text editor
  • .environset - a Dartmouth-specific thing; read by .bashrc
  • .forward - tells Mail where to forwad your email
  • .ssh/ - directory used by inbound ssh connections
  • .vim/ - a directory used by vim text editor
  • .viminfo - used by vim text editor
  • .vimrc - used by vim text editor

Bash shell startup files

The bash shell looks for several files in your home directory:

  • .bash_profile - executed by bash when you log in
  • .bashrc - executed by bash whenever you start a new shell
  • .bash_logout - executed by bash when you log out
  • .bash_history - used by bash to record a history of the commands you’ve typed

The .bashrc file is especially important, because bash reads it every time you start a new bash shell, that is, when you log in, when you start a new interactive shell, or when you run a new bash script. (In contrast, .bash_profile is only read when you login.) In each case,bash reads the files and executes the commands therein. Thus, you can configure your bash experience by having it declare some variables, define some aliases, and set up some personal favorites.

For CS50 we strongly recommend the following customization:

# aliases used for cs50
alias mygcc='gcc -Wall -pedantic -std=c11 -ggdb'
alias myvalgrind='valgrind --leak-check=full --show-leak-kinds=all'

# safety aliases
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# convenience aliases
alias ls='ls -F --color=auto'
alias mkdir='mkdir -p'
alias which='type -all'

The mygcc alias adds some extra options to gcc, which we’ll cover next week.

The myvalgrind alias adds some extra options to valgrind, which we’ll cover later in the term.

The “safety” aliases protect you from accidentally deleting or overwriting files.

The “convenience” aliases give you some handy shortcuts and make the output of some commands more useful.

To make all these changes in your .bashrc, log into your Unix account and copy the above content to your .bashrc file in your home directory. You will need ls -a command in your home directory to see .bashrc file. If you do not have it, create one.

You can then edit ~/.bashrc and ~/.bash_profile to your own taste.

Our top commands

We’ve explored almost four dozen shell commands, below (those with *asterisk will be introduced later). You’ll often need only about half of them.

    alias

    cat, cd, chmod, comm, cp, cut

    date

    echo, emacs, expr, exit

    file, find

    gcc, gdb*, git*, grep

    head

    less, logout, lpr, ls

    make*, man, mkdir, more, mv

    open (MacOS)

    pbpaste, pbcopy (MacOS)

    pwd

    rm, rmdir

    scp, sed, sort, ssh

    tail, tar, touch, tr

    uniq

    whereis, which

    vi, vim

Historical note

Recall we mentioned “teletypes” in the first lecture. It was at Dartmouth that a teletype was actually used to interact with a remote computer - the first ever long-distance terminal, decades before the Internet. Scavenger hunt! Who can find the plaque below, on the Dartmouth campus? Post to Slack when you find it!

Stibitz plaque

Other useful information

We don’t have time in lecture to cover everything you may want to know; read on.

You may have found the man system to be a little challenging to navigate. There is a message that is displayed at the very bottom of the screen when you first enter the command that you might have missed (most people do):

Manual page xxx(n) line 1 (press h for help or q to quit)

If you enter ‘h’ to see the help you will find many more commands than you’re likely to ever use when reading man pages. This is because the man-page reader is actually the less command of Unix. I tend to use only a few:

  • f or space (the spacebar) advances to the next screenful,
  • b goes back to the previous screenful,
  • e or down-arrow advances one more line,
  • y or up-arrow goes back one line,
  • / allows one to type a search phrase and hit return,
  • q quits man, and returns to the shell prompt.

There are shells, shells, and more shells

There are a number of shells available to a Unix user – so which one do you select? The most common shells are:

  • sh : the original shell, known as the Bourne Shell,
  • csh, tcsh : well-known and widely used derivatives of the Bourne shell,
  • ksh : the Korn shell, and
  • bash : the Bourne Again SHell, developed by GNU, is the most popular shell used for Linux.

bash is the default shell for new Unix accounts in our department.

The basic shell operation is as follows. The shell parses the command line; the first word on the line is the command name. If the command is an alias, it substitutes the alias text and again identifies the command. If the command is one built-in to the shell (there are a few, like cd, echo, pwd, and which) it performs that command’s action. Otherwise, the shell looks for the executable file that matches that program name by searching directories listed in the PATH variable. The shell then starts that program as a new process and passes any options and arguments to the program. A process is a running program. You can see a list of your processes with the command ps.

File type

Unix itself imposes almost no constraints or interpretation on the contents of files - the only common case is that of a compiled, executable program: it has to be in a very specific binary format for the operating system (Unix) to execute it. All other files are used by some program or another, and it’s up to those programs to interpret the contents as they see fit. The great power of Unix, and the common shell commands, is that any file can be read by any program; the most common format are plain-text (ASCII) files that are formatted as a series of “lines” delimited by “newline” characters (\n, known by its ASCII code 012).

If you are unsure about the contents of a file (text, binary, compressed, Unix executable, some format specific a certain application, etc.). The file command is useful; it makes an attempt to judge the format of the file.

[plank ~]$ file downloaded
downloaded: POSIX tar archive
[plank ~]$ file fig1
fig1: GIF image data, version 87a, 440 x 306
[plank ~]$ file trash.tar.gz
trash.tar.gz: gzip compressed data, from Unix
[plank ~]$ file public_html/Schedule.*
public_html/Schedule.pdf:  PDF document, version 1.3
public_html/Schedule.xlsx: Microsoft Excel 2007+
[plank ~]$ file /bin/ls
/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=0ac5c509289d650534ce80cdbf5b72744b5c5f3d, stripped
[plank ~]$

Locating files

Many times you want to find a file but do not know where it is in the directory tree (Unix directory structure is a tree - rooted at the / directory) . The find command can walk a file hierarchy:

[plank ~]$ find . -name DougMcIlroy.pdf -print
./public_html/Resources/DougMcIlroy.pdf
[plank ~]$ find public_html -iname reading -print
public_html/Reading
[plank ~]$ find public_html -type d -print
public_html
public_html/Logistics
public_html/Resources
public_html/Labs
public_html/Project
public_html/Lectures
public_html/css
public_html/Reading
[plank ~]$ find public_html -name \*.html -print
public_html/Logistics/index.html
public_html/Resources/index.html
public_html/Resources/Homebrew.html
public_html/Labs/index.html
public_html/Labs/Lab0-Preliminaries.html
public_html/Project/index.html
public_html/Lectures/01-gettingstarted.html
public_html/Lectures/index.html
public_html/index.html
public_html/Reading/index.html
[plank ~]$ find public_html -name \*.png -mtime -1 -print
public_html/Resources/Homebrew1.png
public_html/Resources/Homebrew0.png
[plank ~]$

The first example searches directory . to find a file by a specific name and prints its path name. The second example used -iname (case insensitive search) instead of -name (which is case sensitive) to search public_html for the “reading” directory. The third example searches public_html for any directories (-type d) and prints their path names. The fourth example uses a wildcard * to print path names of files whose name matches a pattern; the backslash \ is there to prevent the shell from interpreting the *, allowing it to be part of the argument to find, which interprets that character itself. The fifth example combines two factors, to print path names of files whose name matches *.png and whose modification time mtime is less than one day -1 in the past.