In this lecture, we discuss the aim and schedule of the course, and take a brief look at Unix, C programming, and computer architecture.

The course

The aim of this course is to develop the necessary systems programming skills in C and Unix as a foundation to tackle the design, implementation, and integration of a large software project working in small teams. The challenge of the course is to quickly get people up to speed so there is sufficient time to get into the details of a complex software design project. The first part of the course serves to develop design, programming and other systems skills such as source-code management, testing, and debugging. The second part of the course is all about the project and team work. Good team work will lead to success. That’s the message.

A note about Unix/Linux: Unix came first. All of the others, including Solaris, AIX, BSD and all the variants of Linux, are derivations of that original effort. So, unless it really matters, I will use the term Unix to refer to them all.

The syllabus in a nutshell:

  • Unix: shell, commands, shell programming
  • C: structure, arrays, pointers, dynamic memory, files
  • Programming tools: gcc (GNU compiler), make (maintain groups of programs), gdb (GNU debugger), git (source code management), and valgrind (profiling and memory leak checker)
  • Software development methodology: design, implement, unit test/debug, integration, demo, maintenance
  • Thematic programming assignments: building a search engine (crawler, indexer, query engine)
  • Team project.

The course includes six Unix shell and C programming assignments for the first part of the course; these assignments are to be done individually. The last part (about 2 weeks) is devoted to the team project. There are no lectures in the last part of the course but the projects are run with design reviews and progress meetings where the team can brainstorm problems and come up with solutions.

Please take note of the reading assigned each week. I will list the reading week by week, rather than lecture by lecture, to give you some flexibility – but please do the reading, you’ll find it to be interesting and valuable!

Logistics

Go over the logistics on the course home page.

About communication: be sure to track Slack and Canvas for announcements and updates.

About engagement: you will learn more from this course if you are actively engaged, where engagement = preparation + participation. Read the lecture notes before the class, practice at home, and participate in classroom activities. The Learning Fellows are here to help with classroom activities, and the Section Leaders are here to help outside class.

About the programming: There is a significant amount of programming in this course requiring a considerable time commitment on the part of the student. You will need to be well organized to complete all the programming assignments and project. It will be challenging, but we hope it will be fun!

About the project: You will be assigned to a team of three or four students and given about two weeks to complete a large project requiring strong collaboration and a problem-solving mindset. Each member is responsible for contributing to the overall system design, implementation, testing, integration, and documentation. The goal of this activity is to help you develop the confidence, skills, and habits necessary to write large computer programs while being part of a multi-person team. You will become conversant in software engineering paradigms, such as source code control with git and other open source tools that ease the software development process. In addition, you will develop vital skills in self-directed learning, problem solving, and communication. The project concludes with a demo and a review of your design and implementation. All members of the team get the same base grade, plus a team-contribution grade determined in part by a short evaluation form completed by all members of each project team.

Goals for today’s lecture

We plan to cover the following in today’s lecture:

  • The concept of a command line;
  • Logging on to a Unix machine;
  • Looking at the home directory and its files;
  • Copying files to/from remote machines;
  • Logging out; and
  • Some housekeeping business.

In the first few lectures we cover Unix, the shell, and shell programming. This is not meant to be a detailed presentation of the Unix OS and its programming tools – it would require a whole term to cover all that material in detail. We need to know enough about Unix and its tools to be able to navigate our way around the system, write some basic shell scripts, and use its programming tools.

It is important that you use these notes as a starting point, but like any budding hacker you need to do some experimenting and read up on the details. You need to go on the web and find information if there are gaps in your knowledge, and then come see the instructor or TAs for help if you’re still stuck. There are many references on the Resources page.

Caveat: Please take note that lecture notes will not always be detailed. You will need to augment these notes with your own comments and by using the references and reading assignments so you can dive deeper into the topic.

The command line

Unix was originally developed for computers with hardwired ‘terminals’, each of which was basically an electronic typewriter - a printer with a keyboard. There were no graphical displays, and the concepts of ‘windows’ and ‘mouse’ had not yet been invented. To interact with the computer, the user types on the keyboard, and the computer echoes the keystrokes on the paper. The system is controlled by typing ‘commands’, most of which print results for the user to see. The interaction occurs over the ‘command line’.

Modern Unix systems support rich graphical user interfaces, but under the hood they all support the command line. In this class, I will demonstrate everything using the MacOS command line, accessed via the MacOS application called Terminal. If you have a Mac, you can follow along. If you use Windows, consider enabling Windows Subsystem for Linux (WSL), which allows running native Linux command-line tools directly on Windows. Check out this link on how to enable this feature. But all of us can (and will) also use Thayer Linux servers.

When I open a new Terminal window on my Mac, I can type Unix commands at the ‘prompt’. This prompt is from the ‘shell’ – a program – that interprets your command line and runs a program representing each command. Your prompt may look different - the shell allows users to customize prompts and you’ll see a few formats in the examples below.

[f002xxx@plank ~]$ ls
Archive/  data/  dotfiles/  public_html/
[f002xxx@plank ~]$ echo Hello CS50
Hello CS50
[f002xxx@plank ~]$

The ls command is the “list directory contents” command. The echo command is like a print statement - it echoes its arguments back.

Most commands quickly produce some output and then exit. Some will run too long - perhaps printing too much output; you can stop (kill) the command, forcing it to exit, by typing control-C ^C at the keyboard. One silly program, yes, just prints an infinite sequence of y characters until you kill it:

[f002xxx@plank ~]$ yes
y
y
y
^C
[[f002xxx@plank ~]$

Some commands ask for your input, and continue to read input until they read an “end of file” (EOF); you can cause the program to detect an EOF by typing control-D ^D at the beginning of an input line. Below I typed three lines of text, the ^D at the start of the fourth input line:

[f002xxx@plank ~]$ cat > test.txt
Hello from cs50
This is a test file
How should we end the input?
[f002xxx@plank ~]$

Here, when we used cat command to create a test file and provide content of the file on the fly. I typed ctrl-D, so the cat program detected an end of file (EOF) on its input (i.e., the keyboard), then exited. You will see a new file test.txt has been created on the current directory with the provided input as the content.

Notice the difference between ^C and ^D; the former kills the program immediately, whereas the latter causes it to detect EOF when it next reads input from the keyboard.

Logging into a remote machine using ssh

The Thayer School of Engineering operates a set of Linux servers. For this course, the primary server we will use is called plank.thayer.dartmouth.edu; you may also use equivalent servers called babylonX.thayer.dartmouth.edu, where X is 1 through 12. (You can check their status here.) Your laptop must first be on the campus network, or on Dartmouth VPN, to access these servers. You should plan to log in to one of those servers for doing the labs.

I’ll login in from my Mac using the secure shell (ssh) Unix command. The ssh command establishes a secure channel and uses public-key cryptography to authenticate the remote computer and the user.

The ssh command replaces the archaic telnet (remote communications with another computer) and rlogin (remote login) because they lack security. The ssh command is exclusively used, these days, because your session is encrypted when it’s transmitted over the network, rather than being sent in clear text.

Assume your your NetID is f002xxx, you will log into the server named plank by giving its full IP name - plank.thayer.dartmouth.edu.

tjp@~> ssh f002xxx@plank.thayer.dartmouth.edu
[f002xxx@plank ~]$ ls
Archive/  data/  dotfiles/  public_html/
[f002xxx@plank ~]$ ls dotfiles/
cs50-home/  fresh-accounts/  MacOS/
[f002xxx@plank ~]$ logout
Connection to plank.thayer.dartmouth.edu closed.
tjp@~>

Tips for speedier login

If you are a Mac or Linux user, you can reduce typing by adding the following to your laptop’s .ssh/config file (using your own NetID); then you can type ssh plank and only have to enter your password to login.

Host plank
   Hostname plank.thayer.dartmouth.edu
   User f002xxx

On MacOS, this file is ~/.ssh/config, and you can create (if it doesn’t already exist) or append to it by opening that file in your favorite editor.

Every time you login, it reminds you about whether you are getting close to consuming all your disk-space quota:

<< You are currently using 47.39M of your 5.00G home directory quota. >>

That’s nice, but it is slow. You can turn off this message by creating a file in your home directory on plank. Although the mere presence of this file is sufficient, put some text in the file so you can remember why it is there:

echo The presence of this file disables login notification of your disk-quota usage. >  ~/.notfsquota

If you later wish to check your disk usage, you can remove this file (then logout and login), or simply run

tfsquota

There is also a simple web portal to check your quotas.

It is also possible to connect to a remote Linux server using X-windows, an early form of graphical user interface that has the capability to run applications on a remote server but put the windows on your local computer. I won’t be using X in this course, but you may wish to explore the option.

There are hundreds of Unix commands – but you probably only need a few dozen to get by. Each Unix command has a short abbreviated command name (e.g., LiSt directory (ls) or Secure SHell (ssh)) and its associated syntax typically includes various arguments, and options; typically, these options/switches are either a single letter preceded by a hyphen (e.g., -l) or one or more words preceded by two hyphens (e.g., --verbose). For example, the format of an ssh command line looks like this:

	ssh [options] [user@] hostname [command]

In ssh -l f002xxx plank.thayer.dartmouth.edu, the switch -l informs the ssh command that the username of the user logging in is f002xxx. Alternately, I used the form ssh f002xxx@plank.thayer.dartmouth.edu earlier.

This is a good time to look at the formatting of these command help texts. The [ ] are used to denote optional things, like

[command]

while the things outside of the [ ] , like hostname, must be specified.

Getting Information using the online manual (man)

If you want the detailed syntax of a Unix command you can use the manual command followed by the command, as in man ssh which produced the following:

    SSH(1)                    BSD General Commands Manual                   SSH(1)

    NAME
         ssh -- OpenSSH SSH client (remote login program)

    SYNOPSIS
         ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec] 
             [-D [bind_address:]port] [-e escape_char] [-F configfile] [-I pkcs11] 
             [-i identity_file] [-L [bind_address:]port:host:hostport] [-l login_name]
             [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] 
                [-R [bind_address:]port:host:hostport] [-S ctl_path]
             [-W host:port] [-w local_tun[:remote_tun]] [user@]hostname [command]

    DESCRIPTION
         ssh (SSH client) is a program for logging into a remote machine and for executing
         commands on a remote machine.  It is intended to replace rlogin and rsh, and
         provide secure encrypted communications between two untrusted hosts over an 
         insecure network.  X11 connections and arbitrary TCP ports can also be forwarded 
         over the secure channel.

         ssh connects and logs into the specified hostname (with optional user name).  The 
         user must prove his/her identity to the remote machine using one of several methods 
         depending on the protocol version used (see below).

         If command is specified, it is executed on the remote host instead of a login shell.

         The options are as follows:

         -1      Forces ssh to try protocol version 1 only.

         -2      Forces ssh to try protocol version 2 only.

         -4      Forces ssh to use IPv4 addresses only.

         -6      Forces ssh to use IPv6 addresses only.

         -A      Enables forwarding of the authentication agent connection.  This can also
                 be specified on a per-host

    ... and a whole lot more

This is just a snippet of the man ssh output (man is short for manual). The manual output includes all the nitty gritty details on options and about the command. For most commands you can use the common option --help (two hyphens) to get a brief breakdown of the command and its switches. This doesn’t work for all commands (including ssh, interestingly), but in that case the use of -help is interpreted as an invalid entry by ssh and it lists of the options anyway.

You can use

man -k keyword

to search through the manual pages for matches on a keyword. For example:

[f002xxx@plank ~]$ man -k shell                                                           
bash (1)             - GNU Bourne-Again SHell
capsh (1)            - capability shell wrapper
chroot (1)           - run command or interactive shell with special root directory
chsh (1)             - change your login shell
CPAN::Admin (3pm)    - A CPAN Shell for CPAN admins
CPAN::Plugin (3pm)   - Base class for CPAN shell extensions
csh (1)              - C shell with file name completion and command line editing
... and much more

Your home directory and its files

Each user has a ‘home directory’. After you have logged in using ssh you are in your home directory - that is, the shell’s notion of your ‘current working directory’ is your home directory.

We can look at our home directory ‘path’ using the pwd (print working directory) command. You can always use the man and info commands to get more information.

[f002xxx@plank ~]$ pwd
/thayerfs/home/f002xxx
[f002xxx@plank ~]$

The tilde (~) above is shorthand for ‘home’. Let’s take a look at the contents of my home directory (using the -l switch which means long format):

[f002xxx@plank ~]$ ls 
Archive/  data/  dotfiles/  public_html/
[f002xxx@plank ~]$ ls -l
total 16
drwxr-x---  8 f002xxx thayerusers 4096 Jun 21 14:58 Archive/
drwxr-xr-x  5 f002xxx thayerusers 4096 Jun 11 06:29 data/
drwxr-xr-x  5 f002xxx thayerusers 4096 Jun 11 06:29 dotfiles/
drwxr-xr-x 10 f002xxx thayerusers 4096 Jun 19 15:15 public_html/
[f002xxx@plank ~]$ cd cs50
-bash: cd: cs50: No such file or directory
[f002xxx@plank ~]$ cs cs50
-bash: cs: command not found
[f002xxx@plank ~]$ cd dotfiles/
[f002xxx@plank ~/dotfiles]$ ls -l
total 12
drwxr-xr-x 2 f002xxx thayerusers 4096 Mar 29 22:16 cs50-home/
drwxr-xr-x 2 f002xxx thayerusers 4096 Mar 29 11:15 fresh-accounts/
drwxr-xr-x 2 f002xxx thayerusers 4096 Apr 13 14:08 MacOS/
[f002xxx@plank ~/dotfiles]$

Yes, I mistyped two commands in that example: cd cs50 and cs cs50. I wanted you to see what the system says when you make a mistake … just in case you make one yourself sometime. ;-)

The cd command changes the current working directory; I finally typed it right and the shell reported that my working directory is ~/dotfiles. The same command, ls -l, listed that directory the second time.

Files can be plain files, directories, or special files (more later). We can see that each file has file permissions and other data associated with it; for example, the directory public_html:

drwxr-xr-x 10 f002xxx thayerusers 4096 Jun 19 15:15 public_html/

You can traverse directory trees assuming you have the appropriate permission.

Unix supports a number of shells (command line interpreters). If we use the echo command we can look at the environment variable that tells us which shell is running. For this course we will use the bash (Born Again SHell) shell.

Again, the shell is the command processor for Unix systems. One way to find out what shell you’re running, try this:

[f002xxx@plank ~]$ echo $SHELL
/bin/bash
[f002xxx@plank ~]$

The first parameter to echo substitutes the value of the variable SHELL. More on the bash shell later.

Another useful command for copying files between machines is the scp - secure copy (remote file copy program) command. Below I find, then copy, of the schedule for CS50 from the course website.

tjp@~> ssh plank
[f002xxx@plank ~]$ ls
Archive/  data/  dotfiles/  public_html/
[f002xxx@plank ~]$ cd public_html/
[f002xxx@plank ~/public_html]$ ls
Comics/  data@      index.html  Lectures/   Reading/   Resources/
css/     examples/  Labs/   Logistics/  README.md  Schedule.pdf
[f002xxx@plank ~/public_html]$ logout
Connection to plank.thayer.dartmouth.edu closed.
tjp@~> scp f002xxx@plank.thayer.dartmouth.edu:~/public_html/Schedule.pdf .
Schedule.pdf                                                            100%   28KB  28.2KB/s   00:00    
tjp@~> open Schedule.pdf

Recall that ~ indicates the home directory, so ~cs50 is the home directory for user cs50. I changed my directory there, then into its public_html, and listed the files there. Now I know the ‘path name’ for the desired file - the directory name followed by slash followed by the filename. The scp command allows me to specify the host and pathname for the source and destination of the copy; here the destination is ., a shorthand for ‘current working directory. Finally, I used the MacOS command open to open the file, which launches Preview to show me the pdf.

Copying with scp

If you want to copy files (or directories) from your laptop to the Linux server, or from the Linux server to your laptop, use the scp command. For our purposes the syntax is

scp [-r] [username=@hostname:]filename [username@hostname:]filename

which specifies the source and destination of the copy. For an argument including a hostname, the filename is relative to the user’s home directory. If you want to specify a file in another directory, write out its pathname (relative or absolute).

In one common case, you might copy a file test.txt from your laptop to plank:

$ scp test.txt netID@plank.thayer.dartmouth.edu:~/cs50/labs/

Or in the opposite direction (here . represents the current directory):

$ scp netID@plank.thayer.dartmouth.edu:~/cs50/labs/test.txt .

You can copy a directory, recursively, with the -r flag:

$ scp -r netID@plank.thayer.dartmouth.edu:~/cs50/labs/lab1 lab1

Logging out

OK, we are ready to logout from our session on plank.thayer.dartmouth.edu.

[f002xxx@plank ~]$ logout
Connection to plank.thayer.dartmouth.edu closed.

Okay, this has been a good start. We have covered a number of important issues that we will revisit in the course.

Housekeeping - things you need to do:

  1. Visit Canvas, join the course Slack channel.

  2. Review all of the class materials on the website (About, Schedule, Reading, Style Guide, Resources), accessible via course web page.

  3. Before next class, use your NetID to go through the above examples: execute all the commands and get a ‘feel’ for the shell.

  4. Choose an editor and become very familiar with it. This is a very important step, as the reading assignment “Learn to use a real editor” (see page 117) says. Your first homework assignment will be released at the next class period, so don’t delay!

One more thing: Remotely accessing Thayer Linux machines

Note the following, depending on your personal computer’s operating system.

Linux: The systems in the Thayer Linux Servers are running Ubuntu Linux. Your Linux will likely be sufficiently compatible that you won’t have any trouble developing on your machine and delivering your assignments on a Thayer machine. That outcome, however, is not guarantied and you should test your solution on a Thayer machine. You may want to try the X-windows connection.

Mac: Mac OSX Unix conforms to the POSIX standard for the C API, shell utilities, and threads and can compile and run your existing code. This is really exciting for Unix/Linux development. The GNU tools we will be using, such as gcc, make, etc., are either included or freely available. (You may need to install command-line tools via Xcode; if so, you will be prompted to do so when you first type gcc or make or related commands.) You can use the Terminal application. You can also use the ssh command to remotely log on to computers, as discussed above.

Windows: Starting from Window 10, Windows Subsystem for Linux (WSL) has been provided to run native Linux command-line tools directly on Windows. Check out this link on how to enable this feature. However, given this is a Unix course you would be far better off doing your assignments on the departmental server. The most successful approaches have been to do your editing and documentation on the Windows system and to upload (using scp, sftp, filezilla, etc.) and build/debug on the lab systems via ssh. Many students have adopted the Sublime Text editor and used its handy sftp integration.

Historical note

The name “Terminal” is still commonly used in Unix parlance, right down to the MacOS app by that name. This term refers to the fact that early Unix computers, like other computers of its time, had a small number of interactive ‘terminals’ connected to it – each directly connected by a dedicated wire to a card within the computer. Each terminal was little more than a keyboard and a screen - or in the early days, a keyboard and a printer that would print each character as it was typed, and print the characters sent by the computer.

The first such terminals were “teletypes”, adapted from the device used by newsrooms around the country, which printed characters transmitted over the phone line from a distant device. This video shows a Teletype model ASR33. From the sound of this machine, can you tell why television news shows’ opening music often has a stocatto tap-tap-tap-tap-tap background theme?