02. Basic Scripting Concepts (Part 1)

The Shell as a Command Line Interpreter

• The shell is an executable program that interacts with the user through the command line

• The first purpose of the shell is to be a command line interpreter: it parses your command, runs the appropriate utility, and sends any output / error to the appropriate destination

• If you have a list of commands that need to be run, you can type each command one at time on the command line. For every command you type in, you see the output before typing the next command.

  • Running a list of commands this way is appropriate when you need to evaluate the output of each command before running the next command.

• Alternatively, you can type the list of commands into a text file. This file becomes a shell script, and the shell can step through each line of the script and run each command without your intervention. 

  • When the shell runs each command in a shell script, it interprets each command in the script the same way it does on the command line

  • Running a list of commands this way is more convenient if you do not need to inspect each command output before running the next command.

• Example of a basic shell script:

  • echo The directory `pwd` has:

  • ls 

  • This shell script will run exactly the same way and produce the same output as if the echo command is typed on a command line, then the ls command is typed on the second command line

  • The script output on screen can be:

    • The directory /home/student/abc has:

    • fileA    fileB

  • When a shell script is made up of common utilities that are run one after another, it is in general portable across different shells. This means a script written to run on the bash shell can also be run by the C shell.

The Shell as a Programming Language

• The shell can do much more than interpreting each command  line by line, whether from the command line or from a script

• The second purpose of the shell is to be a high level language

• This means the shell has built-in programming constructs that support a sophisticate shell script, similar to how a programming language supports its structured programs

• The shell has built-in support for flow control:

  • Loops 

  • Conditional statements

  • Operators

  • Functions

• It also has built-in support for data:

  • Shell variables

  • User defined variables

  • Command line arguments

  • Arrays and strings

• It also supports for structured programming because it supports functions and parameter passing

• To have the built-in programming constructs, the shell has a set of specific syntax to support these constructs

• The syntax dictates how a programmer can write the constructs so that the shell can understand them. For example, to evaluate a logical expression, the bash shell requires that the expression be surrounded by [ ]

• The syntax for the programming constructs are specific to a type of shell. For example, in the tcsh shell the same logical expression discussed above is surrounded by ( )

• Therefore, when a script contains shell programming constructs, it can only run with one type of shell and is not portable to other shells

• We will start with the bash shell, so keep in mind that the syntax in this module and in the next several modules apply to the bash shell

Common Use of Shell Scripts

Some common reasons why a shell script is written

• You have a list of commands that you need to run many times.

  • Rather than typing the list of commands over and over, you type them once in a script. Then it takes one command to run the script, and your entire list of commands will automatically be run

• You need to set up certain environment variables or system conditions before running a specific command. 

  • For example, you want to follow the POSIX standard for sort, but you don’t want the POSIX standard for other commands. You can write a script in which you set LC_ALL to POSIX and then run the sort command. When you run the script, the sort command will run with POSIX standard; and when you run other commands, LC_ALL remains the same default value.

• You want flow control in running the commands, such as running them conditionally or repeatedly, or you want to use complex data structures

  • Using the programming constructs that the shell provides, you can run your list of commands in a more flexible (more ‘intelligent’) way, instead of just running each command one time, from top to bottom of the script

A Shell Script vs. A Program

• A shell script and a program are similar in that they are both:

  • A text file

  • Have variables, constants, data structures, flow control constructs, functions

  • Have instructions that will be executed one by one to do some task

• A shell script and a program are different in how they get executed:

  • A program must be compiled into a binary file, then eventually built into an executable (which is also binary) that is run. 

    • If a program has a syntax error, it is caught in the compile stage, before the program can run.

  • A shell script is interpreted and run line by line, directly from the text file. 

    • If a shell script has syntax error, it will start running until the line with the syntax error is interpreted. Then the syntax error is caught.

Types of shells

• A Linux system usually offers several types of shell

• Historically, after Unix was released from AT&T Bell Lab, it soon developed into 2 major versions: 

  • Unix System V which continued to be developed by AT&T Bell Lab

  • Unix BSD which was developed at UC Berkeley

• As is with all software that are developed separately, each Unix version ended up with a different shell:

  • sh or Bourne Shell: for System V, the original shell that is still available on many systems. It is a basic shell, with a small footprint and few features. 

  • csh or C shell: for Unix BSD, the syntax of this shell resembles the C language. It is favored by C programmers, especially ‘old timers’ with a history of programming on Unix BSD systems.

• More modern Unix / Linux systems now runs the ‘descendants’ of the Bourne and C shells

• Derived from the Bourne shell

  • bash or Bourne Again shell: the GNU shell which is the standard shell for Linux systems. This shell is a superset of the Bourne shell: it starts with the Bourne shell syntax and adds more features. This means that a Bourne shell script can be run with bash. However, the reverse is not true, a bash script cannot always be run with sh.

  • ksh or the Korn shell: appreciated by people with a UNIX background. It is also a superset of the Bourne shell, a Bourne shell script can be interpreted with ksh, but a Korn shell script cannot always be run with sh.

• Derived from the C shell

  • tcsh or TENEX C shell: this shell is a superset of the C shell, with more features, user-friendlier, and faster. It is sometimes called the Turbo C shell. It is meant to replace csh, which can be inconsistent with how it interprets a script

  • The file /etc/shells lists the shells that are available on a particular Linux system

  • First we will learn the concepts of shell programming with the bash shell, using the bash syntax and then later we will keep the same programming concepts but switch syntax so that we write scripts for the Korn and tcsh shells.

Creating a Shell Script

• To create a shell script, use a text editor to create a text file and add any command that can run on the command line

• Typically the commands are listed one per line

• If a command is too long for one line, use \ before the enter key to escape the newline character, then continue with the next line

• For example:    ls -l | awk ‘{total += $5} END {print “Total:”, total}’

•   can be written as 2 lines:     ls –l | awk ‘{total += $5}  \

•          END {print “Total:”, total}’

• Any valid filename can be used for a shell script, but it’s helpful to make the filename descriptive

• Filename extension is not required, but it is common practice to end with a .sh extension 

Making a Shell Script Readable

Documentation

• Just like with any program, a well documented shell script makes it easier to debug and maintain the script

• Documentation generally includes

  • A comment block at the beginning of the program to explain the purpose of the script, the author name, the date, and any other information that introduces the reader to the script

  • For complicated computation or logical decision making, a comment line above or next to the code to explain the code

  • A comment begins with the # token and ends at the end of the line

  • Naming variables and functions

    • Variables (including arrays and strings) should be named descriptively so it is easy to follow them throughout the code

      • For example: userCount and total are easier to read than x and y

    • Likewise, function names should be descriptive by letting the reader know what their task is

      • For example, listFiles is easier to read than function1

• Typically variable and function names are in lowercase, since uppercase characters are for system variables