08. The Shell (Part 1)

Input and Output

• When you give input to a command, you type in from the keyboard. Standard input for the shell is from the keyboard
• When the command has an output, the shell prints it to the screen. Standard output for the shell is to the screen
• When the shell or the command has an error message, the shell prints it to the screen. Standard error for the shell is to the screen
• It is possible to tell the shell to get input from some place other than the keyboard.
• It is also possible to tell the shell to send output and error to some place other than the screen. When this happens, you won’t see any output or any error on screen.

Redirection

• Redirection tells the shell that input, output, and/or error is not from the keyboard or the screen, but from a text file.
• Redirecting input tells the shell to get input from a file
  • Redirecting output tells the shell that output goes to a file Redirecting error tells the shell that error goes to a file
• To redirect input: command < filename
  • command means the full command line, with options and arguments if necessary
  • filename can have a path, it is the input to the command
• To redirect output (overwrite): command > filename
  • To redirect output (append): command >> filename
  • command means the full command line, with options and arguments if necessary
  • filename can have a path, it stores all output from the command
  • If filename doesn’t exist, a new file is created
  • If filename exists, the overwrite option will overwrite the existing file, and the append option will append the output to the file
• To redirect error (overwrite): command 2> filename
  • To redirect error (append): command 2>> filename
  • command means the full command line, with options and arguments if necessary
  • filename can have a path, it stores error messages from the command
  • There is no space between the 2 and the > or >>
  • If filename doesn’t exist, a new file is created
  • If filename exists, the overwrite option will overwrite the existing file, and the append option will append to the file
• A combination of redirection of input, output, and error can be done: command < input_file > output_file 2> err_file where the order of redirecting input, output, error is not important
• To have output and error redirected to the same file: command > output_file 2>&1
  • In OS tradition, the input is assigned the value 0, output is assigned the value 1, and error is assigned the value 2.
  • So 2>&1 means the error (2) is redirected (>) to the address (&) of output (1)

Pipe

• Instead of getting input / output from / to a file, you can also ask the shell to get the input/output from / to another command. This is done through a pipe with the symbol: |
• Format: command1 | command2 | command3
  • command1, command2, command3 are different commands with options and arguments, if necessary
  • The output of command1 does not get printed on screen, instead it is sent to command2 and becomes the input of command2, therefore command2 does not need an input argument.
• Likewise, the output of command2 becomes the input of command3 so command3 does not need an input argument
• Rules for piping:
  • The command before the pipe symbol | needs to have output
  • The command after the pipe symbol | needs to accept input
  • You can pipe as many commands together as needed, as long as there are command output that can be used as input to another command

tee

• So far we can tell the shell to send output to a file or to another command.
• It is also possible to tell the shell to send the output to both a file and another command, by using a tee
• It is called a tee because the shell essentially makes 2 copies of the output and pipes both through a T intersection. At the T intersection, one copy goes to the file and the other copy goes to another command
• Format: command1 | tee filename | command2
  • command1, command2 are different commands with options and arguments, if necessary
  • The filename comes after the tee
  • The output of command1 is piped | into the tee, the first copy goes to filename, and the second copy goes to another pipe | to become the input to command2. command2 does not need an input argument.
  • If there is no piping into command2, then the second copy goes to standard output by default:
    • command1 | tee filename

Multiple Commands on a Command Line

• Commands that work together
  • With piping, the command line contains multiple commands that work together
  • Piping can work with redirection and tee, allowing the shell to do complex coordination between the commands on the same command line
  • Examples:
    • command1 < inFile | command2 | command3 > outFile
      • The input of command1 is inFile, the output of command1 is the input of command2, the output of command2 is the input of command3, the output of command3 is redirected to outFile
    • command1 | tee outFile1 | command2 | tee outFile2
      • The output of command1 goes to outFile1 and is the input to command2, the output of command2 goes to outFile2 and to screen
• Commands that are independent from each other
  • You can have multiple commands on the same command line, but the commands are independent from each other
  • Format: command1 ; command2 ; command3
    • command1, command2, command3 are different commands with options and arguments, if necessary.
    • The command line above runs the same way as if you type the 3 commands on 3 separate command lines
  • Most of the time, each independent command should be put on a separate command line. This causes the output of each command to immediately follow the command line, making it easier to distinguish the output.
  • Putting 2 or more independent commands on the same command line is useful when
    • the 2 commands complement each other and there aren’t that many output lines combined. For example: cd; pwd
    • The independent commands all have output or error that should be redirected together into 1 file

Command Grouping

• When multiple, independent commands need to have all their output or all their error redirected to the same file, use command grouping
• Format: ( command1; command2 ) > outputFile
  • command1, command2 are different commands with options and arguments, if necessary
  • The parentheses ( ) tell the shell to group the commands together
  • The shell runs command1 and then runs command2. The output of command1 is redirected to outputFile and the output of command2 is automatically appended to outputFile, right after the output of command1
  • You can group as many commands together as necessary

Preserving Command Arguments

• Command arguments typically don’t have quotes around them
• But single quotes or double quotes can be used around arguments
• The majority of the time, quoting the arguments doesn’t affect how the command runs, so quoting is not used

There are some cases where quoting the argument is necessary

• (1) To preserve white spaces in the argument:
  • without quotes, extra white spaces will be ignored by the shell and will appear as a single white space
  • you can use single or double quotes
• (2) To preserve quotes in the argument
  • If the argument is a text string with quotes
    • use a pair of ‘ around the “ to preserve the “
    • use a pair of “ around the ‘ to preserve the ‘
    • use \ in front ‘ or “ to preserve both quotes
• (3)To preserve shell special characters in an argument
  • Shell special characters, called metacharacters, cause the shell to interpret the character as special instruction from you
    • For example, > on the command line means redirection, and not a literal > symbol
  • Sometime an argument can contain a metacharacter, and you want to keep the literal meaning of that character
  • To tell the shell to keep the literal meaning of metacharacters in the argument: Use \ in front of each metacharacter
  • Metacharacters are:
    • & ; | * ? ~ ! $ ^ # / \ ‘ “ ` [ ] ( ) { } < > space
    • and newline (the enter key)
  • Difference between single and double quotes
    • In double quotes, the metacharacters $ and ! still have their special meaning, just like with no quotes
    • In single quotes, the characters $ and ! have their literal meaning
    • Generally, if you want all characters to keep their literal meaning, it’s safest to use single quotes around them

Command Substitution

• Command substitution asks the shell to run a command and store the output in a string, which is then used as input to another command or to store in a variable
• Format: command1 `command2` OR command1 $(command2)
  • The ` in the first format is the back quote, it is not a single quote
  • In the second format, there is no space between the $ and the (
  • command1, command2 are commands with options and arguments, if necessary
  • The output of command2 is stored as a string and then used as an input argument for command1
• Another use: variable_name=`command` OR variable_name=$(command)
  • command can have options and arguments, if necessary
  • The output of command is stored as a string in the variable var_name (more about variables in the Variable slides later)