Topics:
Working with the Shell
Input and Output
Redirection
Pipe and tee
Multiple commands and grouping
Quoting rules
Command substitution
Working with the Shell
The shell is the interface between you and the Linux system. So far you’ve observed the shell working in a cycle to support you:
Print the prompt and wait for your command
Parse your command into the utility name, options, arguments
Search for a utility that matches the name
If no match, print an error message and return to step 1
If match, start a process to run the utility, pass to the utility the options and arguments, expand argument names if there are wilcards
Wait for utility to finish and end the process
Print any error message or output from the utility, if any is sent back from the utility
Return to step 1
Input and Output
When you give input to a command, you type in from the keyboard. Standard input for the shell is from the keyboard
When the command has an output, the shell prints it to the screen. Standard output for the shell is to the screen
When the shell or the command has an error message, the shell prints it to the screen. Standard error for the shell is to the screen
It is possible to tell the shell to get input from some place other than the keyboard.
It is also possible to tell the shell to send output and error to some place other than the screen. When this happens, you won’t see any output or any error on screen.
Redirection
Redirection tells the shell that input, output, and/or error is not from the keyboard or the screen, but from a text file.
Redirecting input tells the shell to get input from a file
Redirecting output tells the shell that output goes to a file Redirecting error tells the shell that error goes to a file
To redirect input: command < filename
command means the full command line, with options and arguments if necessary
filename can have a path, it is the input to the command
To redirect output (overwrite): command > filename
To redirect output (append): command >> filename
command means the full command line, with options and arguments if necessary
filename can have a path, it stores all output from the command
If filename doesn’t exist, a new file is created
If filename exists, the overwrite option will overwrite the existing file, and the append option will append the output to the file
To redirect error (overwrite): command 2> filename
To redirect error (append): command 2>> filename
command means the full command line, with options and arguments if necessary
filename can have a path, it stores error messages from the command
There is no space between the 2 and the > or >>
If filename doesn’t exist, a new file is created
If filename exists, the overwrite option will overwrite the existing file, and the append option will append to the file
A combination of redirection of input, output, and error can be done: command < input_file > output_file 2> err_file where the order of redirecting input, output, error is not important
To have output and error redirected to the same file: command > output_file 2>&1
In OS tradition, the input is assigned the value 0, output is assigned the value 1, and error is assigned the value 2.
So 2>&1 means the error (2) is redirected (>) to the address (&) of output (1)
Pipe
Instead of getting input / output from / to a file, you can also ask the shell to get the input/output from / to another command. This is done through a pipe with the symbol: |
Format: command1 | command2 | command3
command1, command2, command3 are different commands with options and arguments, if necessary
The output of command1 does not get printed on screen, instead it is sent to command2 and becomes the input of command2, therefore command2 does not need an input argument.
Likewise, the output of command2 becomes the input of command3 so command3 does not need an input argument
Rules for piping:
The command before the pipe symbol | needs to have output
The command after the pipe symbol | needs to accept input
You can pipe as many commands together as needed, as long as there are command output that can be used as input to another command
tee
So far we can tell the shell to send output to a file or to another command.
It is also possible to tell the shell to send the output to both a file and another command, by using a tee
It is called a tee because the shell essentially makes 2 copies of the output and pipes both through a T intersection. At the T intersection, one copy goes to the file and the other copy goes to another command
Format: command1 | tee filename | command2
command1, command2 are different commands with options and arguments, if necessary
The filename comes after the tee
The output of command1 is piped | into the tee, the first copy goes to filename, and the second copy goes to another pipe | to become the input to command2. command2 does not need an input argument.
If there is no piping into command2, then the second copy goes to standard output by default:
command1 | tee filename
Multiple Commands on a Command Line
Commands that work together
With piping, the command line contains multiple commands that work together
Piping can work with redirection and tee, allowing the shell to do complex coordination between the commands on the same command line
Examples:
command1 < inFile | command2 | command3 > outFile
The input of command1 is inFile, the output of command1 is the input of command2, the output of command2 is the input of command3, the output of command3 is redirected to outFile
command1 | tee outFile1 | command2 | tee outFile2
The output of command1 goes to outFile1 and is the input to command2, the output of command2 goes to outFile2 and to screen
Commands that are independent from each other
You can have multiple commands on the same command line, but the commands are independent from each other
Format: command1 ; command2 ; command3
command1, command2, command3 are different commands with options and arguments, if necessary.
The command line above runs the same way as if you type the 3 commands on 3 separate command lines
Most of the time, each independent command should be put on a separate command line. This causes the output of each command to immediately follow the command line, making it easier to distinguish the output.
Putting 2 or more independent commands on the same command line is useful when
the 2 commands complement each other and there aren’t that many output lines combined. For example: cd; pwd
The independent commands all have output or error that should be redirected together into 1 file
Command Grouping
When multiple, independent commands need to have all their output or all their error redirected to the same file, use command grouping
Format: ( command1; command2 ) > outputFile
command1, command2 are different commands with options and arguments, if necessary
The parentheses ( ) tell the shell to group the commands together
The shell runs command1 and then runs command2. The output of command1 is redirected to outputFile and the output of command2 is automatically appended to outputFile, right after the output of command1
You can group as many commands together as necessary
Preserving Command Arguments
Command arguments typically don’t have quotes around them
But single quotes or double quotes can be used around arguments
The majority of the time, quoting the arguments doesn’t affect how the command runs, so quoting is not used
There are some cases where quoting the argument is necessary
(1) To preserve white spaces in the argument:
without quotes, extra white spaces will be ignored by the shell and will appear as a single white space
you can use single or double quotes
(2) To preserve quotes in the argument
If the argument is a text string with quotes
use a pair of ‘ around the “ to preserve the “
use a pair of “ around the ‘ to preserve the ‘
use \ in front ‘ or “ to preserve both quotes
(3)To preserve shell special characters in an argument
Shell special characters, called metacharacters, cause the shell to interpret the character as special instruction from you
For example, > on the command line means redirection, and not a literal > symbol
Sometime an argument can contain a metacharacter, and you want to keep the literal meaning of that character
To tell the shell to keep the literal meaning of metacharacters in the argument: Use \ in front of each metacharacter
Metacharacters are:
& ; | * ? ~ ! $ ^ # / \ ‘ “ ` [ ] ( ) { } < > space
and newline (the enter key)
Difference between single and double quotes
In double quotes, the metacharacters $ and ! still have their special meaning, just like with no quotes
In single quotes, the characters $ and ! have their literal meaning
Generally, if you want all characters to keep their literal meaning, it’s safest to use single quotes around them
Command Substitution
Command substitution asks the shell to run a command and store the output in a string, which is then used as input to another command or to store in a variable
Format: command1 `command2` OR command1 $(command2)
The ` in the first format is the back quote, it is not a single quote
In the second format, there is no space between the $ and the (
command1, command2 are commands with options and arguments, if necessary
The output of command2 is stored as a string and then used as an input argument for command1
Another use: variable_name=`command` OR variable_name=$(command)
command can have options and arguments, if necessary
The output of command is stored as a string in the variable var_name (more about variables in the Variable slides later)