01. Review of Common Linux Utilities

Overview

  • During this topic you will learn how to write shell scripts (programs)
  • The prerequisites to learn shell scripting are:
    • Introduction to Unix/Linux notes
      • Common Unix/Linux utilities
      • Shell interaction with the commands, such as pipe, redirection, job control
      • More advanced utilities such as egrep, sed, awk, find
    • Any Programming Language
      • Basic programming constructs, such as loops, if else, functions and argument passing
      • Basic data storage concept, such as variables, arrays, strings
      • Basic testing and debugging techniques

Working With The Shell

  • The shell is case sensitive
  • To quickly repeat a previous command:
    • Use the up arrow key to step back the list of commands you’ve run
    • Use the down arrow key to step forward the list of commands you’ve run
    • At the command you want to repeat, hit the enter key
  • Use tab to complete a long filename
  • Use control-u to quickly erase a long command line
  • If a command line is too long, use \ before the enter key. This escapes the newline and the shell will interpret both lines of text as one command line
  • The default secondary prompt is >
    • When you see the secondary prompt, it means the shell is expecting more input on the command line
  • When the shell interprets the command line, it uses 1 or more whitespaces as a delimiter:
    • The first word on the command line is interpreted to be a utility name or an executable file (a compiled binary file or a shell script)
    • All subsequent words preceded by – or + are interpreted as an option or list of options
    • All subsequent words without – or + are interpreted as arguments
    • If the arguments have wildcard characters, the shell expands the wildcards to match existing filenames
    • All options and arguments are passed to the executable without further error checking
    • If there are redirection, pipe, tee, or job control metacharacters, the shell takes the appropriate action with the input or output, and how it runs the executable

Files on Linux

  • A file can mean any of the 7 types of files on Linux, including the 3 types used in this notes: regular files, directories, links
  • Each file has a unique inode number to identify it to the system. Therefore, filenames that are hard linked together have the same inode number
  • Filenames should be descriptive
    • Filenames should not contain any of the following shell metacharacters :
      • & ; | * ? ~ ! $ ^ # / \ ‘ “ ` [ ] ( ) { } < > and space
      • (after this review note, you should be able to explain why each of the characters should not be used for filenames)
  • Wildcards can be used as arguments, in place of or in addition to filenames
    • Wildcard characters are: * ? [ ]

General Purpose Utilities

  • Change a password: passwd
  • Stop the current process (useful for any utility that doesn’t have a quit option, and useful for logging out of the system): exit , control-d
  • Abort (kill) the current process (useful ‘emergency brake’ for cases such as an infinite loop that keeps printing out to screen in a foreground process): control-c
  • Information on users on the system: who, whoami, w, finger
  • Information about your system and your terminal: tty, stty, uname, clear
  • Recording all screen output to a file: script
  • Print a hard copy: lpr
  • Get more information on a utility: man, info
  • Get current date / time: date, cal
  • Print a line of text to screen: echo
  • Spawn a new shell: sh, bash, tcsh, ksh, …

File Utilities

  • Show content of a file: cat, more, less, head, tail
  • List a directory: ls
  • Commonly used options: a, l, F, d, i
  • Create a new file or update the time stamp of an existing file: touch
  • Copy a file: cp
  • Rename or move a file: mv
  • Delete a file: rm
  • To change the access rights of a file: chmod
  • To change default access rights of files: umask
  • To recursively find files that match certain criteria: find
    • To do work on files that match the criteria, use the –exec option of find or the xargs utility
  • To archive multiple files into one archive file: tar
  • To compress a file: gzip, gunzip

Directory Utilities and Link Utilities

  • There are 2 types of directory paths:
    • absolute path (starts at root)
    • relative path (starts from the current directory)
  • Some special directory symbols: . .. ~ ~userID
  • Show current directory: pwd
  • Change directory: cd
  • Create a new directory: mkdir
  • Show the location of a utility: whereis
  • Show the actual utility that runs when typing a command name: which
  • There are 2 types of links:
    • Hard link (actual location of file on hard disk)
    • Symbolic link (logical path to the file location)
  • To create a link: ln

vim

  • Native text editor of Linux
  • 2 modes: command mode to issue command, and insert mode to type in text
    • To go from insert mode to command mode: escape
  • vim is a useful utility for this class since your shell scripts are created by using vim
  • For the voyager system vim has been aliased to vi

Basic commands:

  • To move: arrow keys, h, j, k, l, 0, $, control-f, control-b, G, nG (where n is a number)
  • To add text: i, I, a, A, o, O
  • To delete text: x, dw, dd, D
  • To replace text: r, cw, cc
  • To find and substitute: :s
  • To search: / or ?, then n or N
  • To copy and paste: yw or yy, then p or P
  • To cut and paste: dw or dd, then p or P
  • To join 2 lines: J
  • To repeat the previous command: .
  • To run a command n times: n<command>
  • To undo: u
  • To bring in another file: :r
  • To save: :w
  • To quit: :q

mail

  • Native email utility of Linux
  • 2 modes: send mode and receive mode
  • The mail utility is useful for this class because you will receive feedback of your lab assignment through voyager mail
  • Send mode
    • Command to get into send mode: mail email_addr
    • Command to quit send mode:
      • control-d send
      • control-c exit without send
    • Commands in send mode:
      • ~v use vi to edit the message
      • ~m n n is a number, bring in an existing message n
      • ~r filename bring in a text file with name filename
  • Receive mode
    • Command to get into receive mode: mail
    • Commands in receive mode
      • (n is the optional message number, if there is no message number, the current message is the default message)
      • n enter Show message n
      • R n reply to message n
      • d n delete message n
      • u n undo delete of message n
      • s n file save message n to file
      • q quit out of mail

Features of the Shell

  • Standard input comes the keyboard, standard output and error go to screen
  • To redirect input, output, error from/to a file, use: < (from file) and > or >> (to file)
  • To send output of one command to the input of a second command, use the pipe: |
  • To send the output of one command to both a file and another command, use a tee: tee
  • Multiple commands on a command line can be grouped with ( ) or be separated with ;
  • To get the literal meaning of shell metacharacters, use single quotes ‘ ’, double quotes “ ”, or back slash \
  • To do command substitution, use back single quotes `command` or use $(command)
  • To run a job in the background, add & at the end of the command line
  • A job or process can be in 1 of 4 states: running in foreground, stopped, running in background, or terminated
    • To switch from foreground to background: control-z, then bg
    • To switch from background to foreground: fg
    • To stop a job running in foreground: control-z
    • To terminate a job: control-c (job in foreground), kill (job in background)
  • To customize a command: alias
  • To create a variable for temporary data storage: varName=value
  • To access variable: $varName
  • To see environment variables: env

Filters

  • To see content of a file: more, less, head, tail, cut, paste
  • To filter consecutive identical lines: uniq
  • To count lines, words, characters in a file: wc
  • To see difference in 2 files: diff, cmp
  • To change characters in a file: tr
  • To sort lines of a file: sort
  • To select certain lines of a file: fgrep, grep, egrep
  • To modify manipulate input text lines: sed , awk
    • sed is useful when you want to reformat parts of a line
    • awk is useful when you need to manipulate fields of a line by using programming constructs (such as loops, if statements, arrays…
      • Shell variables can be passed to awk by using the –v option for each variable you need to pass to awk:
        • num=5 # set shell variable num to 5
        • awk –v n=$num ‘{ print n }’ fileA
        • # set awk variable n to the same value as num
        • # awk will print 5 for each line of fileA

Regular Expression

  • Used for pattern matching by certain utilities
  • The common ones are: egrep, sed, awk, and vim substitute
  • Most characters in a regex have literal meaning, but some are metacharacters, as shown:
    • [ ] any one character within the [ ]
    • [^ ] any one character not within the [ ]
    • . any one character, except newline
    • ^ begin of line anchor
    • $ end of line anchor
    • {n} n of the previous character
    • {n,m} n to m of the previous character
    • ? 0 or 1 of the previous character
    • + 1 or more of the previous character
    • * 0 or more of the previous character
    • | alternate (or)
    • ( ) group
    • \ take literal meaning of the next character