NET 138 notes taken by Cat Nilan, Fall 2001 edited/extended by rlh, Winter 2002 [This class was taught as a combined CIS136/138 for a dozen students at Seattle Central Community College. Note that unit numbers given here for assigned readings refer to the unixoutline doc found in /share/138rh and on the back of the syllabus. - rlh] 10/01/01 Logging in to Sextant IP address: 168.156.125.36 username: [first initial + 7 letters of last name] password: [5-8 chars, non-dictionary strings, use non-alpha chars too ] Commands, options, etc. more -> "| more" for one page at a time less -> page backward as well as forward cat -> view file q = quit ls -> list rm -> remove (delete) file use -i switch for confirmation Permissions r (read) = 4 w (write) = 2 x (execute) = 1 chmod = change permissions chgrp = change group permissions chown = change owner permissions (only root can change owner) ex. -> chmod 777 filename (gives full rwx to all) u (user) g (group) o (other) a (all) chmod u+x filename = give user execute permissions Oct 03, 2001 Commands we know: echo = display ls = list cd = change directory man = manual pages finger = user information lookup who = who is logged on w = who is logged on and what they are doing who am i = user information chmod = change permissions mode chgrp = change group permissions chown = change owner permissions cat = concatenate files and show as stdio more = display one screen at a time less = display one screen at a time (forward and backward) date = print/set system date and time cp = copy mv = move (change file name) mkdir = make directory rmdir = remove directory rm = remove file/directory -r = recursively remove directory trees -i = prompt for confirmation ("interactive") df = report filesystem disk space usage du = estimate file space usage free = display free/used memory dmesg = (print kernel ring buffer -- bootup messages) q = quit (more, etc.) | = pipe > filename = redirect stdout to ... (creates or overwrites) >> filename = redirect and append < filename = redirect stdout from ... (input from file) 0 = stdin (keyboard) 1 = stdout (screen) 2 = standard error [stderr] (screen) Save first 100 lines dmesg | head -100 > dmesg.saved Directory listings / = directory * = executable ("x" permission set) @ = link -> = link Links soft link = symbolic link ln -s [destination directory] [target name] ex. "ln -s /share/138rh class" hard link "ln" File Table = links files to addresses In Linux, addresses are "inodes" Network file system Root = "/" /bin /home = "~" /usr /var fstab = static information about the filesystems available mtab = list of filesystems now mounted mount = mount a file system, or w/o args - see what's mounted Oct 05, 2001 On share => sales Filtering and Sorting (through sections 10 & 11) Next week: sxn 12 Next Friday (Oct. 12): 50 point quiz REVIEW: Commands that we should know cd => change directory cd => home cd ~ => home cd / => root cd .. => one directory up To return to a previously entered command => up arrow ls => list contents of directory ls -l => "long" listing (shows permissions, etc.) a "d" in front of the permissions listings indicates directory ex. "drw-r--r--" add [filename] to list single file ls -a => "all" (including hidden files) ls -i => show inode information mv [sourcefilename] [destinationfilename] mv => "move" (copy source to destination) copies file to new filename and erases the original file can be used to rename a file takes less time than "cp" m -i => ask for confirmation cp [sourcefilename] [newfilename] makes a new copy of a file cp => "copy" copied file becomes user's property and permissions can be changed rm [filename] deletes a file rm => delete file rm -i => ask for confirmation before deleting mkdir, rmdir mkdir [directoryname] => make directory rmdir [directoryname] => remove an *empty* directory rm -r => remove a directory that has files in it *be careful with this command* hard links ln [sourcefilename] [targetname] ex. "ln myfile backupmyfile" to view file more [filename] => show file page by page cat [filename] | more => (same) 4 major shell programming utilities: grep awk sed find => grep, egrep, fgrep searches for patterns and returns lines egrep and fgrep do same thing cat [filename] | grep '[string]' ex. => cat sales | grep 'CA' sort cat sales | sort -k2 | more sort by column awk cat sales | awk '{print $2 " -- " $1}' | more switches column 1 and 2 and displays them separated by hyphens add " > [filename]" to end to create script file Writing shell script (in editor) To display the second and first columns (in that order) of sales "cat $1 | awk '{print $2 " -- " $1}' > $2" save as "swap" and make executable run as: swap sales file3 $0 $1 $2 read as: "cat sales | awk '{print [secondcolumn] " -- " [firstcolumn]}' > file3" => find tr uniq Shadow file contains password Only root can access and it will be encrypted Oct 08, 2001 "clear" = clear screen Looking at large log files using pattern searches system log file in: \bin\messages Used by sysadmin to troubleshoot, etc. example: grep 'signal 15' < message > m15 [search for string ('signal 15') in file named 'message' and output to file named 'm15'] grep '[string]' < '[search in filename]' [ > '[write to filename]' ] "Halve" Script #!/bin/sh FILE=$1 # declare local variable (first argument) SIZE=`cat $FILE | wc -l` # determines size of file and puts it in variable # "`" interpreted as a command and returns value to variable HALF=`echo $SIZE/2 | bc` # "$" equals "value of" # "bc" is a calculator # sets HALF = to half of SIZE echo "Total size = $SIZE Half = $HALF" # displays value of variables tail -${HALF} $FILE > $FILE.end # Second half of FILE sent to [filename] head -${HALF} $FILE >$FILE.top # Top half of FILE sent to [filename] ls -l $FILE $FILE.top $FILE.end # list original file and two new files Script is run by typing: "halve [filename]" wc = word count [filename1] [filename2] [filenameN] Displays lines, words, and characters/bytes for file by default -l = display line count only -w = display word count only -c = display characters/bytes only Exercise using share file "msgs.top" [ will make available using "chmod 660 msgs.top" makes this file readable/writable for all of us] -- do a "grep" string search on this file and read results to a new file -- then check for line count and append that information to the new file -- then mail to Rhodes grep '[search string]' < msgs.top > file2 cat file2 | wc -l >> file2 mail rhodes < file2 Writing a script to do this: #!/bin/sh #A first shell script: # get [string] [infile] [outfile] #Declare variables pattern=$1 #the string to search for infile=$2 #the file to search in outfile=$3 #the file to write results to #Test to make sure that the correct number of #parameters have been entered if [ $# -eq 3 ] ; then #Use grep to look for the pattern in the input #file and write results to output file grep $1 < $2 > $3 #Count the number of lines found and append #this result to the output file echo "Number = `cat $3 | wc -l`" >> $3 #Mail the results to Prof. Rhodes #(But DON'T do this until you've finished debugging #this script -- until then, comment out) # mail rhodes < $3 #If incorrect number of parameters entered, output error message else echo "Usage: get [pattern] [input file] [output file]" fi use the file "classgen" as a model ("classgen" is in the share directory) To access msgs.top absolute path: /share/138rh/msgs.top To apply the "get" script to msgs.top, enter the following command: get [a string of your choice] /share/138rh/msgs.top [name of output file in YOUR home directory (and not in share)] 10/10/01 QUIZ: Monday, October 15 *** Using grep to sort contents of files Locate pattern, read lines with pattern into 1 file, rest of lines into a second file Removing NIMDA virus lines from log file pattern = 'host name lookup failure' *** SCRIPT #!/bin/sh #shell directive line #Search/split /var/log/messages to file to dump "nimda" trash #This script must be run by root #Set variables 1 junk=msgs.nimda #filename for nimda-related lines 2 infile=/var/log/messages #source files 3 outfile=msgs #filename for non-nimda lines #Strip out nimda messages, write to 'msgs.nimda' 4 grep 'host name lookup failure' <$infile > $junk --- OR --- cat $infile | grep 'host name lookup failure' > $junk #Extract the remainder to 'msgs' 5 grep -v 'host name lookup failure' > $outfile # -v = *exclude* this pattern --- OR --- cat $infile | grep -v 'host name lookup failure' > $outfile #Copy 'msgs' to /var/log/messages 6 cp msgs /var/log/messages *** How to test if user is root? [This code *didn't* work] if [ $EUID -eq 0 ] ; then ... else ... fi error-message: [ : 0 : unknown operand ] *** [#!/usr/bin/perl => perl script] *** X-Windows Linux GUIs KDE Gnome Word processing, etc. Staroffice [office suite] Wordperfect [word processing] Netscape [browser] Oracle [SQL database] Command line software Joe [text editor] *** System admin with 600 users What sort of problems would you have? User education: forgotten passwords, e-mail doesn't work Wayward users Viruses Server management: clean up junk on drives Backup Missing files Firewall, security Configuring new applications Sharing printers, disk space, bandwidth (CPU time management, RAM, online idle time Useful admin commands w => who is logged on and what they are doing du => estimate file space usage -a => show all files -b => show in bytes -c => total *** ASSIGNMENT xsdspec => find users of excess disk space "xsdspec" is in share directory Copy it to your home directory File consists of comments Assignment is to write the script suggested by the commentsClass Notes 10/12/01 GREP Searches for a pattern, returns lines Variable declaration is *implicit* Use the variable and it exists To access the value of the variable, use "$var" AWK Rearranges columns (symbol-separated words) space is default delimiter "-F:" => delimiter is ":" awk '{ print $n $n }' Will print the nth and nth "columns" of each line in a file When writing scripts use "\" escape (for line breaks) ";" allows you to put multiple lines on a single line (concatenation) sed -d deletes a line Delete a single line from a file and write to a new file cat [file] | sed ld > tmpusrlist mv tmpusrlst userlist "while read" reads one line at a time while read [variable name] touch [filename] Creates/updates a file with current date stamp mkdir work_dir 2> /dev/null SED FIND October 17, 2001 MAIL utilities Mail Mailx Pine Elm To look at man page for "mail", go to "mailx" .forward: to forward mail from one address to another Create .forward file in pico (goes in home directory) one address per line new mail goes to /var/spool/mail read mail goes in mbox [Review of QUIZ] October 19, 2001 Make homework available in a new directory cd ~ mkdir cw cd /home/rhodes/exer cp ex?-[yourusername] ~cw cp mmt-[yourusername] ~cw cd ~ This creates a new directory in your home directory and copies the homework you've turned in to that directory Make your home directory readable and executable by group cd /home/ chmod g+rx [yourusername] [Review of user list script] October 22, 2001 Assignments posted in the "cw" directory should be named as follows Exercises ex1-[username] ex2-[username] etc. Mini midterm mmt-[username] Working through the disk usage report script clear October 24, 2001 Pattern Recognition | Regular Expressions [See p. 552 in Linux in a Nutshell] New homework assignment sent via mail Using "sed" to do substitutions "sed" = "stream editor" Syntax: sed 's/pattern1/pattern2/g' < filename1 > filename2 "s" => substitution "pattern1" => string to be replaced (can be a regular expression) "pattern2" => replacement string "g" => global (replace all occurrences in the line being worked on) "filename1" => file to read from "filename2" => file to read to (if not included, output to screen) Examples: sed 's/cat/dog/g' < feline.txt > canine.txt replaces all occurrences of "cat" in feline.txt with "dog" in a new file named canine.txt sed 's/[0-9]/N/g' < newfile > n3 [0-9] => match any number (a regular expression) [use escape sequence to include a literal "/" => to search for the string "TCP/IP" enter "/TCP\/IP/"] Parameters and Variables "set" => shows shell variables Will show variables set in local script "env" => shows environmental variables to export variables to all shells => "export [variablename]" .profile file #!/bin/bash # Set primary prompt to be host and path. PS1="`hostname` \w " set -b biff n # Set aliases alias bye=exit alias cl=clear alias ll='ls -l' alias rm='rm -i' October 26, 2001 Review of Permissions read write execute Creating a "secret" directory Deny access to an upper directory, but give permissions in a lower directory so that those who know about that directory can access it For class => r and x permissions to your home directory Protect individual files or directories by limiting permissions To deny access to every but yourself: "chmod 700 [directoryname | filename]" Shared with group (read and write) "chmod 750 [directoryname | filename]" HTML => files should be read only (not executable) System Status and Processes ps => processes running for user ps switches (do NOT require hyphen) a => list all users x => daemons as well (a process not associated with a person) ("daemon" => services running in the background [ Possible exam questions: How do you tell a daemon from a regular user? One or more possibilities: -- Has no username -- Name or home ends in "d" -- user id less than 100 -- Has no home (except FTP) -- Has no shell ] u => show user f => "forest" (family tree format) w => who is logged in *and* what they are doing (with CPU usage and what is running) -h => suppress heading and uptime info time notation HH:MM (indicated with "m") MM:SS S.SS (indicated with "s") top => dynamic ps (updates every 5 seconds; shows highest usage first) man top => "ongoing look at processor activity in real time" last => last logins by user or terminal; shows logins since system last came up uptime => how long the system has been running who => who is logged in finger => more information (including real name) whoami => your user name kill => terminate a process kill -9 => kill anything Scheduling tasks at TIME => scheduling a single task at a specified time crontab file => a script that the cron daemon reads and executes at designated times STDOUT to mail (send to dev/null if you don't want mail confirmation) time codes in "man crontab" /var/spool/cron => home of crontab files crontab -e => to create/edit a crontab file one file per user Job Control (Linux in a Nutshell, pp. 498-99) Send tasks to the background while they are running (using job numbers or process id #) jobs => list all running or stopped jobs Process codes S => suspended Z => zombies (should be dead, but it isn't) Variables in the Shell env => displays the current environment (linux) set => *all* variables known to the current shell (bash) October 29, 2001 Cleaning up directories Web server directories Address of the Net 138/140 homepage => http://168.156.125.36 soft link in top level links to file in a lower-level directory /var/lib/httpd/htdocs/index.html (the main homepage) If you have a gateway page linked from the main homepage, there will be a softlink to the file in your home directory => "[yournamehere].html" cnilan.html links to => ~/cnilan/cnilan.html http://168.156.125.36/~[yourusername] => /home/cnilan/public_html/index.html to create your own gateway page, go to /shr/yournamehere.html rename this file "[yourusername].html" and place it in your home directory Edit it with pico, replacing "rhodes" with your name User of browser => identified as "nobody" October 31, 2001 Reviewing the XSD Script New/Useful commands and variables in this script $LOGNAME = current users login name sed -e = edit s = substitute g = global (all occurrences) grep -v = print all lines that *don't* match the pattern ... awk '{seconds += $1 * 60}; END {print seconds}' >> secs for every line in the file, multiply the 1st column by 60 and add that value to seconds (for a cumulative total), and then output results to "secs" file tee -a onlist = output to standard out and "onlist" file mail, mailx mail is interactive mailx isn't (used in scripts) UNIT 20 Reading: sp 159-174, ln: if, while, until, for, case, select Program Flow Control 1. If, then, else -> Bi-directional switch if (test) then (commands) [elif] (commands) [else] (commands) fi 2. While -> operate until test is false while (test) do (commands) done 3. Until -> operate until test is true until (test) do (commands) done 4. For -> Do an operation on a set or array for (name) [in (word1, word2 ...)] do (commands) done 5. Case -> Provide distinct operations for several patterns case (word) in pattern1) [commands];; pattern2) [commands];; esac 6. Select -> generates a menu of user choices select (name) in (word 1, word2 ...) do [commands] done Reading for Friday: units 12, 19, 20 November 2, 2001 In /shr/138rh/reference directory reference files "catcmdref" => a compilation of commands learned by students in a previous class In /var/lib/httpd directory Web server directory (Apache) "httpd" => HTTP daemon htdocs => hypertext documents conf => configuration files access.conf => global access configuration controls access to server, etc. Set of notes from North class Can be ordered for $6 Directory /var/lib/httpd/cgi-bin FIND Syntax: find [pathnames] [conditions] find $2 -type f -exec grep -l $1 {} 1>> $1.file 2> /dev/null \; & $2 => directory to search -type f => type is file -exec => starts subshell (why?) grep -l => search for file name containing $1 => pattern output to file dump error messages \; => end grep? & => run in background To search man pages "/[pattern]" "n" => next screen Search for "read" and send results to file find / -name read 1>> findfile 2> /dev/null & 19: Devices df => "disk free" shows how much space is available du => "disk usage" fstab => file system table mount => displays the name of each mounted device umount => unmount (only user can do this) free => display statistics about memory usage November 05, 2001 MIDTERM: Distributed Friday, due on Monday (11/12/01) Review this week FOR LOOPS for x [in list] do commands done Example: for x in j k l m n o p q r s t do touch file-$x done This will create a set of files name "file-j", "file-k", etc. What should we do with the rest of the semester? Review for exam Working with variables When files (exec), when command lines Writing scripts Loops, if, etc. Grep Sed Awk Permissions Links Quoting QUOTING `` [backticks] => To "fork" a process Forces a new shell to run and then causes it to end We start our sessions in BASH In a script, #!/BIN/SH forces the opening of the Bourne Shell (sh) A new instantiation of sh runs when the scripts hits a backtick and ends when it hits a closing backtick [This also happens in loops, after the test expression] [Why use sh? => Uses less memory; is available on all systems; extra features provided by bash or other more complex shells aren't necessary in scripting] '' [single quotes] => Protect the white space Makes all characters literal; will NOT expand variables "" [double quotes] => Protect the whitespace and expand all variables Takes away the special meaning of all characters except $, backquotes, and these escape sequences: \$, \`, \", \\ PERMISSIONS chmod => to change permissions _ _ _ _ _ _ _ _ _ _ First column: _ (file) d (directory) l (link) c (character special files) b (block special files) First triplet: owner or user [u] Second triplet: group [g] Third triplet: other or all [o] Possible permissions settings rwx => read, write, execute 421 => read, write, execute r w x r w x r w w _ _ _ _ _ _ _ _ _ _ 4 2 1 4 2 1 4 2 1 _ _ _ _ _ _ _ _ _ _ chmod 777 [filename] => gives full permissions to everyone chmod 755 [filename] => all permissions to owner, read and write to users and others chmod a+rx [filename] => gives everyone read and execute permission Directories need to be executable in order to be viewed (files within the directory can be given different permissions) LINKS hard links => "ln [originalfilename] [newfilename]" Absolute Not a file, but a name associated with a disk address (inode) Hard links only work *within* a drive inodes => Every file has an inode inodes contain file administration data, including disk address of the file (and file size, permissions, timestamps) Multiple names can refer to the same inode address When *all* names for a file are deleted, the contents of the file are no longer available soft links => "ln -s [originalfilename] [newfilename]" Symbolic A file that points to another file (contains a path name) Can be used across a network (LAN, etc.) Given full permissions (permissions set at destination) REMINDER: & [at end] => Run in background cd +, cd - : go forward and backward between directories November 16, 2001 How to recognize a daemon? Exists as a file (executable) in /sbin Login != real name Usernumber is under 100 (usually) There is no shell (usually) No home in "/home" (usually) x in password column means the password is in the shadow file * in front of password means that account has been disabled "Nobody" is a guest account used by anyone accessing files via a web-browser "Nobody" has no fixed identity Guest => ftp guest account Generic accounts Nobody => http guest => ftp Guest => anyone Student => special guest account USERS GID = User's group 100 or 500 Real name = login name A home in /home A shell (std) bash, linux, korn, unix Uid 100/500 Does not exist in /sbin Adduser login name: user id: initial group: additional groups: home directory: shell: account expiry date: Account created and then user info is requested Create a "new" password November 21, 2001 A script that creates a backup of the root directory on a separate partition #!/bin/sh # Copy all dirs from root to a backup partition. # expects the target dirname w/o the leading slash # rlh 20011118 bak=$1 cd / ls -1 | grep -v $bak > t0 for dir in \ # Don't include these because they're empty (or devices) auto cd1 cd2 cdrom dos fda fdb part1 part4 zipdrv initrd lost+found proc do cat t0 | grep -v $dir > t1 mv t1 t0 done echo "Is this what you want to copy to /$bak? (yes/no)" cat t0 # Determine how the user has replied read reply if [$reply = yes] then rm -f /$bak/mirror-errs while read item do cp -Rf $item /$bak 2>> /$bak/mirror-errs done < t0 rm t0 mv /$bak/t0 /$bak/mirror-dirlist echo "Job done." else echo "Job aborted." November 26, 2001 FILE CONVERSION Exercise 4: XML file containing grocery list Extract an organized list of fields with all of inventory data minus XML Four Parts to assignment Specification: Write something that interprets what your boss has asked you for -- What is the goal? -- What data is needed? Algorithm (design) as comments Write Code (shell scripting) Results => input file (provided), output file DUE: Monday, December 3rd, 2001 Format: Name, Size, Unit Price, Price, Number, Package Price, Savings output as csv file. http://www.smsys.com/pub/lnxnotes/cweb.html C compilers, etc. November 28, 2001 Internet FTP and Telnet at beginning TCP => Transfer Control Protocol UDP => User Data Protocol ICMP => Internet Control Message Protocol FTP => File Transfer Protocol SMTP => Simple Mail Transfer Protocol Telnet => (remote shell access) HTTP => Hyper Text Transfer Protocol