These are my notes from the Unix and Linux System Administration Handbook, which I strongly recommend. I primarily use Debian, so that's what I focus on here.
The Debian Reference is useful.
$ man 5 shadow $ apropos shadow
1: User-level stuff
2: Sys call, kernel errors
3: Lib calls
4: Device driver, network protocols
5: File formats
6: Games, demos
7: Other files, documents
8: Sys admin commands
9: Obscure kernel stuff
Man pages live in /usr/share/man/. The manpath command shows where man looks for man pages (which can be overridden with export MANPATH=/home/whatever/man:/usr/share/man).
Every process gets at least three channels: STDIN (file descriptor 0), STOUT (file descriptor 1), and STDERR (file descriptor 2).
The < operator tells a command's STDIN to get its input from a file (or file-like thing). The > operator writes the command/s STDOUT o a file (>> appends rather than overwrites).
$ ps ux > myfile.txt $ mysql -u root -p mydatabase < dbdump.sql
The >& operator sends STDOUT and STDERR to the same destination. The 2> operator redirects only STDERR (i.e.—file descriptor 2).
The pipe operator | connects the STDOUT of the first command to the STDIN of a second command.
ls | grep foo
The cut filter selects only certain fields from a delimited records. The default delimiter is TAB.
The uniq filter show only unique or non-unique lines, and can count them.
$ cut -d: -f7 /etc/passwd | sort | uniq -c 14 /bin/bash 10 /bin/false
-b Ignore leading whitespace
-f Case insensitive
-k Sort on column k
-n Sort as integers
-r Reverse order
-t Set delimiter (default is whitespace)
-u Output only unique lines
The tee filter sends its input to both STDOUT and a specified file (or file-like device as shown below).
$ find /home/me -name foo | tee /dev/tty | wc -l
Also: wc, head, tail, grep, and less.
Kernel creates spontaneous user-space processes
The kernel always creates 'init' as PID 1. ps shows these spontaneous processes with [brackets] around their names. If there's a slash and a number after the process name, the number indicates on which core the process is running.
The GRUB config file is /boot/grub/menu.1st. GRUB loads a kernel in /boot. Press [c] to enter command line mode from the GRUB boot screen. GRUB's command line lets you edit config files, pass arguments to the kernel, and load kernels that aren't listed in menu.1st. Press [TAB] for a list of command line options.
Highlight the desired kernel on the GRUB splash screen, and press [a]. Then:
grub append> ro root=LABEL=/ rhgb quiet single
init executes shell scripts in /etc/init.d/. init executes various scripts depending on the systems run level. /etc/inittab tells init what scripts to run for each level.
(Although, there's an additional layer of abstraction in that sym links in /etc/rc0.d/, /etc/rc1.d/, /etc/rc2.d/ &c actually get looked at to determine what runs. The link names start with "S" or "K" to determine what gets started or killed when entering that run level. Further, the sym links are named with a number that determines the order in which they run. These sym links mostly point back to files in /etc/init.d/.)
0: shutdown
1: single-user mode
2-5: normal multiuser mode (Debian's default run level is 2)
6: reboot
The telinit command forces a change in run level (e.g.—telinit 3). telinit q tells init to re-read /etc/inittab.
"There seem to be more run levels defined than are strictly necessary or useful. The traditional explaination for this is that a phone switch had 7 run levels, so it was thought that a UNIX system should have at least that many."
Hah!
Unix has several access control systems.
System calls have access controls. Only root can make certain system calls, for example, and those system calls themselves simply check if the caller is root. Other system calls, such as kill, make more complicated calculations about access, checking the caller's idenity, object owners, and making special allowances for root.
The filesystem has its own access controls. Files have an owner (mapped between username and UID in /etc/passwd) and a group owner (mapped between group name and GID in /etc/group). Groups and users may be recorded in a system like NIS or LDAP rather than in etc files.
Processes have access controls. Process owners can send their processes signals and change (reduce) their priority. Each process has several associated identities:
The superuser 'root' account can perform any valid operation on a process or file, and has the UID zero. Executables can be setuid or setgid to run with privileges other than those of the user who ran them (more elevated privileges). When a user changes his password, for example, the passwd command runs setuid so it has sufficient permission to write to /etc/shadow.
A few special accounts not associated with human users exist besides root. Accounts with UID's below 100 are typically for some particular software or daemon, and those below 10 are core system accounts. To prevent anyone from logging in under these accounts with a password or SSH key, the password field in /etc/shadow is replaced by a star (*) and the shell is set to '/bin/false'.
The previous section describes traditional unix security. Various enhanced security systems have been developed to address real and perceived inadequacies with traditional security, adding features such as auditing or role-based access controls.
SELinux, a distro developed by the NSA, added mandatory access control (MAC), which has since been integrated as an option in the mainline kernel. With MAC, all permission must be assigned only by admins; regular unprivileged users can't delegate their access.
Pluggable authentication modules (PAM) provide authentication rather than authorization. PAM lets authentication worth through modern cryptographic or biometric mechanisms, for example, rather than simply checking a password against /etc/shadow. It's an authentication framework/wrapper for various particular authentication methods/implementations.
Kerberos is one such authentication method. Kerberos client authenticate against a trusted server, which gives the client cryptographic credintials that the client can subsequently present to various services as proof of identity.
Access control lists (ACLs) elaborate on traditional filesystem permissions.
Unlike the "modern" access control embellishments described above, sudo is widely used as an enhancement to traditional unix security. Sudo lets users listed in /etc/sudoers execute commands with root privileges. Sudo logs the commands and who executed them. /etc/sudoers provides some fine-grained control, like letting a user only execute one particular command on a particular host with elevated privileges. Edit /etc/sudoers with visudo (which makes sure no one else is editing it at the time, and validates the entries).
A process is an abstraction for an allocated address space inside the kernel, which includes the program's code, library code it uses, storage for its variables, and information the kernel uses to keep track of the process:
nice and renice.A process can fork itself into threads, which execute on different processor cores, but share the same memory space.
A variety of process-level interrupt requests can be sent to notify a process of an event, kill the process, or act as inter-processes communication. A process can catch a signal if it has an appropriate handler; otherwise, the kernel takes a default action on behalf of the process.
The kill PID command can send any signal to a process, though by default it sends TERM (a catchable, blockable termination). kill -KILL PID terminates the process (uncatchable, unblockable). kill -INT PID is like KILL, but catchable and blockabe; this is the signal sent by typing CTRL-C in a terminal. kill -STOP PID suspends the process (uncatchable, unblockable) until it receives the CONT signal. TSTP is similar to STOP, but gives the process a chance to clean up its state first (catchable, blockable); this is the signal sent by hitting CTRL-Z in a terminal. Many daemons understand kill -HUP PID as a request that they re-read their config files without fully restarting; HUP may also get sent to processes attached to a disconnected terminal.
kill -l gives a complete list of signals.
Examine processes with ps. Two useful sets of flags are ps aux and ps lax. "a" is all processes, "u" formates the output for reading by a user, "x" includes even those without a control terminal, "l" is long output format.
See also top and htop. Note that top includes video memory in the memory used stats as well as memory used by shared libraries, so some processes seem to have an oversized memory footprint. Hitting "f" then "s" in top shows a data column that tends to me a more accurate indicator of memory usage for a given process.
Process information is also available under the /proc/ pseudo-filesystem, including these files of interest:
The strace -p PID command shows all the system calls a process makes. strace -c -p PID tallies all the system calls. strace -T -p PID shows the time spent on each system call.
The fuser and lsof commands help find which process is using a file.
There's also tons of good info via cat /proc/1234/status, where "1234" is the pid.
On unix, everything is a file (i.e.—exposed/represented as a file), including processes processes, devices, and kernel tuning parameters. Consider the filesystem as having four characteristics:
Although linux, like most unices, is basically a monolithic kernel (meaning virtual memory, device drivers, interprocess communication, etc. use the same chunk of reserved memory), it support drivers implemented as modules. Modules load without the need to recompile the kernel and reboot.
Modules get loaded in one of two ways. Either you explicitly tell the kernel to load it, or the kernel detects the device and loads the module on its own. The later happens at boot time or (in the case of many USB devices, for example) while the system is running.
Drivers provide an abstraction layer between the kernel and the hardware. Because drivers are compiled into the kernel or loaded into kernel-space as modules, user access to devices comes through /dev/ files, that map user file operations to driver calls.
Devices have a major number and a minor number:
$ ll /dev |grep sda brw-rw---- 1 root 8, 0 Apr 11 18:18 sda brw-rw---- 1 root 8, 1 Apr 11 18:18 sda1 brw-rw---- 1 root 8, 2 Apr 11 18:18 sda2
The major number ("8" for sda*) identifies the device driver (and, by implication, the type of device). The minor number (0, 1, 2) identifies a particular device from others of its type.
A device might have more than one associated /dev/ file, with each device file supporting different functions (on a sound card, for example, one device might correspond to playback, another to capture).
Traditionally, the admin would create device files like mknod fileName b(lock)|c(haracter) majorNumber minorNumber. Under linux, udevd listens for device status change messages from the kernel, and creates or removes /dev files as necessary.
Some /dev/ files are convenient abstraction that don't correspond to actual devices. The tty* devices are not actual terminals, for example. Other pseudo-devices include /dev/null and /dev/random.
Getting devices working (or working optimally) may require some kernel configuration in one of four ways (listed from least to most annoying): tuning kernel parameters in /proc/sys, loading modules into a running kernel, setting GRUB directives at boot time, or compiling a new kernel from scratch with any necessary patches.
Tune kernel parameters as follows. Note that these tunings do not persist after a reboot if you just cat a value into them. For persistent changes, use the sysctl command or edit /etc/sysctl.conf.
$ cat /proc/sys/dev/cdrom/autoeject 0 $ sudo sh -c "echo 1 > /proc/sys/dev/cdrom/autoeject" $ sudo sysctl dev.cdrom.autoeject=1
Loadable kernel modules let you load or remove a device driver (or whatever) from a running kernel. Loadable modules are in /lib/modules/. The lsmod command shows currently loaded modules. The insmod code inserts new modules into the running kernel, rmmod unloads a running module. The modprobe command may be preferable to insmod in some cases, as it understands a bit about module dendencies and options (using info from /etc/modprobe.d/). The modinfo show the parameters a module accepts; see also /etc/modules. Note that udev will automatically load many modules for you.
There are few reasons these days to compile a custom kernel, but the Debian way is:
# cd /usr/src/linux # make menuconfig # make-kpg clean # make-kpkg --initrd --revision=custom.1.0 kernel_image # dpkg -i ../linux-image-2.6.32-amd64.1.0_i386.deb # shutdown -r now
Udev is a user space device management system that watches for device installation and removal (by watching the virtual filesystem /sys/), and maintains device files in /dev/. The udev management tool is udevadm. Also see /etc/udev/
(/sys/ was designed as a better organized replacement for /proc/. /proc/ is slowly being reverted back to its original purpose of showing only process information.)
© Paul Gorman