DigitalOcean FreeBSD Runbook ============================= Note: to access the DigitalOcean web console, open outbound tcp port 5000. Create a droplet through the DO web tool. Add our ssh keys. Log in via ssh, using the user name 'freebsd': % ssh -l freebsd nnn.nnn.nnn.nnn Add a new user (and make them part of the wheel group): % sudo adduser (sudo is install by default, and wheel are sudoers.) Add ssh keys to new user account: % sudo mkdir /home/myuser/.ssh % sudo cp ~/.ssh/authorized_keys /home/myuser/.ssh/ % sudo chown myuser:myuser /home/myuser/.ssh % sudo chown myuser:myuser /home/myuser/.ssh/authorized_keys Test to sure we can ssh in and sudo with this account. Generate ssh keys for this account: % ssh-keygen -b 4096 Add a minimal .tcshrc for the new account: set promptchars="%#" set prompt = "--- %m %h %c %# " setenv EDITOR vim setenv VISUAL vim setenv PAGER less set autolist alias l 'ls -FG' alias la 'ls -FGa' alias ll 'ls -FGlha' alias h 'history 60 | sort -k2 | uniq -f2 | sort -bn' setenv LSCOLORS "gxfxcxdxbxxggdabagacad" bindkey "^W" backward-delete-word Install updates: % sudo freebsd-update fetch % sudo freebsd-update install % sudo pkg update % sudo pkg upgrade Install out some packages: % sudo pkg install vim-lite git-lite zsh sshguard-ipfw curl tmux alpine python ezjail haproxy wget Edit /etc/rc.conf: hostname="inky.example.com" gateway_enable="YES" sshd_enable="YES" sshguard_enable="YES" sshguard_safety_thresh="30" sshguard_pardon_min_interval="3600" sshguard_prescribe_interval="7200" ntpd_enable="YES" ntpd_sync_on_start="YES" cloned_interfaces="lo1" ifconfig_lo1="inet 172.16.0.1/24" firewall_enable="YES" firewall_script="/etc/ipfw.rules" firewall_logging="YES" firewall_nat_enable="YES" firewall_nat_interface="vtnet0" haproxy_enable="YES" ezjail_enable="YES" Edit /etc/sysctl.conf: net.inet.ip.fw.verbose_limit=5 Edit /etc/ipfw.rules: ipfw -q -f flush add="ipfw -q add" oif="vtnet0" jails="lo1" natout="9000" tcpout="https,http,domain,ssh,ntp,43,67,68" udpout="domain,ntp,67,68" ipfw -q nat 1 config if $oif reset unreg_only same_ports $add 10 allow all from any to any via lo0 $add 20 allow all from any to any via $jails $add 100 deny all from any to 127.0.0.0/8 $add 110 deny ip from 127.0.0.0/8 to any $add 120 deny all from any to ::1 $add 130 deny all from ::1 to any $add 500 deny all from any to any not antispoof in $add 510 deny all from any to 10.0.0.0/8 via $oif $add 520 deny all from any to 172.16.0.0/12 via $oif $add 530 deny all from any to 192.168.0.0/16 via $oif $add 540 deny all from any to 0.0.0.0/8 via $oif $add 550 deny all from any to 169.254.0.0/16 via $oif $add 560 deny all from any to 224.0.0.0/4 via $oif $add 570 deny all from any to 240.0.0.0/4 via $oif # Work main office connection: $add 990 allow tcp from NNN.N.NNN.NNN to me ssh,https setup keep-state # sshguard (dyamic rules are added to table 22): $add 1000 deny ip from table\(22\) to any $add 2000 allow tcp from any to me ssh setup keep-state $add 2100 allow tcp from any to me http,https setup keep-state $add 5000 nat 1 all from any to any in recv $oif $add 5001 check-state $add 6000 skipto $natout tcp from any to any $tcpout out xmit $oif keep-state $add 6100 skipto $natout udp from any to any $udpout out xmit $oif keep-state $add 8999 deny log all from any to any $add 9000 nat 1 all from any to any out xmit $oif $add 9001 allow all from any to any Set the timezone: % sudo tzsetup Select "No" for "Is this machine's CMOS clock set to UTC?". Disable "freebsd" user account: % sudo pw lock freebsd Edit /etc/ssh/sshd_config: PermitRootLogin no Edit /etc/aliases to send system mail to an account we'll read: root: myuser ...and then rebuild the aliases: % sudo newaliases Reboot. Set up jails. Create the ezjail base: # wget http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.2/FreeBSD-10.2-RELEASE-amd64-disc1.iso # mdconfig -a -t vnode -f FreeBSD-10.2-RELEASE-amd64-disc1.iso # mount -t cd9660 /dev/md0 /mnt # ezjail-admin install -h file://mnt/usr/freebsd-dist # umount /mnt # mdconfig -d -u 0 Update base and port for jail base: # ezjail-admin update -u # ezjail-admin update -P Create a first jail for MySQL/MariaDB: # ezjail-admin create mysql.example.com 'lo1|172.16.0.2' # ezjail-admin start mysql.example.com # ezjail-admin console mysql.example.com (It's important that each jail has a FQDN, or weird stuff will go wrong, like local mail delivery.) Basic jail config: root@mysql:~ # echo 'nameserver 8.8.8.8' > /etc/resolv.conf root@mysql:~ # echo 'nameserver 8.8.4.4' >> /etc/resolv.conf root@mysql:~ # tzsetup And set the root password for the jail (`passwd`). # ezjail-admin console mysql root@mysql:~ # pkg install mariadb100-server mariadb100-client root@mysql:~ # echo 'mysql_enable="YES"' >> /etc/rc.conf root@mysql:~ # service mysql-server start Set up a jail for our default website: # ezjail-admin create www.example.org 'lo1|172.16.0.3' # ezjail-admin start www.example.org Set the nameservers, timezone, and root password. root@www:~ # pkg install php56 php56-extensions php56-mysqli php56-mbstring php56-mcrypt nginx vim-lite root@www:~ # echo 'nginx_enable="YES"' >> /etc/rc.conf root@www:~ # echo 'php_fpm_enable="YES"' >> /etc/rc.conf In /usr/local/etc/php-fpm.conf: pm.start_servers = 1 pm.min_spare_servers = 1 pm.max_spare_servers = 2 And: root@www:~ # service php-fpm start root@www:~ # service nginx start Create SSL certs on host (outside jails) # openssl genrsa -rand -genkey -out /etc/ssl/cert.key 2048 # mkdir /etc/ssl/haproxy # openssl req -new -x509 -days 2190 -key /etc/ssl/cert.key -out /etc/ssl/www.example.org.crt -sha256 # openssl req -new -x509 -days 2190 -key /etc/ssl/cert.key -out /etc/ssl/www.example.com.crt -sha256 # cat www.example.com.crt /etc/ssl/cert.key > /etc/ssl/haproxy/www.example.com.pem # cat www.example.org.crt /etc/ssl/cert.key > /etc/ssl/haproxy/www.example.org.pem (We *must* supply the correct FQDN for cert generation when asked by openssl.) Edit /usr/local/etc/nginx/nginx.conf: - location / { - root /usr/local/www/nginx; - index index.html index.htm; - } + root /usr/local/www/example.org; + index index.html index.htm index.php; + autoindex on; + location ~ \.php$ { + include fastcgi_params; + fastcgi_pass 127.0.0.1:9000; + fastcgi_index index.php; + fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; + } And actually do `mkdir /usr/local/www/example.com`. Further nginx considerations: - No Apache-like .htaccess - For basic auth, see: https://www.nginx.com/resources/admin-guide/restricting-access/ - The order of nginx.conf entries matters. E.g., it stops processing at the *first* matching location with a matching regex. - We forbid access to sensitive files in /usr/local/etc/nginx/nginx.conf: - We do only http (80) with nginx. haproxy handles https (443). location = /config.php { deny all; } Do the same thing to set up the second site: www.example.com. Outside the jail, edit /usr/local/etc/haproxy.conf: global daemon maxconn 1024 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http bind 162.243.53.217:80 bind 162.243.53.217:80 ssl crt /etc/ssl/haproxy/ default_backend test.example.org acl test.example.org hdr(host) -i test.example.org acl test.example.com hdr(host) -i test.example.com use_backend test.example.org if test.example.org use_backend test.example.com if test.example.com backend test.example.org server test.example.org 172.16.0.3:80 backend test.example.com server test.example.com 172.16.0.4:80 Create databases and users. # ezjail-admin console mysql root@mysql:~ # mysql -uroot MariaDB [(none)]> CREATE DATABASE blog; MariaDB [(none)]> CREATE DATABASE wp; MariaDB [(none)]> CREATE DATABASE ttrss; MariaDB [(none)]> CREATE USER 'blog'@'localhost' IDENTIFIED BY '****************'; MariaDB [(none)]> GRANT SELECT, INSERT, UPDATE, DELETE ON blog.* TO 'blog'@'172.16.0.3'; MariaDB [(none)]> CREATE USER 'wp'@'172.16.0.4' IDENTIFIED BY '********************'; MariaDB [(none)]> GRANT ALL ON wp.* TO 'wp'@'172.16.0.4'; MariaDB [(none)]> CREATE USER 'ttrss'@'172.16.0.4' IDENTIFIED BY '********************'; MariaDB [(none)]> GRANT ALL ON ttrss.* TO 'ttrss'@'172.16.0.4'; MariaDB [(none)]> FLUSH PRIVILEGES; Restore databases: root@mysql:~ # mysql -u root -p blog < blog-db.sql root@mysql:~ # mysql -u root -p wp < devilghost-wp-db.sql root@mysql:~ # mysql -u root -p ttrss < devilghost-ttrss-db.sql Set MariaDB root password: MariaDB [(none)]> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('***********************'); Create the .htpasswd file: --- www.paulgorman.org 174 ~ # echo -n 'paulgorman:' >> /usr/local/etc/nginx/.htpasswd --- www.paulgorman.org 177 ~ # openssl passwd -apr1 >> /usr/local/etc/nginx/.htpasswd --- www.devilghost.com 16 ~ # echo -n 'paulgorman:' >> /usr/local/etc/nginx/.htpasswd --- www.devilghost.com 17 ~ # openssl passwd -apr1 >> /usr/local/etc/nginx/.htpasswd Set up rewrite rule for example.org/blog/ by adding a location to /usr/local/etc/nginx/nginx.conf: location ~* /blog/(new|save|all|delete) { auth_basic "Login"; auth_basic_user_file /usr/local/etc/nginx/.htpasswd; rewrite ^/blog/((\w|\/)+)$ /blog/index.php?url=$1; } location ~ /blog { rewrite ^/blog/((\w|\/)+)$ /blog/index.php?url=$1; } If we want to add a cron job for a system user in a jail, had root edit the user's crontabl: $ crontab -u www -e Set up backups. Create /root/bin/backup.sh, and make it executable: #!/bin/sh DAY=$(date +'%d') tar -czf /home/paulgorman/backups/backup-inky-${DAY}.tgz \ /root/bin \ /etc/rc.conf \ /etc/ipfw.rules \ /etc/sysctl.conf \ /etc/ssl \ /etc/ssh \ /etc/aliases \ /var/cron \ /home/paulgorman/.ssh \ /home/paulgorman/repo \ /usr/local/etc/ \ /usr/jails/*/etc/rc.conf \ /usr/jails/*/root/bin \ /usr/jails/*/usr/local/etc/ \ /usr/jails/*/var/cron \ /usr/jails/*/usr/local/www/ \ /usr/jails/mysql/root/inky-alldatabases.sql chmod 640 /root/backup-inky-${DAY}.tgz Create a root cron job: 1 3 2,16 * * /root/bin/backup.sh Create the target directory: # sudo mkdir /home/paulgorman/backups # sudo chmod 740 /home/paulgorman/backups Set up a cron job on another machine to fetch the backup files. In the mysql jail, create /root/bin/sqlbackup.sql, and make it executable: #!/bin/sh mysqldump -u root --password='***********************' --all-databases > /root/inky-alldatabases.sql chmod 640 /root/inky-alldatabases.sql In the mysql jail, create a root cron job: 1 2 2,16 * * /root/bin/sqlbackup.sh .tcshrc for jails ------------------- set promptchars="%#" set prompt = "--- %M %h %c %# " setenv EDITOR vim setenv VISUAL vim setenv PAGER less set autolist alias l 'ls -FG' alias la 'ls -FGa' alias ll 'ls -FGlha' alias h 'history 60 | sort -k2 | uniq -f2 | sort -bn' setenv LSCOLORS "gxfxcxdxbxxggdabagacad" bindkey "^W" backward-delete-word .vimrc for jails ------------------ set nocompatible set encoding=utf-8 set viminfo='500,f1,<200,:200,@200,/200,s20,h set t_Co=256 syntax off set ignorecase set smartcase set incsearch set showmatch set wildmenu set autoindent set nosmartindent set tabstop=4 set shiftwidth=4 set expandtab set backspace=indent,eol,start set smarttab set ruler set showmode set nu set rnu let mapleader=" " " Default is \ nmap h :set hlsearch! nmap n :set nu! rnu! nmap o ok nmap O Oj imap ii ## OpenSMTPd ## Add to /etc/rc.conf: sendmail_enable="NO" sendmail_submit_enable="NO" sendmail_outbound_enable="NO" sendmail_msp_queue_enable="NO" smtpd_enable="YES" Stop sendmail, and install opensmtpd: # service sendmail stop # killall sendmail # pkg install opensmtpd Edit /etc/mail/mailer.conf: sendmail /usr/local/sbin/smtpctl send-mail /usr/local/sbin/smtpctl mailq /usr/local/sbin/smtpctl makemap /usr/local/libexec/opensmtpd/makemap newaliases /usr/local/libexec/opensmtpd/makemap Edit /usr/local/etc/mail/smtpd.conf: # See smtpd.conf(5) for more information. # To accept external mail, replace with: listen on all listen on localhost # If you edit the file, you have to run "smtpctl update table aliases" table aliases file:/etc/mail/aliases # Uncomment the following to accept external mail for domain "example.org" #accept from any for domain "example.org" alias deliver to mbox accept for local alias deliver to mbox accept for any relay Start opensmtpd: # service smtpd start