Next I'll be talking about preparing the deployer platform for network OS installs. I've got some servers that need installing, some serial console cables run to them, a webpowerswitch, so we can hard-reset devices if needed.

The tech-tree for the programmatic-os-installs:

platform:deployer ─┐
                   ├ ethernet ─────────────┐
                   ├ ip-network ───────────┤
                   ├ dhcp ─────────────────┤
                   ├ pxe  ─────────────────┤
                   ├ tftp ─────────────────┤
                   ├ http ─────────────────┤
                   ├ serial-communication ─┤
                   └ minicom ──────────────┤
component:usb-2-serial-adapter ────────────┤
component:serial-cable ────────────────────┤
                                           └ programmatic-os-installs:openbsd

The deployer node has both ethernet and wifi. The wifi will be on the same network segment as our workstation, and the ethernet will be it's own "install subnet" of 198.51.100.32/27. This gives us 30 useable IPs, and will allow us to connect crates without collisions. Leave the routers and DNS servers out of the configuration. I want to use WiFi and our local subnet for those.

  • Create an /etc/network/interfaces.d/eth0 file with the following contents
    • auto eth0
iface eth0 inet static
  address 198.51.100.33
  netmask 255.255.255.224
    • I then reboot the deployer to ensure:
      • the ip and netmask on eth0 are set
      • the default gateway and dns are still using wifi and the local area network

    Now I can get things ready to install OpenBSD, I will need (as the technology tree illustrates):</p?

    • a DHCP server
      • apt-get install -y isc-dhcp-server
[ ! -f /etc/dhcp/dhcpd.conf.dist ] && cp /etc/dhcp/dhcpd.conf /etc/dhcp/dhcpd.conf.dist

# /etc/default/isc-dhcp-server
INTERFACESv4="eth0"
INTERFACESv6=""

# /etc/dhcp/dhcpd.conf
ddns-update-style none;
option domain-name "apartment.jameswhite.org";
option domain-name-servers 10.252.0.254 10.252.0.6, 10.252.0.7;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;

subnet 10.252.0.0 netmask 255.255.255.0   { deny unknown-clients; }
subnet 198.51.100.0 netmask 255.255.255.0 { deny unknown-clients; }

[ -f /var/run/dhcpd.pid ] && rm -f /var/run/dhcpd.pid; service isc-dhcp-server start; journalctl -xe
    • a TFTP server
      • apt-get install -y tftpd-hpa
    • pxeboot files
      •
    • a http server
      • apt-get install -y nginx

      SeaBIOS (version rel-1.12.1.3-0-g300e8b70)

Press F10 key now for boot menu

Select boot device:

1. AHCI/0: SATA SSD ATA-11 Hard-Disk (57241 MiBytes)
2. Payload [setup]
3. Payload [memtest]
    • Press the <F10> key
      • default boot order 
      Boot order - type letter to move device to top.

  a USB
  b SDCARD
  c mSATA
  d SATA
  e mPCIe1 SATA1 and SATA2
  f iPXE (disabled)


  r Restore boot order defaults
  n Network/PXE boot - Currently Disabled
  u USB boot - Currently Enabled
  t Serial console - Currently Enabled
  k Redirect console output to COM2 - Currently Disabled
  o UART C - Currently Enabled
  p UART D - Currently Enabled
  m Force mPCIe2 slot CLK (GPP3 PCIe) - Currently Disabled
  h EHCI0 controller - Currently Disabled
  l Core Performance Boost - Currently Enabled
  i Watchdog - Currently Disabled
  j SD 3.0 mode - Currently Disabled
  v IOMMU - Currently Disabled
  y PCIe power management features - Currently Disabled
  w Enable BIOS write protect - Currently Disabled
  x Exit setup without save
  s Save configuration and exit
    • press the <n> key to enable iPXE
    • press the <f><a>,and <b> keys, note pxe will now be enabled and the machine will try USB, then iPXE, then mSATA, then SDCard;
    • new boot order 
      Boot order - type letter to move device to top.

  a USB
  f iPXE
  c mSATA
  b SDCARD
  d SATA
  e mPCIe1 SATA1 and SATA2


  r Restore boot order defaults
  n Network/PXE boot - Currently Disabled
  u USB boot - Currently Enabled
  t Serial console - Currently Enabled
  k Redirect console output to COM2 - Currently Disabled
  o UART C - Currently Enabled
  p UART D - Currently Enabled
  m Force mPCIe2 slot CLK (GPP3 PCIe) - Currently Disabled
  h EHCI0 controller - Currently Disabled
  l Core Performance Boost - Currently Enabled
  i Watchdog - Currently Disabled
  j SD 3.0 mode - Currently Disabled
  v IOMMU - Currently Disabled
  y PCIe power management features - Currently Disabled
  w Enable BIOS write protect - Currently Disabled
  x Exit setup without save
  s Save configuration and exit
    • press the <s> key to enable iPXE
    • make a note of the mac addresses it tries. I'll be using net0 to pxe boot
      • net0: 00:0d:b9:57:17:14 using i211 on 0000:01:00.0 (open)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 00:0d:b9:57:17:14)................. No configuration methods succeeded (http://ipxe.org/040ee119)
net1: 00:0d:b9:57:17:15 using i211 on 0000:02:00.0 (open)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net1 00:0d:b9:57:17:15)................. No configuration methods succeeded (http://ipxe.org/040ee119)
net2: 00:0d:b9:57:17:16 using i211 on 0000:03:00.0 (open)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net2 00:0d:b9:57:17:16)................. No configuration methods succeeded (http://ipxe.org/040ee119)
net3: 00:0d:b9:57:17:17 using i211 on 0000:03:00.0 (open)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net3 00:0d:b9:57:17:17)................. No configuration methods succeeded (http://ipxe.org/040ee119)

Trying next device: 33
Trying next device: 4
Trying next device: 5
No bootable device.  Retrying in 60 seconds.

      I've created a problem when I configured the switches. I put the APU4 nodes on green ports (vlan 256, untagged) and the deployer node is on vlan 1, but I did present all other VLANs to the deployer node in a trunk. So I'll bring up a VLAN interface on the deployer node so it can see the APU$ PXE boot requests.

      I want to use 10.250.0.0/24 for the green subnet, and I'd like the APU4s to be the gateway for that. Traditionally, gateways are at .1 or .254 at the beginning or end of the available IP space, even if they don't have to be, so I'll put the deployer node on .10 just to get it out of the way.

    • I installed the vlan pacage and ensure the 802.11q module is loaded.
      • apt-get install vlan
lsmod | grep 8021q || modprobe 8021q
    • Added the following to:/etc/network/interfaces.d/eth0
      • auto eth0.256
iface eth0.256 inet static
    address 10.250.0.10
    netmask 255.255.255.0
    • Update /etc/default/isc-dhcp-server:
      • INTERFACESv4="eth0 eth0.256"
INTERFACESv6=""
    • And added a subnet declaration to /etc/dhcp/dhcpd.conf and bounced the daemon
      • subnet 10.250.0.0    netmask 255.255.255.0 { deny unknown-clients; }

      service isc-dhcp-server restart
service isc-dhcp-server status

      Now I can run:tail -f /var/log/syslog on the deployer node you'll see requests coming in for DHCP from our two APU4 boxes. There are no free leases because I used `deny unknown-clients` in our dhcp server configuration.

      Sep  8 16:42:35 deployer2 dhcpd[959]: DHCPDISCOVER from 00:0d:b9:57:78:c4 via eth0.256: network 10.250.0.0/24: no free leases
Sep  8 16:42:39 deployer2 dhcpd[959]: DHCPDISCOVER from 00:0d:b9:57:17:14 via eth0.256: network 10.250.0.0/24: no free leases

      OpenBSD

      OpenBSD installs take the form

      1. The host boots using pxe, looks for a dhcp offer
      2. Unpacks the dhcp offer and looks to see if the filename argument passed is auto_install
      3. Also looks for the next-server and server-name arguments passed in the dhcp offer
      4. Retrieves the auto_install file from the TFTP server designated by next-server (this is a symlink to the openbsd pxeboot file)
      5. Executes the preboot execution environment (pxeboot) file
      6. The pxeboot environment etc/random.seed and etc/boot.conf file from the TFTP server designated by next-server
      7. The pxeboot environment then loads the RAM disk specified in the boot.conffile and loads up the installer.
      8. The installer usese the server-name to get the OpenBSD install files and looks for a MACADDR-install.conf file for answering questions

      Booting PXE

      I set the APU4 boxes to boot PXE above using the serial console.

      The DHCP offer, next-server and server-name

      I replaced the empty subnet declaration for 10.250.0.0 setting next-server and server-name to the deployer's IP on that network segment.

      subnet 10.250.0.0    netmask 255.255.255.0 {
  deny unknown-clients:
  option routers 10.250.0.10;

  # OpenBSD Installs
  host hermod { hardware ethernet 00:0d:b9:57:17:14; next-server 10.250.0.10; fixed-address 10.250.0.3; filename "auto_install"; server-name "10.250.0.10"; }
  host vali   { hardware ethernet 00:0d:b9:57:78:c4; next-server 10.250.0.10; fixed-address 10.250.0.2; filename "auto_install"; server-name "10.250.0.10"; }
}

      And bounce the DHCP server service isc-dhcp-server restart

      TFTP files

    • I install the TFTP server with apt-get install -y tftpd-hpa. This creates a /srv/tftp directory. I edit /etc/default/tftpd-hpa and bounce tftpd with service tftpd-hpa restart
      • TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="-4 --secure --create"

      I go over to my local OpenBSD Mirror and grab the install ISO file

    • I get that auto_install file put where DHCP says to find it.
      • mkdir -p /srv/tftp/pxelinux.kernels/openbsd/6.7
wget -O /srv/tftp/pxelinux.kernels/openbsd/6.7/pxeboot ftp://mirror.esc7.net/pub/OpenBSD/6.7/i386/pxeboot
(cd /srv/tftp; ln -s pxelinux.kernels/openbsd/6.7/pxeboot auto_install) # <-- ensure this is a relative and not an absolute path.
    • I grab the installer ramdisk, and set up an /srv/tftp/etc/boot.conf that will point to it. I also create the random.seed
      • wget -O /srv/tftp/pxelinux.kernels/openbsd/6.7/bsd.rd ftp://mirror.esc7.net/pub/OpenBSD/6.7/i386/bsd.rd

mkdir -p /srv/tftp/etc
cat<<EOF>/srv/tftp/etc/boot.conf
set tty com0
stty com0 115200
boot pxelinux.kernels/openbsd/6.7/bsd.rd
EOF

dd if=/dev/random of=/srv/tftp/etc/random.seed bs=512 count=1 status=none

      HTTP files

    • I then install the nginx web server with apt-get install -y nginx. This creates the /var/www/html directory. This can take a few minutes, depending on your mirror.
      • mkdir -p /var/www/html/openbsd/install67/6.7
(cd /var/www/html/openbsd/install67/6.7; wget -nH -P i386 --mirror --cut-dirs=5 --reject fs,iso ftp://mirror.esc7.net/pub/OpenBSD/6.7/i386/)
    • then I need install.conf files. I like to make files named <hostname>-install.conf and then symlink <ma:ca:dd:re:ss:xx>-install.conf to them.
      • 00:0d:b9:57:17:14-install.conf -> install-hermod.conf 
      System hostname = hermod
Terminal type? = vt220
System hostname = hermod
Which network interface do you wish to configure? = em0
IPv4 address for em0 = dhcp
# apt-get install -y apache2-utils; echo "$(htpasswd -bnBC 8 "" bearing-cajole-envision-hew-mangrove-algiers | tr -d ':\n')" # replace the $2y$ with $2a$
Password for root account? = $2a$08$TvzCNv1LKM9U/NGjiMEdTOAww0Gxv/lbYL8COWCp3R4Reu35OhoMq
Public ssh key for root account? = ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICFfZcR+pNjUP84kkmJ9Rh5PTOSC1aBu+PZiA2TqIutu jameswhite@bragi.local
Start sshd(8) by default? = yes
Start ntpd(8) by default? = yes
NTP server? (hostname or 'default') = default
Do you expect to run the X Window System? = no
Change the default console to com0? = yes
Which speed should com0 use? (or 'done') = 115200
What timezone are you in? = UTC
# USB
# Which disk is the root disk? = wd0
# SSD
Which disk is the root disk? = sd0
Setup a user = opt
#  Password for user = thrush-towel-rhizome-iodide-sleek-charisma
# apt-get install -y apache2-utils; echo "$(htpasswd -bnBC 8 "" thrush-towel-rhizome-iodide-sleek-charisma| tr -d ':\n')" # replace the $2y$ with $2a$
Password for user = $2a$08$r4gDYaVXkW1QdLf4odNV1uSEQBBwMu9pvz9pcOs2959ayJb4WXSEa
Public ssh key for user opt? = ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICFfZcR+pNjUP84kkmJ9Rh5PTOSC1aBu+PZiA2TqIutu jameswhite@bragi.local
Allow root ssh login? prohibit-password
Use DUIDs rather than device names in fstab? = yes
Use (W)hole disk or (E)dit the MBR? = W
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? = a
# EXPERIMENTAL: URL to autopartitioning template = http://10.250.0.10/openbsd-net6501-ssd.disklabel
Which disk do you wish to initialize? = done
Location of sets = http
HTTP proxy URL = none
HTTP Server = 10.250.0.10
Server directory = openbsd/install67/6.7/i386/
Use http instead = yes
Set name(s)? = -all bsd bsd.rd bsd.mp base67.tgz comp67.tgz man67.tgz game67.tgz done
Directory does not contain SHA256.sig. Continue without verification = yes
Location of sets? = done
      00:0d:b9:57:78:c4-install.conf -> install-vali.conf 
      System hostname = hermod
Terminal type? = vt220
System hostname = hermod
Which network interface do you wish to configure? = em0
IPv4 address for em0 = dhcp
# apt-get install -y apache2-utils; echo "$(htpasswd -bnBC 8 "" bearing-cajole-envision-hew-mangrove-algiers | tr -d ':\n')" # replace the $2y$ with $2a$
Password for root account? = $2a$08$TvzCNv1LKM9U/NGjiMEdTOAww0Gxv/lbYL8COWCp3R4Reu35OhoMq
Public ssh key for root account? = ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICFfZcR+pNjUP84kkmJ9Rh5PTOSC1aBu+PZiA2TqIutu jameswhite@bragi.local
Start sshd(8) by default? = yes
Start ntpd(8) by default? = yes
NTP server? (hostname or 'default') = default
Do you expect to run the X Window System? = no
Change the default console to com0? = yes
Which speed should com0 use? (or 'done') = 115200
What timezone are you in? = UTC
# USB
# Which disk is the root disk? = wd0
# SSD
Which disk is the root disk? = sd0
Setup a user = opt
#  Password for user = thrush-towel-rhizome-iodide-sleek-charisma
# apt-get install -y apache2-utils; echo "$(htpasswd -bnBC 8 "" thrush-towel-rhizome-iodide-sleek-charisma| tr -d ':\n')" # replace the $2y$ with $2a$
Password for user = $2a$08$r4gDYaVXkW1QdLf4odNV1uSEQBBwMu9pvz9pcOs2959ayJb4WXSEa
Public ssh key for user opt? = ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICFfZcR+pNjUP84kkmJ9Rh5PTOSC1aBu+PZiA2TqIutu jameswhite@bragi.local
Allow root ssh login? prohibit-password
Use DUIDs rather than device names in fstab? = yes
Use (W)hole disk or (E)dit the MBR? = W
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? = a
# EXPERIMENTAL: URL to autopartitioning template = http://10.250.0.10/openbsd-net6501-ssd.disklabel
Which disk do you wish to initialize? = done
Location of sets = http
HTTP proxy URL = none
HTTP Server = 10.250.0.10
Server directory = openbsd/install67/6.7/i386/
Use http instead = yes
Set name(s)? = -all bsd bsd.rd bsd.mp base67.tgz comp67.tgz man67.tgz game67.tgz done
Directory does not contain SHA256.sig. Continue without verification = yes
Location of sets? = done
    • I put them in the root of the html directory
      • root@deployer2:/var/www/html# ls -l
total 16
lrwxrwxrwx 1 root root   19 Sep  8 20:42 00:0d:b9:57:17:14-install.conf -> install-hermod.conf
lrwxrwxrwx 1 root root   17 Sep  8 20:42 00:0d:b9:57:78:c4-install.conf -> install-vali.conf
-rw-r--r-- 1 root root  612 Sep  8 19:41 index.nginx-debian.html
-rw-r--r-- 1 root root 1967 Sep  8 20:41 install-hermod.conf
-rw-r--r-- 1 root root 1943 Sep  8 20:42 install-vali.conf
drwxr-xr-x 3 root root 4096 Sep  8 19:44 openbsd

      Now I reboot the node and watch the install on the serial console. When it's done, we remove the dhcp options of filename,next-server, and server-name from the dhcpd.conf and bouncd the dhcpd service so we don't get a re-install every time we reboot. I'll often comment them out and add new no-install lines for quick install-switching.

      subnet 10.250.0.0    netmask 255.255.255.0 {
  deny unknown-clients;
  option routers 10.250.0.10;
  # OpenBSD Installs
  # host vali   { hardware ethernet 00:0d:b9:57:78:c4; next-server 10.250.0.10; fixed-address 10.250.0.2; filename "auto_install"; server-name "10.250.0.10"; }
  # host hermod { hardware ethernet 00:0d:b9:57:17:14; next-server 10.250.0.10; fixed-address 10.250.0.3; filename "auto_install"; server-name "10.250.0.10"; }
  host vali   { hardware ethernet 00:0d:b9:57:78:c4; fixed-address 10.250.0.2; }
  host hermod { hardware ethernet 00:0d:b9:57:17:14; fixed-address 10.250.0.3; }
}

      Some post-install work I've yet to automate FIXME

      export PKG_PATH="ftp://mirror.esc7.net/pub/OpenBSD/$(uname -r)/packages/$(uname -m)/"
for pkg in curl cyrus-sasl--ldap db-4.6.21p5v0 e2fsprogs gettext git icu4c isc-bind-- isc-dhcp-server json libiconv libidn libxml lzo2 nghttp2 openldap-client-- openldap-server-2.4.45p4 openpam openvpn-2.4.9 quirks rsync-- sudo--gettext-ldap xz; do pkg_add ${pkg};done

sysctl net.inet.ip.forwarding=1
echo "net.inet.ip.forwarding=1" >> /etc/sysctl.conf