You are here: Wiki_Virgo_LSC>IT Web>RTPC_Diskless_Configuration (20 Mar 2020, Dibiase)Edit Attach

RTPC diskless configuration

by Stefano Cortese — last modified 2020-03-20 14:23

Instructions for the first time installation, configuration and kernel recompilation

Elements of the system:

DHCP server
The Internet Consortium ISC dhcpd v3.1ESV has been chosen instead of the default AIX dhcpsd because of the failover features in case of dynamic addresses (even though it does not apply here in that only bootp is used for IP assignment).
In our setup both fs1 and fs2 run a dhcpd instance with the configuration file coming from a shared master copy.

The daemon has been integrated in the AIX subsystem controller (via "mkssys -s dhcpd -p /usr/local/sbin/dhcpd -u 0 -S -n 15 -f 9 -a "-f -cf /etc/dhcpd.conf en0 -G tcpip") in the "tcpip" group so that starts at boot and can be managed with the commands:

startsrc -s dhcpd
lssrc -s dhcpd
stopsrc -s dhcpd

When changing the master configuration file (/egosystem/etc/dhcpd.conf) use the following script on each node to automatically synchronize it and starts or restarts the daemon:

dhcp { start | restart }

TFTP server

The opentftpd v1.62 has been chosen instead of the default AIX tftpd in that allows to bind to a given interface supporting HACMP failover.
It is the same as the for the RIOs boot

NFS server

In our setup the /diskless export is mounted through the "rioboot" alias

Diskless clients ( 1 with disk as the prototype client)

The installation is based on the RedHat Diskless Environment (

Steps to build a first-time diskless image

  1. Perform the OS installation on the system disk (currently SL 4.5) of the prototype client with only 2 partitions: /boot and /
    There is no need of the swap space that can be disabled in the fstab
  2. install the rpm with the tools to build the Redhat Diskless Environment ( currently system-config-netboot-cmd- and syslinux build from syslinux-3.82-1.src.rpm )
    The package offers these commands:
    pxeos: cmd line tool to create an OS image description
    pxeboot: cmd line tool to create the per-client boot configuration file
    system-config-netboot: GUI tool that includes the functions of all the other commands
  3. Insure that the /tftpboot directory exists on the prototype client. There, in the subdir linux-install, will be created the pxeboot files that will have to be moved to the TFTP server
  4. Create the NFS exports on the server
    The root tree contains the OS part common to all clients (that can be read-only), the snapshot tree contains the client dependent part of the OS (read-write will be populated with subdirectories with the names of the clients)
    In the AIX case:
    /diskless/rtpcOS/root -root=rtpc,rw,anon=65534,vers=3

    /diskless/rtpcOS/snapshot -root=rtpc,rw,anon=65534,vers=3
    Note: the root export must be rw otherwise the first install process will not work
  5. Copy the OS installation from the prototype client to the root export (preserving everything, for example with tar)
    You may want to empty the machine dependent information like the log files (this can be done also afterwards)
  6. Mount the root export on the client in rw
  7. Start the system-config-netboot GUI on the client with the following steps:
  • select "diskless installation"
  • choose a name as the identifier of the diskless image (for example rtpcOS)
  • give the NFS server name and path of the root export (for example /diskless/rtpcOS/root)
  • select the kernel for the diskless image (from the /boot subdirectory of the root export); note that the kernel is the same of the disk installation and does not need to be modified

  • Copy the contents of the /tftpboot/linux-install directory in the tftp server directory ( in our configuration /diskless/tftpdir/rtpc which is shared by fs1 and fs1 and is pointed by the /tftpboot/rtpc symlink on each server)
    the directory contains these files:
    pxelinux.? : the pxe kernel that manages the boot phases
    pxelinux.cfg : the pxe directory that contains one pxe configuration file for each client
    rtpcOS : the directory that contains the initrd and kernels of the given image

At this point the root file system on the root NFS export has been adjusted to serve as a single image for the diskless client (the network configuration is changed, the original files have been renamed as prediskless-filename .

Steps to add the clients

The first client can be created with the system-config-netboot GUI. The result is the PXE config file for the client, placed under the pxelinux.cfg directory.
The name of the file is a string with the IP address of the client in hexadecimal characters.
This file must be moved to the corresponding directory on the tftp server.

The successive clients can be configured manually simply by cloning the PXE config file and changing the reference to the client name inside ( the SNAPSHOT variable).
The name of the file can be found easily with the command:

# gethostip hostname_of_the_client

NOTE: the snapshot directory does not need to be created, it is done automatically at the first boot by the disklessrc script.

In our setup the pxelinux.cfg directory structure has been modified manually to offer a boot-time menu at the console as a common part as follows:
# Configurazione menu comune a tutti i client
# Contiene le entry di menu per il boot locale e diskless
# Questo file viene incluso dai file pxelinux.cfg/XXXXXXXX
# la cui linea di append e' invece diversa per ogni client perche'
# contiene la variabile SNAPSHOT con il nome del singolo client
# (Nota: la linea di append non puo' essere splittata)
DEFAULT menu.c32
MENU TITLE PXE boot menu

LABEL local
localboot 0

MENU LABEL ^Hardware Detection
kernel hdt.c32

MENU LABEL ^Diskless
kernel rtpcOS/vmlinuz

The comboot files with .c32 extension are modules that can be executed by pxelinux and are included in the syslinux package.
They have to be copied to the tftpboot directory ( /diskless/tftpdir/rtpc/ )

Client specific file:

INCLUDE pxelinux.cfg/

# INITRD_DBG fornisce una shell in vari punti di disklessrc (da 1 a 4)
# append INITRD_DBG=4 initrd=rtpcOS/initrd.img root=/dev/ram0 init=disklessrc ramdisk_size=40960 SNAPSHOT=rtpctest1

append vmalloc=512M noirqbalance isolcpus=1,2,3 root=/dev/ram0 initrd=rtpcOS/initrdhack.img init=disklessrc ramdisk_size=44032 SNAPSHOT=rtpctest1 apic=debug

To make the management easier after having created the client file inside /diskless/tftpdir/rtpc/pxelinux.cfg please create a symlink to it in the directory /diskless/tftpdir/rtpc/config with the hostname of the client (not fully qualified):

cd /diskless/tftpdir/rtpc/config ; ln -s ../pxelinux.cfg/hexip_of_the_clienthostname_f_the_client

Customization of disklessrc

The disklessrc script is the boot script executed from the initrd ramdisk image that is loaded by kernel.
This script uses some of the kernel command line parameters to mount the diskless root and snapshot NFS images, creating the structure of the snapshot directory at the first run.

The script makes also dhcp/bootp queries to configure the network.

In order to overcome the problem of the limited length of the kernel command line, the variables needed in this step, like the NFSROOT parameter, have been moved as far as possible to the dhcp server and to retrieve them the disklessrc script has been customized and the version as of 12-11-2011 can be found here

The best way to modify the script is to retrieve the initrd image directly from the tftp server and open it on a linux machine as follows:

# mv initrd.img initrd.gz
# gunzip initrd.gz
# losetup /dev/loop0 initrd
# mkdir initrd_dir
# mount /dev/loop0 ./initrd_dir
# vi initrd_dir/disklesrc
# umount initrd_dir
# losetup -d /dev/loop0
# gzip initrd
# mv initrd.gz initrd.img

and copy back initrd.img to the tftp server

Note: if the image has gotten larger you may want to increase the ramdisk_size kernel command line parameter

12-11-2011: removed the NFS mounting of files in the /lib/modules/kernelversion/.. directory because seems unnecessary and breaks the make modules_install step on the read-write client

Further cleaning of the image

You may want to do the following from a single read-write client:

  • Review crontab entries: move updatedb, prelink and similar jobs into a NOcron subdirectory of cron.{hourly,daily,weekly,monthly}
    These jobs can be run manually from the read-write client as needed (i.e. after software installation)

How to install additional software on the diskless image

RPMs and tarballs can be installed using a single diskless client and the installed files will be seen at once by all the other clients with the following caveats:

  • runtime actions triggered by the installation (i.e. the start of a service) must be repeated on all the other clients
  • the installation paths must be checked, if some of them fall in the local snapshot directory they need to be replicated on the other clients via mount -bind option and inserted in the custom_files list to be available also at the next boot
  • per-client configuration files (that fall for example inside /etc) containing per-client information must be moved from the root tree to the snapshot tree

The client to choose for the installation may be whichever. Once logged in as root user the root filesystem must be remounted read-write with the command:

# mount -o remount /

and then proceed as it were a normal standalone installation.

When finished you may want to run updatedb, prelink or makewhatis to update the image databases.

Note about the RPM installation

RPMs installation is special in that the rpm commands use read or write locks on the Berkeley DB databases in /var/lib/rpm to control concurrent acccess.
It is not clear whether rpm use the NFS locking ( fcntl() posix ) but anyway the diskless NFS image is mounted with the nolock option that cannot be turned off with the -o remount option (and removing it from disklessrc initrd boot sequence makes the NFS mount fail because of the absence of the portmapper).
Therefore care must be taken to use only one client for the installation and don't issue multiple rpm commands that may corrupt the rpm database.
In case of problems with rpm commands complaining about locks do the following:

Check if there are hanging rpm locks:
# /usr/lib/rpm/rpmdb_stat -h /var/lib/rpm -CA

If any, clean the locks removing the __db lock files:

# rm /var/lib/rpm/__db*

How to upgrade/modify the kernel of a diskless image

The modifications/upgrades can be performed on the read-write client and don't affect the production configuration because the new kernel and modules are created aside of the running ones and are not activated until they are inserted in the dhcp/pxelinux configuration (the /boot directory is not used by the diskless clients).

If you want to operate in a completely new test environment, duplicate the root tree and also the snapshot tree for the hosts that will be the testbed, then update the /etc/exports file with the new trees and enable them one by one via exportfs and modify /egosystem/etc/dhcp.conf adding the new root tree only for the testbed hosts

Starting from the source code the steps on the read-write client are the following:

  1. if it is a new version, take the kernel and modules from the Virgo sw repository via cvs; if instead you want to recompile the current kernel (maybe with some new modules or to change the kernel configuration) start directly from the working tree located for example in /usr/src/Rtai/v3r81
    Even if starting from scratch the configuration parameters proper of this RTAI installation and which should be kept are in
  2. for a hint on the procedure look at this document; note that it is thought for a standard standalone disk installation
  3. as a user with a VCS login profile do the "cmt config" and "make" steps in the ./cmt directory
  4. as the same user or root do the "make" and "make modules" steps in the OS subtree (for example ./Linux-i686-SL4/linux- and the "make" step in the rtai subtree (for example ./Linux-i686-SL4/rtai-3.8.1) (note that for this last step to works there should be defined the symlink /usr/src/linux to the kernel source tree, i.e. /usr/src/Rtai/v3r81/Linux-i686-SL4/linux- )
  5. as root do the "make modules_install" and "make install" steps in the same subtrees as above
  6. if the kernel is not a new version simply take the bzImage in the source subtree (for example /usr/src/Rtai/v3r81/Linux-i686-SL4/linux- and place it in the right /diskless/tftpdir/rtpc subdirectory on the tftp server
    If the network modules have not been changed there is no need to change the initrd.img or initrdhack.img image because all the other modules are loaded from the root filesystem; the kernel image is therefore ready to run
  7. if the kernel is a new version a new initrd image must be created from scratch; start the system-config-netboot GUI and define a new diskless image for the new version (with the same NFS export root as the current one or with the new testbed root) selecting the new kernel
    1. once the new version is defined the new vmlinuz and initrd.img files will have been created in the /tftpboot/linux-install directory (note that vmlinuz is renamed from exactly the same file as the kernel in the /boot directory while the initrd.img file is derived by the one in /boot and it is formatted as a compressed ramdisk image instead of a compressed cpio archive).
      Hack the initrd image as described above and put it with the vmlinuz file in a directory like /diskless/tftpdir/rtpc/rtpcOS_new_version_name
  8. configure the pxelinux.cfg files for one or all the clients to point to the new kernel at the next reboot by changing the "kernel" and "initrd" parameters in the proper places.
    Note that you may need to increase the "ramdisk_size" value if the new initrdhack.img is larger than in the previous version

To test the rtai modules load them using the script in the source tree corresponding to the kernel, i.e. /usr/src/Rtai/v3r81/Linux-i686-SL4/LoadModules- script and check that the modules are laoded correctly

See also the:
Topic attachments
I Attachment Action Size Date Who Comment
disklessrcEXT disklessrc manage 12 K 20 Mar 2020 - 13:30 Main.Dibiase disklessrc
Topic revision: r1 - 20 Mar 2020, Dibiase
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Wiki_Virgo_LSC? Send feedback