Linux System Init: The Case of the Disappearing CD-ROM Device

The software created by my company requires a two-step installation process with a reboot in between.   First, you install the base operating system from a CD.  Then you reboot and install the product software from another CD.  (There's a reason for doing it this way, but it's not relevant here.)

After installing the base OS, we tell the user/installer to reboot the system, interrupt the boot loader, and add "single enforcing=0" to the command line.  In other words, "boot to single user mode with SELinux in permissive mode".

Then we tell the user to insert the product CD and issue the mount /dev/cdrom /media command in order to proceed with the installation.

In moving from a Red Hat Enterprise Linux 5 base to RHEL 6, an annoying problem popped up: the /dev/cdrom device disappeared when the CD was inserted.  It was there before the CD went in, and it was gone afterward.

The problem and the solution are both related to udev.  Not knowing the innermost workings of udev, all I can describe is the visible indications that lead to the fix:

1. Since /dev/cdrom is a symlink, the CD can still be accessed as /dev/sr0.
2. Once installation is complete and the system is booted to runlevel 5, /dev/cdrom is back and the problem never occurs again.
3. The fully booted system contains a file called 70-persistent-cd.rules in <>code>/etc/udev/rules.d. This file was not there in single user mode.
4. There is an interesting script called udev-post in /etc/init.d.

It became apparent that udev behavior is changed in RHEL6.  This has something to do with "failed events" and persistent device naming.  I repeated the installation of the base OS, booted to single user mode, and looked in /dev/.udev/rules.d.  Sure enough, there was a temporary file containing the persistent CD device naming rules, which is supposed to be moved to it's permanent place in the /etc/ hierarchy by udev-post.

But we boot into single-user mode, so no init scripts (more rightly called "upstart scripts" in RHEL6) are being started.

I went back and did a little research to compare runlevel S to runlevel 1 but I only came up with vague references as to the original SysV purpose of these guys.  Some references even state that runlevel 1 is single-user mode.  And that is essentially correct.  Runlevel 1, in RHEL5 and RHEL6, simply starts the LVM monitor and then executes telinit S. Ah, but in RHEL6 they slipped a little fixer-upper: udev-post.

It seems that something in UdevLand is a little broken in RHEL6.  I believe this hack has gone away Fedora, or it's buried deeply in the various udev services of systemd.  Anyway, for us, the upshot is that we will switch from single user mode as the initial state to runlevel 1 for product installation.