Ocassionally things go bad(tm). This has bit me for the second time in approx 3 years now, but generally one needs to load the old FreeBSD kernel to start debugging and going through a box with a fine toothcomb.

The following snipbit gives you an idea what steps to take when rebooting the server so that you can load the previous working copy of the FreeBSD kernel:
When the boot menu appears hit the spacebar to stop the countdown.
Press "6" for "to escape to loader prompt"
unload
load /boot/kernel.old/kernel
boot

Now the old working FreeBSD kerenl is booting up. It would be recommended to copy the last working version to /boot/kernel.last for example so that you can "load /boot/kernel.last/kernel", especially if you are going to be building your kernel multiple times on a server.

Install CVSup

cd /usr/ports/net/cvsup-without-gui
make install distclean

Make and populate the CVSup config file

touch /root/cvsup-stable-src.sup
echo '*default host=cvsup14.us.FreeBSD.org' >> /root/cvsup-stable-src.sup
echo '*default base=/var/db' >> /root/cvsup-stable-src.sup
echo '*default prefix=/usr' >> /root/cvsup-stable-src.sup
echo '*default release=cvs tag=RELENG_5' >> /root/cvsup-stable-src.sup
echo '*default delete use-rel-suffix compress' >> /root/cvsup-stable-src.sup
echo ' src-all' >> /root/cvsup-stable-src.sup

Update the /usr/src/ tree

cvsup /root/cvsup-stable-src.sup

Get rid of any old "worlds" and make a new one

rm -rf /usr/obj/usr
cd /usr/src/
make buildworld

Make changes to /usr/src/sys/i386/conf/GENERIC and name it was what you want.

Build the kernel, install the kernel, verify it and dot.old in /boot/, run mergemaster, and install the new world.

make buildkernel KERNCONF=GENERIC
make installkernel KERNCONF=GENERIC
mergemaster -p
make installworld
mergemaster
ls -l /boot/
shutdown -r now

Note, we often run in a securelevel of 1 and have immutable binaries in the system folders. You'll need to edit rc.conf.

nano /etc/rc.conf

kern_securelevel_enable="NO"
kern_securelevel="1"

Reboot

shutdown -r now

Then make things mutable

chflags noschg /bin/*
chflags noschg /sbin/*
chflags noschg /bin
chflags noschg /sbin
chflags noschg /usr/bin/*
chflags noschg /usr/sbin/*
chflags noschg /usr/bin
chflags noschg /usr/sbin