[DCN-TechOps] wheel outage 2009-3-13

[Dave Zavatson]

Greetings,

wheel was unavailable Friday, March 13 from 1120 till 1310 due to a
destroyed /etc/shadow file.  The shadow file is the companion to the
passwd file.  It is where all encrypted passwords are kept.

All users were locked out of wheel.  Mail delivery continued, since
the users still existed as defined in the passwd file.  However,
nobody could check their mail of do radius authentication.

I luckily had a root window to wheel open and was able to copy a base
shadow file back that allowed root, help, and mothra logins.  omsoft
staff loaded the backup tape and restored a shadow file dated 3/11.  I
put that file in place around 1310, and logins began working once
more.

The outage was most likely due to /tmp filling up due to high mail
download volume and a password change occurring while /tmp was full.
The operating system was not able to write out the new shadow file and
clobbered the original.  This is backed by logs showing that /tmp was
full and with verification by omsoft staff that they did a password
change for a user right around 1120.  There is no reason to believe
the system was compromised.

However, it does show the precarious setup of wheel.  Wheel is running
on an end of lifed OS that hasn't been patched in years.  There are
most certainly security vulnerabilities on the system which would
allow for root compromises.  We still allow telnet access to the shell
which sends passwords in clear text.  This is a disaster waiting to
happen.

I'll work more on the hardware upgrade proposal.  We should move on
the purchase of a new server for public access soon.

Dave