Strange changes - any ideas?

[fc () all net (Fred Cohen)]

We run a change-controlled environment, which means that we should be
aware of all changes.  To crosscheck this, we regularly do automated
change detection.  This morning, I made some minor changes to some user
areas and ran the change control checks only to find the changes listed
below. (Here are some select extracts)

        *** '/bin/newgrp' has changed as follows:
           The contents (md5 checksum) changed. Any change in content can trigger this.

Note that while the content changed, none of the times changed,
the space remained the same, etc.

        *** '/etc/motd' has changed as follows:
           The size changed. This indicates an addition or removal.
           The modification time changed. This indicates a file edit or similar change.
           The status change time changed. Any change should trigger this.
           The content did not change! This could be the result of a reboot or a crash.

Here's one where everything indicates a change, but the content is
unchanged! Sort of hard to believe - there were several of these.

These changes would normally indicate a massive corruption, a disk
crash, total system collapse, or takeover by bad-people.  I checked the
log files that would indicate any intrusions and found nothing to
indicate any out-of-the-ordinary usage.  I found an apparent file in a
directory listing - but when I tried to see it, it did not actually
exist.  I did a cat of /etc/motd (described above) and found that it had
a partial syslog entry appended to it - very strange stuf considering
that the MD5 checksum was unchanged!

Within a few minutes, I rebooted the system.  When it came back up, I
ran a complete check again, only to find NONE of these changes! I
suspect some sort of memory cache problem but wanted to get some other
opinions.

The security implications? Good gracious.  During a corruption such as
this I could have (I actually did at one point) modify files that should
not have been accessible to me - apparently because permissions were
also corrupt (as cached?).  System: SunOS on a Sun 4/330.

Just so we all understand, here are some extracts of the things that
"unchanged" after reboot:

====================================================
Tracer Starting Engines on all.net by fc.
     Sat Jun  8 08:52:43 EDT 1996
The system type is SunOS Unix
Copyright (c), 1985-6 Management Analytics
          All Rights Reserved
====================================================

======>> Start:Checking for changes in system files.
Change control database found and being used.
Checking for changes in existing files.
Checking /var
Checking /bin
*** '/bin/newgrp' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/login' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/crontab' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/atq' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/atrm' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/cu' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/tip' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/iostat' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/ipcs' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/lp' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/lpstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/cancel' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/ypcat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/ypmatch' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/yppasswd' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/fusage' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/nsquery' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uucp' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uulog' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uuname' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uupick' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uusend' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/uustat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uuto' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/uux' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/lpstat.FCS' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/cancel.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/mail.orig' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/at.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/lpstat.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/cancel.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/iostat.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/login_orig' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/rnews' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/bin/screen' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/bin/passwd.old' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
Checking /usr/bin
*** '/usr/bin/newgrp' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/login' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/crontab' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/atq' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/atrm' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/cu' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/tip' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/iostat' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/ipcs' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/lp' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/lpstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/cancel' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/ypcat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/ypmatch' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/yppasswd' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/fusage' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/nsquery' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uucp' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uulog' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uuname' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uupick' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uusend' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/uustat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uuto' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/uux' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/lpstat.FCS' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/cancel.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/mail.orig' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/at.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/lpstat.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/cancel.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/iostat.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/login_orig' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/rnews' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/bin/screen' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/bin/passwd.old' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
Checking /usr/ucb
*** '/usr/ucb/lpr' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/ucb/lpq' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/ucb/lpq.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/vmstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/rdist' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/lpr.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/lprm.FCS' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/ucb/lprm' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/lpr.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/lprm.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/ucb/lpq.101434-01' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
Checking /etc
*** '/etc/ld.so.cache' has changed as follows:
   The modification time changed. This indicates a file edit or similar change.
   The status change time changed. Any change should trigger this.
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/arp' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/crash' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/etc/dkinfo' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/etc/dmesg' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/etc/dump' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/ypbind.lock' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/motd' has changed as follows:
   The size changed. This indicates an addition or removal.
   The modification time changed. This indicates a file edit or similar change.
   The status change time changed. Any change should trigger this.
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/mtab' has changed as follows:
   The modification time changed. This indicates a file edit or similar change.
   The status change time changed. Any change should trigger this.
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/psdatabase' has changed as follows:
   The modification time changed. This indicates a file edit or similar change.
   The status change time changed. Any change should trigger this.
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/pstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/etc/rdump' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/shutdown' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/etc/syslog.pid' has changed as follows:
   The modification time changed. This indicates a file edit or similar change.
   The status change time changed. Any change should trigger this.
   The content did not change! This could be the result of a reboot or a crash.
*** '/etc/ttys' has changed as follows:
   The modification time changed. This indicates a file edit or similar change.
   The status change time changed. Any change should trigger this.
   The content did not change! This could be the result of a reboot or a crash.
Checking /usr/etc
*** '/usr/etc/dump' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/rdump' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/shutdown' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/arp' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/dmesg' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/dkinfo' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/pstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/crash' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/keyenvoy' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/eeprom' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/auditd' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/chill' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/dumpfs' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/kgmon' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/trpt' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/devinfo' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/lpc' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/nfsstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/rfsetup' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/in.uucpd' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/lpc.FCS' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/pac.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/ypserv.orig' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/lpc.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/etc/pac.101434-01' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/etc/pppd' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
Checking /usr/kvm
*** '/usr/kvm/pstat' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/kvm/crash' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/kvm/getcons' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/kvm/eeprom' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/kvm/ps.FCS' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
Checking /usr/lib
*** '/usr/lib/lpd' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/lib/exrecover' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/lib/expreserve' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/lib/lpd.FCS' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/lib/expreserve.FCS' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/lib/lpd.101434-01' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/lib/sendmail.real' has changed as follows:
   The contents (md5 checksum) changed. Any change in content can trigger this.
*** '/usr/lib/sendmail.mx.fcs' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
*** '/usr/lib/sendmail.fcs' has changed as follows:
   The content did not change! This could be the result of a reboot or a crash.
Checking for files in the database but not in the system.
<<=== End:Done checking for changes in system files.


====================================================
Tracer done - Sat Jun  8 08:57:19 EDT 1996
====================================================

-> See: Info-Sec Heaven at URL http://all.net/
Management Analytics - 330-686-0090 - PO Box 1480, Hudson, OH 44236