Monday, March 26, 2012

Stale NFS file handle .... WHAT !!!! (And im not even using NFS)

Today I experienced a very weird situation.

Nagios was telling me, that my hourly mysql backup process failed.

So looking at the LVM, of where we store our backups.

What do I see ...

root@sql02:/data_backup# ls -la
ls: cannot access hourly_Tue_06_mysql.tar.gz: Stale NFS file handle
ls: cannot access hourly_Mon_14_mysql.tar.gz: Stale NFS file handle
ls: cannot access hourly_Mon_20_mysql.tar.gz: Stale NFS file handle
ls: cannot access hourly_Mon_18_mysql.tar.gz: Stale NFS file handle
ls: cannot access hourly_Mon_16_mysql.tar.gz: Stale NFS file handle
total 81737756
drwxr-xr-x  3 root root       4096 2012-03-26 12:54 .
drwxr-xr-x 24 root root       4096 2010-02-04 12:39 ..
-rw-r--r--  1 root root 4645355531 2012-03-26 06:45 hourly_Mon_06_mysql.tar.gz
-rw-r--r--  1 root root 4646191698 2012-03-26 08:44 hourly_Mon_08_mysql.tar.gz
-rw-r--r--  1 root root 4647695788 2012-03-26 10:49 hourly_Mon_10_mysql.tar.gz
-rw-r--r--  1 root root 4649504933 2012-03-26 12:44 hourly_Mon_12_mysql.tar.gz
-?????????  ? ?    ?             ?                ? hourly_Mon_14_mysql.tar.gz
-?????????  ? ?    ?             ?                ? hourly_Mon_16_mysql.tar.gz
-?????????  ? ?    ?             ?                ? hourly_Mon_18_mysql.tar.gz
-?????????  ? ?    ?             ?                ? hourly_Mon_20_mysql.tar.gz
-rw-r--r--  1 root root 4644413050 2012-03-24 14:43 hourly_Sat_14_mysql.tar.gz
-rw-r--r--  1 root root 4644347702 2012-03-24 16:43 hourly_Sat_16_mysql.tar.gz
-rw-r--r--  1 root root 4644389185 2012-03-24 18:43 hourly_Sat_18_mysql.tar.gz
-rw-r--r--  1 root root 4644334737 2012-03-24 20:43 hourly_Sat_20_mysql.tar.gz
-rw-r--r--  1 root root 4644340846 2012-03-24 22:43 hourly_Sat_22_mysql.tar.gz
-rw-r--r--  1 root root 4644322947 2012-03-25 06:45 hourly_Sun_06_mysql.tar.gz
-rw-r--r--  1 root root 4644647579 2012-03-25 08:43 hourly_Sun_08_mysql.tar.gz
-rw-r--r--  1 root root 4645116133 2012-03-25 10:44 hourly_Sun_10_mysql.tar.gz
-rw-r--r--  1 root root 4645206124 2012-03-25 12:43 hourly_Sun_12_mysql.tar.gz
-rw-r--r--  1 root root 4645299884 2012-03-25 14:44 hourly_Sun_14_mysql.tar.gz
-rw-r--r--  1 root root 4645487729 2012-03-25 16:43 hourly_Sun_16_mysql.tar.gz
-rw-r--r--  1 root root 4645544082 2012-03-25 18:43 hourly_Sun_18_mysql.tar.gz
-rw-r--r--  1 root root 4645580059 2012-03-25 20:43 hourly_Sun_20_mysql.tar.gz
-rw-r--r--  1 root root 4645698797 2012-03-25 22:43 hourly_Sun_22_mysql.tar.gz
-?????????  ? ?    ?             ?                ? hourly_Tue_06_mysql.tar.gz
drwx------  2 root root      16384 2010-02-12 15:18 lost+found

Flip knows how that happened. I umount the LVM and run
 root@sql02:/# e2fsck -f -v -p /dev/mapper/vg0-data_backup
/dev/mapper/vg0-data_backup: Entry 'hourly_Mon_14_mysql.tar.gz' in / (2) has deleted/unused inode 17.  CLEARED.
/dev/mapper/vg0-data_backup: Entry 'hourly_Mon_20_mysql.tar.gz' in / (2) has deleted/unused inode 18.  CLEARED.
/dev/mapper/vg0-data_backup: Entry 'hourly_Mon_18_mysql.tar.gz' in / (2) has deleted/unused inode 270338.  CLEARED.
/dev/mapper/vg0-data_backup: Entry 'hourly_Tue_06_mysql.tar.gz' in / (2) has deleted/unused inode 21.  CLEARED.
/dev/mapper/vg0-data_backup: Entry 'hourly_Mon_16_mysql.tar.gz' in / (2) has deleted/unused inode 270337.  CLEARED.

      29 inodes used (0.00%)
       0 non-contiguous inodes (0.0%)
         # of inodes with ind/dind/tind blocks: 18/18/18
20893815 blocks used (79.70%)
       0 bad blocks
      19 large files

      18 regular files
       2 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
      20 file


 e2fsck to the rescue  !!!!

2 comments: