Virtualization Team

vCenter Appliance fsck failed

While I have been working with a customer on upgrading their infrastructure to vSphere 5.5, they had a power outage during the night which got their full virtual infrastructure down including storage (Not sure what the story with their UPS/Power generator). After they have brought their virtual infrastructure backup up both their vCenter Appliance and VDP appliance and few other Linux VMs has failed the fsck check while it was booting and it was stuck at the following error:

=========================================

fsck failed.  Please repair manually and reboot.  The root
file system is currently mounted read-only.  To remount it
read-write do:

bash# mount -n -o remount,rw /

Atention:  Only CONTROL-D will reboot the system in this
maintanance mode.  shutdow or reboot will not work.

Give root password for maintenance
(or type Control-D to continue):

==========================================

Below is an actual screenshot of the error for you to compare it with what you got:

VMware vCSA fsck failed

Below is the steps I have followed to fix this on both vCSA and VDP Advanced:

  1. Stay Calm & Don’t Panic!
  2. Take a backup/snapshot in case the below steps does not work for you and you need to revert it back to the current state when you call support or call for help. (Hopefully you won’t need this, but it is like your car insurance policy you pay for it but you hope you will never need to use it).
  3. Enter your Root Password.
  4. PRESS ENTER
  5. Type: mount -n -o remount,rw
  6. PRESS ENTER
  7. You will get some complain and that you should check with: e2fsck  (Check the below screenshot to see what it looks like)
  8. Type: e2fsck -y /dev/sda3
  9. PRESS ENTER
  10. It will check for error and ask you if you want to fix errors (Make sure you answer with yes to fix each of the errors).
  11. press: CTRL+D to exit
  12. press: CTRL+D to  restart the system.
  13. This should get your vSphere Data Protection Appliance or vCenter Appliance into an operational state and should get you back on track.

The below screenshots show how I executed the above procedure and the expected output:

VMware vCSA running e2fsck

 e2fsck disk repair completed

As I mentioned earlier I was able to use the same procedure above to fix:

  • vCenter Appliance (vCSA) fsck failed error
  • vSphere Data Protection (VDP) fsck failed error
  • Other Linux appliance and VMs that had the same fsck failed error.

While at my customer this error was caused by a power outage, this same error can happen after any type of any non proper shutdown of your Linux VMs. Hope this help get you back on track. If this fix your problem please share your result in the comment. If it does not and you had to do things differently, please share it in the comments below so every one can benefit.

Eiad Al-Aqqad

,

35 responses to “vCenter Appliance fsck failed”

  4. Thank you for the great write up. It was an exact keyword match with my issue. But unfortunately, the client does not have the root password for the VDP appliance. I attempted to reset the VDP password using this link:
    http://www.vladan.fr/how-to-reset-root-password-in-vdp/

    But that failed as the file system came up in read only mode. I tried the default “changeme” password but unfortunately – that didn’t work either. So the implementation engineer must have changed the root password -but it was not properly documented.

    Amir

  5. Hi,

    I get exactly this error when deploying a brand new vMA from OVF. I can’t enter a root password as I have never configured one. Any hints?

    Thanks in advance

  7. > 5. Type: mount -n -o remount,rw
    This should give you an error – no mount point parameter. And also you should not do re-mount in R/W if you want to make fsck – this can be dangerous depending which file system should be fixed.

  8. Very nice. This helped tremendously today. Now life is good again with vCenter.

  10. THanks for this!

    Just as a note, my vcenter did not boot properly after fixinf sda3, needed to check the rest too 🙂

  13. Thanks – this fixed my issue. I’m currently running a lab environment and every time I make progress on the course I’m following, I cleanly shut down the appliance to take a snapshot. Without fail, every time I do this I get this error on boot up again – weird. This has saved me a a massive headache. Thanks again.

  14. Worked like a champ for my home lab following a power outage. This blog was at the top of my Bing search (yes, Bing) and I’ve shared with my co-workers who have also occasionally experienced a failed vCenter Server Appliance fsck failure.

    Thank you for the great write up!

    Dave

  18. Just needed to use this with a vCenter 6 installation that experienced a power outage. The “shell.set –enabled true” command wasn’t working and I had no SSH access so I followed the second part of KB2069041 @ http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069041 in order to get booted directly into bash. Forwhat ever reason I needed to go through this process twice until e2fsck said that the volume was clean. Thanks for your help.

  20. Found this today after running into same problem on my VCSA6 VM, thanks for the write up.

  23. You rock Brother. Had a SAN Storage disconnect that poot me in the fsck failed loop.
    Thanks for awsome posts!

  26. My root password isn’t working. I reinstalled vcenter 6 few times and it get stuck here and then my root password for maintenance don’t work.

  27. You just saved my arse with a powersupply that killed my host. When I replaced it, I got this exact error. Your steps years later did the trick and my lab environment is coming back online.

    Thank you sir,

    Ron

  29. This article saved my lab! My NAS/SAN died and after I brought it back up, I was getting this error. I had shell disabled for my VCSA instance, so I had to do a workaround of getting into shell through modifying the boot process and then run these commands. It ended up working in the end and saved my bacon. For instructions on how to get into shell from the boot menu, I used an article on resetting the root password for when it gets locked out (but I didn’t reset the root password.)

  30. i got FSCK error when i powered on VCSA… i unable to enter the root password… i just typed it but says login failed.. any one help me in that

  33. Hi, i followed your steps in this block but very unfortunatly mine didn’t work. after typing:
    “e2fsck -y /dev/sda3” it says No such file or directory while trying to open /dev/sda3.
    Then it says something like this “The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193

    I am running suse 11.1
    Please help, i have been trying to get this work for some days now.

  34. What if you forgotten the root password at this time..? Is there any way to solve this problem when you don’t know the root password?

Leave a Reply

Your email address will not be published. Required fields are marked *