Nov 042018
 

This weekend I came across a big issue with my HPe MSA 2040 where one of the SAN controllers became unresponsive, and appeared to had failed because it would not boot.

It all started when I decided to clean the MSA SAN. I try to clean the components once or twice a year to remove dust and make sure it’s not getting all jammed up. Sometimes I’ll shut the entire unit down and remove the individual components, other times I’ll remove them while operating. Because of the redundancies and since I have two controllers, I can remove and clean each controller individually at separate times.

Please Note: When dusting equipment with fans, never allow the fans to spin up with compressed air as this can generate current which can damage components. Never allow compressed air flow to spin up fans.

After cleaning out the power supply’s, it came time to clean the controllers.

The Problem

As always, I logged in to the SMU to shutdown controller A (storage). I shut it down, the blue LED illuminated it was safe for removal. I then proceeded to remove it, clean it, and re-insert it. The controller came back online, and ownership of the applicable disk groups were successfully moved back. Controller A was now completed successfully. I continued to do the same for controller B: I logged in to shutdown controller B (storage). It shut down just like controller A, the blue LED removable light illuminated. I was able to remove it, clean it, and re-insert it.

However, controller B did not come back online.

After inserting controller B, the status light was flashing (as if it was booting). I waited 20 minutes with no change. The SMU on controller B was responding to HTTPS requests, however you could not log on due to the error “system is initializing”. SSH was functioning and you could log in and issue commands, however any command to get information would return “Please wait while this information is pulled from the MC controller”, and ultimately fail. The SMU on controller A would report a controller fault on controller B, and not provide any other information (including port status on controller B).

I then tried to re-seat the controller with the array still running. Gave it plenty of time with no effect.

I then removed the failed controller, shutdown the unit, powered it back on (only with controller A), and re-inserted Controller B. Again, no effect.

The Fix

At this point I’m thinking the controller may have failed or died during the cleaning process. I was just about to call HPe support for a replacement until I noticed the “Power LED” light inside of the failed controller would flash every 5 seconds while removed.

This made me start to wonder if there was an issue writing the cache to the compact flash card, or if the controller was still running off battery power but had completely frozen.

I tried these 3 things on the failed controller while it was unplugged and removed:

  1. I left the controller untouched for 1 hour out of the array (to maybe let it finish whatever it was doing while on battery power)
  2. There’s an unlabeled button on the back of the controller. As a last resort (thinking it was a reset button), I pressed and held it for 20 seconds, waited a minute, then briefly pressed it for 1 second while it was out of the unit.
  3. I removed the Compact Flash card from the controller for 1 minute, then re-inserted it. In hoping this would fail the cache copy if it was stuck in the process of writing cache to compact flash.

I then re-inserted the controller, and it booted fine! It was not functioning and working (and came up very fast). Looking at the logs, it has no record of what occurred between the first shutdown, and final boot. I hope this post helps someone else with the same issue, it can save you a support ticket, and time with a controller down.

Disclaimer

PLEASE NOTE: I could not find any information on the unlabeled button on the controller, and it’s hard to know exactly what it does. Perform this at your own risk (make sure you have a backup). Since I have 2 controllers, and my MSA 2040 was running fine on Controller A, I felt comfortable doing this, as if this did reset controller B, the configuration would replicate back from controller A. I would not do this in a single controller environment.

Update – 24 Hours later

After I got everything up and running, I checked the logs of the unit and couldn’t find anything on controller B that looked out of ordinary. However, 24 hours later, I logged back in and noticed some new events showed up from the day before (from the day I had the issues):

MSA 2040 Code 549

MSA 2040 Code 549

You’ll notice the event log with severity error:

Recovery from internal processor fault detected on controller.
Code 549

One thing that’s very odd is that I know for a fact the time is wrong on the error severity log entry, this could be due to the fact we had a daylight savings time change last night at midnight. Either way it appears that it finally did detect that the Storage controller was in an error state and logged it, but it still would have been nice for some more information.

On a final note, the unit has been running perfectly for over 24 hours.

Update – April 2nd 2019

Well, in March a new firmware update was released for the MSA. I went to upgrade and the same issue as above occurred. During the firmware upgrade, at one point of the firmware update process a step had failed and repeated 4 times until successful.

The firmware update log (below was repeated):

Updating system configuration files
System configuration complete
Loading SC firmware.
STATUS: Updating Storage Controller firmware.
Waiting 5 seconds for SC to shutdown.
Shutdown of SC successful.
Sending new firmware to SC.
Updating SC Image:Remaining size 6263505
Updating SC Image:Remaining size 5935825
Updating SC Image:Remaining size 5608145
Updating SC Image:Remaining size 5280465
Updating SC Image:Remaining size 4952785
Updating SC Image:Remaining size 4625105
Updating SC Image:Remaining size 4297425
Updating SC Image:Remaining size 3969745
Updating SC Image:Remaining size 3642065
Updating SC Image:Remaining size 3314385
Updating SC Image:Remaining size 2986705
Updating SC Image:Remaining size 2659025
Updating SC Image:Remaining size 2331345
Updating SC Image:Remaining size 2003665
Updating SC Image:Remaining size 1675985
Updating SC Image:Remaining size 1348305
Updating SC Image:Remaining size 1020625
Updating SC Image:Remaining size 692945
Updating SC Image:Remaining size 365265
Updating SC Image:Remaining size 37585
Waiting for Storage Controller to complete programming.
Please wait...
Please wait...
Please wait...
Please wait...
Storage Controller has completed programming.
Got an error (138) on firmware packet
CAPI error: Firmware Update failed. Controller needs to reboot.
Waiting 5 seconds for SC to shutdown.
Shutdown of SC successful.
Sending new firmware to SC.
Updating SC Image:Remaining size 6263505
Updating SC Image:Remaining size 5935825
Updating SC Image:Remaining size 5608145
Updating SC Image:Remaining size 5280465
Updating SC Image:Remaining size 4952785
Updating SC Image:Remaining size 4625105
Updating SC Image:Remaining size 4297425
Updating SC Image:Remaining size 3969745
Updating SC Image:Remaining size 3642065
Updating SC Image:Remaining size 3314385
Updating SC Image:Remaining size 2986705
Updating SC Image:Remaining size 2659025
Updating SC Image:Remaining size 2331345
Updating SC Image:Remaining size 2003665
Updating SC Image:Remaining size 1675985
Updating SC Image:Remaining size 1348305
Updating SC Image:Remaining size 1020625
Updating SC Image:Remaining size 692945
Updating SC Image:Remaining size 365265
Updating SC Image:Remaining size 37585
Waiting for Storage Controller to complete programming.
Please wait...
Please wait...
Storage Controller has completed programming.
Got an error (138) on firmware packet
CAPI error: Firmware Update failed. Controller needs to reboot.
Waiting 5 seconds for SC to shutdown.
Shutdown of SC successful.
Sending new firmware to SC.
Updating SC Image:Remaining size 6263505
Updating SC Image:Remaining size 5935825
Updating SC Image:Remaining size 5608145
Updating SC Image:Remaining size 5280465
Updating SC Image:Remaining size 4952785
Updating SC Image:Remaining size 4625105
Updating SC Image:Remaining size 4297425
Updating SC Image:Remaining size 3969745
Updating SC Image:Remaining size 3642065
Updating SC Image:Remaining size 3314385
Updating SC Image:Remaining size 2986705
Updating SC Image:Remaining size 2659025
Updating SC Image:Remaining size 2331345
Updating SC Image:Remaining size 2003665
Updating SC Image:Remaining size 1675985
Updating SC Image:Remaining size 1348305
Updating SC Image:Remaining size 1020625
Updating SC Image:Remaining size 692945
Updating SC Image:Remaining size 365265
Updating SC Image:Remaining size 37585
Waiting for Storage Controller to complete programming.
Please wait...
Please wait...
Storage Controller has completed programming.
Got an error (138) on firmware packet
CAPI error: Firmware Update failed. Controller needs to reboot.
Waiting 5 seconds for SC to shutdown.
Shutdown of SC successful.
Sending new firmware to SC.
Updating SC Image:Remaining size 6263505
Updating SC Image:Remaining size 5935825
Updating SC Image:Remaining size 5608145
Updating SC Image:Remaining size 5280465
Updating SC Image:Remaining size 4952785
Updating SC Image:Remaining size 4625105
Updating SC Image:Remaining size 4297425
Updating SC Image:Remaining size 3969745
Updating SC Image:Remaining size 3642065
Updating SC Image:Remaining size 3314385
Updating SC Image:Remaining size 2986705
Updating SC Image:Remaining size 2659025
Updating SC Image:Remaining size 2331345
Updating SC Image:Remaining size 2003665
Updating SC Image:Remaining size 1675985
Updating SC Image:Remaining size 1348305
Updating SC Image:Remaining size 1020625
Updating SC Image:Remaining size 692945
Updating SC Image:Remaining size 365265
Updating SC Image:Remaining size 37585
Waiting for Storage Controller to complete programming.
Please wait...
Please wait...
Storage Controller has completed programming.
Updating SC Image:Remaining size 0
Storage Controller has been successfully updated.
STATUS: Current CPLD firmware is up-to-date.
CPLD update not required.
==========================================
Software Component Load Summary:
MC Software:    SUCCESSFUL
SC Software:    SUCCESSFUL
EC Software:    NOT ATTEMPTED
CPLD Software:  NOT ATTEMPTED
==========================================

During the Storage Controller restarting process, the controller never came back up. I removed the controller 1 hour, re-inserted and the above fix did not work. I then tried it after 2 hours of disconnection.

At this point I contacted HPe, who is sending a replacement controller.

The following day (12 hours of controller removed), I re-inserted it again and it actually booted up, was working with the new firmware, and then did a PFU (Partner Firmware Update) of controller A.

While it is working now, I’m still going to replace the controller as I believe something is not functioning correctly.

  15 Responses to “HPe MSA 2040 Unresponsive, Dead, or Failed Controller, Controller won’t boot”

  1. Thank You for the great post it saved my life ๐Ÿ™‚
    I had exactly the same issue. It has begun with a host port IP setting.
    After a few minutes controller B showed up the CopactFlash fault.
    First i tryed to reboot the controller and aftert this step instant controller failure.
    After it i unplugged the controller ad did the reset via your advice and a miracle is happened ๐Ÿ™‚

  2. Hi Barnabas,

    Glad to hear it helped! Always be careful though, doing this will corrupt or clear the flash and compact flash on the controller… Only do this if the other controller is valid, responding, functioning properly, and has an OK status.

    Cheers,
    Stephen

  3. Dear Steve,
    We are migrating from Netapp to a HPE MSA 2050. I’ve just started to work with it and I got the error message. I have no experience with the HPE, but it looks fairly straight forward in installing, much better then the Netapp. I will be visiting your blog in the next couple of weeks whilst setting up the HPE. ๐Ÿ˜‰
    By the way is there a big difference between the 2040 and 2050?

    Have a great day,
    Andrรฉ

  4. Glad to hear, the MSA storage arrays are beautiful products!

    There is a few differences, the main being that the 205x series are the next generation agyer the 204x series.

  5. Hi Stephen! My name is Alex, i got some problem with my MSA, I hope You can give me good advice. So, i Have two MC on HP MSA, each of MC connected via FibreChannel modules to HP Proliant Servers throught SAN switch. And there are dozen of VMs on servers. I need to replace MC B, without stopping (power off) whole MSA. How do You think, is it possible without data loss?

  6. Hi Alex,

    As long as you configured the unit properly, and have the proper channels of redundancy (which you likely do, if the controller has failed and everything is still working), you just simply follow the standard procedures for swapping the controller.

    No data loss should occur if the unit was configured properly, and if you follow standard procedures.

    Stephen

  7. Stephen, thanx for fast answer, yeah, looks like everything configured good, but i have to be sure. Well, i do not really know, witch MC is bad, and a problem i have – a can’t see any mappings via web-interface or CLI, but DG, volumes and other stuff i can see clear. So, HP sent me new MC, i whatnt to make hot replace, but data in MSA very important…

  8. You need to identify which controller is bad first.

    Have you logged on to the SMU on both controllers? One should have good information.

  9. No, i can\t see any mappings via A and via B MC. And in CLI – too, but “show volume-maps” – still works. And other one symptom – web-interface often unavailable, just disconnecting.

  10. Have you tried to restart one of the management controllers (not the storage controller)?

    What did HPe advise?

    Are there any amber lights on the back of the storage controllers?

  11. There is no any amber lights. All HP recommendations i made at all (restart of MSA, FW upgrade and downgrade, etc,) – no result. They sent me new MC and wants me to replace it, but i cant stop MSA in september…

  12. Did you restart the Storage Controllers, or Management Controllers?

    The storage controllers handle the actual storage, whereas the management controllers provide the CLI and SMU.

    Sometimes the management controllers can crash, but everything still works. In this case it would require a restart of only the management controllers.

    However, if you’ve restarted the SC’s, the MCs, and the entire SAN with no effect, I’m wondering if there could be an issue with the midplane or something.

    When you log on to vSphere, are all paths active? If some paths are offline, you can use this to determine which storage controller may be problematic. In this case, if there are paths offline, I’d replace the storage controller that has the failed paths.

  13. Yes, Stephen, i restarted MCs and SCs, and all the paths online…HPe 2 and 3 level support do not have descision still. As for me – some issue, that common for MCs: Compact Flash card trouble, Firmware trouble…Maybe the fact important that I use virtual volumes.

  14. Hi Stephen,

    I have HP MSA 2040 storage which we are going to repurpose. The device is stacked but I am unable to take its serial and it always says COM4/7 doesn’t exists. So, I reset the device and in next attempt I was able to connect through serial port and when i tried to reset it to default settings it started giving me below error.

    (none) login: restoredefaults
    Password:

    2019-09-09 14:59:05: Running MC-only restoredefaults on partner controller.
    2019-09-09 14:59:05: Running MC-only restoredefaults on this controller.
    2019-09-09 14:59:05: Perform Restore Defaults
    /app/default/restoreDefaults.sh: source: line 96: can’t open ‘/mem/relinfo/features.env’

    So I dont know its password and i cant reset it as well. Any idea how to fix it?

    Regards,
    Sandeep

  15. Hi Sandeep,

    Can you try reflashing the firmware using the restoredefaults account? I’m not familiar with that account/method, but I’m curious if it’s possible.

    Do you have active warranty? I’d check the serial number, and if it has warranty, try submitting a ticket with HPe.

    Cheers,
    Stephen

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)