Well, recently one of the servers I monitor and maintain in a remote oil town recently started throwing out a Windows event log warning:
Event ID: 129
Source: HpCISSs2
Description: Reset to device, \Device\RaidPort0, was issued.
The server is an HP ML350p Gen8 (Windows Server 2008 R2) running latest firmware and management software. It has 2 RAID Arrays (RAID1, and RAID5), and a total of 6 disks.
Researching this error, I read that most people had this occur when running the latest HP WBEM providers, as well as anti-virus software. In our case, I actually tried to downgrade to an older version, but noticed the warning still occurs. While we do have anti-virus, it’s not actively scanning (only weekly scheduled scans).
In the process of troubleshooting, I noticed that under the HP Systems Management Homepage, one of the drives in the RAID1 array, had the following stats:
Hard Read Erros: 150
Recovery Read Errors: 7
Total Seeks: 0
Seek Errors: 0
I found these numbers to be very high in my experience. None of the other drives had anything close to this (in 4 years of running, only one other disk had a read error (a single one), this disk however had tons. For some reason the drive is still reporting as operational, when I’d expect it to be marked as a predicted failure, or failed.
While all online documentation was pointing towards at locks on the array by software, from my own experience I think it was actually the array waiting for a read operation on the array, and it was this single disk that was causing a threshold to be hit in the driver, that caused a retry to recover the read operation.
Called up HPE support, I mentioned I’d like to have the drive replaced. The support engineer consulted her senior engineer and reviewed the evidence I presented (along with ADU reports, and Active Monitoring health reports), the senior engineer concurred that the drive should be replaced.
Replacing the drive resolved the issue. I’m also noticing a performance increase on the array as well.
Make sure to always check the stats on the individual components of your RAID arrays, even if everything is operating sound.
While most of us frequently deploy new ESXi hosts, a question and task not oftenly discussed is how to properly decommission a VMware ESXi host. Some might be surprised to… Read More
This guide will outline the instructions to Disable the VMware Horizon Session Bar. These instructions can be used to disable the Horizon Session Bar (also known as the Horizon Client… Read More
Normally, any VMs that are NVIDIA vGPU enabled have to be manually migrated with manual vMotion if a host is placed in to maintenance mode, to evacuate the host. While… Read More
You may experience GPU issues with the VMware Horizon Indirect Display Driver in your environment when using 3rd party applications which incorrectly utilize the incorrect display adapter. This results with… Read More
Today we're going to cover a powerful little NAS being used with VMware; the Synology DS923+ VMware vSphere Use case and Configuration. This little (but powerful) NAS is perfect for… Read More
Today we'll go over how to install the vSphere vCenter Root Certificate on your client system. Certificates are designed to verify the identity of the systems, software, and/or resources we… Read More
View Comments
I found this to be the issue and solution.
https://blogs.technet.microsoft.com/kevinholman/2013/06/21/event-id-129-storachi-reset-to-device-deviceraidport0-was-issued/
Hi Erik,
Since the time of this original post, I've discovered that this is a very generic error/warning, and can be caused by a whole bunch of different issues.
Essentially it's a warning that something isn't right with the storage system. It could be hardware failure, hardware misconfiguration, issues with systems that the hardware uses (such as the issue you posted in your comment), failed drives, back planes, even malfunctioning drivers...
I had this similar error come up on a Windows 2012 terminal server and it had me stumped for days. File explorer was slow and the RDP client took minutes to load and would ultimately fail. I went and reviewed the Windows system logs for the very first instance of this warning and right before the very first one was an Event ID 7 error "CdRom0, has a bad block". I initially ignored this error thinking that it was harmless. When I noticed that error on second viewing a couple days later, I went to the server and ejected a disc that had been in the server for some time. Upon doing that the 129 errors stopped and the system responded as expected.
First time I've ever solved a problem of that nature simply by ejecting a CD from the CD ROM drive.
Hopefully this helps some people who were as lost as I was.
thanks for the heads-up! one drive had 18000 read & 4 writes & 2 rebuilds, but was still listed as "OK" in the SSA/iLo. Crazy!!