HP Proliant DL360p Gen8 - UID LED on
So you want to add NVMe storage capability to your HPE Proliant DL360p Gen8 (or other Proliant Gen8 server) and don’t know where to start? Well, I was in the same situation until recently. However, after much research, a little bit of spending, I now have 8TB of NVMe storage in my HPE DL360p Gen8 Server thanks to the IOCREST IO-PEX40152.
Unsupported you say? Well, there are some of us who like to live life dangerously, there is also those of us with really cool homelabs. I like to think I’m the latter.
PLEASE NOTE: This is not a supported configuration. You’re doing this at your own risk. Also, note that consumer/prosumer NVME SSDs do not have PLP (Power Loss Prevention) technology. You should always use supported configurations and enterprise grade NVME SSDs in production environments.
Update – May 2nd 2021: Make sure you check out my other post where I install the IOCREST IO-PEX40152 in an HPE ML310e Gen8 v2 server for Version 2 of my NVMe Storage Server.
Update – June 21 2022: I’ve received numerous comments, chats, and questions about whether you can boot your server or computer using this method. Please note that this is all dependent on your server/computer, the BIOS/EFI, and capabilities of the system. In my specific scenario, I did not test booting since I was using the NVME drives purely as additional storage.
DISCLAIMER: If you attempt what I did in this post, you are doing it at your own risk. I won’t be held liable for any damages or issues.
There’s a number of reasons why you’d want to do this. Some of them include:
Well, after all that research I mentioned at the beginning of the post, I installed an IOCREST IO-PEX40152 inside of an HPE Proliant DL360p Gen8 to add NVMe capabilities to the server.
At first I was concerned about dimensions as technically the card did fit, but technically it didn’t. I bought it anyways, along with 4 X 2TB Sabrent Rocket 4 NVMe SSDs.
The end result?
IMPORTANT: Due to the airflow of the server, I highly recommend disconnecting and removing the fan built in to the IO-PEX40152. The DL360p server will create more than enough airflow and could cause the fan to spin up, generate electricity, and damage the card and NVME SSD.
Also, do not attempt to install the case cover, additional modification is required (see below).
Installing the card inside of the PCIe riser was easy, but snug. The metal heatsink actually comes in to contact with the metal on the PCIe riser.
You’ll notice how the card just barely fits inside of the 1U server. Some effort needs to be put in to get it installed properly.
There are ribbon cables (and plastic fittings) directly where the end of the card goes, so you need to gently push these down and push cables to the side where there’s a small amount of thin room available.
Unfortunately, just when I thought I was in the clear, I realized the case of the server cannot be installed. The metal bracket and locking mechanism on the case cover needs the space where a portion of the heatsink goes. Attempting to install this will cause it to hit the card.
The above photo shows the locking mechanism protruding out of the case cover. This will hit the card (with the IOCREST IO-PEX40152 heatsink installed). If the heatsink is removed, the case might gently touch the card in it’s unlocked and recessed position, but from my measurements clears the card when locked fully and fully closed.
I had to come up with a temporary fix while I figure out what to do. Flip the lid and weight it down.
For stability and other tests, I simply put the case cover on upside down and weighed it down with weights. Cooling is working great and even under high load I haven’t seen the SSD’s go above 38 Celsius.
The plan moving forward was to remove the IO-PEX40152 heatsink, and install individual heatsinks on the NVME SSD as well as the PEX PCIe switch chip. This should clear up enough room for the case cover to be installed properly.
I went on to Amazon and purchased the following items:
4 x GLOTRENDS M.2 NVMe SSD Heatsink for 2280 M.2 SSD
1 x BNTECHGO 4 Pcs 40mm x 40mm x 11mm Black Aluminum Heat Sink Cooling Fin
They arrived within days with Amazon Prime. I started to install them.
And now we install it in the DL360p Gen8 PCIe riser and install it in to the server.
You’ll notice it’s a nice fit! I had to compress some of the heat conductive goo on the PFX chip heatsink as the heatsink was slightly too high by 1/16th of an inch. After doing this it fit nicely.
Also, note the one of the cable/ribbon connectors by the SAS connections. I re-routed on of the cables between the SAS connectors they could be folded and lay under the card instead of pushing straight up in to the end of the card.
As I mentioned above, the locking mechanism on the case cover may come in to contact with the bottom of the IOCREST card when it’s in the unlocked and recessed position. With this setup, do not unlock the case or open the case when the server is running/plugged in as it may short the board. I have confirmed when it’s closed and locked, it clears the card. To avoid “accidents” I may come up with a non-conductive cover for the chips it hits (to the left of the fan connector on the card in the image).
And with that, we’ve closed the case on this project…
One interesting thing to note is that the NVME SSD are running around 4-6 Celsius cooler post-modification with custom heatsinks than with the stock heatsink. I believe this is due to the awesome airflow achieved in the Proliant DL360 servers.
I’ve been running this configuration for 6 days now stress-testing and it’s been working great. With the server running VMware ESXi 6.5 U3, I am able to passthrough the individual NVME SSD to virtual machines. Best of all, installing this card did not cause the fans to spin up which is often the case when using non-HPE PCIe cards.
This is the perfect mod to add NVME storage to your server, or even try out technology like VMware vSAN. I have a number of cool projects coming up using this that I’m excited to share.
While most of us frequently deploy new ESXi hosts, a question and task not oftenly discussed is how to properly decommission a VMware ESXi host. Some might be surprised to… Read More
This guide will outline the instructions to Disable the VMware Horizon Session Bar. These instructions can be used to disable the Horizon Session Bar (also known as the Horizon Client… Read More
Normally, any VMs that are NVIDIA vGPU enabled have to be manually migrated with manual vMotion if a host is placed in to maintenance mode, to evacuate the host. While… Read More
You may experience GPU issues with the VMware Horizon Indirect Display Driver in your environment when using 3rd party applications which incorrectly utilize the incorrect display adapter. This results with… Read More
Today we're going to cover a powerful little NAS being used with VMware; the Synology DS923+ VMware vSphere Use case and Configuration. This little (but powerful) NAS is perfect for… Read More
Today we'll go over how to install the vSphere vCenter Root Certificate on your client system. Certificates are designed to verify the identity of the systems, software, and/or resources we… Read More
View Comments
Hello,
This is a great tuto,
I'm in the same situation with, I have installed a NVME PCI card and installed an ESXi on it but on reboot I cannot boot on.
can you please share with me how you did boot from NVME pci card on the DL360P GEN8 ?
Best regards,
Diogène.
Hi Diogene,
I don't boot off my card, but it should be possible. You just need to set the boot priority on the server to boot which NVMe stick you want.
Cheers
Stephen
Awesome!!! Was just looking for a way to add internal ssd storage to my DL380P Gen8 to free up two 3.5-bays in my unRaid server. If this is compatible for unraid cache I am golden. Thank you for sharing. As much as I love my HP servers in the homelab, they don't like to pair with non hp components.
Hope the post helped!
That’s really great and that’s what I’m looking for. Looking at the first picture, I can’t exactly see if there is the standard 4 Port Ethernet card there (what are the black cables). You’ve a 2 Port 10Gbit card (HP 560SFP) instead in use? Do I have to remove the 4P ort card, otherwise the Iocrest card may not fit? In the underlined text you’ve mentioned a critical situation which I can’t follow. What exactly is the situation there? What do you mean with “I have confirmed when it’s closed and locked, it clears the card. To avoid accidents I may come up with a non-conductive cover…”
I went through the Iocrest documentation, however, dou you know if I can mix Size (TB) and NVME manufacture on the board?
Hi itarch,
In the first image, I have two servers stacked. The black cables are SFP+ DAC cables going to the server below the server that's being discussed in this post.
As for your 2nd question, due to the size of the card and the heatsink that ships with it, the heatsink get's in the way of the mechanics of the case. With the standard IOCREST heatsink installed, you cannot close the case as it's in the way of the locking pin. Without the heatsink, it may come in contact with the board, so I'd recommend powering off the server when opening or closing the case to avoid it rubbing against the card.
The card is simply a PCIe card with a PCIe switch chip. You can use any mix of size and manufacturer you'd like.
Cheers,
Stephen
Hi Stephen,
First, I would like to thank you for the post (and of course for your feedback) and of course wish you Merry Christmas. In my first comment, very focused on the technical stuff, I forgot to appreciate your effort, sorry for that.
Yes, I took a look inside the server (DL360p Gen8) and understand now what you mean (Pic 5, the locking pin). And even after modifications there is still potential danger to scratch the IOcrest board when opening the case...
Now, I’m googling around to find this board and the only option is Aliexpress, where I’ve never bought before and have to overcome now …
Cheers,
Itarch
Hi itarch,
I'm glad if the post helps! Merry Christmas!!!
As for purchasing the card, I also purchased it from Aliexpress. If you use the link in my other post, it goes directly to the IOCREST genuine store. It's safe, painless, and I received the card very quickly.
Cheers,
Stephen
Hello! I have a problem. Installed NVMe m.2 samsung 970 evo plus via pci adapter. But the server doesn't see the pci. In iLo4, there is nothing on the pci slot. Do you know how to fix the situation? Don't scold. Translated by Google.
Hi eres,
The IOCREST card should not be seen inside of iLO. The PCI card simply has a PCIe switch chip. This is normal and expected behaviour.
Cheers,
Stephen
Hello Stephen,
Great article, exactly what I was hoping to find. Did you run any benchmarks on the card that you can share?
Hi Stefanos,
Inside of the post, there's a few links to other posts discussing the card and NVMe drives. I believe there should be some benchmarks on those posts. In all cases, the limiting factor for me was the server hardware (CPU), lol. I haven't been able to max out the NVMe drives or the card as of yet.
Cheers,
Stephen
Hi Stephen,
great article...but you should also describe the basic enviroment (bios version and configuration )
I'm trying to do the same with a single nvme (samsung 970) and a pcie adapter (ATLANTIS LAND A06) but I cannot understand how to set the SDD as boot drive because It doesn't appear into the bios, I've already tryied any pcie slot....Any suggestion please?
Thanks Carette,
On older systems you should expect that you won't be able to boot off an NVME drive. You might be able off of one, but with this NVMe card you won't be able to if you're using software RAID unless you have a different source to boot.
Cheers,
Stepehn
Hey all and hey Stephen!
Just to add to Stephen's great write-up. I can confirm the IO-PEX40152 and the IO-PEX40129 both work in DL380p G8 and DL380 G9 machines. I also can confirm the SI-PEX40152 and SI-PEX40129 (Syba branded / US based IO distributor on Amazon) works as well. I can also say that the StarTech.com does as well. I've tested a few others and can confirm Switchtec PFX-L 32xG3 and ASMedia's ASM2824 chipsets for bifurcation work on these devices, so I would imagine other cards (Asus Hyper, etc) work as long as they have these chipsets (might be more, but those are only two I've checked personally).
The one thing to ABSOLUTELY sure on is if the storage does NOT appear in ESXi, or in my case, booting to Ubuntu live and doing a GPARTED or lsblk via terminal, then don't sit and try to troubleshoot why its not showing up. The cards work great IF the NVME's are working. If the slots don't populate, get a different M.2 / NVME drive. I went through 2-3 off the shelf Amazon offerings and they didn't show up, but as soon as I switched to Sabrent or Samsung, the drives started showing. I don't know if some of the cheaper NVME's don't recognize, or maybe they are junk (they didn't even show up on my gaming motherboard in Windows).
I can also confirm on DL380 G9's that with the latest SPP (2020 or 2021 SPPs) that the M.2 NVME to PCIe 3.0 adapters you find on Amazon work directly. Granted you only get 1 per slot, if you want to save a few bucks, they will work! One box I have is running 2x of these (your mileage may very): https://www.amazon.com/gp/product/B07JJTVGZM
Thanks again Stephen for doing a ton of the legwork on this, definitely a great resource!
Hey Jon,
Thanks for taking the time to leave a comment and for that excellent write up on the other cards and alternatives! :) It's much appreciated and I'm sure it'll help a ton of other people!
It's interesting how many times I've been contacting with issues regarding bad NVMe drives. I'm wondering if there's a big batch of faulty ones, or if this might be drives used for CHIA mining that are circulating after being used.
Either way, I'm glad you're up and running! Feel free to keep us updated with what you're working on!
Cheers,
Stephen
Hi guys!
So is it confirmed that possible to boot from this pcie - ssd expansion card from a hp DL 380 G8?
I have the same server! it looks great, instead of that I'm adding a GPU, I just wondered, why not removing the lock from the cover?
Before I start purchasing anything. Would it be possible to boot from the PCI-E NVME from a HPE DL 380p Gen8?
Hi Ninja,
I'm not sure. I didn't test it, but one of the NVME did show up on the boot order list (if I remember correctly). However I'd recommend against booting even if it does work.
If there's is a chance to boot from nvme and you do not recommend to boot from nvme then where is point to install nvme. And if we can't use nvme as boot then they are useless to keep them just a storage. It's like waste of time and money to increase performance for server. I use SSD on my server to get better performance. And all backup are stored on HDD.
Hi Shadow,
Typically in servers the OS or hypervisor is loaded on to a small drive, USB key, or flash stick.
The storage (and data) is usually separate and loaded on to high performance media (such as Flash, NVMe, SSD).
This is why we install the NVMe in to servers and don't necessarily care about booting from it, as it's usually incorporated in to the design.
Cheers,
Stephen
Hi Stephen Wagner
Your training is very useful. Thank you
I have two questions? 1- Is it possible to replace the heatsink with the following model because there does not seem to be enough space from the top and sides to be able to use all 4 slots
https://www.amazon.com/dp/B07KDDKDNN?th=1&psc=1
2- Is it possible to use RAID in this model and method or not?
Thanks again
Hi Milad,
I'm not sure about what you mean with space, but if you do choose to remove the PCIe Card heatsink, and use your own, you'll also need to find a way to cool the card.
If you're installing this in a normal computer, just use the factory PCIe card heatsink and cooling as it works great. I only had to use my own custom solution because of the specific server I was using.
To use RAID, you'll need to utilize a software RAID solution. The PCIe card only provides passthrough.
Cheers,
Stephen
Hi im tried use HP Z Turbo Drive Quad Pro with two Samsung 1TB 980 PCIe 3.0 x4 M.2 nvme drives, but I can't get the OS to recognize the hard drives. should something be activated in the bios? or any special recommendation
Hi Ruben,
Sorry but I'm unfamiliar with the HP Z Turbo Drive. I'd recommend checking their documentation as it might have the answers you're looking for.
Cheers,
Stephen
NVMe NOT showing up after upgrading from ESXi 6.5U2 may be due to consumer drives not supporting NVMe protocol 1.3 (ie. not on the VMware HCL). I personally replaced the NVMe driver in U3 with the one from U2. My procedure was: https://williamlam.com/2019/05/quick-tip-crucial-nvme-ssd-not-recognized-by-esxi-6-7.html#comment-58645
NVMe Protocol
Samsung 970 EVO 500Gb 1.3
Seagate 510 1Tb 1.3
Intel 660p 1Tb No
NB Technically, the 660p supports 1.3 but only appeared in ESXi after 1.2 driver downgrade
Hope this saves someone from any trial & error NVMe buying