May 262020
 

So you want to add NVMe storage capability to your HPE Proliant DL360p Gen8 (or other Proliant Gen8 server) and don’t know where to start? Well, I was in the same situation until recently. However, after much research, a little bit of spending, I now have 8TB of NVMe storage in my HPE DL360p Gen8 Server thanks to the IOCREST IO-PEX40152.

Unsupported you say? Well, there are some of us who like to live life dangerously, there is also those of us with really cool homelabs. I like to think I’m the latter.

PLEASE NOTE: This is not a supported configuration. You’re doing this at your own risk. Also, note that consumer/prosumer NVME SSDs do not have PLP (Power Loss Prevention) technology. You should always use supported configurations and enterprise grade NVME SSDs in production environments.

DISCLAIMER: If you attempt what I did in this post, you are doing it at your own risk. I won’t be held liable for any damages or issues.

Use Cases

There’s a number of reasons why you’d want to do this. Some of them include:

  • Server Storage
  • VMware Storage
  • VMware vSAN
  • Virtualized Storage (SDS as example)
  • VDI
  • Flash Cache
  • Special applications (database, high IO)

Adding NVMe capability

Well, after all that research I mentioned at the beginning of the post, I installed an IOCREST IO-PEX40152 inside of an HPE Proliant DL360p Gen8 to add NVMe capabilities to the server.

IOCREST IO-PEX40152 with 4 x 2TB Sabrent Rocket 4 NVME

At first I was concerned about dimensions as technically the card did fit, but technically it didn’t. I bought it anyways, along with 4 X 2TB Sabrent Rocket 4 NVMe SSDs.

The end result?

Picture of an HPE DL360p Gen8 with NVME SSD
HPE DL360p Gen8 with NVME SSD

IMPORTANT: Due to the airflow of the server, I highly recommend disconnecting and removing the fan built in to the IO-PEX40152. The DL360p server will create more than enough airflow and could cause the fan to spin up, generate electricity, and damage the card and NVME SSD.

Also, do not attempt to install the case cover, additional modification is required (see below).

The Fit

Installing the card inside of the PCIe riser was easy, but snug. The metal heatsink actually comes in to contact with the metal on the PCIe riser.

Picture of an IO-PEX40152 installed on DL360p PCIe Riser
IO-PEX40152 installed on DL360p PCIe Riser

You’ll notice how the card just barely fits inside of the 1U server. Some effort needs to be put in to get it installed properly.

Picture of an DL360p Gen8 1U Rack Server with IO-PEX40152 Installed
HPE DL360p Gen8 with IO-PEX40152 Installed

There are ribbon cables (and plastic fittings) directly where the end of the card goes, so you need to gently push these down and push cables to the side where there’s a small amount of thin room available.

We can’t put the case back on… Yet!

Unfortunately, just when I thought I was in the clear, I realized the case of the server cannot be installed. The metal bracket and locking mechanism on the case cover needs the space where a portion of the heatsink goes. Attempting to install this will cause it to hit the card.

Picture of the HPE DL360p Gen8 Case Locking Mechanism
HPE DL360p Gen8 Case Locking Mechanism

The above photo shows the locking mechanism protruding out of the case cover. This will hit the card (with the IOCREST IO-PEX40152 heatsink installed). If the heatsink is removed, the case might gently touch the card in it’s unlocked and recessed position, but from my measurements clears the card when locked fully and fully closed.

I had to come up with a temporary fix while I figure out what to do. Flip the lid and weight it down.

Picture of an HPE DL360p Gen8 case cover upside down
HPE DL360p Gen8 case cover upside down

For stability and other tests, I simply put the case cover on upside down and weighed it down with weights. Cooling is working great and even under high load I haven’t seen the SSD’s go above 38 Celsius.

The plan moving forward was to remove the IO-PEX40152 heatsink, and install individual heatsinks on the NVME SSD as well as the PEX PCIe switch chip. This should clear up enough room for the case cover to be installed properly.

The fix

I went on to Amazon and purchased the following items:

4 x GLOTRENDS M.2 NVMe SSD Heatsink for 2280 M.2 SSD

1 x BNTECHGO 4 Pcs 40mm x 40mm x 11mm Black Aluminum Heat Sink Cooling Fin

They arrived within days with Amazon Prime. I started to install them.

Picture of Installing GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
Installing GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
Picture of IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME
IOCREST IO-PEX40152 with GLOTRENDS M.2 NVMe SSD Heatsink on Sabrent Rocket 4 NVME

And now we install it in the DL360p Gen8 PCIe riser and install it in to the server.

You’ll notice it’s a nice fit! I had to compress some of the heat conductive goo on the PFX chip heatsink as the heatsink was slightly too high by 1/16th of an inch. After doing this it fit nicely.

Also, note the one of the cable/ribbon connectors by the SAS connections. I re-routed on of the cables between the SAS connectors they could be folded and lay under the card instead of pushing straight up in to the end of the card.

As I mentioned above, the locking mechanism on the case cover may come in to contact with the bottom of the IOCREST card when it’s in the unlocked and recessed position. With this setup, do not unlock the case or open the case when the server is running/plugged in as it may short the board. I have confirmed when it’s closed and locked, it clears the card. To avoid “accidents” I may come up with a non-conductive cover for the chips it hits (to the left of the fan connector on the card in the image).

And with that, we’ve closed the case on this project…

Picture of a HPE DL360p Gen8 Case Closed
HPE DL360p Gen8 Case Closed

One interesting thing to note is that the NVME SSD are running around 4-6 Celsius cooler post-modification with custom heatsinks than with the stock heatsink. I believe this is due to the awesome airflow achieved in the Proliant DL360 servers.

Conclusion

I’ve been running this configuration for 6 days now stress-testing and it’s been working great. With the server running VMware ESXi 6.5 U3, I am able to passthrough the individual NVME SSD to virtual machines. Best of all, installing this card did not cause the fans to spin up which is often the case when using non-HPE PCIe cards.

This is the perfect mod to add NVME storage to your server, or even try out technology like VMware vSAN. I have a number of cool projects coming up using this that I’m excited to share.

  13 Responses to “Add NVMe capability to an HPE Proliant DL360p Gen8 Server”

  1. […] In this post I’ll be reviewing the IOCREST IO-PEX40152, providing information on how to buy, benchmarks, installation, configuration and more! I’ve also posted tons of pics for your viewing pleasure. I installed this card in an HPE DL360p Gen8 server to add NVME capabilities. […]

  2. Hello,

    This is a great tuto,
    I’m in the same situation with, I have installed a NVME PCI card and installed an ESXi on it but on reboot I cannot boot on.

    can you please share with me how you did boot from NVME pci card on the DL360P GEN8 ?

    Best regards,
    Diogène.

  3. Hi Diogene,

    I don’t boot off my card, but it should be possible. You just need to set the boot priority on the server to boot which NVMe stick you want.

    Cheers
    Stephen

  4. Awesome!!! Was just looking for a way to add internal ssd storage to my DL380P Gen8 to free up two 3.5-bays in my unRaid server. If this is compatible for unraid cache I am golden. Thank you for sharing. As much as I love my HP servers in the homelab, they don’t like to pair with non hp components.

  5. Hope the post helped!

  6. That’s really great and that’s what I’m looking for. Looking at the first picture, I can’t exactly see if there is the standard 4 Port Ethernet card there (what are the black cables). You’ve a 2 Port 10Gbit card (HP 560SFP) instead in use? Do I have to remove the 4P ort card, otherwise the Iocrest card may not fit? In the underlined text you’ve mentioned a critical situation which I can’t follow. What exactly is the situation there? What do you mean with “I have confirmed when it’s closed and locked, it clears the card. To avoid accidents I may come up with a non-conductive cover…”
    I went through the Iocrest documentation, however, dou you know if I can mix Size (TB) and NVME manufacture on the board?

  7. Hi itarch,

    In the first image, I have two servers stacked. The black cables are SFP+ DAC cables going to the server below the server that’s being discussed in this post.

    As for your 2nd question, due to the size of the card and the heatsink that ships with it, the heatsink get’s in the way of the mechanics of the case. With the standard IOCREST heatsink installed, you cannot close the case as it’s in the way of the locking pin. Without the heatsink, it may come in contact with the board, so I’d recommend powering off the server when opening or closing the case to avoid it rubbing against the card.

    The card is simply a PCIe card with a PCIe switch chip. You can use any mix of size and manufacturer you’d like.

    Cheers,
    Stephen

  8. Hi Stephen,

    First, I would like to thank you for the post (and of course for your feedback) and of course wish you Merry Christmas. In my first comment, very focused on the technical stuff, I forgot to appreciate your effort, sorry for that.
    Yes, I took a look inside the server (DL360p Gen8) and understand now what you mean (Pic 5, the locking pin). And even after modifications there is still potential danger to scratch the IOcrest board when opening the case…
    Now, I’m googling around to find this board and the only option is Aliexpress, where I’ve never bought before and have to overcome now …

    Cheers,
    Itarch

  9. Hi itarch,

    I’m glad if the post helps! Merry Christmas!!!

    As for purchasing the card, I also purchased it from Aliexpress. If you use the link in my other post, it goes directly to the IOCREST genuine store. It’s safe, painless, and I received the card very quickly.

    Cheers,
    Stephen

  10. Hello! I have a problem. Installed NVMe m.2 samsung 970 evo plus via pci adapter. But the server doesn’t see the pci. In iLo4, there is nothing on the pci slot. Do you know how to fix the situation? Don’t scold. Translated by Google.

  11. Hi eres,

    The IOCREST card should not be seen inside of iLO. The PCI card simply has a PCIe switch chip. This is normal and expected behaviour.

    Cheers,
    Stephen

  12. Hello Stephen,

    Great article, exactly what I was hoping to find. Did you run any benchmarks on the card that you can share?

  13. Hi Stefanos,

    Inside of the post, there’s a few links to other posts discussing the card and NVMe drives. I believe there should be some benchmarks on those posts. In all cases, the limiting factor for me was the server hardware (CPU), lol. I haven’t been able to max out the NVMe drives or the card as of yet.

    Cheers,
    Stephen

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)