Dec 072016
 

When upgrading VMware vSphere and your ESXi hosts to version 6.5 or 6.7 you may experience an error similar to: “The upgrade contains the following set of conflicting VIBs: Mellanox_bootbank_net.XXXXversionnumbersXXXX. Remove the conflicting VIBs or use Image Builder to create a custom ISO.”. This is due to conflicting VIBs on your ESXi host. This post will go in to detail as to what causes it, and how to resolve it.

The issue

After successfully completing the migration from vCenter 6.0 (on Windows) to the vCenter 6.5 Appliance, all I had remaining was to upgrade my ESXi hosts to ESXi 6.5.

In my test environment, I run 2 x HPe Proliant DL360p Gen8 servers. I also have always used the HPe customized ESXi image for installs and upgrades.

It was easy enough to download the customized HPe installation image from VMware’s website, I then loaded it in to VMware Update Manager on the vCenter appliance, created a baseline, and was prepared to upgrade the hosts.

I successfully upgraded one of my hosts without any issues, however after scanning on my second host, it reported the upgrade as incompatible and stated: “The upgrade contains the following set of conflicting VIBs: Mellanox_bootbank_net.XXXXversionnumbersXXXX. Remove the conflicting VIBs or use Image Builder to create a custom ISO.”

The fix

I checked the host to see if I was even using the Mellanox drivers, and thankfully I wasn’t and could safely remove them. If you are using the drivers that are causing the conflict, DO NOT REMOVE them as it could disconnect all network interfaces from your host. In my case, since they were not being used, uninstalling them would not effect the system.

I SSH’ed in to the host and ran the following commands:

esxcli software vib list | grep Mell

(This command above shows the VIB package that the Mellanox driver is inside of. In my case, it returned “net-mst”)

esxcli network nic list

(This command above verifies which drivers you are using on your network interfaces on the host)

esxcli software vib remove -n net-mst

(This command above removes the VIB that contains the problematic driver)
After doing this, I restarted the host, scanned for upgrades, and successfully applied the new vCenter 6.5 ESXi Customized HPe image.

Hope this helps! Leave a comment!

  49 Responses to “VMware vSphere 6.5 and 6.7 – Conflicting VIBs when upgrading ESXi to 6.5 or 6.7”

  1. […] A note on a problem I dealt with during ESXi 6.5 upgrade (conflicting VIBs marking image as incompatible when deploying HPe customized image) can be found here: http://www.stephenwagner.com/?p=1120 […]

  2. Just went through this myself, thanks for the post! A few notes:

    The VIBs my esxi hosts complained about were different, but still not used, so I had to change the grep/remove commands to match

    The VIB remove commands take a loooong time on my boxes to execute, like 5+ min, and I thought they were stalled/doing something horrible. Nope, they just take a long ass time.

  3. Thank you! Same problem. Worked like a charm.

  4. Hi All,

    Need your help. I tried to upgrade our host Esxi 6.0.0, 3380124 to ESXi-6.5.0-20170104001-standard but got this error “The upgrade contains the following set of conflicting VIBs:” I am fully aware that I should delete if conflicting VIB but on this one it doesn’t display the conflicting Vib.

    Thanks,
    JC

  5. Hi JC,

    Is this happening on one, or multiple hosts?

    If it’s one, maybe burn an ISO of the install image and try upgrading that way.

    Have you checked any other logs to see what package is causing the conflict?

    Out of curiosity, are you using a custom image, have you loaded any special VIBs or drivers on your install?

    Cheers

  6. Hi JC,

    My problem is a bit different, I wasn’t upgrading to a higher version of VM Ware vSphere, I was simply installing ESXi 6.5 on my first lab ESXi host as part of my VMware training, and got the following error:

    <CONFLICTING_VIBS_ERROR: Vibs on the host are conflicting with vibs in metadata. Remove conflicting vibs or use Image Builder to create a custom ISO providing newer versions of the conflicting vibs. ['Hewlett-Packard_bootbank_char-hpcrv_5.5.6.6-10EM-550.0.0.1198610'
    'Mellanox_bootbank_net-mst_2.0.0.0-10EM.550.0.0.600000', etc, etc

    Thanks for your help. Jorge

  7. Hi Jorge,

    It sounds like your install image is corrupt, or you’re maybe installing on to a previous install?

    I would recommend zero’ing the drive you are installing to, as well as re-downloading the ESXi installer images.

    Cheers

  8. In my case my error was:
    HUAWEI_bootbank_hio

    So i ran this command:
    esxcli software vib remove -n hio

    After confirming it wasnt being used. Thanks for the article.

  9. Hello I found your article very helpful. I had bootbank vib conflicts on HP DL360p G8s.

  10. Ever seen it went there were about 15 conflicting vibs? I am upgrading a different ESXI environment and one of my host has about 15 vibs conflicting.

  11. Hi Joe,

    That’s crazy. I’m curious what the conflicts are?

    Have you tried doing an upgrade using the install media (instead of vSphere Update Manager)?

    If that was me, I may just blow away the host and do a fresh install after trying to two methods above. For that to happen, it sounds like the host is either old, or has had quite a bit of work done to it while it’s been in production!

    Cheers,
    Stephen

  12. I had this exact same problem and your solution fixed it. Thanks!

  13. Same issue – but 3 identical hosts 1st one upgraded fine the second two come up with the VIB issues. All kit 1 year old and super fast. going to try the ISO system

  14. This helped me aswell – thank you so much!

    I had 3 VIBS that I removed on 6 hosts and the upgrade to ESXi 6.5 went smooth.

  15. How would one determine if the conflicting VIB is in use or not by the host? I’d like to remove the conflicts and just do an upgrade via Update Manager rather than rebuild the OS.

    Our environment happens to have 11 hosts complaining about;
    LSI_bootbank_scsi-mpt3sas_04.00.00.00.1vmw-1OEM.500.0.0.472560
    LSI_bootbank_scsi-mpt3sas_04.00.00.00.1vmw-1OEM.500.0.0.472560

  16. Hi Robert,

    In your case I think those VIBs are needed as they have to do with your SAS/SCSI/HBA Host Adapter (you may need to verify this).

    In some cases, the issue is recoverable… In other’s it’s not and a full re-install may be needed…

    Cheers,
    Stephen

  17. Thank you for the reply. I am going to try 1 redundant host to see what happens. All of our hosts have redundant internal SD cards with the ESXi installation on them so hopefully we wouldn’t need the host adapter/controller as we’re not using any 3.5 internal storage.

  18. HI Robert

    In that case you should be fine. I’m pretty sure that driver is for internal storage only.

    Let me know how you make out…

    Cheers

  19. Thanks so much. Worked for me

  20. Hi Robert,
    Thanks for wonderful article

    I get the below conflicts for one of the host
    ________________________________________
    DEBUG:root:Potentially conflicting vibs : set(['VMware_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.600.0.0.2494585', 'VMware_bootbank_sata-ahci_3.0-26vmw.600.3.57.5050593', 'VMware_bootbank_ehci-ehci-hcd_1.0-4vmw.600.3.69.5572656', 'MEL_bootbank_nmlx4-en_3.15.5.5-1OEM.600.0.0.2768847', 'VMware_bootbank_net-e1000e_3.2.2.1-2vmw.600.3.57.5050593', 'VMware_bootbank_vsan_6.0.0-3.69.5568629', 'VMware_bootbank_esx-ui_1.19.0-5387100', 'VMware_bootbank_vsanhealth_6.0.0-3000000.3.0.3.69.5572665', 'VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-4vmw.600.3.57.5050593', 'MEL_bootbank_nmlx4-core_3.15.5.5-1OEM.600.0.0.2768847', 'MEL_bootbank_nmlx4-rdma_3.15.5.5-1OEM.600.0.0.2768847', 'VMware_bootbank_xhci-xhci_1.0-3vmw.600.3.69.5572656', 'QLogic_bootbank_ima-qla4xxx_500.2.01.31-1vmw.0.3.100400', 'QLogic_bootbank_scsi-qla4xxx_644.55.36.0-1OEM.550.0.0.1331820', 'NetApp_bootbank_NetAppNasPlugin_1.1.0-0', 'VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.600.2.43.4192238', 'VMware_bootbank_vmware-fdm_6.0.0-3634794', 'VMware_bootbank_esx-xserver_6.0.0-3.66.5485776', 'VMware_bootbank_qlnativefc_2.1.50.0-1vmw.600.3.57.5050593', 'VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-3vmw.600.3.57.5050593', 'VMware_bootbank_esx-tboot_6.0.0-3.57.5050593', 'VMware_bootbank_misc-drivers_6.0.0-3.69.5572656', 'VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-3vmw.600.3.57.5050593','VMware_bootbank_esx-base_6.0.0-3.69.5572656'])
    ____________________________________________________________

    Do you think I will be a good idea to remove all the vibs and reboot the host.

  21. So this was the same problem at my site – but how to see this coming before you run the ESXi update? Because after i have run the update and i get this error I have to reboot the host and then i need to start the installation process from scratch (no update option)! Any ideas for future upgrades?

  22. This message is from the logs after upgrade was done.
    So now there is an error for this host on in update manager. need to fix that

  23. Thanks for your help very useful 🙂

    Used commands to resolve conflicts with Dell PowerEdge servers 720/410 upgrading to 6.5.0 update 1

  24. Just a quick note for the googlers – I ran into this issue when I tried to go from 6.0 U1 to 6.5 on the Dell Custom ISO. Updating the host to 6.0U3 then 6.5 worked for me without having to remove the VIB. YMMV.

  25. Thank you!

    Worked for me when updating 5.5 to 6.5 with HPE custom iso

  26. I found the “VMware-ESXi-5.5.0-Update3-3568722-HPE-550.9.6.5.9-Dec2016-depot.zip” update for my HP ESXi 5.5 server that appeared to have the required files/vibs that was needed to do the upgrade to 6.5

  27. I came across this on one of my Dell R710 servers that is currently on 6.0. Trying to update to 6.5 (from Dell Bootable Media), I get the error message about conflicting VIBs. Now, I downloaded this Dell ISO image for an R720 server, and in the past, have been able to use the same media to upgrade both the R720s and R710s with it. When I went to the Dell compatability matrix for 6.5 on Dell servers, the R710 is not even listed anywhere, so I assume the hardware isn’t supported any longer. Not a huge deal as these servers were repurposed from being CASHUB Exchange servers a few years ago and only have a couple VM’s that users connect to for a few specific applications, so if I can’t upgrade them, it’s not the end of the world. Hopefully this helps someone looking for the info on this same server. It’s unsupported by dell for ESXi 6.5. Dell matrix: http://topics-cdn.dell.com/pdf/vmware-esxi-6.5.x_reference%20guide_en-us.pdf

  28. Thanks dude. Managed to upgrade my HPE microserver gen7 N54L from esxi 6.0 update-2 to 6.5 update-1, thanks to your info.
    Cheers mate!!!

  29. Hello Robert,

    First, thanks for this post as it sent me in the right direction. For me, the difficult part was to understand the various command output, because I have never dealt with them before.

    So if this can help someone:
    1- The command : esxcli software vib list | grep “xxxx”
    “xxxx” is the name of the package. For example, in the error: “Brocade_bootbank_net-bna_3.0.3.0-10EM.5000.0.47250” the required portion of the VIB name is “Brocade”.

    2- In the previous error, we can already see the driver name, while this is not evident. In this case the driver name is “net-bna”, followed by the version number.

    3- The command “esxcli network nic list” is fine to list the driver loaded for the NIC cards, but it does not cover the other errors I had related to SCSI adapter.

    4- The output of the command, “esxcli software vib list | grep Brocade” produces:

    net-bna 3.0.3.0-1OEM.500.0.0.472560 Brocade VMwareCertified 2016-10-17
    scsi-bfa 3.0.3.0-1OEM.500.0.0.472560 Brocade VMwareCertified 2016-10-17

    The first field of the line is the driver name. In this example, I needed to remove the net-bna driver (the error in step 1). It took me a bit to understand what was the driver name because of the format of the output, the limited screen real estate I had at the time, and the fact I was wondering if there were missing lines in the output. So, if you are like me and want to get the smallest details ironed out before proceeding, then know that the first field IS the driver name.

    5- I had another error pertaining to a SCSI driver. I found that I could check if it was loaded using the command “esxcfg-info”. This produces a very long output, so I copied the results to my workstation, opened and search through it using notepad++. Once I was satisfied that the SCSI driver was not found, in the file, making multiple searches (LSI, mpt3sas, any other string I thought was relevant), I removed it.

    6- Needless to say, I moved all the VM off the host before starting and set it in maintenance mode. I took screenshots of the config in case I blew something up and be forced to start from scratch, but in the end, everything went well.

    7- Everything I`ve read regarding this issue, the drivers are always unused. What happens if they are in use? I couldn’t find any solutions, so I guess reformatting the installation partition and starting from scratch, is the only solution I can think of. If someone went through this and has something to add, please do. Fortunately, that was not the case for me.

    8- One final note, VMWare complains that the host must be rebooted for the changes to take effect, I did not reboot between the removal, but I rebooted and let the server go until it was completely booted before proceeding with the update. I felt it necessary regardless if the updates are done from VUM or directly from the custom CD.

  30. Well after waiting forever to get a maintenance window I was able to removed the following VIBs from our hosts;

    LSI_bootbank_scsi-mpt3sas_04.00.00.00.1vmw-1OEM.500.0.0.472560
    LSI_bootbank_scsi-mpt3sas_04.00.00.00.1vmw-1OEM.500.0.0.472560

    After rebooting the hosts and scanning for updates/upgrades it was listing the 6.5 upgrade as Not Compliant rather than Not Compatible now. I was able to successfully upgrade to 6.5 without having to do a full fresh install of esxi on our 11 hosts.

    I don’t remember ever using a customized esxi installer, however, I cannot explain otherwise how those drivers were supplied on our hosts even though they are not used. Either way I was able to remove the hurdle and get things upgraded without any issues.

    Thanks for the great article Stephen!

  31. Same Problem with Upgrade to 6.5 from 6.0 with Fujitsu Hardware RX300 S8

    net-nx-nic 5.0.626-3vmw.500.0.0.472560 VMware VMwareCertified 2015-10-27

    i did not use this network driver, so i removed it.

    Thanks 🙂

  32. Found I had to remove ehci-ehci-hcd which had dependencies on xhci.

    Running both on the same command resolved

    esxcli software vib remove -n ehci-ehci-hcd -n xhci-xhci

  33. On our system the vibs in conflict all are in use, is fresh install the only option available?
    Thanks
    Haider

  34. Hi Haider,

    Maybe try upgrading the individual VIBs with their respective upgrade packages before trying to upgrade ESXi itself.

  35. Thanks Stephen
    I will download newer drivers from Vmware web site for the 2 cards we use ( Intel and Broadcom) and hopefully they will come with newer vib, I will update here how it went. I need a downtime as the server in the only Esx server in the VCenter in the branch office.

  36. Just wanted to let you know that on 03/15/18 this information is still VERY relevant and useful. I had a similar machine to the one in the article with VIB issues during an upgrade. Your process was right on target and I was able to find, verify and remove a Qlogic and Emulux VIB that was hanging me up.

    Thank you for the article and the education.

    Regards,

    Bob

  37. Very useful, the fix was to download updated drivers for the nic as they are in use and can’t be removed. Once drives upgraded ( vibs upgraded) then the upgrade of the VSphere host was possible.
    Thanks a lot.

  38. Thanks for this post. I had the same issue with the HUAWEI “hio” vib as an earlier poster and followed the instructions to remove the vib and rechecked to see it gone. I then reboot and attempt to upgrade from 6.0 -> 6.5 and I still get the conflicting vibs error even though when I search the vib list there is no HUAWEI match. I have run through multiple reboot cycles. Any thoughts?

  39. Yes its working for me. Thanks

  40. Hi Kevin,

    After removing the hio driver, what does the error message say now? Or is it still complaining about the specific driver that you already removed?

    Stephen

  41. Your solution worked like a charm upgrading my home lab ESXI from 6.0 to 6.5 (the problematic driver was DLink-528T)

    THANK YOU Stephen! 🙂

  42. Hi Stephen,

    Thanks for the reply. I was able to get past this and upgrade. Here is my best analysis. I hope it helps someone else. I got the same error on “reboot” and interestingly enough the hio vib was still on the list but I missed it because the name was different (just “hio”) from the original on the vib list output “HUAWEI_bootbank_hio”. What I think I traced it down to was that there are two boot banks and the initial vib delete removes from the first but not the second in order to be able to rollback on an unsuccessful reboot. When I removed the vib, instead of rebooting into my 6.0, to save time I immediately rebooted into the 6.5 iso for upgrade. Apparently this is not considered a “successful” reboot and so the “hio” vib saved off on the second boot bank was restored. Pilot error missed it in the list the second time.

  43. Thanks for the article. Doing a Dell R720 upgrade in VCenter from ESXi 5.5 to 6.5U2 with Dell ISO and got the mpt3sas VIB conflict message. Removed the mpt3sas via command line, rebooted, rescanned and upgrade proceeded successfully.

  44. Hi
    Thanks helped me. not the same drivers but same process

  45. Thanks very much for an excellent article.
    I followed your instruction and got my issue resolved . I was able to upgraded my ESXi hosts from 5.5 to 6.5

  46. Hi ! I am upgrading the HOST from 6 update 3 to 6.5 but getting error –

    Unsupported devices [103c:3230 103c:3235] found on the host.
    The upgrade contains the following set of conflicting VIBs:
    VMware_bootbank_ehci-ehci-hcd_1.0-4vmw.600.3.69.5572656

    Running the command – esxcli network nic list
    gives the following output:

    Name PCI Device Driver Admin Status Link Status Speed Duplex MTU Description
    —— ———— —— ———— ———– —– —— —————– —- ————————————————————————-
    vmnic0 0000:03:00.0 bnx2 Up Up 1000 Full 1500 QLogic Corporation NC373i Integrated Multifunction Gigabit Server Adapter
    vmnic1 0000:05:00.0 bnx2 Up Up 1000 Full 1500 QLogic Corporation NC373i Integrated Multifunction Gigabit Server Adapter
    vmnic2 0000:0b:00.0 e1000e Up Down 0 Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
    vmnic3 0000:0b:00.1 e1000e Up Down 0 Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

    So can I remove this VIB ?
    ehci-ehci-hcd 1.0-4vmw.600.3.69.5572656 VMware VMwareCertified 2018-05-23

  47. Thanks so much for taking the time to post this. I’m pretty sure I would have needed to call support if not for this posting. My upgrade of an R710 from ESXi 6.0 to 6.5 U2 that had net-bna, scsi-bfa and scsi-qla2xxx VIBs went fine after first checking that the drivers were not in use and then removing them! Thanks again.

  48. Howard, I was able to get my dell r710 upgraded to ESXi 6.5 U2 but i’ve had random reboots ever since on 2 out 4 hosts. Have you had any issues with this? Dell support hasn’t been able to help as it’s unsupported and vmware keeps referring me back to Dell.

  49. Just wanted to say thanks Stephen! I had this same issue on a HP Gen8 server upgrade from ESXi 5.5 to 6.5U2 and had to remove the VIB files below to get the offline bundle upgrade to work, after that the upgrade went perfect!! d-(^_^)-b

    net-mlx4-core
    net-mlx4-en
    net-mst

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)