The CIMC fiasco

I went from hanging out upgrading the CIMC and BIOS today to reclining back waiting on it to raging to the point I wanted to throw a laptop out the window.  The process to complete the upgrades is fairly simple and straight forward but the issues in between can make you want go drop a car into a canyon with a crane.

I started the upgrade today at 17:00 / 5:00PM.  I didn't get out of the office until 20:15 / 8:15PM.  Let me explain a few things that can and probably will occur.  First, Java is the biggest piece of crap since Netscape.  If you don't know what Netscape is, you might not be old school.  The CIMC has a built in KVM client that lets you view the ESXi and boot process, this is particularly useful when you are in my position and the site is 2 hours away and there isn't anyone there at the time of night I was working.

Long story short, Java....Java....Java....  I did the initial upgrade from 1.4(7) to 1.5(4) and the KVM worked fine.  In fact, I was surprised since the KVM is Java based and Cisco doesn't ever test the latest release of Java.  Why is this a problem?  Well, most organizations aren't in the habit of running old Java since it's the biggest security leak you could possibly have.  That and Java itself doesn't run well most of the time on anything I try to use, Unity Connection call recording included.

I was using Java 8 with the latest release.  The first upgrade processed and I saw the firmware update but not the BIOS.  I began immediately crapping bricks since CIMC and the BIOS need to be on the same release or you end up getting a server that isn't booting.  While you may still have a KVM, you need to unhose what you did.  The HUU .ISO works incredibly well, it has it's own GUI and is bootable.  Little to no effort is needed to use this tool.  See below for a few images of what it looks like.

Basically, once it boots after you shut all of your VMs down you get the banner in the KVM screen. That takes about 5 minutes at most and then you see the next screen where you can select what you want to upgrade.  You can pick and choose or just do everything.  There is also the HDD firmware upgrade.  Once you hit go, it gives you a warning to let you know what you are doing is rather permanent and then it is on its way.  

The progress bar is something that needs fixed.  I sat there for a good 5 minutes and it barely moved.  I figured I was in for the long haul (I was anyways, just hadn't run into the java and cosmetical alarm issues yet.).  So basically, here is the process so far:

1. Open KVM in CIMC
2. Go to Virtual Media
3. Add your .iso image
4. Map it to the virtual dvd thing
5. Restart the server from the CIMC
1. Please for the love of god make sure your VMs are not set to auto-boot.  While this shouldn't be a problem, I like to be safe than sorry.
6. Let her rip, get the banner and upgrade what you want.

If you do the CIMC and BIOS you end up having to turn the server back on, at least I did the first time.  The second time it seemed to start back up on it's own.  Granted the first time I did this, I was crapping bricks because I had never done a CIMC upgrade, who really needs to?  Upon boot I saw the firmware and was happy.  I then saw the BIOS was wrong and started the fecal cycle again.  I refreshed the screen and so did the BIOS version and I felt better.  Next, I started taking a peek at stuff and saw a giant red X on the main status screen and was like now what the....  

The above is the error I saw.  The KVM stopped working and I started thinking catastrophic things were happening.  What was really going on was two things at the same time.  First, the cosmetically error is just that, cosmetically.  I spent an hour and a half with TAC and they deduced that the new version does this when you are not running a dual SD FlexDrive RAID. I asked why it was a major alarm and never really got an answer other than because it does.  The SD card wasn't in use anyways and it's only real use was to hold an operating system or whatever else you might want on there.  But again, it wasn't in RAID so it crapped the bed.

The second issue, the KVM kept not working and giving me java errors.  I was getting mad at this point but figured I knew how bad Java is.  I uninstalled Java, removed all associated programs and relaunched FireFox and BAM nothing.  No website, nothing.  I launched IE and logged in.  No Java prompt.  I had to go back into Java and force it to prompt.  This entire process was long as I kept logging in and out of the CIMC and nothing was working, all the while I got a Publisher server down.  

Eventually, I got java to work and KVMed back in.  Started the second upgrade which went so smoothly I felt like a champ.  However, before logging in I couldn't even log in.  I got a max sessions exceeded from the CIMC.  The problem here, is CIMC doesn't record you logging out immediately if you just close the window and leaves it hang.  There are two sure fire ways of getting this to clear.  

1. Slam the server
2. Use the SSH and clear it (Yes, do it this way hotshot)

As you can see in the image above, I had to SSH in, set the session and terminate it.  I did a few to get the hang of it and if Java gave me more grief.  Overall, this was a great learning experience for me and I hope it gives y'all a heads up too.  If this was a TLDR (Too Long Didn't Read) situation take these important facts home with you:

1. Use the .iso its easier, I was skeptical at first too
2. Make sure you got Java 7 or the latest tested release.  8 isn't as new anymore but it isn't tested.
3. Give the entire process some time.  10 to 15 minutes should do the trick for the BIOS/CIMC
4. Verify the upgrade occurred
5. Make sure no errors exist 
1. Cosmetically ones may apply, as they did in my situation
6. Power on and off from the CIMC works great