At last, we are here with the final post on the CIMC and SD card fiasco that has riddled me with holes. Over the weekend I finished up the SD card project in which I needed to install 2 SD cards in a UCS C200 M3 server. You might think this would require a simple amount of time to accomplish but it has been nothing but simple. Cisco's document hasn't been updated to reflect that 32 GB SD cards are now supported. How did I find this out you may ask? I contacted TAC when I had a problem, the new SD cards were 32 GB and the old end of sale ones were 16 GB.
This might not seem like a problem at first, in fact, I didn't even consider that they were different as I just dropped them in and went about my way only to find out that RAID 1 was not going to work. For those of you that don't know, RAID 1 is a mirrored configuration of two disks. This means that one disk is identical to the other and if one fails, it isn't a big deal. Well, two different size disks means no RAID 1. So I called TAC to see if there was a way I could get around this and they didn't have an answer for me. I asked them why they thought it was acceptable to have 32 GB cards only now and still didn't get an answer.
I pondered for a bit, stewed for a bit, and then came up with a brilliant solution. Let's take the existing 16 GB SD card from the sub and put it in the pub! This means I got two 16 GB SD cards in the pub and two 32 GB SD cards in the sub. As soon as I did that I was able to synchronize the SD cards in the CIMC but I still got a "degraded RAID". What gives you might ask? Well, you still need to run the SCU ISO for format the SD cards which will let you put ESXi on them. Now this sounds easy and it really is but I had quoted 3.5 hours of labor and ended up with 13.5 because of issue after issue. First, the SD card size mismatch, second getting someone to go out there and spend an hour and a half between two sites pulling SD cards and me shutting down and turning on VMs and servers. Second, the TAC case took some time, and third the actual process of getting the SCU uploaded and ran takes forever.
No where in Cisco's documentation does it say that an SD card RAID array can take between 1-2 hours each! I had assumed since they were flash devices, this would be done in 10-20 minutes but no, the first 16 GB array took upwards to an hour while the 32 GB array took about an hour and a half. What makes it worse, it said it was done and wasn't. There is a pop-up message on the SCU that states the RAID array is done building and you can reboot. Well, the first time on the 16 GB card this went off without a hitch so I rebooted the machine and all the VMs came back up no problem. The 32 GB server lied to me. I rebooted, checked the CIMC and bam "RAID Rebuilding" then "RAID Degraded" because I rebooted and it wasn't done. Doh!
So I started the long process all over again. By the time I was done rebuilding the array it was 2am. Granted most of this was me just checking on it time to time and not really doing anything but still. When it said the RAID array was complete, I checked CIMC before I rebooted and it had actually completed this time around. Now why it didn't work the first time I don't know but it did this time. Lesson learned, if you plan on using the SD cards, get some chips, a computer game, and keep tabs on it while it rebuilds the RAID array.
I think at this point my blog is the most comprehensive guide to the entire CIMC and SD card setup. There is more information here than in the Cisco docs with the exception of the step by step guides that are helpful but don't tell you all the small things. It has been a long few months getting this done, but it has been an experience, one that I will remember and remind customers that the SD cards are a useless add-on that you don't need.
No comments:
Post a Comment