Search This Blog

Wednesday, April 1, 2015

Cisco TMS Woes

Today I had some spare time at work and started to work on our telepresence upgrade again.  I have already got all of our endpoints on the same version, albeit old ones, but still, at least they are on the same software release.  The MCU I haven't upgraded yet as I am still doing research on what will break if I upgrade it.  But the TMS is something we have but don't even use. 

Our current TMS is old as dirt, 13.1 is the release and I have the software for 14 from our service contract.  The problem is, the service contract is now expired so any support we had is gone.  To further this, the box it is currently on is also old as dirt.  TMS shipped on a physical box which is beginning to be unheard of these days since everything is now virtualized.  After managing to backup the SQL database, which was a complete pain in the butt, I dropped it onto a brand new 2008 R2 VM.  I installed the old 13.1 as I planned to step upgrade to 14 and got beat down hard.  I found out that there are two different versions of TMS, one is appliance driven and the other is software driven(VM). 

So the ultimate issue is that I would have to pay to get the other version.  The other issue is I am having problems getting the SQL backup to push into the new TMS since Windows is well...Windows.  So just a heads up, anyone looking to migrate, make sure you got a valid service contract to get a new release key and make sure if you are moving to VM you are on the 80A and not the 80B serial number as 80B is the appliance version.

Tuesday, March 31, 2015

The dust in the wind

Not much going on at work lately but I figured I would post something anyways just to show I didn't abandon this blog.  For starters, I am selling my CCNP Voice lab on Ebay.  I am at a point now where I just need to be working on the latest and greatest and running CUCM 10 just isnt viable in demo mode for me. 

On the tech side of things at work, as I said, things are a bit slow but I have been keeping busy with learning Python and voice stuff.  Last Friday I had a customer inform me that his 911 test call out the local FXO was not showing the correct ANI to the local PSAP.  I took a look and I believe one of two things happened.  First, 911 somehow reported the wrong number or two, the POTS line attached to his gateway isn't the number he thinks it is.

I did some checks on CUCM and found nothing wrong, after all, this site is set up in the exact same manner as the other 12 sites.  Each one has a dedicated voice gateway for FXO dialing out which is 911 only while the rest of the calls hit the T1 PRI.  Easy enough you might say, well try explaining that to someone who isn't as familiar with voice.  So after a bit of discussion I took it upon myself to check RTMT.  RTMT never lies and will show you how the call left and how it appeared.  Sure enough, the call completed as it should have which indicates one of the two things I noted before.

In fact, I just got an email confirming that there was a cross-connect issue.  I cannot stress enough the proper troubleshooting techniques as an engineer.  Following a set standard, whether it yours or someone else's (that works of course), is paramount.  Keep your head clear, systematically eliminate one thing at a time until you come to the last possibility and then that must be the problem.  As someone once said, "Consider, and when you have considered, then act."

Tuesday, March 24, 2015

The Cisco ELM (Enterprise License Manager) and the rehost debacle

It is of no surprise by now that the same UCS box I have been fighting with over the last few weeks still has on-going issues.  Last night I finished an ELM rebuild.  For those of you that do not know what ELM, it is basically the PLM of CUCM 9.X.  So what is PLM then?  Well if you haven't any experience with CUCM 9-10 then that would be a good reason why you don't know.  Basically, Cisco went away from device licenses on CUCM 8 to an ELM/PLM user based license system.  There is still a licensing server and manager, it's just that the server is now either a dedicated VM or sits along side with CUCM or Unity.  The good thing about the ELM in standalone mode is that if your CUCM eats it or Unity craps the bed nothing is affected, just do a restore.  With ELM you upload the PAKs to the server or still go do it the old school way through Cisco and then put the information in the ELM/PLM. 

The major problem with ELM is that if it eats the dirt you have to rebuild it from scratch.  This isn't normally a problem until you run into a deployment where they didn't statically set the MAC on the VM.  Guess what is tied to the hash which includes the MAC on the VM?  You guessed it, the licensing.  I rebuilt the ELM and didn't bother to check to see if their existing ELM was statically set.  I was happy, everything stood back up the restore worked and I could see the keys.  I changed over the ELM control to the new one and BAM, right in the kisser, licensing conflicts. 

I could have just changed the MAC on the new VM to what the old dynamic one was set to but you can't if the old ELM is still turned off and sitting on the ESXi.  I never delete old VMs until I am absolutely sure I no longer need them and thank goodness I didn't.  I wrote down the info, shutdown the new ELM and fired the old one back up.  It works, just goes into read-only mode every 4 days...  So now, I need to get in contact with Cisco and see if they can re-host the license.  The second barrier is this unit doesn't have software SmartNet on the contract, so I might be out of luck.

It's funny how everything melts down with this particular site.  I'm not sure if it's just a run of hardcore bad luck or if the UCS boxes are cursed from Cisco saying upgrade me to 10.6 (10.5.2).  Either way, I'll be glad once this debacle is over.

Friday, March 20, 2015

Cisco and Python

I attended the Cisco Geekfest in Austin a few weeks ago since we are pretty close. The opening introduction on day 2 had Cisco preaching we, as engineers need to know some basic programming.  The code they were showing off for us to know was python and some java.  I have personally dabbled a little in Java but to me that programming language is a mess and got difficult quick.  I think part of the problem was me not being able to draw out the loops.

Python on the on the other hand I started picking up two days ago.  I'm already to the point where I can write a while or for loop with little to no issues.  Now I haven't done anything crazy yet since I am still in the crawl phase but it's coming along well.  For those of you that don't know any programming like me, I suggest Python as a first language as it is extremely forgiving and tells you exactly what the problem is.  Look below for an example on something I screwed up in my code.

c:\Python34>python calculator.py
Traceback (most recent call last):
  File "calculator.py", line 84, in <module>
    main()
  File "calculator.py", line 69, in main
    print(longstring)
NameError: name 'longstring' is not defined

As you can see, I forgot to remove a variable called "longstring".  I put some ASCII art in my program and decided to pull it out as it was making it hard for me to read some stuff half way down the program.  I made a basic calculator that does your standard 4 math functions as well as calculate areas the radius of shapes.  The CBT Nuggets course helped me quite the bit but a YouTube channel with a guy named onestopprogramming really propelled me as he explains every little detail and the "why" as well as the "how".  Their website also goes a long way and provides other little practice examples with the full code following if you need help or want to see the solution.

I can see how programming could be useful for us engineers but at the same time, keep it to the basics unless you have a desire to learn something new.  I personally found programming to be fun as you frequently encounter problems and need to find the solution.  With VoIP, this also happens but is usually a long complex drawn out process unless you just fat fingered a route pattern or screwed up a CSS.  Anyways, just felt that if you didn't already know, some basic programming stuff is coming our way in the years to come and you might want to at least be able to read and understand whats going on behind the scenes.

...I figured I would post my calculator program I built.  90% of this is all my work while the other 10% I got help from the videos when I initially started off since it was part of the learning experience.  In fact, I was never told nor shown how to make an advanced menu that spits you back out to the main menu when you completed a task(Go me...).  Below is the code, if you can figure it out then you are already ahead of the game (Not that its hard).  I included everything here and a copy/paste in Python 3.X will work great.

# Returns the sum of num1 and num2
def add(num1, num2):
    return num1 + num2

 #Returns the result of subtracting num1 - num2
def sub(num1, num2):
    return num1 - num2

 #Returns the result of multiplying num1 * num2
def mul(num1, num2):
    return num1 * num2

 # Returns the result of dividing num1 / num2
def div(num1, num2):
    return num1 / num2

 # This is the basic math function that is called after the main menu (option 1 on the menu)   
def basicMath():
    operation = input('What do you want to do? (+,-,*,/):')
    if(operation != '+' and operation != '-' and operation != '*' and operation != '/'):
        #invalid operation
        print('You must enter a valid operation, HAULTING!!!')
    else:
        var1 = int(input('Enter num1: '))
        var2 = int(input('Enter num2: '))
        if(operation == '+'):
            print(add(var1, var2))
            main()
        elif(operation == '-'):
            print(sub(var1, var2))
            main()
        elif(operation == '*'):
            print(mul(var1, var2))
            main()
        else:
            print(div(var1, var2))
            main()

 #This is the area math function that is called after the main menu (option 2 on the menu)
def areaMath():
    print('Choose an option:')
    print('1: Area of a Circle')
    print('2: Area of a Rectangle')
    print('3: Area of a Square')
    print('4: area of a Triangle')
    operation = input('What do you want to do?:')
   
    if(operation == '1'):
        circle = int(input('Enter the radius of the circle: '))
        print('Your area based off of the circle number input is:', 3.14 * circle ** 2)
        main()
    elif(operation == '2'):
        rectangleLen = int(input('Enter the length of the rectangle: '))
        rectangleWid = int(input('Enter the width of the rectangle: '))
        print('The area of the rectangle based off of your input is:', rectangleLen* rectangleWid)
        main()
    elif(operation == '3'):
        square = int(input('Enter the length of the side of the square: '))
        print('The area of the square based off of your input is: ', square ** 2)
        main()
    else:
        triangleBase = int(input('Enter the base length of the triangle: '))
        triangleHeight = int(input('Enter the height of the triangle: '))
        print('The area of the triangle based off of your input is: ', (triangleBase / 2) * triangleHeight)
        main()
           
 #This is the main function of the program
def main():
    print('')
    print('Please choose an option or press enter to exit')
    print('1: Basic Math')
    print('2: Area Calculations (i.e. diameter of circles or areas of squares)')
    menu = input('Please choose an option now: ')
    if(menu == '1'):
        basicMath()
    elif(menu == '2'):
        areaMath()
    else:
        print('')
        print('You are recieving this message because you either pressed "enter" or did not enter a 1 or 2. Haulting Program now!')
   
main()

Tuesday, March 17, 2015

the CIMC SD Card

Well, I had to re-seat and swap the SD cards on a UCS C220 M3 as my earlier posts had been leading up to.  I figured I would post a short one today with images showing where the SD cards are on this particular model.  I would like to say first that you need to have purchased these in order for them to be present, in fact, you should have bought two of them.  Below are some images of where they are and what they look like in case, for some reason, you have been living in a cave and don't know what an SD card is.



 As you can see, they are easily accessible.  I know the second picture shows nothing and this is what it would look like if you didn't have one while the third is if you did.

Thursday, March 12, 2015

Cisco FlexFlash and it's Role

So the customer I replaced the RAID controller for as well as upgraded their CIMC is asking how to use the FlexFlash and how to get a second card.  I did some research and found that if you already have one then adding a second 16 GB by ordering directly from Cisco is easy.  You cannot use third party SD cards, or at least it isn't supported and the only benefit from having a second one is RAID 1.
For those of you that arn't familiar with RAID, basically each RAID level defined does something with storage.  RAID 0 is striping where it takes two drives and turns them into one with better performance.  The caveat to this is, if one drive fails, your entire system dumps. This isn't a big deal in a lab or non-production environment but on a server, hell to the no....  RAID 10 also known as RAID 1+0 would be your go to level if you wanted expanded storage as well as redundancy.  Basically it takes RAID 1 and 0 and combines them.  RAID 1 is mirroring.  Mirroring does exactly what you would think, it duplicates the drive to another providing redundancy in the event of a backup.  With RAID 10, you have striping and mirroring for the ultimate setup.

The Flexflash contains 4 partitions and on the newer versions 1.5+ I believe they reduced this to one partition.  Each partition has a different utility.  Firmware, upgrades, drivers for the chassis itself, and a lone partition for whatever you want to use it for.  The link provided is a more in depth understanding of what the FlexFlash is and does.

http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/unified-computing/whitepaper_C11-718938.pdf

In the end you need to use the SCU on the existing SD card to set up the new one.  This will let you get the RAID going as well as verify everything is in proper working order.  I would take it one step further and make sure everything is on the latest release and keep a backup release that you know works on the last partition just so you got something to work with.  SCU stands for Server Configuration Utility by the way.  It's basically a GUI driven bootable image on the SD card that makes life easy.  Who wants to dig around the BIOS setting stuff up like its 1999?

Below are some additional links I found useful on FlexFlash:

https://supportforums.cisco.com/discussion/12421051/flexflash-questions

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/OL-32097-01.html#pgfId-125827

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C220/install/C220/replace.html

https://supportforums.cisco.com/discussion/12002036/flexflash-vs-usb-esxi-install

http://www.cisco.com/en/US/docs/unified_computing/ucs/c/sw/gui/config/guide/1.5/b_Cisco_UCS_C-series_GUI_Configuration_Guide.151_chapter_011.html#concept_7DC7EAB11DCA4B4D8F08F697398A960E

As always, I hope this blog post was helpful.  Still not a lot of views and no comments.  Hopefully that changes within the next few months.  I really want this blog to turn into something helpful and a one stop source for those on off situations where Cisco isn't of much help unless you get in contact with TAC.  Since Smartnet is expensive, this could be a temporary filler.

Tuesday, March 10, 2015

The CIMC fiasco

I went from hanging out upgrading the CIMC and BIOS today to reclining back waiting on it to raging to the point I wanted to throw a laptop out the window.  The process to complete the upgrades is fairly simple and straight forward but the issues in between can make you want go drop a car into a canyon with a crane.

I started the upgrade today at 17:00 / 5:00PM.  I didn't get out of the office until 20:15 / 8:15PM.  Let me explain a few things that can and probably will occur.  First, Java is the biggest piece of crap since Netscape.  If you don't know what Netscape is, you might not be old school.  The CIMC has a built in KVM client that lets you view the ESXi and boot process, this is particularly useful when you are in my position and the site is 2 hours away and there isn't anyone there at the time of night I was working.

Long story short, Java....Java....Java....  I did the initial upgrade from 1.4(7) to 1.5(4) and the KVM worked fine.  In fact, I was surprised since the KVM is Java based and Cisco doesn't ever test the latest release of Java.  Why is this a problem?  Well, most organizations aren't in the habit of running old Java since it's the biggest security leak you could possibly have.  That and Java itself doesn't run well most of the time on anything I try to use, Unity Connection call recording included.

I was using Java 8 with the latest release.  The first upgrade processed and I saw the firmware update but not the BIOS.  I began immediately crapping bricks since CIMC and the BIOS need to be on the same release or you end up getting a server that isn't booting.  While you may still have a KVM, you need to unhose what you did.  The HUU .ISO works incredibly well, it has it's own GUI and is bootable.  Little to no effort is needed to use this tool.  See below for a few images of what it looks like.




Basically, once it boots after you shut all of your VMs down you get the banner in the KVM screen. That takes about 5 minutes at most and then you see the next screen where you can select what you want to upgrade.  You can pick and choose or just do everything.  There is also the HDD firmware upgrade.  Once you hit go, it gives you a warning to let you know what you are doing is rather permanent and then it is on its way.  

The progress bar is something that needs fixed.  I sat there for a good 5 minutes and it barely moved.  I figured I was in for the long haul (I was anyways, just hadn't run into the java and cosmetical alarm issues yet.).  So basically, here is the process so far:
  1. Open KVM in CIMC
  2. Go to Virtual Media
  3. Add your .iso image
  4. Map it to the virtual dvd thing
  5. Restart the server from the CIMC
    1. Please for the love of god make sure your VMs are not set to auto-boot.  While this shouldn't be a problem, I like to be safe than sorry.
  6. Let her rip, get the banner and upgrade what you want.
If you do the CIMC and BIOS you end up having to turn the server back on, at least I did the first time.  The second time it seemed to start back up on it's own.  Granted the first time I did this, I was crapping bricks because I had never done a CIMC upgrade, who really needs to?  Upon boot I saw the firmware and was happy.  I then saw the BIOS was wrong and started the fecal cycle again.  I refreshed the screen and so did the BIOS version and I felt better.  Next, I started taking a peek at stuff and saw a giant red X on the main status screen and was like now what the....  

The above is the error I saw.  The KVM stopped working and I started thinking catastrophic things were happening.  What was really going on was two things at the same time.  First, the cosmetically error is just that, cosmetically.  I spent an hour and a half with TAC and they deduced that the new version does this when you are not running a dual SD FlexDrive RAID. I asked why it was a major alarm and never really got an answer other than because it does.  The SD card wasn't in use anyways and it's only real use was to hold an operating system or whatever else you might want on there.  But again, it wasn't in RAID so it crapped the bed.

The second issue, the KVM kept not working and giving me java errors.  I was getting mad at this point but figured I knew how bad Java is.  I uninstalled Java, removed all associated programs and relaunched FireFox and BAM nothing.  No website, nothing.  I launched IE and logged in.  No Java prompt.  I had to go back into Java and force it to prompt.  This entire process was long as I kept logging in and out of the CIMC and nothing was working, all the while I got a Publisher server down.  

Eventually, I got java to work and KVMed back in.  Started the second upgrade which went so smoothly I felt like a champ.  However, before logging in I couldn't even log in.  I got a max sessions exceeded from the CIMC.  The problem here, is CIMC doesn't record you logging out immediately if you just close the window and leaves it hang.  There are two sure fire ways of getting this to clear.  
  1. Slam the server
  2. Use the SSH and clear it (Yes, do it this way hotshot)
As you can see in the image above, I had to SSH in, set the session and terminate it.  I did a few to get the hang of it and if Java gave me more grief.  Overall, this was a great learning experience for me and I hope it gives y'all a heads up too.  If this was a TLDR (Too Long Didn't Read) situation take these important facts home with you:
  1. Use the .iso its easier, I was skeptical at first too
  2. Make sure you got Java 7 or the latest tested release.  8 isn't as new anymore but it isn't tested.
  3. Give the entire process some time.  10 to 15 minutes should do the trick for the BIOS/CIMC
  4. Verify the upgrade occurred
  5. Make sure no errors exist 
    1. Cosmetically ones may apply, as they did in my situation
  6. Power on and off from the CIMC works great