Cisco 8841 to Analog Paging Choppy Audio

I know it's been awhile but things have been a bit slow around here so I've been using my spare time to learn Linux inside and out the best I can.  Figured another technology under my belt surely can't hurt anything.  Last week however was a change of pace with a strange issue.

I have a customer that is experiencing choppy audio when dialing their analog paging system.  They are on 10.5(2) so everything in place works and is supported even prior to their upgrade.  Their 8841s go off hook and dial a paging number that hits a voice gateway and then pushes out an FXO port to their paging system.  Every time they call the number the overhead paging sounds like crap.  At first, they thought the paging system was bad but with a buttset hooked up to the POTS line going to the paging system, everything sounded fantastic.  That leaves Cisco's side of things.

I had to rule out the DSPs and the FXO port.  I had them make a call and after a "show voice call x/x/x" and saw nothing out of the ordinary.  Logs on the gateway were also fine so what the heck?  They only have a two port FXO card so I wasn't about to ask them to pull their primary line inbound to test.  I instead asked a coworker to go on-site and with a spare FXO card that we know works.  We swapped out the card and the same thing still happened, so piss!

After that fiasco, we went back to the drawing board and tried dialing from a phone that isn't an 8841 and everything sounded good.  That old 7912 from over ten years ago still works and pointed the finger to the 8841.  I don't know if it's the G.722 codec, firmware, or somehow and someway SIP is screwing up the call setup using a weird codec but I will need to pull logs to find out for sure.  I need to schedule another test call with them and pull logs relevant only to that time period so I can get a fresh cap.

Hopefully this pans out to be firmware but with it suddenly happening who knows.  I don't see SIP screwing up since it's just a SIP phone pushing to an H.323 gateway.  At that point CUCM should have already done the heavy lifting of protocol conversion.

Out of Compliance on CUCM and need to delete some devices?

I discovered today that CUCM does the same thing Unity does if you let it lock out due to being out of compliance in the PLM.  If you reboot the CUCM you get 24 hours to remove whatever may be causing you to be out of compliance.  I discovered this through a TAC case as a customer had let their system lock out after the grace period and I was in a hard place to tell them either buy more licenses or leave it.  I knew Unity in fact let you reboot and fix the problem and apparently this is also a think as of CUCM 10.x.  Good news, didn't have to make a customer mad!

Cisco UCS Password

Well the holidays are over and everyone is back to the daily grind.  I know it's been awhile but I was reading an article that was a bit stunning.  Apparently Cisco changed the UCS default password on the C-Series UCS servers from "password" to "Cisco1234".  As of a few days ago this was brought to light and now people can stop calling TAC.  Why was this done and then not propogated down to the users?  Who knows.  Here is a link to the article that explains it all.  Glad someone posted it!

Voice PCM capture

I figured two articles in one day would be cool....  Anyways, in my last article I wrote about a PCM capture.  This essentially lets you capture the voice stream after digitization from the DSPs and play it back.  I know that can be scary since you can listen to anyone's conversation but it was required and authorized in my case since I needed to find out if everything was worky worky on the PRI side of things.  So below is the script you need to capture voice, please use this responsibly.

1. Debug mgcp packet (if applicable)
2. Debug isdn q931
3. voice hpi capture buffer 1000000
4. voice hpi capture destination flash:pcm.dat
5. exit
6. test voice port 0/0/0:23.23 pcm-dump caplog 7 30
7. :::::::::Make Test Call::::::::::::;
8. test voice port 0/0/0:23.23 pcm-dump disable
9. conf t
10. no voice hpi capture buffer 1000000
11. no voice hpi capture destination flash:pcm.dat

Note this is for prior to IOS 15.  Also, your voice ports are going to be different based on whichever channel gets opened for the bearer channel  for IOS 15 and later:

voice pcm capture buffer 200000
voice pcm capture destination flash:
exit
!
test voice port 0/1/3 pcm-dump caplog 7


:::::::::Make Test Call::::::::::::;

test voice port 0/1/3 pcm-dump disable


conf t
no voice pcm capture buffer 200000
no voice pcm capture destination flash:


On dial peer:
conf t
voice pcm capture buffer 80000000
voice pcm capture destination flash:
dial-peer voice 1
pcm-dump caplog FFFFFFF 254
!

After the all is done, to stop:

conf t
no voice pcm capture buffer 80000000
no voice pcm capture destination flash:
dial-peer voice 1
pcm-dump caplog disable
!

Intermittent one-way audio issues Resolved!

So if something is 99% of the time related to a certain issue it is probably the cause right?  Well, being stubborn and convinced it wasn't that 99% I found out today it was.  I did a PCM capture of the phone call exhibiting one way audio and you can hear both parties talking but yet only one way audio.  I did a debug mgcp packets and saw the channel in use and everything looked good there.  ISDN Q931 was clean and there were no transcoders or MTPs involved so what is the issue you might ask?

Well, 99% of one-way audio issues are network related.  I didn't adhere to that rule since I was so sure it was something else.  I figured since the issue followed the phone it couldn't possibly be the network!  I didn't stop to consider that when pushing it out the PRI, the network in use is still that same gateway.  It turns out the customer had two default ip route statements, one pushing to an IP address and another pushing it over the wire via interface.  We pulled the 0.0.0.0 0.0.0.0 fa x/x route out of there and everything started to magically work.  Now the strange part is that it really shouldn't have mattered unless there is something I am missing?  I'm not a CCNP R/S so I might be talking out by bum here.

Lesson learned: If its 99% possible that something is the cause, go check the damn 99%!

Intermittent One-Way Audio on Select phones

So today I was working a one-way audio issue that ultimately received a band-aid and not a solution.  A client has older 7940 phones and only two are acting up at a particular location.  All other phones are on that subnet and PRI are fine.  At first, I wanted to blame the network since almost all one-way audio issues are network related but I find that hard to believe if the rest of the phones are working fine.  Then, I wanted to blame the phones themselves and sure enough the problem follows the phone.  The only problem is that these phones have been replaced only for the problem to show up a day later or a week later or whenever.

At this point I am thinking PRI.  I push the call out another PRI and the phones are fine.  This nixes the phones being bad as they worked immediately after switching from one PRI to another.  This goes back to the PRI being part of the problem but I can see packets being transmitted and received at the proper G.711u-law codec.  Ultimately, I am going to have to see if the carrier is willing to troubleshoot since the problem is most likely something sporadic on their side.  This problem has been a hindrance since before a major upgrade and followed it afterwards.  It is strange for something like this to be popping up like this.  More information will follow as I find out the issue.

Time between posts

I know it's been awhile between posts but the holidays are here and having just ended Thanksgiving things are still a bit slow.  I'm trying to figure out what I could post that would be good for educational content as lessons learned are at an all time low as they usually are around this time of year.

I just wanted to put a post up showing I am still actively here but just haven't had any good content to put up that would be worth reading.  I do have a pending outbound call issue that could potentially be useful once I resolve the issue but the ticket hasn't landed yet so I don't know what all is involved.

For what it's worth, the Cisco PEC GoldLabs for UC/Collab are actually fairly well designed.  It's clear some of the labs were built around physically being on site but you can do all of them remotely via RDP.  There are some errors here and there in the workbooks but nothing that causes issues with following along with the exception of a VoH/MediaSense portion that wanted me to configure an EX90 that the lab left out completely.  Even with auto-registration on, you need to be able to access the EX90 and it's IP address is completely left out of the documentation.  The image they show that auto-registered isn't accurate as I tried that IP and got nothing.