Search This Blog

Thursday, December 17, 2015

Voice PCM capture

I figured two articles in one day would be cool....  Anyways, in my last article I wrote about a PCM capture.  This essentially lets you capture the voice stream after digitization from the DSPs and play it back.  I know that can be scary since you can listen to anyone's conversation but it was required and authorized in my case since I needed to find out if everything was worky worky on the PRI side of things.  So below is the script you need to capture voice, please use this responsibly.


  1. Debug mgcp packet (if applicable)
  2. Debug isdn q931
  3. voice hpi capture buffer 1000000
  4. voice hpi capture destination flash:pcm.dat
  5. exit
  6. test voice port 0/0/0:23.23 pcm-dump caplog 7 30
  7. :::::::::Make Test Call::::::::::::;
  8. test voice port 0/0/0:23.23 pcm-dump disable
  9. conf t
  10. no voice hpi capture buffer 1000000
  11. no voice hpi capture destination flash:pcm.dat

Note this is for prior to IOS 15.  Also, your voice ports are going to be different based on whichever channel gets opened for the bearer channel  for IOS 15 and later:



voice pcm capture buffer 200000
voice pcm capture destination flash:
exit
!
test voice port 0/1/3 pcm-dump caplog 7


:::::::::Make Test Call::::::::::::;

test voice port 0/1/3 pcm-dump disable


conf t
no voice pcm capture buffer 200000
no voice pcm capture destination flash:


On dial peer:
conf t
voice pcm capture buffer 80000000
voice pcm capture destination flash:
dial-peer voice 1
pcm-dump caplog FFFFFFF 254
!

After the all is done, to stop:

conf t
no voice pcm capture buffer 80000000
no voice pcm capture destination flash:
dial-peer voice 1
pcm-dump caplog disable
!

Intermittent one-way audio issues Resolved!

So if something is 99% of the time related to a certain issue it is probably the cause right?  Well, being stubborn and convinced it wasn't that 99% I found out today it was.  I did a PCM capture of the phone call exhibiting one way audio and you can hear both parties talking but yet only one way audio.  I did a debug mgcp packets and saw the channel in use and everything looked good there.  ISDN Q931 was clean and there were no transcoders or MTPs involved so what is the issue you might ask?

Well, 99% of one-way audio issues are network related.  I didn't adhere to that rule since I was so sure it was something else.  I figured since the issue followed the phone it couldn't possibly be the network!  I didn't stop to consider that when pushing it out the PRI, the network in use is still that same gateway.  It turns out the customer had two default ip route statements, one pushing to an IP address and another pushing it over the wire via interface.  We pulled the 0.0.0.0 0.0.0.0 fa x/x route out of there and everything started to magically work.  Now the strange part is that it really shouldn't have mattered unless there is something I am missing?  I'm not a CCNP R/S so I might be talking out by bum here.

Lesson learned: If its 99% possible that something is the cause, go check the damn 99%!

Tuesday, December 15, 2015

Intermittent One-Way Audio on Select phones

So today I was working a one-way audio issue that ultimately received a band-aid and not a solution.  A client has older 7940 phones and only two are acting up at a particular location.  All other phones are on that subnet and PRI are fine.  At first, I wanted to blame the network since almost all one-way audio issues are network related but I find that hard to believe if the rest of the phones are working fine.  Then, I wanted to blame the phones themselves and sure enough the problem follows the phone.  The only problem is that these phones have been replaced only for the problem to show up a day later or a week later or whenever.

At this point I am thinking PRI.  I push the call out another PRI and the phones are fine.  This nixes the phones being bad as they worked immediately after switching from one PRI to another.  This goes back to the PRI being part of the problem but I can see packets being transmitted and received at the proper G.711u-law codec.  Ultimately, I am going to have to see if the carrier is willing to troubleshoot since the problem is most likely something sporadic on their side.  This problem has been a hindrance since before a major upgrade and followed it afterwards.  It is strange for something like this to be popping up like this.  More information will follow as I find out the issue.

Tuesday, December 1, 2015

Time between posts

I know it's been awhile between posts but the holidays are here and having just ended Thanksgiving things are still a bit slow.  I'm trying to figure out what I could post that would be good for educational content as lessons learned are at an all time low as they usually are around this time of year.

I just wanted to put a post up showing I am still actively here but just haven't had any good content to put up that would be worth reading.  I do have a pending outbound call issue that could potentially be useful once I resolve the issue but the ticket hasn't landed yet so I don't know what all is involved.

For what it's worth, the Cisco PEC GoldLabs for UC/Collab are actually fairly well designed.  It's clear some of the labs were built around physically being on site but you can do all of them remotely via RDP.  There are some errors here and there in the workbooks but nothing that causes issues with following along with the exception of a VoH/MediaSense portion that wanted me to configure an EX90 that the lab left out completely.  Even with auto-registration on, you need to be able to access the EX90 and it's IP address is completely left out of the documentation.  The image they show that auto-registered isn't accurate as I tried that IP and got nothing.

Tuesday, November 17, 2015

ESXi and NIC Teaming gone wrong

Lately I have had the pleasure of discovering a new issue that has risen up from the 1s and 0s.  Currently, there is a network that is not using LACP but using NIC teaming, this is fine.  The weird part is, one of their UC VMs randomly stopped talking outside the subnet.  Rebooting the VM did nothing and it kept looking like it was more and more of a network problem.  A TAC case was opened and even Cisco was pointing the finger to the network.

Without being to in detail on ESXi, since I'm no guru, I took a look at the network side of the house and everything looked like it was in working order as well.  Pings to and from the ESXi were working but for some strange reason on VM was sending data out one NIC and receiving it on the other.  This was the only VM doing this and I think if LACP was enabled it would never have happened.  The end result was a VM that wasn't talking correctly and couldn't communicate with it's subscriber.  We ended up moving one of the NICs from active to standby and ran a test.  Everything magically worked itself out as all traffic was being forced over one lane.  We then put it back into active mode and everything still was working fine.  Weird.... How did the TCP/UDP data streams end up getting hosed up?

I checked again today and traffic was still evenly distributed and no weirdness was going on behind the scenes with one VM sending out one port and receiving on the other.  I'm not sure this was ever a problem as again, I'm not an ESXi expert but it seemed to have resolved the issues and they haven't returned.  This also seemed to have fixed a second issue they were getting which was calls going to voicemail but then terminating after 5 seconds.  If this was indeed the resolution, nothing was being shown as wrong anywhere on the network and no errors were being thrown.  Just a heads up for any of you that run into this issue yourselves.  Just bounce the port or go standby then back to active, it should end with the same result.

Friday, November 13, 2015

7940 / 7960 DST change and CUCM 6.x - 7.x

Last week I was working on a set of problem phones on a really old version of CUCM.  The problem was that when daylight savings time ended, all the newer phones rolled back an hour with the change of NTP based on the Date/Time group but the older 7940s and 7960s did not.  This is a constant yearly issue with this particular client's CUCM as they are on 6.x.  After doing some research I found a bug that affects the 6.x train and also appears to go all the way to 8.0.  This bug essentially holds the 40s and 60s back and you either have to manually change the time, update the CUCM with a .cop file, or just wait a week for them to change.

The weird part about all of this is that I changed it earlier this year to bump it ahead an hour and I didn't have to change it again afterwards.  I cycled the time back to -7 time since I am on central time and everything was fine.  Later last week I got a call that their phones had once again went forward an hour.  This didn't make sense at all.  I went in and did a screen cap of the 7940 and sure enough, it was showing eastern time again.  I set the date/time group back to central time and this ultimately resolved the issue.

So the lesson learned is:

  1. Upgrade your CUCM
  2. Get rid of those ancient phones
  3. Upgrade and get rid of the phones
  4. Wait a week for the time to change
  5. Update the CUCM with a COP file

Why this issue still persists is beyond me.  This is something that was going on back when 6.x and 7.x were still supported over 4 years ago or so.  It should have been fixed then but never was.  Still, all the more reason to get off of an MCS server and move over to a BE6k or BE7k!

Wednesday, November 11, 2015

T1s being T1s

So there I was, end of the day and a ticket came in for a long distance code not working for a customer.  Easy, probably doesn't have a FAC on CUCM for some reason, just as the others didn't.  Well, long story short, there was a FAC and the call was hitting the gateway and I was getting a 41 cause code via RTMT.  DSPs were fine and the T1 looked good.  Came in this morning figuring whatever the carrier had issues with was fixed but nope, still hosed.  More tickets were coming in saying their long distance codes stopped working.

This was the last 24 hours of my work life.  Everything looked good, no errors, no issues except randomly failing calls out a particular site.  Sure, I could have re-routed the entire site out another gateway but that would have taken significant time, even with BAT changing out the CSS to route out the other direction.  That would also put a huge load on the other T1 bundle and soak up a crap ton of ports and DSPs.

So I sat there, in discontent wondering what was going on.  Logs looked good, traces were fairly clean and still Cause Code 41.  I tried rapid long distance dialing and got a few calls to work.  My thoughts immediately went back to the carrier.  I even tried changing out the gateways but since both T1 CASs were failing that was also doomed to fail.  I was talking with another voice guy and he figured we should try to reset the T1 CAS.  I thought..why didn't I try this?  So simple...  My thought process was that it was broken however, and now I know better.  Reset it even if it is appearing to be working, who cares since long distance isn't working anyways right?  I reset the port made about 10 calls and had the customer call me back and everything is hunky doory.

So what was the problem?  Damned if I know.  Neither of us did anything and the carrier sure didn't do anything.  A reset seemed to fix an invisible hangup on the T1.  Note to self, just reset the damn T1, even if it doesn't look like it needs it!  Also, who the heck still uses T1 CAS?  That's the first time I saw a CAS in about 8 years.

Tuesday, November 3, 2015

Unity Connection integration with an analog PBX system

Recently I got the awesome experience of integrating Unity Connection into an analog setup.  While the customer does indeed use Cisco, the non-Cisco side is still in place for a few reasons and needs to continue to work.  Their old Unity box was decommissioned but then brought back on.  When the phones received calls, they would send the voicemail to the Unity Connection box as you would expect.  The problem is, the old Unity box was still for some reason triggering when the messages button would be pressed on the non-cisco phones.

After poking around a bit I found they have a DMG or Digital Media Gateway.  This had routes to Unity Connection but was listed second in the top down list.  The first thing to fix was this and get rid of the Unity route completely.  The issue is, being unfamiliar with the DMG, I was rolling the dice.  The configuration menus were not too difficult to figure out but I still didn't full understand everything.  It turns out however, my solution was correct.

The next stop was to make PIMG ports on the Unity Connection box.  This will allow the analog phones to get their MWI and access their voicemail on demand when pressed.  Now the only thing I still don't fully understand in the first place is how the Unity Connection was being used by the DMG as voicemails were being delivered there somehow without the PIMG ports.  My guess is that it hit a route elsewhere when it got sent to the Cisco side.  However, being on a time schedule to getting something fixed since it was paid per hour, I didn't have too much time to try to figure that aspect out.

Friday, October 23, 2015

CCNA Lab #3

One more CCNA R/S lab for those folks still following my data rant.  Configure this one up the best you can and get it working in record time.


And here are the instructions:


I'm going to record me doing this lab so that people can follow along.  Keep in mind this might be a bit long as I'm bound to make mistakes.  I think this will help out those who need it though.  Recording will be posted once I get it made and uploaded.


Apparently I am not going to be able to record the session at the moment.  Windows as usual is acting up and making life difficult.  I will still try to get it recorded but I am not making any promises.  I may have to get this complete on my iMac since I never have issues on OS X.


UPDATE: I made a boo boo and Oklahoma should be 192.168.3.0 /26 and 192.168.3.64 /26.

Monday, October 19, 2015

A revisit on SIP

Month's ago I had put a post up on the basics of SIP and said that I was later going to get a little more in detail on the protocol. That post was mainly a very high level view with the core components. Now what I would like to do is take it one step further and start explaining some of the headers and how to read the SIP messages correctly.

I think one of the biggest issues with SIP is the pure amount of messages that are sent for every transaction.  Every time you setup a call you get an INVITE, every time you go on hold, INVITES are sent with certain SDP characteristics announcing a hold status so the call doesn't get torn down.  Every time something registers you get a REGISTER message.  Basically, everything is recorded down on what is going on.

Below is a basic SIP header from an INVITE, we will go over each header field that is relevant and annotate which ones are mandatory.  The thing is with SIP, you are required to have certain SIP headers regardless of what is going on.  While others may be optional, headers like To:, From:, Via:, Call-ID:, etc. are all necessary components to the SIP protocol.

Via: SIP/2.0/UDP 10.0.0.4:5060;branch=z9hG4bKs3uc2t20dgj04lkn51s1.1
From: <sip:2226541300@10.0.1.3:51088;user=phone>;tag=127.0.0.1+1+a5380157+652ceee3
CSeq: 818076443 INVITE
Expires: 180
Min-SE: 3600
Session-Expires: 3600
Supported: replaces, timer
Content-Length: 313
Request-Disposition: fork, sequential
Allow: INVITE, BYE, REGISTER, ACK, OPTIONS, CANCEL, SUBSCRIBE, NOTIFY, PRACK, INFO, REFER, UPDATE
Call-ID: 9AB0E358-1FFEC2pzsxRAnf9OsjW3of1wROpUYiBxisHiIwiB6zBrtY79UsAJ92FonzIYhJx1BEJKAr
P-Asserted-Identity: <sip:2226541300@10.0.1.3;user=phone>
Privacy: none
Max-Forwards: 66
Content-Type: application/sdp
User-Agent: Alcatel-Lucent 5020 MGC-8 8.1.0.11.SP1.1
v=0
o=root 2085757204 2085757204 IN IP4 10.0.0.4
s=session
c=IN IP4 10.0.0.4
t=0 0
m=audio 34732 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=silenceSupp:off - - - -
a=ptime:20
a=sendrecv
So with the above, I highlighted a few lines that I will go over.  First and foremost this is a INVITE meaning an endpoint is sending a request.  I am capitalizing this information because all requests are always CAPITALIZED, and will always get a response back, regardless if it is a success or failure.  The six requests that you can get in SIP are INVITE, CANCEL, BYE, REGISTER, ACK, and OPTIONS.  Responses would be something like a 200 ok, ringing, trying, etc.  These are always in response to a SIP REQUEST.  Now, yes I know there are others like a PRACK and SUBSCRIBE but those are not the six main requests/methods.  Again, everything starting in CAPITAL letters is a request and anything starting with a number is a response such as a 180 ringing.  See below for a response to a SIP request.  Please note this one does not coincide with the SIP message I posted above.

SIP/2.0 180 Ringing
Via: SIP/2.0/UDP 182.168.250.49:5060;branch=z9hG4bK786d156c;rport=5060
Contact: <sip:9011@182.168.246.213:35134;rinstance=7af05ded7e7e49e6>
To: <sip:9011@182.168.246.213:35134;rinstance=7af05ded7e7e49e6>;tag=9a00d038
From: "9012"<sip:9012@182.168.250.49>;tag=as66995cd4
Call-ID: 7cebe5d1060b11452571a22e0e2cb919@192.168.250.49
CSeq: 102 INVITE
User-Agent: X-Lite release 1002tx stamp 29712
Content-Length: 0

Now I wish I had a way of generating some SIP traffic right now but I don't have access to a test lab for SIP at the moment since my demo license expired and I'm not in the habit of using a production network for labbing and testing unless absolutely necessary.  Anyways, as you can see, the above message is a response as it starts with a number and generally is not capitalized.  You still see all the same basic headers here only without the SDP.  Please note these are SIP messages I have used from the web and I have changed some of the information intentionally just in case there is a confidentiality issue.

Now a bit more on the OPTIONS request.  This is becoming more and more common as we move forward into a pure IP telephony environment.  Amongst other things, OPTIONS is used heavily for something called an OPTIONS PING which monitors the status of a egress point on a gateway.  If it doesn't respond it can mark it as dead and go out another route.  The great thing about this is, before SIP, this wasn't really possible and you have to wait for timeouts to occur in the H.323 and MGCP world or worse, manually change it over.  The rest of the requests I have already gone over in the other post and can be easily discovered on the web.  At this point, I am assuming you have a high level knowledge of SIP. 

I'd like to go over some of the SDP markers in the first SIP message.  As you can see, it is quite extensive.  The bad part is, with voice, video, and presentations that list will double in size to create even more of an eye sore.  Starting with the "c=" line in the SDP we can see an IP address and that it is IPv4.  This line identifies where media is coming from and where it should be sent to.  This can be a good place to look to make sure that your network is working the way it should be.

Originator or "o=" is an indicator of who is initiating the connection.  This has a username (if applicable), their IP, and who they are.  It is as simple as that.  There are more stipulations as to how the field should be populated and those can be found in RFC 4566 if you want to get into the nitty gritty.

Next is the all important "a=" line which marks what codecs you are going to be advertising and accepting.  PCMA/8000 is G.711a-law and PCMU/8000 is G.711u-law.  In the United States we use u-law inclusively. Another way to remember it is u-law for the U<---nited States.  (Thanks Jeremy Ciaroa for that one!).  Other codecs can also be listed here like G729 as listed above, iLBC, etc.  

You will also see things like "a=sendrcv".  This can be an indication of what is going on with the media.  Is it expecting to send only, receive only, do both, or go inactive completely?  There are four different  attributes in this particular that you will commonly see:

  1. a=sendrecv
  2. a=sendonly
  3. a=recvonly
  4. a=inactive
Normally, in a hold situation you will see a a=sendonly followed by a response of a=recvonly.  This is indicative of a stream going on hold.  A new transaction is sent just for the hold feature.  Now, this isn't the only time you will see these tags but the most common in a phone environment is hold/resume.  so why is one send only and the other receive only?  Well the user placing someone on hold is presumed to be send only because they are going to be streaming music on hold from CUCM or something of the sort while the distant end will only be receiving and not sending and voice packets back.  You should only see your RX on your Cisco phone bumping substantially during this time.  The other possiblity during a hold session is a=inactive.  This indicates nothing is being sent or received and basically dead air is all that remains.  SIP has to maintain the connection somehow, so you will continue to see SIP INVITES and 200 ok messages with a=inactive until the session is resumed.  How else would you maintain that call?

Well, looking at everything above I think I will cut this post here.  I will continue on with more headers and SDP lines in a future post.  I don't want to risk someone falling asleep on me now since it is imperative you understand SIP as the replacement technology for H.323...if we were get there in our lifetime.  It seems like PRIs are still everywhere while SIP is being slowly implemented in more and more places.





Friday, October 16, 2015

GNS3 CCNA R/S Lab #2

EDIT: Note that I made a change to the topology. I decided two links to Salado was stupid since if that core went down Jim was hosed. Still create GLBP links on the inside but instead connect to Austin as the new image shows. Yep, still studying for the new CCNA R/S and whipped up a second lab that you can try on your own and see how far you get. This one is a little more complex than the other one. Also, on the other one, if the texas remote office couldn't ping across its because I had a tunnel setup and the remote computer had to send an ICMP or some sort of data to the distant end before the crypto tunnel would stand up. See below for the latest lab image. I won't bother adding in configs for you as this is supposed to be done from the ground up. I may add more to this later, I can't think of any exam objectives I would be missing at this point other than switching with STP/SRTP/PVST+....Come to think of it, I will post a file when I am 100% satisfied I have configured this lab to my satisfaction so you can get a working model and compare.



Please note that this lab may require in excess of 3-5 GB of RAM. My laptop has 8 GB and I was sitting right around 5.16 GB of RAM chewed up from this lab. I'm not taking into account all of my other programs open however, I had email, a ticketing system, and notepad++ open. For Jim's network, it is up to you to design a proper subnet for the GLBP and WAN links. Part of learning is doing something yourself! Whip up a DHCP server while you are at it for him too. IPv6 lab will be coming soon.

  • ensure the line console doesn't interrupt your typing
  • setup telnet and ssh except Salado-Exec, ssh should only be enabled for this gateway
  • configure both the console and vty lines for a password of cisco and enforce logging in
  • create a username of cisco and a secret password of cisco
  • create a secret enable password of cisco
  • make sure random words don't force DNS lookups if you make a mistake typing
  • configure all ip interfaces as shown
  • configure EIGRP in AS 1 and only send hello packets on interfaces that you own or networks that are completely behind the GW you are on
  • Configure the frame relay interfaces according to the DLCI chart
  • Configure the frame relay switch
  • EIGRP should be establishing adjacencies at this point, verify by pinging across
  • configure DHCP for 10.x.x.x networks and exclude .1 to .9 while setting DNS to 8.8.8.8 and the default-router to the .1 address
  • make sure all clients get their respective DHCP addresses and other information such as gateway and dns
  • configure NAT for the additional router added into the Salado portion to use PAT, ensure nat translations are occuring on the inside global address to the inside local
  • configure GLBP for Jim's setup and ensure the timers are set to miliseconds of 50 and 160 dead
  • configure static routes for Jim's network to Salado-Exec and beyond
  • configure default routes that all point to Corp-HQ in case a route is not known
  • configure a proper route in the correct gateways to ensure data gets routed statically to Jim's computer since he is a loner and no one likes him
  • configure eigrp md5 authentication between corp HQ and all other cores
  • configure PPP Chap between Jim's GWs
  • deny telnet to all gateways except from the engineers and IT networks
  • deny ssh to all gateways except from the engineers and IT networks
  • ONLY deny the CEO and CFO, permit the CTO to do his job with all engineering tasks like icmp, telnet,ssh
  • deny ICMP on all host networks except IT and engineering
  • deny http traffic from the corp sales department, they are committing shenangians on youtube

Wednesday, October 14, 2015

A CCNA Lab for those that need extra practice

I've already had my rant about not knowing CCNA R/S for new engineers or even existing ones. After several weeks of intense study I think I will be ready to go sit the new CCNA R/S exam. Yes I know, some may say it is easy, and I'm sure it is for those that do data all the time but I am a UC guy that is sliding back into the data realm slowly. Bottom line is, I miss data and all the cool routing and wireshark captures to find out problems and ways to fix them. I spent some time today finishing up a lab that I had made from scratch that tests most of the CCNA topics but not all. STP is difficult since you can't replicate the ASICS and I don't have an router IOS with a switch module to at least attempt to get STP up. I also haven't implemented GLBP / HSRP/ VRRP either even though they are fairly straight forward. I probably will add to the diagram but my back is killing me today and I haven't been in the mood to really build anything new aside from what I have already done. The below image is what I challenge you to replicate and get working 100% of the way.



At a minimum, here is what you need to configure:
  1. Connect all devices as shown
  2. Configure all ports with approrpriate addresses and bring them online
  3. Configure ssh access only with a password of cisco
  4. Configure a username of cisco and secret of cisco
  5. Configure an enable secret of cisco
  6. Ensure that the routers will not mistaken letters for DNS names
  7. Disable enable timeouts
  8. Set hosts with IPs so you can use their names instead of IP addresses
  9. Configure EIGRP and advertise networks based on subnet with no automatic summarization, do not configure Canada nor advertise its networks!
  10. Exclude the first 10 addresses in all ranges in the private networks for DHCP
  11. Configure DHCP for all Offices with a /24 subnet based on the network assigned
  12. Assign default routes to point to the local primary gateway of each site (not the office GWs)
  13. Ensure all PCs get their DHCP addresses assigned to them and they can ping to their local GW
  14. Configure the Frame Relay DLCIs as shown and map them as indicated
  15. The FR-Switch should not have any other configuration as it is acting like a cloud
  16. Assign static routes to get to Canada based on location (Texas will be a end all be all GW if other primary gateways don't know the correct path
  17. Configure NAT for Canada to use the internet IP with PAT
  18. Ensure all devices can ping to the private network in Canada
  19. Ensure Canada can ping from internal to any CORP based internal IP (i.e. 10.0.3.11, 10.0.0.1)
  20. Configure EIGRP md5 authentication for all links
  21. Make sure that any loopback interfaces assigned are being seen as their subnet in the routing tables
  22. Block all telnet traffic from all locations except Texas
  23. Block all SSH traffic on private LANs on all sites except Texas
  24. Verify NAT on the Canada GW
  25. Enable password encrpytion globally so that any plain text passwords are garbled at minimum security.
If you can do the above without having to look anything up, you are probably good to go or still competent in data. I myself had to look up how to do md5 authentication because it's just not something I do everyday...or at all really. I'm not a network engineer I'm a UC engineer. I challenge you to complete this as fast as you can and report your times with honesty. I will probably wipe this entire topology and start from scratch and do a YouTube recording when I get the chance. I want to see how fast I can go under pressure and what I do and do not forget. Keep in mind I set all DLCI interfaces to a /24, you can modify as you want.

Monday, October 5, 2015

The Importance of CCNA in the UC Realm

I've been on my soapbox before about knowing data if you are going to be a specialty engineer, whether that be UC, wireless, security, data-center, or whatever else. I had some time on my hands so I built out a nice GNS3 lab to test my data skills. Granted I did cheat a bit since I had gone through the new CCNA R/S course on CBT nuggets, but it is still necessary to know the underlying foundation. My biggest gripe right now is that Cisco wiped out the need for a CCNA R/S as a pre-requisite to CCNA Collab and CCNP Collab. With this new road map, new engineers will be coming into the field for UC not knowing the different between router and switch. Ok, maybe that was a bit too much, but still, the problem remains. Ask them to troubleshoot STP or check a VLAN and you are going to get a blank stare. Anyways, below is a GNS3 image of what I built out. It is still a work in progress and I plan on using this to gauge other engineers's capabilities with data.
The above diagram hosts a variety of different situations that need to be configured. All the internal sites are using OSPF and the WAN connections are using EIGRP. Yes, I could have used OSPF or EIGRP across the board but part of knowing CCNA R/S is route redistribution. Thus, I used two protocol for that part. The other part is the connection to the ISP, I have a static route in place but also put in BGP as I plan on expanding that diagram to another company and bridging together from there. While BGP really isn't touched on much in the CCNA world, it is still mentioned since it is heavily used at the provider level. Additionally, different networks for each gateway per floor that go in sequence as well as ACLs to limit ICMP, web access, ssh, and telnet accordingly. Finally, when egressing to the internet, all gateways will NAT to their public IP (once I get the other ISP gateways put in there). I will also probably add some security on the routers for neighbor authentication for the hell of it as well. It's stuff like this that is important, if the ASICs could be emulated I would be running real switches as well. The switch modules available can do the job though if I really wanted to run STP and get a loop free topology while using etherchannel. Just in case anyone is doubting the configuration:
I know this wasn't really voice related and more of a rant, but I haven't run into anything interesting or different on CUCM lately. I'll try to break something on accident and post about it :)

Friday, September 25, 2015

Modifying device line buttons saying "Fixed Feature"

So today I was whipping up a new DID scheme for a user since they have been getting to many calls on their current lines since they were not being masked correctly. Easy enough. I got a new set of DIDs for them in the pool and assigned them appropriately and let the know that their voice mail would be changed since the DIDs were changing and everything is LDAP synced. Yes, technically I could do it the hard way but we don't hand code anything unless necessary for voice mail. I had a few phones that had a message on them when I was modifying the line order saying "Fixed Feature Button". This didn't make much sense to me since nothing on that line is a feature other than the line itself and the other phones didn't have this problem. After investigating a little more I saw people completely deleting that line and to me, that isn't an acceptable solution. I ended up disassociating that line with the phone then adding it back in without redoing everything. Since this happened on three different phones, deleting and re-adding would have taken to much time. Once I disassociated the line on the phones then put the line back on all was good. Not a huge deal, but a I guess this was a bug from 9.X that still hasn't been resolved...

Tuesday, September 22, 2015

Call Pickup Group "Feature is Unavailable" Message

So today I was presented with a very simple yet difficult problem. A customer had a problem with their call pickup group. Any time they would try to pick up a call after seeing the visual alert it would say "Feature is Unavailable". After doing some research on the problem I came up empty handed with others having this same problem. I made a few test calls and GCallPickup and then dialing the ringing number worked yet CallPickup didn't. Perplexed, I had the customer go try another set of phones in another pickup group and everything worked as it should. Absolutely nothing was different between the two configurations other than a the pickup group number (obviously...). So after an hour or so of wasted time troubleshooting and checking everything I created an entirely new pickup group and assigned the phones to it and voila, everything worked. I then deleted the old pickup group and changed the new one back to the old one's name as well as it's number and guess what, everything worked. My only assumption as to why this randomly started to fail all of a sudden is a database issue and removing it while creating a new one was the only solution. No other settings like extension mobility were in play and certainly all of the settings were correct such as the CSS and partitions. Weird...but at least it is fixed. I highly doubt TAC would have an answer other than the solution I ultimately came up with.

Thursday, September 17, 2015

The Hunt Pilot Timer

Everyone has those moments where stupidity overrides thought processes. Yesterday I spent about 30 minutes on something that should have taken less than 3. I was providing support for a customer after I migrated their CUCM from a physical box to a virtual host and they wanted to do some additional configuration on their main line calling in. No big deal, slapped a second line group in and added it to the Hunt list and adjusted timers accordingly. After some fine tuning we got the line groups to ring properly but the second line group would always cut off prematurely. After trying to figure out what was going on and checking things I knew were right, it hit me. After you rang out it would hit a voice mailbox which just happened to be read by Cisco's voice and I knew exactly what was happening. The hunt pilot timer was set to 15 seconds and that was seconds more than then the second line group would hit which only allowed a very brief ring on the second group. I adjusted that timer to 30 seconds and bam, everything worked flawlessly. Sometimes you just need a small sample of stupid to realize that everything isn't complicated unless you make it that way. I think part of the reason I was forgetting the Hunt Pilot timer was due to the lack of sleep from the migration on the night prior. Who knows, maybe I'm making excused for a novice mistake?

Tuesday, September 15, 2015

Unsupported ESXi License that works

Up on the roadmap to be complete today is an MCS to ESXi migration for a clients CUCM. While it isn't a UCS box which makes the entire design not supported, their ESXi is not running a supported version either. They are currently running on 5.5 which, by all intents and purposes, works great. We all know we can load up a ESXi box and load CUCM on it with no issues. Heck, you can load it up on a box with an intel processor then migrate to AMD and it works if you really wanted to but PCD, the migration software that does all the magic in the background specifies certain license sets of ESXi due to certain issues pertaining to access and such. It turns out VMWare Essentials Plus 5.5 works fine despite not being listed in the supported sets of ESXi licensing on Cisco's site. I thought this important as some may need to migrate in the situation that I am in and not know if the ESXi is supported. One sure fire way of finding out is trying to add the ESXi host to the inventory in PCD. It either works or it doesn't. The good thing is, from here on out, I know an extra piece of information that can help out in the future and I wanted to share that with everyone here as well. The other option is to get a key or get a trial license and temporarily apply that to ESXi then bounce back when you are done since PCD is just a temporary situation to begin with.

Tuesday, September 8, 2015

Unity Connection Greeting Recording

This is a post I have been meaning to put up for a few weeks now. Basically, Cisco refuses to get with the times and use another method to record meetings via the web interface on unity connection. Java is and always will be a big issue with anything Cisco related. When you try to record on your PC you usually get SSL errors and other errors, even with IE. The only way around this is to downgrade your java so far down the ladder, it becomes a security risk. This essentially forces you to stand up a VM or use another box just for recording Unity greetings. You might say just record via phone, which is the best option. For those of you using a VPN connection or have other issues and can't go that route you are stuck with the web interface. You can either create the WAV file by computer recording or upload it from your desktop if you already have the file. When doing this you get the SSL errors I was talking about earlier. The way that I have found to get around the errors is to record via computer but do not save the recording after finishing. Instead, save a copy to your desktop and reload the page. From here, upload the file to the media player again, select the "phone" playback option and have it dial whatever number you want and leave "computer" as the recording option. Open the file up via the web player and hit play with those options set. Answer the phone and hit save while it is playing through your CIPC or phone and you will not get the error. From here, reload the page and hit play to verify it saved and you are done. Now why do we have to go through this for those of us on company laptops that have stringent security requirements and cannot downgrade java?

Wednesday, September 2, 2015

UCCX and UCCE with Voice and Video

Video Call and Video Chat for Cisco UCCX/UCCE
Video in contact center is something we’re hearing more and more about lately. Video call and video chat will most likely become conventional contact center channels thanks to WebRTC and zealous marketing from major UC&CC vendors.
Let’s not put here marketing stuff about how cool it is. Instead, we’ll consider ways to enable video in Cisco UCCX/UCCE.
The first thing that comes in mind is Cisco Jabber Guest – a native solution from Cisco that helps web-visitors interact with enterprise workers, including contact center agents. Users click the URL link and their browser open an audio/video session to wherever Jabber Guest Server routes them. On the back-end you’ll need the Cisco UCM, Cisco Jabber Guest server with Cisco Expressway-C/-E and internal video-enabled endpoint.
As guests, your website visitors are offered:
  • a point-to-point audio/video,
  • in-call control through a keypad,
  • the ability to mute video, audio, or both.
While this could be enough to offer a guest access to your employees (for associates, suppliers, partners etc), the use of Jabber Guest in the contact center is a big question because of two major issues:
  1. Jabber Guest requires a browser plug-in to be installed before making a call. A client will hardly agree to spend his time installing the plug-in to make a single call to your contact center. He heeds a “click-to-video” feature, but not the “click-to-install-to-video” one.
  2. When calling through Jabber Guest a client has no collaboration features. The dual video definitely improves the communications  – you see the person you’re talking to and how he reacts to what you’re saying. But to provide the real LIVE assistance your agents need to collaborate with clients like they’re in person – exchange the docs, co-work on the website, show, point, discuss etc.
So, when thinking about video call / video chat channel for your contact center you should also consider third party solutions designed specifically for the customer service. The key features that make them different from Jabber Guest are:
  • no downloads or plug-ins are required even for mobile browsers (thanks to WebRTC),
  • integration with contact center software,
  • web-collaboration features – chat, app sharing, co-browsing,
  • interaction recording.
There are several vendors of video chat solutions on the market, of which Aurus RichCall is the only that claims to be designed specifically for Cisco UCCX/UCCE:
  • it inherits UCCX/UCCE agent accounts, skill groups and routing rules,
  • the video call bypasses the IVR menu and gets routed right to the appropriate skill group,
  • the agent’s UI is integrated with Cisco Finesse or Cisco Agent Desktop (CAD),
  • each video chat session is  reflected in Cisco Unified CCX Historical Reports and Cisco Unified Intelligence Center,
  • the audio is recorded by the call recording software already deployed in contact center.
Seems to be quite a niche solution, but developed by Cisco Solution Partner for Cisco contact center platforms. So, if you’re running the UCCX/UCCE-based contact center, it may be of your interest.

Tuesday, September 1, 2015

Customer service done correctly



I spent some time today thinking.  I have been told by numerous sources how I am good in front of customers.  They tend to like me, request that I do things, and overall I improve the relationship between people.  Now I'm sure I'm not the only one doing this in my current company, but I wanted to share some things as an engineer that I think are what contributes to my success so far.  I know this isn't voice related but it is professional related to any field out there.  The more you use the foundation of being a human being and professional the better the results.  Remember, integrity, ethics, courteousness, and knowledge!

-----------------------------------------------------------------------------------------------------------------------------


Customer Service Done Correctly

Over the last twelve years I have experienced much and if it is one thing I have learned, the more humanly you treat people, the better the results.  This thought process didn't radiate at first with me.  When I was in my adult teens and early to twenties I felt like I had something to prove.  I think that might be a stage that everyone goes through; I just went through it quicker.  If it's one thing the military does, it is get you into a mature state of mind earlier.  Having seen the world and the plights of different cultures, one thing has remained a constant, treating people like human beings.  You would be surprised at how well people with completely different beliefs or thought processes can get along.
Since my exit from the military due to medical issues, I have gained insight as to how to handle customers, even when they are upset.  The mindset I keep can turn a relatively rocky start into a blooming relationship in which a customer may request you to perform a service you might not be qualified for.  The reason behind the request is that they are both comfortable with you and know your ethics and standards.  Every professional in any field should have a high level of both of those areas, ethics for doing the right thing despite the consequences and standards for getting something done correctly and efficiently while maintaining proper records of everything for later.
When it comes to servicing requests from customers, everyone involved should always be one-hundred percent informed.  Even if it was something as simple as a ticket already completed by someone else, someone should know.  When someone knows, someone can act properly in the event something else occurs.  If someone already removed a phone description and replaced it with something else that was requested, record that it was already done, follow up with the customer and move on to the next task.  Every time you inform someone that something is done, the concept of work is being done is felt.  With that feeling, a relationship begins to form.  Continuing on that relationship is entirely up to you and how you handle future encounters.
That brings me to the next idea that has popped off since talking about customer service.  The more detailed and well-presented information, the more likely you are going to give the customer, co-workers, and manager the warm and fuzzy.  Keep things presentable and neat as well as well documented.  If you start annotating times you did things, great, keep at it but remember to only get as detailed as you can within the allotted time.  My way of documenting and reporting may be completely different from the way you do it but as long as it is presented well and detailed everything should flow.  A good example of what not to do when handling documentation is to be vague or brief such as the following:
·         "Fixed the directory number."
·         "Added phone"
·         "Created new dial plan for end user to accommodate request"

The above examples are worthless as anyone can write those sentences.  Again, you need to be detailed and well presented.  Different circumstances will force different documentation.  For instance, putting information in a spreadsheet and sent in an email is fantastic but might not work so well in a service ticket unless you attach the file.  You may need to make a bulleted list of the things you did.  Instead of doing what I showed in the examples above, you should be getting more detailed like such:
·         "Removed directory number xxx-xxxx as it was the wrong DN assigned and in the wrong partition"
·         "Assigned a new DN (xxx-xxxx) to the device and placed it in the correct partition of 'PT-Blah'"
·         "Added one Cisco 7965 with a MAC of AAAA.AAAA.AAAA and a DN of xxx-xxxx"
·         "Called customer to confirm everything is as it should be, (customer name) confirmed that everything is functioning properly, closing ticket"
The above example is everything I would normally put into one ticket if it was pertaining to an IP Phone configuration problem or add/change/remove.  When you get that detailed, people will know what happened, why it happened, and how to fix it in the future if you are not around.  Furthermore, that level of detail shows time, care, and knowledge.  If you took the time to write detailed information, you probably care about the issue and want to make sure that the customer is taken care of.  Additionally, making that final call to the customer for confirmation will make them happy, even in the worst of days.  I had one customer that was absolutely irritated at the world when I was assigned a ticket that involved rebuilding a call server.  I got it all completed, updated them, and showed them screenshots of me fixing everything every step of the way.  I called them to let them know it was fixed and even invited them into a WebEx Meeting (online screen sharing, video, and audio meeting) to show them that everything is the status quo.  The next thing I know, I am going out to that very location to do a review of what was done on an entirely different project not even related to my profession.
When presented with a problem, go out and tackle it head first.  If you mess something up, let the customer know and reassure them that you are taking the proper corrective actions to remedy the situation.  No one ever grows and gets better by staying idle in their small corner that they are comfortable with.  No one is perfect in this world, we all make mistakes.  Some may say that is easy to say but hard to deal with and I disagree.  The more you learn and try, the better you become.  I once deleted a directory number in a server that wiped out 20 phones.  I could have sworn it wasn't me as I wouldn't make a mistake like that.  I went back and checked and ultimately found out I did, despite not remembering.  Either I was really tired that day or slammed with tickets.  I went back into my ticket log, found my actions and referenced that with the audit log of changes and sure enough I screwed up.  I informed the correct people and the issue was resolved.  Don't have fear of screwing up and don't have fear of telling someone the truth, the more you do it, the easier it gets.  This doesn't mean that you should go busting up servers then telling people, just be honest and in return you will learn and be treated well.
All of this leads back to treating people like a human being.  Despite some peoples’ bad behavior or attitudes, you are there to do a job and get it done correctly.  Being courteous, efficient and detailed, people will calm down and start talking.  I do understand that some people won't fit this bill, some are just downright rude, but you still have to try and you still must be courteous, efficient, and detailed.  If someone is being hard to deal with and you still get the job done, they will probably come back assuming everything is understood and aren’t misled or misinformed as to what occurred during the entire job.  Those that are disrespectful, you can simply choose to not do business with them again.  The smile, candor, and professionalism should always be present; you owe it to the customer because they are paying your bills for both yourself and the company that you represent.