Time between posts

I know it's been awhile between posts but the holidays are here and having just ended Thanksgiving things are still a bit slow.  I'm trying to figure out what I could post that would be good for educational content as lessons learned are at an all time low as they usually are around this time of year.

I just wanted to put a post up showing I am still actively here but just haven't had any good content to put up that would be worth reading.  I do have a pending outbound call issue that could potentially be useful once I resolve the issue but the ticket hasn't landed yet so I don't know what all is involved.

For what it's worth, the Cisco PEC GoldLabs for UC/Collab are actually fairly well designed.  It's clear some of the labs were built around physically being on site but you can do all of them remotely via RDP.  There are some errors here and there in the workbooks but nothing that causes issues with following along with the exception of a VoH/MediaSense portion that wanted me to configure an EX90 that the lab left out completely.  Even with auto-registration on, you need to be able to access the EX90 and it's IP address is completely left out of the documentation.  The image they show that auto-registered isn't accurate as I tried that IP and got nothing.

ESXi and NIC Teaming gone wrong

Lately I have had the pleasure of discovering a new issue that has risen up from the 1s and 0s.  Currently, there is a network that is not using LACP but using NIC teaming, this is fine.  The weird part is, one of their UC VMs randomly stopped talking outside the subnet.  Rebooting the VM did nothing and it kept looking like it was more and more of a network problem.  A TAC case was opened and even Cisco was pointing the finger to the network.

Without being to in detail on ESXi, since I'm no guru, I took a look at the network side of the house and everything looked like it was in working order as well.  Pings to and from the ESXi were working but for some strange reason on VM was sending data out one NIC and receiving it on the other.  This was the only VM doing this and I think if LACP was enabled it would never have happened.  The end result was a VM that wasn't talking correctly and couldn't communicate with it's subscriber.  We ended up moving one of the NICs from active to standby and ran a test.  Everything magically worked itself out as all traffic was being forced over one lane.  We then put it back into active mode and everything still was working fine.  Weird.... How did the TCP/UDP data streams end up getting hosed up?

I checked again today and traffic was still evenly distributed and no weirdness was going on behind the scenes with one VM sending out one port and receiving on the other.  I'm not sure this was ever a problem as again, I'm not an ESXi expert but it seemed to have resolved the issues and they haven't returned.  This also seemed to have fixed a second issue they were getting which was calls going to voicemail but then terminating after 5 seconds.  If this was indeed the resolution, nothing was being shown as wrong anywhere on the network and no errors were being thrown.  Just a heads up for any of you that run into this issue yourselves.  Just bounce the port or go standby then back to active, it should end with the same result.

7940 / 7960 DST change and CUCM 6.x - 7.x

Last week I was working on a set of problem phones on a really old version of CUCM.  The problem was that when daylight savings time ended, all the newer phones rolled back an hour with the change of NTP based on the Date/Time group but the older 7940s and 7960s did not.  This is a constant yearly issue with this particular client's CUCM as they are on 6.x.  After doing some research I found a bug that affects the 6.x train and also appears to go all the way to 8.0.  This bug essentially holds the 40s and 60s back and you either have to manually change the time, update the CUCM with a .cop file, or just wait a week for them to change.

The weird part about all of this is that I changed it earlier this year to bump it ahead an hour and I didn't have to change it again afterwards.  I cycled the time back to -7 time since I am on central time and everything was fine.  Later last week I got a call that their phones had once again went forward an hour.  This didn't make sense at all.  I went in and did a screen cap of the 7940 and sure enough, it was showing eastern time again.  I set the date/time group back to central time and this ultimately resolved the issue.

So the lesson learned is:

1. Upgrade your CUCM
2. Get rid of those ancient phones
3. Upgrade and get rid of the phones
4. Wait a week for the time to change
5. Update the CUCM with a COP file

Why this issue still persists is beyond me.  This is something that was going on back when 6.x and 7.x were still supported over 4 years ago or so.  It should have been fixed then but never was.  Still, all the more reason to get off of an MCS server and move over to a BE6k or BE7k!

T1s being T1s

So there I was, end of the day and a ticket came in for a long distance code not working for a customer.  Easy, probably doesn't have a FAC on CUCM for some reason, just as the others didn't.  Well, long story short, there was a FAC and the call was hitting the gateway and I was getting a 41 cause code via RTMT.  DSPs were fine and the T1 looked good.  Came in this morning figuring whatever the carrier had issues with was fixed but nope, still hosed.  More tickets were coming in saying their long distance codes stopped working.

This was the last 24 hours of my work life.  Everything looked good, no errors, no issues except randomly failing calls out a particular site.  Sure, I could have re-routed the entire site out another gateway but that would have taken significant time, even with BAT changing out the CSS to route out the other direction.  That would also put a huge load on the other T1 bundle and soak up a crap ton of ports and DSPs.

So I sat there, in discontent wondering what was going on.  Logs looked good, traces were fairly clean and still Cause Code 41.  I tried rapid long distance dialing and got a few calls to work.  My thoughts immediately went back to the carrier.  I even tried changing out the gateways but since both T1 CASs were failing that was also doomed to fail.  I was talking with another voice guy and he figured we should try to reset the T1 CAS.  I thought..why didn't I try this?  So simple...  My thought process was that it was broken however, and now I know better.  Reset it even if it is appearing to be working, who cares since long distance isn't working anyways right?  I reset the port made about 10 calls and had the customer call me back and everything is hunky doory.

So what was the problem?  Damned if I know.  Neither of us did anything and the carrier sure didn't do anything.  A reset seemed to fix an invisible hangup on the T1.  Note to self, just reset the damn T1, even if it doesn't look like it needs it!  Also, who the heck still uses T1 CAS?  That's the first time I saw a CAS in about 8 years.

Unity Connection integration with an analog PBX system

Recently I got the awesome experience of integrating Unity Connection into an analog setup.  While the customer does indeed use Cisco, the non-Cisco side is still in place for a few reasons and needs to continue to work.  Their old Unity box was decommissioned but then brought back on.  When the phones received calls, they would send the voicemail to the Unity Connection box as you would expect.  The problem is, the old Unity box was still for some reason triggering when the messages button would be pressed on the non-cisco phones.

After poking around a bit I found they have a DMG or Digital Media Gateway.  This had routes to Unity Connection but was listed second in the top down list.  The first thing to fix was this and get rid of the Unity route completely.  The issue is, being unfamiliar with the DMG, I was rolling the dice.  The configuration menus were not too difficult to figure out but I still didn't full understand everything.  It turns out however, my solution was correct.

The next stop was to make PIMG ports on the Unity Connection box.  This will allow the analog phones to get their MWI and access their voicemail on demand when pressed.  Now the only thing I still don't fully understand in the first place is how the Unity Connection was being used by the DMG as voicemails were being delivered there somehow without the PIMG ports.  My guess is that it hit a route elsewhere when it got sent to the Cisco side.  However, being on a time schedule to getting something fixed since it was paid per hour, I didn't have too much time to try to figure that aspect out.

CCNA Lab #3

One more CCNA R/S lab for those folks still following my data rant.  Configure this one up the best you can and get it working in record time.

And here are the instructions:

I'm going to record me doing this lab so that people can follow along.  Keep in mind this might be a bit long as I'm bound to make mistakes.  I think this will help out those who need it though.  Recording will be posted once I get it made and uploaded.

Apparently I am not going to be able to record the session at the moment.  Windows as usual is acting up and making life difficult.  I will still try to get it recorded but I am not making any promises.  I may have to get this complete on my iMac since I never have issues on OS X.

UPDATE: I made a boo boo and Oklahoma should be 192.168.3.0 /26 and 192.168.3.64 /26.

A revisit on SIP

Month's ago I had put a post up on the basics of SIP and said that I was later going to get a little more in detail on the protocol. That post was mainly a very high level view with the core components. Now what I would like to do is take it one step further and start explaining some of the headers and how to read the SIP messages correctly.

I think one of the biggest issues with SIP is the pure amount of messages that are sent for every transaction.  Every time you setup a call you get an INVITE, every time you go on hold, INVITES are sent with certain SDP characteristics announcing a hold status so the call doesn't get torn down.  Every time something registers you get a REGISTER message.  Basically, everything is recorded down on what is going on.

Below is a basic SIP header from an INVITE, we will go over each header field that is relevant and annotate which ones are mandatory.  The thing is with SIP, you are required to have certain SIP headers regardless of what is going on.  While others may be optional, headers like To:, From:, Via:, Call-ID:, etc. are all necessary components to the SIP protocol.

INVITE sip:3212@10.0.0.153:5060;user=phone SIP/2.0

Via: SIP/2.0/UDP 10.0.0.4:5060;branch=z9hG4bKs3uc2t20dgj04lkn51s1.1

From: <sip:2226541300@10.0.1.3:51088;user=phone>;tag=127.0.0.1+1+a5380157+652ceee3

To: <sip:3212@97.208.31.204:5060;user=phone>

CSeq: 818076443 INVITE

Expires: 180

Min-SE: 3600

Session-Expires: 3600

Supported: replaces, timer

Content-Length: 313

Request-Disposition: fork, sequential

Allow: INVITE, BYE, REGISTER, ACK, OPTIONS, CANCEL, SUBSCRIBE, NOTIFY, PRACK, INFO, REFER, UPDATE

Call-ID: 9AB0E358-1FFEC2pzsxRAnf9OsjW3of1wROpUYiBxisHiIwiB6zBrtY79UsAJ92FonzIYhJx1BEJKAr

P-Asserted-Identity: <sip:2226541300@10.0.1.3;user=phone>

Privacy: none

Max-Forwards: 66

Contact: <sip:2226541300@10.0.1.3;user=phone>

Content-Type: application/sdp

User-Agent: Alcatel-Lucent 5020 MGC-8 8.1.0.11.SP1.1

v=0

o=root 2085757204 2085757204 IN IP4 10.0.0.4

s=session

c=IN IP4 10.0.0.4

t=0 0

m=audio 34732 RTP/AVP 0 8 18 101

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:18 G729/8000

a=fmtp:18 annexb=no

a=rtpmap:101 telephone-event/8000

a=fmtp:101 0-16

a=silenceSupp:off - - - -

a=ptime:20

a=sendrecv

So with the above, I highlighted a few lines that I will go over.  First and foremost this is a INVITE meaning an endpoint is sending a request.  I am capitalizing this information because all requests are always CAPITALIZED, and will always get a response back, regardless if it is a success or failure.  The six requests that you can get in SIP are INVITE, CANCEL, BYE, REGISTER, ACK, and OPTIONS.  Responses would be something like a 200 ok, ringing, trying, etc.  These are always in response to a SIP REQUEST.  Now, yes I know there are others like a PRACK and SUBSCRIBE but those are not the six main requests/methods.  Again, everything starting in CAPITAL letters is a request and anything starting with a number is a response such as a 180 ringing.  See below for a response to a SIP request.  Please note this one does not coincide with the SIP message I posted above.

SIP/2.0 180 Ringing
Via: SIP/2.0/UDP 182.168.250.49:5060;branch=z9hG4bK786d156c;rport=5060
Contact: <sip:9011@182.168.246.213:35134;rinstance=7af05ded7e7e49e6>
To: <sip:9011@182.168.246.213:35134;rinstance=7af05ded7e7e49e6>;tag=9a00d038
From: "9012"<sip:9012@182.168.250.49>;tag=as66995cd4
Call-ID: 7cebe5d1060b11452571a22e0e2cb919@192.168.250.49
CSeq: 102 INVITE
User-Agent: X-Lite release 1002tx stamp 29712
Content-Length: 0

Now I wish I had a way of generating some SIP traffic right now but I don't have access to a test lab for SIP at the moment since my demo license expired and I'm not in the habit of using a production network for labbing and testing unless absolutely necessary.  Anyways, as you can see, the above message is a response as it starts with a number and generally is not capitalized.  You still see all the same basic headers here only without the SDP.  Please note these are SIP messages I have used from the web and I have changed some of the information intentionally just in case there is a confidentiality issue.

Now a bit more on the OPTIONS request.  This is becoming more and more common as we move forward into a pure IP telephony environment.  Amongst other things, OPTIONS is used heavily for something called an OPTIONS PING which monitors the status of a egress point on a gateway.  If it doesn't respond it can mark it as dead and go out another route.  The great thing about this is, before SIP, this wasn't really possible and you have to wait for timeouts to occur in the H.323 and MGCP world or worse, manually change it over.  The rest of the requests I have already gone over in the other post and can be easily discovered on the web.  At this point, I am assuming you have a high level knowledge of SIP. 

I'd like to go over some of the SDP markers in the first SIP message.  As you can see, it is quite extensive.  The bad part is, with voice, video, and presentations that list will double in size to create even more of an eye sore.  Starting with the "c=" line in the SDP we can see an IP address and that it is IPv4.  This line identifies where media is coming from and where it should be sent to.  This can be a good place to look to make sure that your network is working the way it should be.

Originator or "o=" is an indicator of who is initiating the connection.  This has a username (if applicable), their IP, and who they are.  It is as simple as that.  There are more stipulations as to how the field should be populated and those can be found in RFC 4566 if you want to get into the nitty gritty.

Next is the all important "a=" line which marks what codecs you are going to be advertising and accepting.  PCMA/8000 is G.711a-law and PCMU/8000 is G.711u-law.  In the United States we use u-law inclusively. Another way to remember it is u-law for the U<---nited States.  (Thanks Jeremy Ciaroa for that one!).  Other codecs can also be listed here like G729 as listed above, iLBC, etc.  

You will also see things like "a=sendrcv".  This can be an indication of what is going on with the media.  Is it expecting to send only, receive only, do both, or go inactive completely?  There are four different  attributes in this particular that you will commonly see:

1. a=sendrecv
2. a=sendonly
3. a=recvonly
4. a=inactive

Normally, in a hold situation you will see a a=sendonly followed by a response of a=recvonly.  This is indicative of a stream going on hold.  A new transaction is sent just for the hold feature.  Now, this isn't the only time you will see these tags but the most common in a phone environment is hold/resume.  so why is one send only and the other receive only?  Well the user placing someone on hold is presumed to be send only because they are going to be streaming music on hold from CUCM or something of the sort while the distant end will only be receiving and not sending and voice packets back.  You should only see your RX on your Cisco phone bumping substantially during this time.  The other possiblity during a hold session is a=inactive.  This indicates nothing is being sent or received and basically dead air is all that remains.  SIP has to maintain the connection somehow, so you will continue to see SIP INVITES and 200 ok messages with a=inactive until the session is resumed.  How else would you maintain that call?

Well, looking at everything above I think I will cut this post here.  I will continue on with more headers and SDP lines in a future post.  I don't want to risk someone falling asleep on me now since it is imperative you understand SIP as the replacement technology for H.323...if we were get there in our lifetime.  It seems like PRIs are still everywhere while SIP is being slowly implemented in more and more places.