A change of pace

I know that those who do bother to read my blog for informational purposes or research are probably wondering why I haven't posted in a few months. I'm just going to put it out there now, I lost my job due to financial cut backs at my last company. I've been searching for new work and have a tentative start date with a new company as in instructor, where I first started out after exiting the US Army. It hasn't been easy, but I've managed all these weeks. Unfortunately, my voice career is probably over as I am going back to Route/Switch. I will be slowly converting this blog over to match that change. It isn't something I wanted to do but it was an excellent opportunity for me to go back to being an instructor. Maybe I will end up teaching VoIP/Telepresence in the future, I don't know. For now, those that want to keep reading, I will be posting more educational content rather than one off issues since as an instructor, I will encounter far less field issues and far more educational things that people may not know or fully understand. I hope that the few readers that I had will not abandon this blog. In fact, I will probably be posting more content than ever since the CCNA R/S books don't really make it easy to learn all the material. CBT Nuggets and Lynda.com are valid other resources but it's always good to see someone's personal perspective. Until later!

Cisco 8841 to Analog Paging Choppy Audio

I know it's been awhile but things have been a bit slow around here so I've been using my spare time to learn Linux inside and out the best I can.  Figured another technology under my belt surely can't hurt anything.  Last week however was a change of pace with a strange issue.

I have a customer that is experiencing choppy audio when dialing their analog paging system.  They are on 10.5(2) so everything in place works and is supported even prior to their upgrade.  Their 8841s go off hook and dial a paging number that hits a voice gateway and then pushes out an FXO port to their paging system.  Every time they call the number the overhead paging sounds like crap.  At first, they thought the paging system was bad but with a buttset hooked up to the POTS line going to the paging system, everything sounded fantastic.  That leaves Cisco's side of things.

I had to rule out the DSPs and the FXO port.  I had them make a call and after a "show voice call x/x/x" and saw nothing out of the ordinary.  Logs on the gateway were also fine so what the heck?  They only have a two port FXO card so I wasn't about to ask them to pull their primary line inbound to test.  I instead asked a coworker to go on-site and with a spare FXO card that we know works.  We swapped out the card and the same thing still happened, so piss!

After that fiasco, we went back to the drawing board and tried dialing from a phone that isn't an 8841 and everything sounded good.  That old 7912 from over ten years ago still works and pointed the finger to the 8841.  I don't know if it's the G.722 codec, firmware, or somehow and someway SIP is screwing up the call setup using a weird codec but I will need to pull logs to find out for sure.  I need to schedule another test call with them and pull logs relevant only to that time period so I can get a fresh cap.

Hopefully this pans out to be firmware but with it suddenly happening who knows.  I don't see SIP screwing up since it's just a SIP phone pushing to an H.323 gateway.  At that point CUCM should have already done the heavy lifting of protocol conversion.

Out of Compliance on CUCM and need to delete some devices?

I discovered today that CUCM does the same thing Unity does if you let it lock out due to being out of compliance in the PLM.  If you reboot the CUCM you get 24 hours to remove whatever may be causing you to be out of compliance.  I discovered this through a TAC case as a customer had let their system lock out after the grace period and I was in a hard place to tell them either buy more licenses or leave it.  I knew Unity in fact let you reboot and fix the problem and apparently this is also a think as of CUCM 10.x.  Good news, didn't have to make a customer mad!

Cisco UCS Password

Well the holidays are over and everyone is back to the daily grind.  I know it's been awhile but I was reading an article that was a bit stunning.  Apparently Cisco changed the UCS default password on the C-Series UCS servers from "password" to "Cisco1234".  As of a few days ago this was brought to light and now people can stop calling TAC.  Why was this done and then not propogated down to the users?  Who knows.  Here is a link to the article that explains it all.  Glad someone posted it!

Voice PCM capture

I figured two articles in one day would be cool....  Anyways, in my last article I wrote about a PCM capture.  This essentially lets you capture the voice stream after digitization from the DSPs and play it back.  I know that can be scary since you can listen to anyone's conversation but it was required and authorized in my case since I needed to find out if everything was worky worky on the PRI side of things.  So below is the script you need to capture voice, please use this responsibly.

1. Debug mgcp packet (if applicable)
2. Debug isdn q931
3. voice hpi capture buffer 1000000
4. voice hpi capture destination flash:pcm.dat
5. exit
6. test voice port 0/0/0:23.23 pcm-dump caplog 7 30
7. :::::::::Make Test Call::::::::::::;
8. test voice port 0/0/0:23.23 pcm-dump disable
9. conf t
10. no voice hpi capture buffer 1000000
11. no voice hpi capture destination flash:pcm.dat

Note this is for prior to IOS 15.  Also, your voice ports are going to be different based on whichever channel gets opened for the bearer channel  for IOS 15 and later:

voice pcm capture buffer 200000
voice pcm capture destination flash:
exit
!
test voice port 0/1/3 pcm-dump caplog 7


:::::::::Make Test Call::::::::::::;

test voice port 0/1/3 pcm-dump disable


conf t
no voice pcm capture buffer 200000
no voice pcm capture destination flash:


On dial peer:
conf t
voice pcm capture buffer 80000000
voice pcm capture destination flash:
dial-peer voice 1
pcm-dump caplog FFFFFFF 254
!

After the all is done, to stop:

conf t
no voice pcm capture buffer 80000000
no voice pcm capture destination flash:
dial-peer voice 1
pcm-dump caplog disable
!

Intermittent one-way audio issues Resolved!

So if something is 99% of the time related to a certain issue it is probably the cause right?  Well, being stubborn and convinced it wasn't that 99% I found out today it was.  I did a PCM capture of the phone call exhibiting one way audio and you can hear both parties talking but yet only one way audio.  I did a debug mgcp packets and saw the channel in use and everything looked good there.  ISDN Q931 was clean and there were no transcoders or MTPs involved so what is the issue you might ask?

Well, 99% of one-way audio issues are network related.  I didn't adhere to that rule since I was so sure it was something else.  I figured since the issue followed the phone it couldn't possibly be the network!  I didn't stop to consider that when pushing it out the PRI, the network in use is still that same gateway.  It turns out the customer had two default ip route statements, one pushing to an IP address and another pushing it over the wire via interface.  We pulled the 0.0.0.0 0.0.0.0 fa x/x route out of there and everything started to magically work.  Now the strange part is that it really shouldn't have mattered unless there is something I am missing?  I'm not a CCNP R/S so I might be talking out by bum here.

Lesson learned: If its 99% possible that something is the cause, go check the damn 99%!

Intermittent One-Way Audio on Select phones

So today I was working a one-way audio issue that ultimately received a band-aid and not a solution.  A client has older 7940 phones and only two are acting up at a particular location.  All other phones are on that subnet and PRI are fine.  At first, I wanted to blame the network since almost all one-way audio issues are network related but I find that hard to believe if the rest of the phones are working fine.  Then, I wanted to blame the phones themselves and sure enough the problem follows the phone.  The only problem is that these phones have been replaced only for the problem to show up a day later or a week later or whenever.

At this point I am thinking PRI.  I push the call out another PRI and the phones are fine.  This nixes the phones being bad as they worked immediately after switching from one PRI to another.  This goes back to the PRI being part of the problem but I can see packets being transmitted and received at the proper G.711u-law codec.  Ultimately, I am going to have to see if the carrier is willing to troubleshoot since the problem is most likely something sporadic on their side.  This problem has been a hindrance since before a major upgrade and followed it afterwards.  It is strange for something like this to be popping up like this.  More information will follow as I find out the issue.

Time between posts

I know it's been awhile between posts but the holidays are here and having just ended Thanksgiving things are still a bit slow.  I'm trying to figure out what I could post that would be good for educational content as lessons learned are at an all time low as they usually are around this time of year.

I just wanted to put a post up showing I am still actively here but just haven't had any good content to put up that would be worth reading.  I do have a pending outbound call issue that could potentially be useful once I resolve the issue but the ticket hasn't landed yet so I don't know what all is involved.

For what it's worth, the Cisco PEC GoldLabs for UC/Collab are actually fairly well designed.  It's clear some of the labs were built around physically being on site but you can do all of them remotely via RDP.  There are some errors here and there in the workbooks but nothing that causes issues with following along with the exception of a VoH/MediaSense portion that wanted me to configure an EX90 that the lab left out completely.  Even with auto-registration on, you need to be able to access the EX90 and it's IP address is completely left out of the documentation.  The image they show that auto-registered isn't accurate as I tried that IP and got nothing.

ESXi and NIC Teaming gone wrong

Lately I have had the pleasure of discovering a new issue that has risen up from the 1s and 0s.  Currently, there is a network that is not using LACP but using NIC teaming, this is fine.  The weird part is, one of their UC VMs randomly stopped talking outside the subnet.  Rebooting the VM did nothing and it kept looking like it was more and more of a network problem.  A TAC case was opened and even Cisco was pointing the finger to the network.

Without being to in detail on ESXi, since I'm no guru, I took a look at the network side of the house and everything looked like it was in working order as well.  Pings to and from the ESXi were working but for some strange reason on VM was sending data out one NIC and receiving it on the other.  This was the only VM doing this and I think if LACP was enabled it would never have happened.  The end result was a VM that wasn't talking correctly and couldn't communicate with it's subscriber.  We ended up moving one of the NICs from active to standby and ran a test.  Everything magically worked itself out as all traffic was being forced over one lane.  We then put it back into active mode and everything still was working fine.  Weird.... How did the TCP/UDP data streams end up getting hosed up?

I checked again today and traffic was still evenly distributed and no weirdness was going on behind the scenes with one VM sending out one port and receiving on the other.  I'm not sure this was ever a problem as again, I'm not an ESXi expert but it seemed to have resolved the issues and they haven't returned.  This also seemed to have fixed a second issue they were getting which was calls going to voicemail but then terminating after 5 seconds.  If this was indeed the resolution, nothing was being shown as wrong anywhere on the network and no errors were being thrown.  Just a heads up for any of you that run into this issue yourselves.  Just bounce the port or go standby then back to active, it should end with the same result.

7940 / 7960 DST change and CUCM 6.x - 7.x

Last week I was working on a set of problem phones on a really old version of CUCM.  The problem was that when daylight savings time ended, all the newer phones rolled back an hour with the change of NTP based on the Date/Time group but the older 7940s and 7960s did not.  This is a constant yearly issue with this particular client's CUCM as they are on 6.x.  After doing some research I found a bug that affects the 6.x train and also appears to go all the way to 8.0.  This bug essentially holds the 40s and 60s back and you either have to manually change the time, update the CUCM with a .cop file, or just wait a week for them to change.

The weird part about all of this is that I changed it earlier this year to bump it ahead an hour and I didn't have to change it again afterwards.  I cycled the time back to -7 time since I am on central time and everything was fine.  Later last week I got a call that their phones had once again went forward an hour.  This didn't make sense at all.  I went in and did a screen cap of the 7940 and sure enough, it was showing eastern time again.  I set the date/time group back to central time and this ultimately resolved the issue.

So the lesson learned is:

1. Upgrade your CUCM
2. Get rid of those ancient phones
3. Upgrade and get rid of the phones
4. Wait a week for the time to change
5. Update the CUCM with a COP file

Why this issue still persists is beyond me.  This is something that was going on back when 6.x and 7.x were still supported over 4 years ago or so.  It should have been fixed then but never was.  Still, all the more reason to get off of an MCS server and move over to a BE6k or BE7k!

T1s being T1s

So there I was, end of the day and a ticket came in for a long distance code not working for a customer.  Easy, probably doesn't have a FAC on CUCM for some reason, just as the others didn't.  Well, long story short, there was a FAC and the call was hitting the gateway and I was getting a 41 cause code via RTMT.  DSPs were fine and the T1 looked good.  Came in this morning figuring whatever the carrier had issues with was fixed but nope, still hosed.  More tickets were coming in saying their long distance codes stopped working.

This was the last 24 hours of my work life.  Everything looked good, no errors, no issues except randomly failing calls out a particular site.  Sure, I could have re-routed the entire site out another gateway but that would have taken significant time, even with BAT changing out the CSS to route out the other direction.  That would also put a huge load on the other T1 bundle and soak up a crap ton of ports and DSPs.

So I sat there, in discontent wondering what was going on.  Logs looked good, traces were fairly clean and still Cause Code 41.  I tried rapid long distance dialing and got a few calls to work.  My thoughts immediately went back to the carrier.  I even tried changing out the gateways but since both T1 CASs were failing that was also doomed to fail.  I was talking with another voice guy and he figured we should try to reset the T1 CAS.  I thought..why didn't I try this?  So simple...  My thought process was that it was broken however, and now I know better.  Reset it even if it is appearing to be working, who cares since long distance isn't working anyways right?  I reset the port made about 10 calls and had the customer call me back and everything is hunky doory.

So what was the problem?  Damned if I know.  Neither of us did anything and the carrier sure didn't do anything.  A reset seemed to fix an invisible hangup on the T1.  Note to self, just reset the damn T1, even if it doesn't look like it needs it!  Also, who the heck still uses T1 CAS?  That's the first time I saw a CAS in about 8 years.

Unity Connection integration with an analog PBX system

Recently I got the awesome experience of integrating Unity Connection into an analog setup.  While the customer does indeed use Cisco, the non-Cisco side is still in place for a few reasons and needs to continue to work.  Their old Unity box was decommissioned but then brought back on.  When the phones received calls, they would send the voicemail to the Unity Connection box as you would expect.  The problem is, the old Unity box was still for some reason triggering when the messages button would be pressed on the non-cisco phones.

After poking around a bit I found they have a DMG or Digital Media Gateway.  This had routes to Unity Connection but was listed second in the top down list.  The first thing to fix was this and get rid of the Unity route completely.  The issue is, being unfamiliar with the DMG, I was rolling the dice.  The configuration menus were not too difficult to figure out but I still didn't full understand everything.  It turns out however, my solution was correct.

The next stop was to make PIMG ports on the Unity Connection box.  This will allow the analog phones to get their MWI and access their voicemail on demand when pressed.  Now the only thing I still don't fully understand in the first place is how the Unity Connection was being used by the DMG as voicemails were being delivered there somehow without the PIMG ports.  My guess is that it hit a route elsewhere when it got sent to the Cisco side.  However, being on a time schedule to getting something fixed since it was paid per hour, I didn't have too much time to try to figure that aspect out.

CCNA Lab #3

One more CCNA R/S lab for those folks still following my data rant.  Configure this one up the best you can and get it working in record time.

And here are the instructions:

I'm going to record me doing this lab so that people can follow along.  Keep in mind this might be a bit long as I'm bound to make mistakes.  I think this will help out those who need it though.  Recording will be posted once I get it made and uploaded.

Apparently I am not going to be able to record the session at the moment.  Windows as usual is acting up and making life difficult.  I will still try to get it recorded but I am not making any promises.  I may have to get this complete on my iMac since I never have issues on OS X.

UPDATE: I made a boo boo and Oklahoma should be 192.168.3.0 /26 and 192.168.3.64 /26.

A revisit on SIP

Month's ago I had put a post up on the basics of SIP and said that I was later going to get a little more in detail on the protocol. That post was mainly a very high level view with the core components. Now what I would like to do is take it one step further and start explaining some of the headers and how to read the SIP messages correctly.

I think one of the biggest issues with SIP is the pure amount of messages that are sent for every transaction.  Every time you setup a call you get an INVITE, every time you go on hold, INVITES are sent with certain SDP characteristics announcing a hold status so the call doesn't get torn down.  Every time something registers you get a REGISTER message.  Basically, everything is recorded down on what is going on.

Below is a basic SIP header from an INVITE, we will go over each header field that is relevant and annotate which ones are mandatory.  The thing is with SIP, you are required to have certain SIP headers regardless of what is going on.  While others may be optional, headers like To:, From:, Via:, Call-ID:, etc. are all necessary components to the SIP protocol.

INVITE sip:3212@10.0.0.153:5060;user=phone SIP/2.0

Via: SIP/2.0/UDP 10.0.0.4:5060;branch=z9hG4bKs3uc2t20dgj04lkn51s1.1

From: <sip:2226541300@10.0.1.3:51088;user=phone>;tag=127.0.0.1+1+a5380157+652ceee3

To: <sip:3212@97.208.31.204:5060;user=phone>

CSeq: 818076443 INVITE

Expires: 180

Min-SE: 3600

Session-Expires: 3600

Supported: replaces, timer

Content-Length: 313

Request-Disposition: fork, sequential

Allow: INVITE, BYE, REGISTER, ACK, OPTIONS, CANCEL, SUBSCRIBE, NOTIFY, PRACK, INFO, REFER, UPDATE

Call-ID: 9AB0E358-1FFEC2pzsxRAnf9OsjW3of1wROpUYiBxisHiIwiB6zBrtY79UsAJ92FonzIYhJx1BEJKAr

P-Asserted-Identity: <sip:2226541300@10.0.1.3;user=phone>

Privacy: none

Max-Forwards: 66

Contact: <sip:2226541300@10.0.1.3;user=phone>

Content-Type: application/sdp

User-Agent: Alcatel-Lucent 5020 MGC-8 8.1.0.11.SP1.1

v=0

o=root 2085757204 2085757204 IN IP4 10.0.0.4

s=session

c=IN IP4 10.0.0.4

t=0 0

m=audio 34732 RTP/AVP 0 8 18 101

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:18 G729/8000

a=fmtp:18 annexb=no

a=rtpmap:101 telephone-event/8000

a=fmtp:101 0-16

a=silenceSupp:off - - - -

a=ptime:20

a=sendrecv

So with the above, I highlighted a few lines that I will go over.  First and foremost this is a INVITE meaning an endpoint is sending a request.  I am capitalizing this information because all requests are always CAPITALIZED, and will always get a response back, regardless if it is a success or failure.  The six requests that you can get in SIP are INVITE, CANCEL, BYE, REGISTER, ACK, and OPTIONS.  Responses would be something like a 200 ok, ringing, trying, etc.  These are always in response to a SIP REQUEST.  Now, yes I know there are others like a PRACK and SUBSCRIBE but those are not the six main requests/methods.  Again, everything starting in CAPITAL letters is a request and anything starting with a number is a response such as a 180 ringing.  See below for a response to a SIP request.  Please note this one does not coincide with the SIP message I posted above.

SIP/2.0 180 Ringing
Via: SIP/2.0/UDP 182.168.250.49:5060;branch=z9hG4bK786d156c;rport=5060
Contact: <sip:9011@182.168.246.213:35134;rinstance=7af05ded7e7e49e6>
To: <sip:9011@182.168.246.213:35134;rinstance=7af05ded7e7e49e6>;tag=9a00d038
From: "9012"<sip:9012@182.168.250.49>;tag=as66995cd4
Call-ID: 7cebe5d1060b11452571a22e0e2cb919@192.168.250.49
CSeq: 102 INVITE
User-Agent: X-Lite release 1002tx stamp 29712
Content-Length: 0

Now I wish I had a way of generating some SIP traffic right now but I don't have access to a test lab for SIP at the moment since my demo license expired and I'm not in the habit of using a production network for labbing and testing unless absolutely necessary.  Anyways, as you can see, the above message is a response as it starts with a number and generally is not capitalized.  You still see all the same basic headers here only without the SDP.  Please note these are SIP messages I have used from the web and I have changed some of the information intentionally just in case there is a confidentiality issue.

Now a bit more on the OPTIONS request.  This is becoming more and more common as we move forward into a pure IP telephony environment.  Amongst other things, OPTIONS is used heavily for something called an OPTIONS PING which monitors the status of a egress point on a gateway.  If it doesn't respond it can mark it as dead and go out another route.  The great thing about this is, before SIP, this wasn't really possible and you have to wait for timeouts to occur in the H.323 and MGCP world or worse, manually change it over.  The rest of the requests I have already gone over in the other post and can be easily discovered on the web.  At this point, I am assuming you have a high level knowledge of SIP. 

I'd like to go over some of the SDP markers in the first SIP message.  As you can see, it is quite extensive.  The bad part is, with voice, video, and presentations that list will double in size to create even more of an eye sore.  Starting with the "c=" line in the SDP we can see an IP address and that it is IPv4.  This line identifies where media is coming from and where it should be sent to.  This can be a good place to look to make sure that your network is working the way it should be.

Originator or "o=" is an indicator of who is initiating the connection.  This has a username (if applicable), their IP, and who they are.  It is as simple as that.  There are more stipulations as to how the field should be populated and those can be found in RFC 4566 if you want to get into the nitty gritty.

Next is the all important "a=" line which marks what codecs you are going to be advertising and accepting.  PCMA/8000 is G.711a-law and PCMU/8000 is G.711u-law.  In the United States we use u-law inclusively. Another way to remember it is u-law for the U<---nited States.  (Thanks Jeremy Ciaroa for that one!).  Other codecs can also be listed here like G729 as listed above, iLBC, etc.  

You will also see things like "a=sendrcv".  This can be an indication of what is going on with the media.  Is it expecting to send only, receive only, do both, or go inactive completely?  There are four different  attributes in this particular that you will commonly see:

1. a=sendrecv
2. a=sendonly
3. a=recvonly
4. a=inactive

Normally, in a hold situation you will see a a=sendonly followed by a response of a=recvonly.  This is indicative of a stream going on hold.  A new transaction is sent just for the hold feature.  Now, this isn't the only time you will see these tags but the most common in a phone environment is hold/resume.  so why is one send only and the other receive only?  Well the user placing someone on hold is presumed to be send only because they are going to be streaming music on hold from CUCM or something of the sort while the distant end will only be receiving and not sending and voice packets back.  You should only see your RX on your Cisco phone bumping substantially during this time.  The other possiblity during a hold session is a=inactive.  This indicates nothing is being sent or received and basically dead air is all that remains.  SIP has to maintain the connection somehow, so you will continue to see SIP INVITES and 200 ok messages with a=inactive until the session is resumed.  How else would you maintain that call?

Well, looking at everything above I think I will cut this post here.  I will continue on with more headers and SDP lines in a future post.  I don't want to risk someone falling asleep on me now since it is imperative you understand SIP as the replacement technology for H.323...if we were get there in our lifetime.  It seems like PRIs are still everywhere while SIP is being slowly implemented in more and more places.

GNS3 CCNA R/S Lab #2

EDIT: Note that I made a change to the topology. I decided two links to Salado was stupid since if that core went down Jim was hosed. Still create GLBP links on the inside but instead connect to Austin as the new image shows. Yep, still studying for the new CCNA R/S and whipped up a second lab that you can try on your own and see how far you get. This one is a little more complex than the other one. Also, on the other one, if the texas remote office couldn't ping across its because I had a tunnel setup and the remote computer had to send an ICMP or some sort of data to the distant end before the crypto tunnel would stand up. See below for the latest lab image. I won't bother adding in configs for you as this is supposed to be done from the ground up. I may add more to this later, I can't think of any exam objectives I would be missing at this point other than switching with STP/SRTP/PVST+....Come to think of it, I will post a file when I am 100% satisfied I have configured this lab to my satisfaction so you can get a working model and compare.



Please note that this lab may require in excess of 3-5 GB of RAM. My laptop has 8 GB and I was sitting right around 5.16 GB of RAM chewed up from this lab. I'm not taking into account all of my other programs open however, I had email, a ticketing system, and notepad++ open. For Jim's network, it is up to you to design a proper subnet for the GLBP and WAN links. Part of learning is doing something yourself! Whip up a DHCP server while you are at it for him too. IPv6 lab will be coming soon.

• ensure the line console doesn't interrupt your typing
• setup telnet and ssh except Salado-Exec, ssh should only be enabled for this gateway
• configure both the console and vty lines for a password of cisco and enforce logging in
• create a username of cisco and a secret password of cisco
• create a secret enable password of cisco
• make sure random words don't force DNS lookups if you make a mistake typing
• configure all ip interfaces as shown
• configure EIGRP in AS 1 and only send hello packets on interfaces that you own or networks that are completely behind the GW you are on
• Configure the frame relay interfaces according to the DLCI chart
• Configure the frame relay switch
• EIGRP should be establishing adjacencies at this point, verify by pinging across
• configure DHCP for 10.x.x.x networks and exclude .1 to .9 while setting DNS to 8.8.8.8 and the default-router to the .1 address
• make sure all clients get their respective DHCP addresses and other information such as gateway and dns
• configure NAT for the additional router added into the Salado portion to use PAT, ensure nat translations are occuring on the inside global address to the inside local
• configure GLBP for Jim's setup and ensure the timers are set to miliseconds of 50 and 160 dead
• configure static routes for Jim's network to Salado-Exec and beyond
• configure default routes that all point to Corp-HQ in case a route is not known
• configure a proper route in the correct gateways to ensure data gets routed statically to Jim's computer since he is a loner and no one likes him
• configure eigrp md5 authentication between corp HQ and all other cores
• configure PPP Chap between Jim's GWs
• deny telnet to all gateways except from the engineers and IT networks
• deny ssh to all gateways except from the engineers and IT networks
• ONLY deny the CEO and CFO, permit the CTO to do his job with all engineering tasks like icmp, telnet,ssh
• deny ICMP on all host networks except IT and engineering
• deny http traffic from the corp sales department, they are committing shenangians on youtube

A CCNA Lab for those that need extra practice

I've already had my rant about not knowing CCNA R/S for new engineers or even existing ones. After several weeks of intense study I think I will be ready to go sit the new CCNA R/S exam. Yes I know, some may say it is easy, and I'm sure it is for those that do data all the time but I am a UC guy that is sliding back into the data realm slowly. Bottom line is, I miss data and all the cool routing and wireshark captures to find out problems and ways to fix them. I spent some time today finishing up a lab that I had made from scratch that tests most of the CCNA topics but not all. STP is difficult since you can't replicate the ASICS and I don't have an router IOS with a switch module to at least attempt to get STP up. I also haven't implemented GLBP / HSRP/ VRRP either even though they are fairly straight forward. I probably will add to the diagram but my back is killing me today and I haven't been in the mood to really build anything new aside from what I have already done. The below image is what I challenge you to replicate and get working 100% of the way.



At a minimum, here is what you need to configure:

1. Connect all devices as shown
2. Configure all ports with approrpriate addresses and bring them online
3. Configure ssh access only with a password of cisco
4. Configure a username of cisco and secret of cisco
5. Configure an enable secret of cisco
6. Ensure that the routers will not mistaken letters for DNS names
7. Disable enable timeouts
8. Set hosts with IPs so you can use their names instead of IP addresses
9. Configure EIGRP and advertise networks based on subnet with no automatic summarization, do not configure Canada nor advertise its networks!
10. Exclude the first 10 addresses in all ranges in the private networks for DHCP
11. Configure DHCP for all Offices with a /24 subnet based on the network assigned
12. Assign default routes to point to the local primary gateway of each site (not the office GWs)
13. Ensure all PCs get their DHCP addresses assigned to them and they can ping to their local GW
14. Configure the Frame Relay DLCIs as shown and map them as indicated
15. The FR-Switch should not have any other configuration as it is acting like a cloud
16. Assign static routes to get to Canada based on location (Texas will be a end all be all GW if other primary gateways don't know the correct path
17. Configure NAT for Canada to use the internet IP with PAT
18. Ensure all devices can ping to the private network in Canada
19. Ensure Canada can ping from internal to any CORP based internal IP (i.e. 10.0.3.11, 10.0.0.1)
20. Configure EIGRP md5 authentication for all links
21. Make sure that any loopback interfaces assigned are being seen as their subnet in the routing tables
22. Block all telnet traffic from all locations except Texas
23. Block all SSH traffic on private LANs on all sites except Texas
24. Verify NAT on the Canada GW
25. Enable password encrpytion globally so that any plain text passwords are garbled at minimum security.

If you can do the above without having to look anything up, you are probably good to go or still competent in data. I myself had to look up how to do md5 authentication because it's just not something I do everyday...or at all really. I'm not a network engineer I'm a UC engineer. I challenge you to complete this as fast as you can and report your times with honesty. I will probably wipe this entire topology and start from scratch and do a YouTube recording when I get the chance. I want to see how fast I can go under pressure and what I do and do not forget. Keep in mind I set all DLCI interfaces to a /24, you can modify as you want.

The Importance of CCNA in the UC Realm

I've been on my soapbox before about knowing data if you are going to be a specialty engineer, whether that be UC, wireless, security, data-center, or whatever else. I had some time on my hands so I built out a nice GNS3 lab to test my data skills. Granted I did cheat a bit since I had gone through the new CCNA R/S course on CBT nuggets, but it is still necessary to know the underlying foundation. My biggest gripe right now is that Cisco wiped out the need for a CCNA R/S as a pre-requisite to CCNA Collab and CCNP Collab. With this new road map, new engineers will be coming into the field for UC not knowing the different between router and switch. Ok, maybe that was a bit too much, but still, the problem remains. Ask them to troubleshoot STP or check a VLAN and you are going to get a blank stare. Anyways, below is a GNS3 image of what I built out. It is still a work in progress and I plan on using this to gauge other engineers's capabilities with data.

The above diagram hosts a variety of different situations that need to be configured. All the internal sites are using OSPF and the WAN connections are using EIGRP. Yes, I could have used OSPF or EIGRP across the board but part of knowing CCNA R/S is route redistribution. Thus, I used two protocol for that part. The other part is the connection to the ISP, I have a static route in place but also put in BGP as I plan on expanding that diagram to another company and bridging together from there. While BGP really isn't touched on much in the CCNA world, it is still mentioned since it is heavily used at the provider level. Additionally, different networks for each gateway per floor that go in sequence as well as ACLs to limit ICMP, web access, ssh, and telnet accordingly. Finally, when egressing to the internet, all gateways will NAT to their public IP (once I get the other ISP gateways put in there). I will also probably add some security on the routers for neighbor authentication for the hell of it as well. It's stuff like this that is important, if the ASICs could be emulated I would be running real switches as well. The switch modules available can do the job though if I really wanted to run STP and get a loop free topology while using etherchannel. Just in case anyone is doubting the configuration:

I know this wasn't really voice related and more of a rant, but I haven't run into anything interesting or different on CUCM lately. I'll try to break something on accident and post about it :)

Modifying device line buttons saying "Fixed Feature"

So today I was whipping up a new DID scheme for a user since they have been getting to many calls on their current lines since they were not being masked correctly. Easy enough. I got a new set of DIDs for them in the pool and assigned them appropriately and let the know that their voice mail would be changed since the DIDs were changing and everything is LDAP synced. Yes, technically I could do it the hard way but we don't hand code anything unless necessary for voice mail. I had a few phones that had a message on them when I was modifying the line order saying "Fixed Feature Button". This didn't make much sense to me since nothing on that line is a feature other than the line itself and the other phones didn't have this problem. After investigating a little more I saw people completely deleting that line and to me, that isn't an acceptable solution. I ended up disassociating that line with the phone then adding it back in without redoing everything. Since this happened on three different phones, deleting and re-adding would have taken to much time. Once I disassociated the line on the phones then put the line back on all was good. Not a huge deal, but a I guess this was a bug from 9.X that still hasn't been resolved...

Call Pickup Group "Feature is Unavailable" Message

So today I was presented with a very simple yet difficult problem. A customer had a problem with their call pickup group. Any time they would try to pick up a call after seeing the visual alert it would say "Feature is Unavailable". After doing some research on the problem I came up empty handed with others having this same problem. I made a few test calls and GCallPickup and then dialing the ringing number worked yet CallPickup didn't. Perplexed, I had the customer go try another set of phones in another pickup group and everything worked as it should. Absolutely nothing was different between the two configurations other than a the pickup group number (obviously...). So after an hour or so of wasted time troubleshooting and checking everything I created an entirely new pickup group and assigned the phones to it and voila, everything worked. I then deleted the old pickup group and changed the new one back to the old one's name as well as it's number and guess what, everything worked. My only assumption as to why this randomly started to fail all of a sudden is a database issue and removing it while creating a new one was the only solution. No other settings like extension mobility were in play and certainly all of the settings were correct such as the CSS and partitions. Weird...but at least it is fixed. I highly doubt TAC would have an answer other than the solution I ultimately came up with.

The Hunt Pilot Timer

Everyone has those moments where stupidity overrides thought processes. Yesterday I spent about 30 minutes on something that should have taken less than 3. I was providing support for a customer after I migrated their CUCM from a physical box to a virtual host and they wanted to do some additional configuration on their main line calling in. No big deal, slapped a second line group in and added it to the Hunt list and adjusted timers accordingly. After some fine tuning we got the line groups to ring properly but the second line group would always cut off prematurely. After trying to figure out what was going on and checking things I knew were right, it hit me. After you rang out it would hit a voice mailbox which just happened to be read by Cisco's voice and I knew exactly what was happening. The hunt pilot timer was set to 15 seconds and that was seconds more than then the second line group would hit which only allowed a very brief ring on the second group. I adjusted that timer to 30 seconds and bam, everything worked flawlessly. Sometimes you just need a small sample of stupid to realize that everything isn't complicated unless you make it that way. I think part of the reason I was forgetting the Hunt Pilot timer was due to the lack of sleep from the migration on the night prior. Who knows, maybe I'm making excused for a novice mistake?