Troubleshooting IPSec site to site VPN tunnels on Cisco

First, double-check everything (examples below are what you should see in the config). The majority of problems with IPsec tunnels are down to a mis-match in the settings, so it pays to double check all settings before debugging.

Check the peer IP addresses are the correct far end peer IP address:
on Cisco routers check

crypto map ipsectunnels 10 ipsec-isakmp
set peer

and

crypto isakmp key ****** address

On Cisco firewalls check

and


Check the IPsec transform sets match:
on Cisco routers check

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

NB: In the line above, 'ESP-3DES-SHA' is the name of the transform set you have created by using this command, and 'esp-3des esp-sha-hmac' are the actual ipsec methods allowed in the named transform set you have created. You later assign 'ESP-3DES-SHA' to a crypto map (see below)

and

crypto map ipsectunnels 10 ipsec-isakmp
set transform-set ESP-3DES-SHA

On Cisco firewalls check

and


Check the ISAKMP policies match:
on Cisco routers check

crypto isakmp policy 10
encr 3des
authentication pre-share
group 2

On Cisco firewalls check


Re-enter the shared-secret (to make sure it isnt a typo):
on Cisco routers check

crypto isakmp key ****** address

On Cisco firewalls check

and

NB: On Cisco firewalls, the shared-secret is blanked out with **** when you do a normal show run. Make sure you don't just copy the ***


Check the crypto map is correct and matches the far end:
on Cisco routers check

crypto map ipsectunnels 10 ipsec-isakmp
set peer
set security-association lifetime seconds 86400
set transform-set ESP-3DES-SHA
match address 155

In the above, note the match address 155 - this references an access-list, make sure the access lists, both full source and destination IP range AND subnet mask, match EXACTLY for both ends of the tunnel.

and

crypto map ipsectunnels
The above should be applied to the outbound interface of the router where packets should be encrypted - on an ADSL router this would be the dialer interface that has the public IP (assuming its a public ADSL line).

On Cisco firewalls check

and




If that is all ok, check the status of the tunnels:

Router#show crypto ipsec sa
Router#show crypto isakmp sa

Do an extended ping to force traffic down the tunnels in case your expected traffic isn't making it to the tunnel for some reason

Finnaly, have a look at the debug:

Router#debug crypto ipsec

Router#debug crypto isakmp

No debugging output seen on a terminal session

If you've enabled debugging, issued a term mon command and are still not seeing any debugging output, then the logging level for the monitor (what you're asking to be showing when you issue term mon) may be set to a lower level than debug.

You can see what logging levels are set by using the show log command. Look for the Console logging: and Monitor logging: lines. If it does not say level debugging then you wont see any debug output for that output type.

To enable debug level output for the monitor, type:

Router(config)#logging monitor debug

The just enter

Router#term mon

and you should start seeing the debugging output (assuming you have turned debugging on for something!)

Cisco IOS Naming Conventions (12.4)

Just had a really interesting read(!) of the Cisco IOS naming convention whitepaper, so to save other peoples sanity I thought I would share my notes :)

The non 'T' train of 12.4 is there to provide the most stable releases. Maintenance deployment (MD) releases of 12.4 (non 'T') are the MD of the 12.3T train. This implies these are the most stable releases available. New releases constitute mostly software fixes

The 'T' train of 12.4 is there to provide new releases with the latest feature and hardware support. Early Deployment (ED) releases of 12.4T introduce new functionality that is not available in the 12.4 train. They also receive software fixes. There will be the occasional MD release of 12.4T

The release number (digits that follow 12.4, e.g. 12.4.xx) are unique across both 12.4 and 12.4T, e.g. 12.4.10a -> 12.4.11T1 -> 12.4.12a -> 12.4.12b -> 12.4.13T1

The rebuild letter follows the release number in the 12.4 train, e.g. 12.4.10b denotes the second rebuild of 12.4.10

The rebuild number follows the release number in the 12.4T train, e.g. 12.4.13T4 denotes the 4th rebuild of 12.4.13T4

In 12.4, 'Releases' include mostly software fixes, and occasionally minimal new feature support and minimal additional hardware support

In 12.4T, 'Releases' include software fixes, new features and additional hardware support

The rebuild designation (a letter for 12.4 and a number for 12.4T) shows that there has been a rebuild of the software to improve stability. The criteria on what changes are made in a rebuild are very strict.

In summary, choose a 12.4 (MD) with the highest release and rebuild letter for the most stable release available and choose 12.4T (ED) with the highest release and rebuild number for the IOS with the latest features and hardware support.

For full details of the IOS naming conventions, see the following link:

http://www.cisco.com/web/about/security/intelligence/ios-ref.html

Debugging Cisco ADSL

Use

debug atm events

to see the atm training process. This should show a detailed output as the atm interface comes up

Use

debug ppp auth

to view the dialer auth process. It should be clear from the output if the username/pass is bad

Use

sh dsl int atm0

to view detailed info on the DSL interface atm0. Look out for a very low (close to 1dB) or negative Noise Margin. This is difference between the maximum acceptable noise for ADSL profile assigned by the carrier and the actual noise on the line. The number shown is the actual noise subtracted from the maximum acceptable noise, so a positive number is good.

Some handy notes on Cisco IOS commands for Serial interface debugging

Router#debug serial interface

and

Router#debug serial event

and obviously...

Router#term mon

...if you're not on the console.

These will show the progress of the interface going up/down and the packet sequence between the router and the carriers NTE.

Some example debug messages you may see:

Jan 7 14:02:30: Serial0/1/0(out): StEnq, myseq 1, yourseen 0, DTE down

This line above represents the router sending a message to the NTE '...(out)...' and that the message is number 1 in the sequence '...myseq 1...' and that it has received zero messages back from the NTE '...yourseen 0...' and that it thinks the NTE is down '...DTE down'. The yourseen field will only increment when the router starts to receive packets from the carrier NTE.


Jan 7 14:24:29: Serial0/1/0(out): StEnq, myseq 209, yourseen 207, DTE up
Jan 7 14:24:29: Serial0/1/0(in): Status, myseq 209, pak size 14
Jan 7 14:24:39: Serial0/1/0(out): StEnq, myseq 210, yourseen 208, DTE up
Jan 7 14:24:39: Serial0/1/0(in): Status, myseq 210, pak size 14

The lines above show the normal output to expect when you enable debug serial event. You can see there is inbound and outbound traffic, and both the sent (myseq) and received (yourseen) packet counts are incrementing.

Finally, check out the following pages on the Cisco website for really detailed debugging info various serial interface types:

http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1915.html

http://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a758d.shtml

Editing access lists line by line on Cisco routers

If you're using number access lists there is a really easy way to edit the access lists line by line on your router without having to remove and add the whole access list each time. Note: This feature is not available on older versions of IOS.

Router#conf t
Router(config)#ip access-list ext

Now you are in access list editing mode. To get a quick look at your access lists:

Router(config-ext-nacl)#do show access-list

Output will look similar to the following:
10 permit tcp any host 192.168.1.1 established log
20 permit udp any host 192.168.1.1
30 permit ip host 10.10.10.10 host 192.168.250.250

Note the numbers next to each line for each access list. To remove one of the lines on your access list just type:

Router(config-ext-nacl)#no

You can also add a line to your access list. To insert a line between lines 20 and 30 above, just start your new line with 25 (or any other number between 20 and 30):

Router(config-ext-nacl)#25 permit udp any host 192.168.200.100

BT Business Broadband with Cisco 877 and first ping is dropped on Cisco 877 ADSL

I recently got BT Business Broadband via work for home office use and I'm currently using a Cisco 857 for the ADSL modem & router

The connection came with 5 useable public IPs which is really handy, however due to the weird way BT actually assign the IP addresses to the dialer interface, you can't just leave this on negotiate and expect it to get the IP address BT have assigned to the router. If you do you get a dynamic IP from BT which is a pain.

To work around this you need to create a loopback interface, e.g. loopback0, and configure this with your allocated IP range, e.g.:

interface loopback0
ip address 80.12.34.56 255.255.255.248

You then need to assign this loopback to the dialer interface:

interface dialer1
ip unnumbered loopback0

You also need to remember to NAT traffic using the IP address of the loopback0:

ip nat inside source list 10 interface loopback0 overload

That all seems to work pretty nicely however I did run into a problem where if I ping a public IP from the router itself or an attached client PC, the first ping is dropped. This is because the router is doing an ARP request on the public interface to try and find who has the public IP you are pinging, an while this is happening the first ping is dropped. To get round this you need to configure the following on the dialer interface:

no ip proxy-arp

This resolved my issue where the first packet was dropped when talking to any 'new' public IP. The issue also manifested itself as webpages taking a couple of extra seconds before the page started loading.

IP Phone (Mitel) drops traffic when the connected PC changes its network card speed

If a user is in a phone call on a Mitel 5220 handset and they reboot their PC (in this case a Lenovo desktop PC), the user experiences around 2 seconds of silence twice while the PC is booting - once just before the BIOS screen is displayed and once just before the Windows 7 logon screen is displayed.

This is caused by the PC changing the NIC speed during bootup from 100Mbps to 10Mbps, which in turn causes the Mitel 5220 handset to change its network speed from 100Mbps to 10Mbps, and during this change voice packets are dropped (the user experiences silence).

On the Lenovo PC in question this issue can be resolved by either setting 'Shutdown Wake-on-LAN' to disabled in the properties of the network card on the Advanced tab, or by changing the 'WOL & Shutdown Link Speed' setting to 100Mbps (default is 10Mbps).

You can demonstrate the same issue by connecting any PC with a 10/100 NIC to the back of the Mitel 5220 phone and within Windows manually setting the speed of the network card to 10Mb. Obviously you need to be in a phone call on the Mitel handset at the time to notice the 2 second silence.