Well after 5 days of CCIE Troubleshooting tickets with 20 other students and Brian Dennis in London this week, I am back home and just trying to process what I have learnt!
Day 1 – full write up here – was an introduction to troubleshooting, basic strategy and some technology basics, and then attempting to build the entire 30 router topology and starting at 6pm the first 10 tickets.
The next 4 days took the format of reviewing the previous evenings tickets through the day, taking at least 45 minutes per ticket and not just going through the solution but exploring all possible solutions. If you want to see the style of the teaching there some videos from INE here
This has certainly been a drinking from the fire hose week where there wasn’t a moment where you were not learning something new.
If you have never been to a ccie bootcamp I would highly recommend it – they are not cheap but totally worth it.
Mixed in with Brians humour and views and insight into the future of networking this was for me and I am sure everyone else that attended an invaluable week in preparation for the troubleshooting section of ccie r&S lab.
I am not going to blog here today about everything I learnt as this will come in the next few weeks with some more specific posts on different technologies, but I will list a few points below
- Do not be afraid to remove any feature to prove if it is causing a problem.
- What this means is if you isolate a problem to what you think might be being caused by a feature simply remove that feature and see if it fixes the ticket. (obviously copy the config to notepad so you can re-apply it) If it does resolve the problem you can then look at how you can change that feature to make the ticket work.
- Step back from the problem and try to work out what the actual problem is, if your ticket says R1 cannot telnet to R9’s loopback address, this does not mean that this is a telnet problem it could be caused by routing, layer 2, filtering, firewall, private vlans etc etc etc
- Be 100% sure you understand what the output of all the show commands should be, if you are trying to troubleshoot and you do a show command you need to be 100% confident that when you see the output you know that it looks wrong or something is missing. This is essential to isolate your problem quickly.
- Build a large scale topology from scratch. This was a great tip from Brian were he suggests renting a Troubleshooting Rack and simply loading on the base config (IP’s / Vlans etc) and then configure everything else to get full reachability across the topology. Doing this will firstly ensure you are confident configuring everything but also ensures you know how every feature works. You don’t want to be trying to troubleshoot an issue with a technology that you have only every read about once. Nothing in the ccie lab and in particular the troubleshooting section should be a surprise to you.
- Only do a show run when you have isolated the problem, if your first troubleshooting technique is going to be a show run on the affected devices you are going to looking for a needle in a haystack. The routers in the troubleshooting section are configured for multiple topologies and will contain a lot of configuration which has nothing to do with your lab. You could be trying to fix a problem that looks like it could be your issue but actually has nothing to do with it.
- Do not remove any feature – change it. If you have isolated the problem to be an access-list do not just remove the access-list. Although this would probably fix your ticket you will have violated the restrictions. You need to understand what part of the access-list is causing problem and then change it.
- Have a plan – some people want to go in and go through the tickets one at a time, others want to tackle the easy ones first, others want to do the hard ones first. A technique that was suggested this week was to look at ticket 1 and after 5 minutes make a few notes of what you think the problem could be and the rough area it could be in, move onto ticket 2 and do the same, if you can fix it in 5 minutes then do. Hopefully by the first pass, you will be an hour in and will have resolved quite a few tickets and will have some very good notes on what all the other problems could be. On the second pass you may see things that you did not before. A story from a passing candidate said he did not see a problem until the third pass using this technique. At all costs do not spend 30 minutes trying to fix one ticket, you have to move on and come back to it with a fresh pair of eyes.
- Finally – know all the technologies – if you don’t know how something works, there is no way you will be able to work out if it is broken and how to fix it.
