How to build CCIE V5 Lab using ESXi and Cisco Cloud Services Router CSR 1000V
This post will detail how to build a CCIE v5 lab using Vmware and the Cisco CSR1000v cloud services routers. It will detail the issues I faced which should hopefully help you make this build a lot easier.
The basis of this topology is centered around the INE Hardware build which is detailed here – INE CCIE V5 Hardware Topology
In simple terms you are going to build a VMWare ESXi server and run multiple instances of the Cisco CSR1000v router – this can be connected to some physical switches and you can have a 20 router / 4 switch topology that will provide for all your labbing needs. This post is focussed on the router build only and a later post will focus on the switch build and connecting them together.
Virtual Rack – Shopping List
To build your virtual ccie rack you will need:
- 1 x server or high power desktop with at least 16GB ram for 10 routers and 32GB ram for 20 routers
- 1 x Cisco CSR1000v ova file
- 1 x VMWare ESXi 5.1
If you really want to know what is going on I recommend reading the CSR100V configuration guide
Server or High Power Desktop
I have built this out twice now. Once on a high end DL360 G8 server and the second time on a Dell Optiplex 790 desktop. Once they are built and up and running there is no performance difference in the two boxes. So I would recommend unless you have access to some big servers then a good desktop PC is more than suitable.
Which ESXi version to use for ccie lab setup?
For the purpose of this tutorial and my home lab setup I am using ESXI 5.1 This is based on the fact that I have tested this and it works. I tried to use ESXi 5.5 but got hung up with the web client so went back to 5.1 and the install and setup worked without a problem. As they say, if it ain’t broke, don’t fix it!
Cisco CSR1000v
You will need a copy of the Cisco CSR 1000v virtual router this can be downloaded from the Cisco Website
Information about the CSR Router – www.cisco.com/c/en/us/products/routers/cloud-services-router-1000v-series
Download link – Cisco CSR 1000v download
You will want to download the OVA file
A few people have asked me about the Cisco CSR 1000v price. For lab use you can download and use the router for free, it will run with a limited bandwith but that is fine for just labbing. If you want to use it in production then you need to purchase a license which will increase the throughput of the cloud services router.
CSR1000v requirements are very straightforward, as it is a Virtual Machine your server just needs to support multiple versions of the router, each router requires the following:
- 4 Virtual CPUs
- 4GB RAM (you can run with 2GB)
- 8GB Hard Drive
1 x VMWare ESXi 5.1
You can use VMWare ESXi 5.5 ( I have not tested) but for the purpose of this document I am using 5.1
Download a copy of the VMWare ESXi 5.1 Hypervisor from here
You will need to create an account to download.
Installing VMWare ESXi
Now you have everything you need to install the hypervisor onto your chosen platform. I am not going to detail installing the hypervisor but if you want a good tutorial there are many good tutorials out there. In simple terms make a bootable USB stick, copy the ISO file onto your USB stick and put into your server / desktop and boot the hardware from the USB.
The only two pieces of information you are going to need to provide is a password and an IP address.
For the purpose of this build my IP is 192.168.1.1
So you should now have your VMWare ESXi hypervisor up and running and once logged in should be looking at a screen like this.
Yours will be empty so we will now go through the installation of the Cisco CSR1000v router.
Installing the Cisco Cloud Services Router – CSR1000v
From the VMWare client click on File / Deploy OVF template
Browse to the location that you saved your CSR1000v installation package
Click Next
Notice the Size on disk: If you deploy this router as Thick Provision it will require 8.3GB of disk space. Most of this space is unused. If you deploy as Thin Provisioned it will only take the disk space required but it will grow. This is selected later on. Click Next
Now give your router a name – this is used to identify all your routers so make them unique i.e CSR-1, CSR-2, Click Next
For the purpose of a virtual ccie rack you have two choices, Small or Medium
- Small – 1vCPU, 2.5GB RAM
- Medium – 2vCPUs, 4GB RAM
Again this is all dependent on your hardware platform. On the Server I selected Medium and on my PC install I selected Small.
Once the router is running, it actually uses <1GB of RAM but it does need a bit more to get started.
Select your Configuration and Click Next
This screen is where you select Thick or Thin Provisioned
- Use Thick Provision if you have lots of disk space
- Use Thin Provision if you don’t
- I used Thick provision on the big server install where I had lost of disk space
- I used Thin provision on my PC install
- Both work
Basically Thick provisioned allocates a larger chunk of disk space which will not grow, and thin provisioned allocates a smaller amount of disk space which can grow. For a typical 20 router ccie virtual rack you should be fine with a 128GB drive – SSD is better if you can. Select your choice and Click Next, Next again and then Finish
Your router will now be deployed
Virtual Serial
On your vSphere Client you should now see your newly deployed router it will not have green arrow beside it because it is not powered on.
Before you power the router on we are going to add a Virtual Serial Port
Right Click on your router and select Edit Settings – then click Add
Click on Serial Port and Click Next
Select Connect via Network and click Next
You need to change Network Backing to Server
In the Port URI add telnet : // server-ip:port
In my case I would configure telnet : // 192.168.1.1:2013
Ensure Connect at power on and Yield CPU on poll are ticked and click Next then Finish
Enable Serial port over Network in ESXi firewall
By default the ESXi server will block access to the VM Serial port over network. You need to enable this in the Security Profile.
Click on your Server and then click the Conguration tab and then properties
Scroll the window down a bit until you see VM serial port connected over network – it will be un-ticked – Tick this box and click OK
ESX Server License Type
NOTE: In order to be able to use the VM Serial Port functionality for your CSR Rack you will need the Enterprise License.
If you are using the 60 day trial license for ESXi this will be enabled.
To check your license version log into your ESXi server and click on Configuration/Licensed Features.
My server is running the VMware vSphere 5 Enterprise Plus Licensed for 1 physical CPUs (unlimited cores per CPU)
If your server does not show Enterprise then your telnet to the routers will not work.
You can now power on the first Virtual router in your ccie virtual rack.
Powering on the Cisco CSR 1000v
Right click on your router and select power on. You should now be able to connect to your router using your chosen terminal client and see it boot up.
Be patient as the first time you boot the router it will take quite a few minutes to start.
I hope you enjoyed the first post on how to build ccie v5 lab
I am now focussed automation, please check out my posts on Network Automation using Ansible & Python
Naz
Thanks Roger – great post!
How many instances were you able to run on the Dell Optiplex 790?
Roger Perkin
Naz, I am comfortably running 12 routers with 16Gb of RAM, I have not looked into any paging optimisation on the Hypervisor but this box is doing exactly what I want from it – which is much better than the 30U rack I used to have in my spare room!
Rob
I dont understand why you need so much memory. Also why do you need so much power for lab. Each router has like 32 meg requirement. In a CCIE lab you could have 2600 routers and 1841 routers. None have more then 64 megs of ram and maybe 32 megs of flash. Even some 3600 or 3800 routers have 128 or 256 megs of ram. Multiply that by 20 and you get like 5 gigs. So I don’t get why you need double of that.
Roger Perkin
Rob,
These are not hardware routers they are software routers and Cisco recommend 2-4Gb for each router, however you can easily run each router with <1Gb of ram. If I run up my lab with 10 routers it consumes about 12Gb of ram.
Remember these are Virtual machines and not hardware routers.
BundyboyUK
Hi, thanks for this post,
I tried this with a ESXi 5.5 and the issue is that I am not able to edit the router config after install as apparently from 5.5 you need the web client to edit the settings of a V10 VM…Is there a chance you can post an example of a VMX file so I can copy/past the serial config part?
Many thanks in advance
Roger Perkin
I originally started with ESXi 5.5 but was hit with needing the web client, so I reverted back to what I knew with 5.1 and had no issues.
mohammed
dear Roger
thanks a lot for your efforts,
I guess by “Use Think Provision if you don’t” you meant thin not think.
I wondering when you will post the new article, because I have some queries like:
1) do we need two NIC cards on the EXSI server ? one for the vsphere client and one to connect to the physical rack ? or one is enough ?
2) shall we use only one ethernet interface on each virtual router to build the topology ? also, can we build a L3 topology without physical switches !?, if yes then how will we connect the routers with each other ?
thank you very much
mohammed
Roger Perkin
Thanks! Post updated – now reads thin and not think
Sam
Hello, is more than one physical network port required on the ESX host?
Roger Perkin
No, you only need one port on your ESXi host This is to connect to the server, if you are connecting it to switches you will need 2
Tim
Hi Roger,
The work I have done with this build is very similar, but I have a question with licensing. You have not made mention of the license you’re using. As far as I can see, I need to activate the 60 day enterprise trial license to get virtual serials working. At the moment I’m using vrf’s for access which work well. I know there are methods to roll back the trial license, deleting licence files and restarting services, etc. I was wondering what method you have found works well for you.
Cheers,
Tim
Roger Perkin
Tim,
You need to enable the premium license on all the routers using the command below
R1(config)#license boot level premium
Then reload – this will enable all the licensed features for 60 days – it’s the equivalent to having a K9 image, you need this to get all the IPSEC stuff working.
After 60 days you will retain all the licensed features but throughput drops to 2.5Mbps – which is fine for a lab environment.
Roger
Julian West
I installed the router with “small” configuration (2.5gb, 1cpu) and I was able to easily run 20 instances of these with no disk swapping on an ESXi 5.5 server with 64gb RAM (v5.5 removes the arbitrary 32gb limit so I added RAM to get there).
I’m left with a good amount of RAM to run VMs with GNS3 0.8.6 or the new in-progress GNS3 1.0 Alpha as well (for switching). Really fantastic little router for CCIE R/S studies and, like you say, beats needing to have a 30U rack and the power consumption that goes along with it!
I do have an issue I’m working through with the serial/telnet – I followed the instructions above, and I am able to successfully get an active telnet connection to the URI “telnet://host-ip:port” to the serial/telnet URI just fine; however my terminal client window just sits there blank. ESXi Firewall is correctly configured to allow connections and I have done serial port/telnet URI in ESXi before with a Linux VM — but so far no success getting this to work. Just a blank telnet session. Anyone ever seen this in ESXi 5.x?
Julian West
Update: remote console/serial via telnet works now! I had all of my CSR VMs restarting in w/ default blank configs ( via snapshot) and you must set enablepass and have startup-config with “platform console serial” set. On a fresh reload the CSR VMs now allow remote console/telnet via ESXi serial as specified above. May want to add that minor step to that section, although I figure most experienced people know to set this! Thanks again for the instructions here — looking forward to labbing with this.
Michael Grann
Hello, I’ve followed the guide and I’m able to boot the CSR and go to enable/config prompt using the vSphere Client’s Console tab for the CSR. However, when I try to telnet to the server-ip:port (192.168.1.10:2001) using windows or Putty, the Telnet session is immediately closed. My Windows client has disabled the Firewall, I’ve allowed the virtual serial in VM’s Security Profile, also configured an enable password on the router, and also configured “platform console serial” as described above. I’ve also applied the temporary “license boot level premium”. All with no luck telnetting to the console port.
After doing a “write” and “reload”, my CSR VM’s no longer boot. They are stuck in “INIT:Entering runlevel: 2” followed by “INIT: Switching to runlevel: 3”.
I have a dual quad-Xeon with 32Gb RAM and 270Gb HD space.
Any help is greatly appreciated.
Roger Perkin
Michael,
The fact that the routers are saying “INIT:Entering runlevel: 2” followed by “INIT: Switching to runlevel: 3”. means you have the virtual console setup, you won’t be able to use the VM console at this point. So just close it. There can only be two things stopping you telnetting to your routers.
1: You have the telnet settings setup wrong it should be telnet: //IP-ADDRESS:port
2: The license on your VM Server does not support Virtual Console
Can you confirm the telnet settings are correct the license you have on your VM box.
Your routers have booted, you just need to telnet to them via putty now
Carlton
Hello Roger
Great post.
I came across your post as I’ve been having an identical problem as Michael Grann September 27, 2014
I have tried everything but can’t seem to telnetting to console port.
My telnet settings are telnet://64.x.x.x:9018.
As Michael, I have configured all of the firewall settings in Security Profile.
My license is as follows:
Product: VMware vSphere 5 Hypervisor Licensed for 1 physical CPUs (unlimited cores per CPU)
License Key: 0N49J-A2H04-78H49-0C90K-8E356
Expires: Never
Product Features:
Up to 8-way virtual SMP
Can you please help shed some light on why I can’t telnet
Cheers
Carlton
Roger Perkin
It has to be an Enterprise License for Telnet to work, when you click on license does it say Enterprise?
PJ
Hi Roger
Thanks for this very good tutorial. Just one question really. On the INE they use subinterfaces – in ESXI you only see the one Ethernet adatptor how does the networking aspect work on this when you setup the v/switching.
Thanks
PJ
Roger Perkin
Create one V Switch, I called my V5-Switch, you only need one interface from each router to be connected to it. Under the V Switch properties under VLAN ID ensure it is set to All (4095)
Each router interface will then trunk to the switch and you just run sub-interfaces for each link you want between each router, once you see the INE workbooks and configs it will all make sense.
Chris Anderson
Hey Roger, great write up and thank you!
Not familiar as much with vmware and the cisco CSR 1000v but I assume we can use them without any licensing costs? I see above that performance is toned down after 60 days but this is just for a lab environment.
Roger Perkin
Chris, The version I used to write this post had a 60 day limit on the license, the latest version 03.13.00.S will run forever with a 100k limit which is perfect for your CCIE Lab
Jason Briones
How do you go about building a topology once you have all the appliances running?
Do you just mess around with standard vswitches or something?
Roger Perkin
Building the topology is all covered in the INE workbooks, but in simple terms all the Routers are connected to a V Switch which trunks all vlans. Then on each router you want to connect to another just create a sub-interface e.g Gi1.46 and then on the other router you want to connect to do the same. Both routers will then trunk via the V Switch. Once you see the INE configs it is all very straightforward
Chris
Hi Roger,
first of all thanks a lot for the great post, it helps me a lot because I m not so experienced in VMware especially ESXi.
Where I m still struggling with, is the point, if I can also use a “high end” desktop PC (which meets the CSR1000v minimum requirements) with VMware Workstation for building the INE R&S v5 Labs or is it better to buy a Workstation (Dell Precision T7500) and use ESXi as the hypervisor?
Do I have any advantages when i use ESXi or limitation when using VMware Workstation?
Thanks a lot Roger,
BR Chris
Roger Perkin
Don’t try and run CSR1000v on VMWare workstation, you are running virtualisation inside virtualisation and it will only give you grief.
A half decent PC with 16GB Ram will run 10 routers easily on ESXi
Roger
Edward Boggs
I found the fix to using ESXi 5.5 and using Vsphere 5.5 desktop client. The problem is that the copied .OVA image for VM is version “VM10”
csr1000v-universalk9.03.12.00.S.154-2.S-std.ova. upload it into your ESXi host like normal.
then download the “VMware-converter-en-5.5.2-1890136.exe” which is the vm converter and down grade it to “VM9” and all will work. see the following website for reference.
http://techhead.co/vmware-esx-how-to-downgrade-a-vms-vm-versionhw-level-from-7-4-0-to-4-3-x/
Vsphere client 5.5 desktop is working great now and you can edit the vRouters.
let me know how it works.
Sam B
Hey Roger,
Cheers for the post, I skim read it first time and just put on ESXI 5.5 (this is a first time setup for me) & what a pain trying to get into the settings off the machines with web client. I tried Workstation, changing the .vmdk from version 10 to 9 but in the end i just gave up. Suffice to say once I threw in the towel and used 5.1 it span it up in no time!!
Also I noticed on 5.5 I had to spend a long time custom packing the ISO file to get my NIC card to work, lost of Google time tonight.
For a first time user of ESXI have to say my initial impression of 5.5 are not great now, however I will chalk this up to part of the learning experience!
But I just wanted to say thanks for the post, it certainly pointed me in the right direction of where to go!
For others that read this & have spend a long time trying to find a cheap Rig to use, here is the setup I got of Amazon for a mere 500 pounds:
Warrior FX-8350 Gaming/Home PC (AMD FX-8350 8 Core Vishera CPU, AMD Radeon 6670 2GB Graphics Card, 1TB Hard Drive, 32GB DDR3 Memory, HDMI 1080p, USB 3.0) (No Operating System)
I cant rate it yet as i only set it up this morning,
Good luck all,
Sam
David D
Hello Roger,
This is an awesome post and thanks a bunch for putting this together.
I have little problem with my csr1000v. I’m using ESXi 5.5, I was able to go through the entire setup even though I wasn’t able to edit the guest machine (scr1000v) I used vSphere vcenter converter stand alone and it works perfectly.
My only issue is after running the license boot level premium command I’m still not able to telnet on 10.10.10.1:2006. The firewall setting, serial port setting and everything else is in place. Any advice? Thanks again
Roger Perkin
If you have followed my post through then I can only think it is a license issue on your ESXi server or a setting wrong somewhere. Send me over some screen shots of your Serial Port config for a router and the firewall settings on the ESXi roger at rogerperkin.co.uk
It might also be a license issue on your ESXi – what license are you running?
mrredmeat
Roger,
Thanks for the write up. I’m one of the many people out there confused on how to start to build a lab. I bookmarked this post and will be trying it out as soon as I get my server.
A few questions i fyou don’t mind.
When you say the throughput drops to 2.5, does it mean your GigE ports can’t run at 10/100 or 1000 anymore and actually just 2.5Mbps?
When you activate the premium license, do you get serial ports to use or are you still stuck with only ethernet ports?
It’s so confusing where to start
Roger Perkin
If you get the latest version of of CSR it does not have a 60 day eval license, it will run at 100k throughput forever. This is fine for a lab environment. The CSR 1000v routers are designed to put put into hosted environments where you would want to push some traffic through them, it is then you would purhcase a 10, 20, 50, 100Mb throughput license. For your CCIE version 5 Lab the free version is all you need.
Tomasz Krupa
Hello Roger,
Have you tried to run CSR1kv on Linux/KVM? I am interested to try that just to see how it performs. I will give it a try and I will share some info regarding performance.
At this moment i have:
2 x Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
32GB RAM
Regards
Roger Perkin
Hi Tomasz,
I have only run up the CSR’s on Vmware – I would be intereseted to know if they run any different on other Hypervisors, but right now the routers are running and my CCIE V5 Lab is working so I am not going to put any more time into that and focus on my studies. I spent many hours a while back building a full physical rack, if I had my time again I would have just gone for rack rentals, but this new setup is suiting me fine. Let me know your findings for other Hypervisors?
Jezz
Hello Roger,
Were you able to run 20 routers on your Dell Optiplex 790 with 16Gb of RAM? I believe the Dell Optiplex 790 can only take a max of 16Gb of RAM but in your post you say that you need 32Gb of RAM to run 20 routers.
Kind Regards
Jezz
Roger Perkin
Jezz,
You are correct the Dell will only take 16GB. I have not run up to 20 routers yest, but have run 12 quite happily on 16GB of RAM, needing 32GB of RAM to run 20 routers is a rough guide, but the best thing is to just keep starting them up and see when performance drops. The ESXi server will start to do disk swapping and this is not an exact science. In an ideal world you need a big server at work!
Jezz
Roger,
Thanks for that. I also will be going down the INE road. I’ve noticed that in the V5 workbook you only seem to need 10 routers at most and yet INE state that you need 20 to build your lab topology. Is that because you will need 20 later on when it comes to full practice labs and should we expect there to be as many as 20 routers in the real CCIE lab? I want to make sure that I get the right HW for the job, I don’t see the point in buying something that will only take 16 Gb of RAM if it means junking it and having to upgrade to 32Gb later on.
Do you have to pay for the VMWare ESXi 5.1 Hypervisor download or is it a free licence?
Kind Regards
Jezz
Roger Perkin
Jezz,
Most of the INE workbooks are based around 10 routers, I believe there are going to be a lot more in the lab and troubleshooting sections. Personally I am running with my 10 router setup and will use rack rentals for anything larger than that. If you can get hardware that will take 32Gb of RAM then you will be covered to run 20 routers or more.
VMWare ESXi 5.1 is free and you can get a 60 day trial license. There is a lot of info on the internet about how to restore your configuration after you rebuild it after 60 days
Andrew Roderos
Good update to INE’s blog post on how to install CSR 1000v! The INE’s blog post is a bit dated so some of the screenshots no longer apply with the new OVAs. Anyway, just want to share that I was able to run 20 CSR 1000v instances with 16GB RAM but I wouldn’t recommend in doing so. Here’s mine http://bit.ly/1qEKKFP
Roger Perkin
Andrew,
Nice post on your CCIE R&S v5 Home Lab setup, I picked up a few tips myself!
Roger
Jair R
Very nice guide! Thanks for sharing it. though I had to keep jumping back and forth a lot to figure out how to install everything, but it was fun! I’m going to INE’s CCIE R&S bootcamp soon (yay!), so this helped me out a lot when trying to figure out what to do to prep. Unfortunately I don’t have Cisco switches since at work Juniper dominates.
I’ve been trying to buy some switches off ebay, but I’m not sure if a license file is needed on 3560-E switches like it is on ISR G2 routers to run 15.0 code and enable IP Services feature. I changed jobs right when 15.0 was introduced and only was familiar with 12.4 on switches and barely at all on ISR G2 routers.
Roger Perkin
Jair, where are you doing the INE bootcamp? It is certainly worth doing but don’t expect it to be the magic bullet. It is 2 weeks of concentrated learning but you still need to prep before and after to pass the lab.
Mark Wallis
ESXi5.1 is free forever, not just for 60 days. Just register for a free license, mine has been up and running far longer than that.
Edson
I’ve been going through the INE Labs for a few weeks now on my newly built HP-Z800 server, but every now and then then my console session to the CSRs disconnects sometimes saying: “The semaphore timeout period has expired.” but sometimes it just doesnt say anything. If you wait long enough the prompt comes back again. Anyone come across this?
Roger Perkin
Sorry Edson, I have not seen this before. My connections to the routers are solid. I would put this down on the way you are connectig to the routers, wired / wireless / vpn from work etc rather than the server.
Tilden
Hello Roger,
I’m in the process of building out my CCIE lab (just waiting FEDEX), and I wanted to know do you have to assigned a different port number for each router?
Thanks,
Roger Perkin
Yes you would configure a different port for each router
R1=2001, R2=2002, R3=2003 etc
kamil
Hi Roger, I have been using CSR1000V more than 4 month, unfortunately today I found that CSR 1000V premium licence does not support mpls:((
DO you have any idea? Until now I was using trial license and took the sanpshot of CSR routers. Every 2 month I was reinstalling the Routers. But I am not certain how to deal with this problem.
Roger Perkin
What do you mean does not support mpls?
I have installed the latest version of the CSR1000v It removes the licence nag and also supports mpls.
Version 03.13.00.S
R1(config)#int g1
R1(config-if)#ip add 1.1.1.1 255.255.255.0
R1(config-if)#mpls ip
R1#sh mpls interface
Interface IP Tunnel BGP Static Operational
GigabitEthernet1 Yes No No No No
Kalpesh Shah
Hi
I have a question about connecting a switch directly to the Esxi. i have used one interface on hte server and connected it to the interface on the switch. I used regular config on the switch; configuring it as trunk, and then created a vswitch in the vsphere. I created a subinterface on the router and a vlan interface on the switch. I am trying to ping either way and I am not able to get any success.
I am a newbie with Esxi, and it would really be nice to get some insight into this.
thanks in advance,
kalpesh
Kalpesh Shah
No worries about it, found what I was missing. Thanks,
Faz
Hi Roger,
Thanks a lot for this post. It helped a lot. I am in process to getting a server. I am a bit confused about the RAM requirement for running 20 CSR routers. I have researched a lot. Some blog say 32 GB while some say 64GB as one CSR needs 2.5GB of ram. So 2.5*20=50. Can you please advice ? Have you or anyone tried to run 20CSR on 32GB RAM server.
Also, is that correct that ESXi 5.1 is free and supports 32 GB. Anything above like 5.5 support 64GB RAM but it’s not free?
I will appreciate your reply
Many thanks..
Roger Perkin
I run 10+ routers on 16GB so I would recommend 32Gb is sufficient to run 20 routers. The CSR1000v routers will take 2-3Gb when they boot but once they settle down they use just over 1Gb.
5.1 is free for 60 days – as I have said before there are many resources on the internet about how to extend this. I personally run a licensed version. You do need the enterprise license though to make sure the serial ports work.
Debbie Westall
Excellent Roger! Thank you for all your efforts!!
Roger Perkin
Glad it has been some help – let me know if you need any more help with anything
nick
Excellent tutorial! Thanks a lot for your valuable help!
My environment consists of 4 Catalysts (1 x 3650 , 3 x 3750G) plus a server with the latest version of CSR1000v (x 10 routers).
I have only to think about my strategy now, since I have only one shoot for my lab exam.
Thanks and best regards,
Nick
Roger Perkin
Thats great, please pm me if you need any help with strategy, the ccie is a marathon and you need a strategy
Kamil
Hi Roger, i found the problem. I deployed OVA again, changed licence level to premium.It works. Moreover older versions of SR1000V have a bug. Over the dmvpn , multicast traffic is not passing.csr1000v-universalk9.03.13.01.S.154-3.S1-ext.ova this version is more stable
Roger Perkin
Great, glad you got it working
HeyGuy
I am probably overthinking this but how do you setup multiple CSR1001v’s
I have installed VM Esxi and have created one vm with CSR1001v and it works fine but it seems that I cant create another using the same ova image
Do I have to create multiple copies w/ different names for all 10 routers I wish to use or is their a better way to make multiple vm instances using one OVA Image ?
Roger Perkin
Its all described in the post, you either just install another router using the same ova and give it a different name or go into the storage and copy the router files multiple times.
The time to do each one is about the same, I prefer to install the router multiple times others like to copy the files
Arwin Reprakash
I’ve posted something similar but it goes a little bit deeper in a 3 part series on “How to build the CCIE v5 lab”
I’ve included scripts as well in Part 3 which will allow you to auto load initial config on all the routers easily.
Roger Perkin
Thanks Arwin, I have included links to your 3 part series at the bottom of this post.
Great resources!
Gbolahan
Thanks for this post. I have been able to setup all my VMs. One question though, do you have any post on how to connect the switches to ESXi 5.5 yet?
Roger Perkin
This is a post I have meaning to do for a while now as a lot of people have asked how to connect the switches to the esxi server for their ccie lab. In summary you just create an interface on your ESXi server and trunk it to a port on switch 1 – you are then just passing the vlans between the switches and the routers.
Carlos
Hi Roger,
Nice guide, thank you very much for your efforts writing this down. Quick question:
I’m trying to decide between Intel (Xeon Quad-Core E-1225V3) or AMD (8 cores FX-8750E). What I really like of AMD is the number of Cores, but Intel is in the compatibility list.
With any of these 2 processors and 32 GB RAM and SSD drive, Do you think I will be able to run 20 CSR1000V for this lab?
Appreciate in advance your comments.
Roger Perkin
Carlos, Thank you for your question, to be honest I can’t advise if one core is better than the other, either is going to fine. If you are running 32GB RAM and an SSD drive you will have twice the RAM of my server and a lot more CPU and I am running 12 routers happily on 16G and a 4 Core PC. Don’t spend too much time on this just get a server spin it up get the routers running and start learning!
BR
Hi Roger,
Thanks much for this! After 8hours of troubleshooting on how to make the telnet work it finally worked! When I used the license from vmware website not the trial version then I cannot telnet to it even though the telnet://:port was correct! So I decided to reinstall the ESXi but with the trial version for 60 days then I was able to telnet into it! LOL! long day!
Roger Perkin
Yes, you will need the enterprise license on your ESXi server to enable telnet functionality. If you run the 60 day trial license it has all features enabled, the license you were using must not be an enterprise one.
I will add this to the post as it is a common problem that people come across
Thanks for letting me know
BR
By the way what is the normal bootup time for the CSR routers?
Roger Perkin
The first time you bootup the CSR routers they could take up to 20 mintutes to boot, but once they have booted once – they boot up really quickly under 30 seconds, faster if you have a fast server.
Roger
John
Hello,
First I want to say thanks for this excellent write up, I have been able to get my CSR1000v’s installed and running.
I am running into a bit of a snag regarding the telnet over serial function.
I have enabled the serial over telnet and allowed it in the firewall as you demonstrated however the serial connection is not functioning properly.
When I telnet to the IP and port specified, my telnet session opens and I see some output from the router but not all when comparing to what is printed on the vmware console. For example I cannot see the router> prompt. In addition when I hit enter nothing happens.
I am using the 60 day trail of ESXi 5.1, but when I check the license level it states I am at evaluation. How do I get the license level to say enterprise so I can get the serial over telnet to work?
Thank you for your help.
Roger Perkin
Boot the router and in the vmware console enter this command platform console serial. Save and reload and you should be able to telnet to your routers, you will not see enterprise in your license if you are running an evaluation. However in the evaluation license all the features are enabled
Mian
Hi Roger
I am planning to build a dedicated machine for CCIE Lab with 32 GB RAM, to run 20 CSR1000V.
I would be very grateful if you please suggest which processor/s would be good enough which serve the purpose or it would be even better if you please tell the specs of your machine
Thanks
Mian
Roger Perkin
Mian,
I am just running a Dell Optiplex 7010 with 16GB of ram it runs 20 routers fine.
It is an Intel i5-3570 CPU @ 3.4GHz
To be honest it doesn’t really matter too much focus on the technology – you can learn so much with 5 routers on GNS. So many people myself included spend too much time focussing on their lab and not as much on actually learning the technology. This works fine for me.
Mian
Hi Roger
Thanks for you kind suggestions and sorry for the “late” thanks
I completely agree with you that lot of people { I am included too } wasting too much time focusing on making lab …while they/we should be focusing on studying . learning …….I believe we network engineers enjoy playing with hardware and its config ): aren’t we ?
Anyways after wasting few week I bought the following from eBay for £172 including postage )……but not fully tested yet
2 X QAUD-CORE XEON E5530 2.4 64-bit ,
(Dual Capable, BOTH Installed, 8cores /logical Procs /16 Threads)
RAM: 32GB of DDR3 SD RAM
Hard Drives: 2 X 146gb 10000rpm Hotplug SFF SAS Drive
Backplane: 1 x 8 Bay 2.5”SAS / SATA
RAID: HP Smart array P400 RAID Controller , 256mb cache
Optical: Not Included
Power Supply: 1 x Power Supply**
NIC: 2 x GB NICs
Remote Management: iLO2 ,Integrated Lights-Out Standard
Daniel
Great tutorial, i’m running ESXi 5.5 on a workstation with 20GB however, every time i deploy the OVF and add the serial with the telnet settings and either power off or reload the CSR, it gets stuck with this message
IF i add the serial on boot the CSR gets stuck on this message.
unable to open /dev/varied/
unable to open /dev/cdrom0
unable to open /dev/cdrom1
INIT: Entering runlevel: 2
INIT: Switching to runlevel: 3
Roger Perkin
This is the default behaviour, at this point you just connect to the router via Telnet to Serial – the VMWare console will stick at this point after you have enabled virtual serial
keith
Great write up, thanks for the effort. It helped me alot
Roger Perkin
That’s great Keith, I am glad it helped – once it is setup it will run like a dream!
Sasha
Great post Roger. Thanks.
I am doing a BSc Networking final year project at university and want to compare a virtual network throughput compared to a physical network. My lecturer wants me to avoid gns3 and use VMware virtual routers and switches, but the more I investigate the more questions I have raised.
1. How can I do a true comparison if the IOS on virtual and physical are different.
2. How many routers will be needed in the network to wow my lecturers
3. Is it possible to connect the virtual network to physical and send data from one to the other. comparing input vs output.
You help will be appreciated.
Thanks.
Sasha
Roger
Sasha,
To answer your questions;
1. If you are using an ASR1000 series router then the IOS will be the same, but I doubt you will have many of them in your lab. So the IOS will be different.
2. You can wow your lecturers with 2 routers – it just depends what you are doing with them. Why not setup a free Amazon account and you can host a server with them and also spin up a CSR1000v – then run a DMVPN tunnel to it – the possibilities are endless.
3. Yes – on your ESXi server ensure you have a physical NIC that connects to the real world and then connect routers to that.
Your issue on throughput will be license on the CSR’s as the free license you get limits the bandwidth, to open this up costs money as you have to license it as a real router, which if you want big bandwidth can be as expensive as buying the physical router.
Good luck, glad the tutorial helped.
I am using my topology right now for testing OTV
Sasha
Thanks
Edo Sunday
Please I need assistance on how to clone routers on ESXI 5.5 . I installed my Esxi 5.5 on VM workstaion because i had the usual network error associated with realtek NIC on help systems. I am sourcing where to buy intel NIC card to enable me do bare metal installation. My main challenge is how to build more routers and also V-switch to start practising the lab.
Roger Perkin
To clone a router just Browse the Datastore on your ESXi server and copy R1 folder as many times as you want.
Then refresh to add to Inventory all the new routers will show up
hansari
Roger,
first of all i have to send my great thanks for your very helpful post on how to build ccie lab .it is great .but i have one problem .bfd is not available in my csr1000v . i would appriciate if you can give any comment on it.
hansari
Roger Perkin
Hansari,
What do you mean BFD is not available on your CSR1000v?
How are you configuring it?
This is the config from mine
R1(config-if)#
R1(config-if)#bfd int
R1(config-if)#bfd interval ?
<50-999> Milliseconds
R1(config-if)#bfd interval
I am running
Cisco IOS XE Software, Version 03.13.00.S
voor
Hi Roger,
Thanks for the writeup, it’s very helpful. In one of your recent replies, you mention that you’re running 3.13.00S. How were you able to download this version from Cisco’s website? Do you have a service contract with them? For anything newer like 3.12.1S(ED), I’m getting the following message when trying to download after logging in:
Service Contract Required
To Download this software, you must have a valid service contract associated to your Cisco.com profile.
voor
Roger Perkin
Hi Voor, I do work for a partner and have access to these downloads. Sorry I can’t help you source the images, I can only advise on how to use them.
mutaz Eldrdere
i would like to know how to connect 4 physical switches to ESXI 6 for ccie lab v5
Roger Perkin
Mutaz,
You simply need a second NIC in your ESXi box and trunk it to your switches.
More info here – http://ieoc.com/forums/p/31497/251164.aspx
swapnil
can i install ESXi server 5.5 into the virtual box 4.3 and then upload all the CSR1000v & IOS-XR VM . i have intel i5 6th generation processor / 32 GB DDR4 RAM. did it work ??
Roger Perkin
You would be trying to install a virtualisation platform inside another virtualisation platform.
I have not tried it but you could try and install the CSR1000V inside Virtual Box?
ESXi is designed to be installed as a bare metal install.
The ideal solution is a dedicated box running ESXi i.e Server or High Powered PC
Roger
DON
i’M CURIOUS WHY ALL THE FUSSING ABOUT VIRTUAL SERIAL PORTS. tHE VSPHERE CLIENT ( wEb OR OTHERWISE) WORKS FINE FOR ME FOR ACCESSING THE ROUTER CONSOLES. iT’S NOT AS SLICK AS SECURE crt OR SOMETHING, BUT IT’S A SIMPLE THING TO CREATE A MANAGEMENT NETWORK IN ESXI AND HAVE EACH ROUTER WITH ITS OWN DEDICATED MGMT NIC AND THEN CONNECT TO EACH ONE WITH YOUR TERMINAL PROGRAM OF CHOICE AT THAT POINT.
UNLESS i’M MISSING SOMETHING. nEEDING AN ENTERPRISE LIC OF VMWARE IS A PRETTY BIG DEAL FOR HOME LAB ENTHUSIASTS.
p.s. i HOPE THIS DOES NOT APPEAR IN ALL CAPS. i COULD NOT GET IT TO USE REGULAR CASE. i TRIED TWO DIFFERENT BROWSERS ACROSS TWO PCS.
Roger Perkin
There is no reason why you cannot use the ESXi console, but most people like to use SECURECRT and have all the routers in one place.
Jg
Hello roger, very nice post.
Sorry that everything is on cap, i am routing from my phone tryed to change but apparently is not working.
I have been trying to get this working on my lab, but it seem to be impossible and very frustrating.
I have a poweredge r630, 64gb ram with esxi 5.5.x
I Installed de Csr routers following your post everything is fine, i can telnet into the routers with no iSsue.
But i can’t ping any Other router in the same vswitch or outside de vswitch, not even the host ip Configuring and ip on thAt range and vSwitch. (create a vswitch for router and activated promiscous mode), On the routers i turnet on the interface attached to the vswitch, created sub-interface, specifified the vlan (encapsulation dot1d 10) put ip there.
I have windows server machine on that same hosT i is able to ping outside the Host and ping the host ip, but not any Csr router.
Is this something that you can shed some light on, or any of your reader.
Any help would be very apPreciated.
Thanks you very much for the post.
IMRAN SHAHID
Hi roger,
i used the same version of 1000v on bare metal esxi 6.5 and it has the diag issue. any ideas. thanks
imran
(Brighton , east sussex)
Roger Perkin
Imran, what is the diag issue?
IMRAN SHAHID
hi,
that’s where it gets stuck.
Unable to open /dev/varied
unable to open /dev/cdrom0
UNABLE TO OPEN /DEV/CDROM1
INIT: entering runlevel: 2
INIT: ENTERING RUNLEVEL: 3
does not go anywhere from here
Roger Perkin
Now connect to the Router via the serial connection you setup. The VMWare console will stop at this point.