Capstone: monitor the branch, then reconstruct an outage from its logs

Steps

1. Open ccna-cap3-branch-routing.pkt and at once save it under this lab's name (File > Save As, ccna-cap5-branch-monitoring.pkt), so the domain 3 file stays as it was. Check the branch is healthy before you change anything:

R1# show standby brief
C:\> ping 10.30.0.10

R1 should be Active and the ping should succeed from PC1. 2. On Server1's Services tab, turn SYSLOG On and NTP On. Note the date and time the NTP page shows: the routers will take it. 3. Point all three routers at Server1 for time. On each router first note that show clock starts with *, then:

R3# configure terminal
R3(config)# ntp server 10.30.0.10
R3(config)# end
R1# configure terminal
R1(config)# ntp server 10.30.0.10
R1(config)# end
R2# configure terminal
R2(config)# ntp server 10.30.0.10
R2(config)# end

After a minute or two (Fast Forward Time helps), show ntp status on each reads Clock is synchronized and show clock has lost its *. Correlating R1's and R2's messages later depends on this. 4. Apply the starting logging policy on all three routers, the same lines on each. It contains the planted fault, the trap level:

R1# configure terminal
R1(config)# service timestamps log datetime msec
R1(config)# logging host 10.30.0.10
R1(config)# logging trap warnings
R1(config)# end

Repeat on R2 and R3 at their own prompts. 5. Make every router an SNMP agent with a read-only, lab-only community, and no read-write community at all:

R1# configure terminal
R1(config)# snmp-server community lab-only-ro RO
R1(config)# end

Repeat on R2 and R3. 6. Build the inventory from PC2, the stand-in monitoring station. Open Desktop > MIB Browser, click Advanced, set Read Community lab-only-ro and SNMP Version v2c, and get sysName from 192.168.10.2, 192.168.10.3 and 10.30.0.1 in turn, changing the Address each time. Write down the three answers, R1, R2 and R3, against their addresses. Then get sysUpTime from 10.30.0.1 and convert the timeticks to days, hours and minutes. 7. The outage, at the starting policy. Fail the switch port facing R1, wait one full minute, then read both branch routers' consoles:

S1> enable
S1# configure terminal
S1(config)# interface g0/1
S1(config-if)# shutdown
S1(config-if)# end

R1 logs the link and line protocol going down, an OSPF adjacency change and an HSRP state change. R2 logs its own HSRP takeover and, later, an OSPF adjacency change. Now open Server1's SYSLOG list: from all of that, only R1's %LINK-3-UPDOWN message for GigabitEthernet0/0/1 arrived. The operations team would see a link go down and nothing about the gateway moving to R2. 8. Break and fix: diagnose with the inclusive rule. Warnings (4) sends 0 to 4; OSPF adjacency changes are 5, and HSRP state changes are 5 on current IOS XE (older releases log them as %STANDBY-6-STATECHANGE, level 6). Read the severity from your own routers' messages, then choose informational (6), which covers both, on all three:

R1# configure terminal
R1(config)# logging trap informational
R1(config)# end
R1# show logging

Repeat on R2 and R3. show logging now reads Trap logging: level informational. 9. Restore the port, then repeat the outage with the corrected policy:

S1# configure terminal
S1(config)# interface g0/1
S1(config-if)# no shutdown
S1(config-if)# end

Wait until show standby brief on R1 shows Active again (R1 preempts). Then shut S1's g0/1 again, wait one full minute, and restore it once more. 10. Decode the burst. On Server1's SYSLOG list, and on R1's and R2's consoles, take each message from the second outage apart into facility, severity and mnemonic, and put them in one timeline by their timestamps. You should find, in order: R1's %LINK-3-UPDOWN for GigabitEthernet0/0/1 and its line-protocol message; R1's OSPF message for neighbor 10.255.255.2 giving Interface down or detached; about 10 seconds later, R2's HSRP change to Active (the hold time); and about 40 seconds after the link failed, R2's OSPF message for neighbor 10.255.255.1 giving Dead timer expired (the dead interval), because R2's own link never went down. 11. Write the incident note in three lines, as you would for a ticket: the root cause (layer 1 on R1's LAN link, the cable or the S1 port), the effect (R2 took over the gateway after about 10 seconds; users kept working), and the consequences that need no action (the OSPF adjacency changes). Name the facility and severity of each message you relied on.