Thursday, 28 May 2015

Troubleshoot Duplicate Route ID

Troubleshooting

The troubleshooting was done with a Cisco IOS software release released before the Cisco bug ID CSCdr61598 (registered customers only) and Cisco bug ID CSCdu08678 (registered customers only) integration.

Single Area Network

This image is a representation of the single area network described in these steps.
duplicate_router_id_ospf1.gif
  1. Issue the show proc cpu | include OSPF command. This allows you to see the OSPF processes that utilize the CPU.
    
r4#show proc cpu | include OSPF
    

    
   3        4704       473       9945  1.38%  0.81%  0.68%   0 OSPF Hello       
    

    
  71        9956      1012       9837  1.47%  1.62%  1.41%   0 OSPF Router
    As seen in the previous example, there is high CPU for OSPF. This shows that there must be something wrong with either the link stability or a duplicate router-id.
  2. Issue the show ip ospf statistics command. This allows you to see if the SPF algorithm is run more than ordinary.
    
r4#show ip ospf statistics
    

    
  Area 0: SPF algorithm executed 46 times
    
SPF calculation time

    
Delta T   Intra D-Intra Summ    D-Summ  Ext     D-Ext   Total   Reason
    
00:01:36   0    0       0       0       0       0       0       N, 

    
00:01:16   0    0       0       0       0       0       0       R, N, 
    
00:01:26   0    0       0       0       0       0       0       R, N, 

    
00:00:56   0    0       0       0       0       0       0       R, N, 
    
00:01:06   0    0       0       0       0       0       0       R, N, 
00:00:46   0    0       0       0       0       0       0       R, N, 

    
00:00:26   0    0       0       0       0       0       0       R, N, 
    
00:00:36   0    0       0       0       0       0       0       R, N, kmbgvc
00:00:16   0    0       0       0       0       0       0       R, N, 

    
00:00:06   0    0       0       0       0       0       0       R, N,
    The show ip ospf statistics command shows that recalculation of SPF is done every 10 seconds, as seen in the previous example. It is triggered by the router and network LSA. There is a problem in the same area as the current router.
  3. Issue the show ip ospf database command.
    
r4#show ip ospf database 
    


    
            OSPF Router with ID (50.0.0.4) (Process ID 1)
    
Router Link States (Area 0)


    
Link ID         ADV Router      Age         Seq#       Checksum Link count
    
50.0.0.1        50.0.0.1        681         0x80000002 0x7E9D   3

    
50.0.0.4        50.0.0.4        705         0x80000003 0x83D    4
    
50.0.0.2        50.0.0.2        674         0x80000004 0x2414   5
50.0.0.5        50.0.0.5        706         0x80000003 0x5C24   6

    

    
50.0.0.6        50.0.0.6        16          0x80000095 0xAF63   6
50.0.0.7        50.0.0.7        577         0x80000005 0x86D5   8
                Net Link States (Area 0)


    
192.168.2.6     50.0.0.6        6           0x8000007A 0xABC7
    

    
Link ID         ADV Router      Age         Seq#       Checksum
    The show ip ospf database command shows that one LSA is newer (age 16) and its sequence number is much higher then the other LSAs in the same OSPF database. You need to figure out which router sent this LSA. Since it is in the same area, the advertising router id is known (50.0.0.6). It is more probable that this router ID is duplicated. You need to find out which other router has the same router-id.
  4. This example shows several instances of the show ip ospf database command.
    
r4#show ip ospf database router adv-router 50.0.0.6
    


    
            OSPF Router with ID (50.0.0.4) (Process ID 1)
    
Router Link States (Area 0)

  LS age: 11

    
  Link State ID: 50.0.0.6
    
Options: (No TOS-capability, DC)
  LS Type: Router Links
  Advertising Router: 50.0.0.6

    
    Link connected to: a Transit Network
    
LS Seq Number: 800000C0
  Checksum: 0x6498
  Length: 72
  Number of Links: 4


    
      Number of TOS metrics: 0
    
(Link ID) Designated Router address: 192.168.2.6
     (Link Data) Router Interface address: 192.168.2.6
       TOS 0 Metrics: 10

    
     (Link ID) Neighboring Router ID: 50.0.0.7
    
Link connected to: another Router (point-to-point)

    

    
(Link Data) Router Interface address: 192.168.0.21

    

    
Number of TOS metrics: 0
       TOS 0 Metrics: 64

    
     (Link ID) Network/subnet number: 192.168.0.20
    
Link connected to: a Stub Network
     (Link Data) Network Mask: 255.255.255.252

    
    Link connected to: a Stub Network
    
Number of TOS metrics: 0
       TOS 0 Metrics: 64

     (Link ID) Network/subnet number: 50.0.0.6

    
r4#show ip ospf database router adv-router 50.0.0.6
    
(Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metrics: 1


    

    

    
            OSPF Router with ID (50.0.0.4) (Process ID 1)
    
Router Link States (Area 0)

  LS age: 7

    
  Link State ID: 50.0.0.6
    
Options: (No TOS-capability, DC)
  LS Type: Router Links
  Advertising Router: 50.0.0.6

    
  LS Seq Number: 800000C7  
    

    

    

    
!--- The sequence number has increased.
    

    
 
    

    
Checksum: 0x4B95 
    
Length: 96 

    

    
Number of Links: 6 

    


    

    
!--- The number of links has increased although the network has been stable. 
    

    

    

    
Link connected to: a Stub Network
    

    
     (Link ID) Network/subnet number: 192.168.3.0
    
(Link Data) Network Mask: 255.255.255.0

    
       TOS 0 Metrics: 10
    
Number of TOS metrics: 0


    
     (Link ID) Neighboring Router ID: 50.0.0.5
    
Link connected to: another Router (point-to-point)

    
       TOS 0 Metrics: 64
    
(Link Data) Router Interface address: 192.168.0.9
      Number of TOS metrics: 0


    
     (Link Data) Network Mask: 255.255.255.252
    
Link connected to: a Stub Network
     (Link ID) Network/subnet number: 192.168.0.8
      Number of TOS metrics: 0

    
     (Link ID) Neighboring Router ID: 50.0.0.2
    
TOS 0 Metrics: 64

    Link connected to: another Router (point-to-point)
     (Link Data) Router Interface address: 192.168.0.2

    
     (Link ID) Network/subnet number: 192.168.0.0
    
Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link Data) Network Mask: 255.255.255.252

    
     (Link Data) Network Mask: 255.255.255.255
    
Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 50.0.0.6
      Number of TOS metrics: 0

    
       TOS 0 Metrics: 1
  5. If you know your network, you can find which router advertises those links. The first previous output shows that the LSAs are sent by a router with OSPF neighbors 50.0.0.7, whereas the second output shows neighbors 50.0.0.5 and 50.0.0.6. Issue the show ip ospf command in order to find those routers and access them in order to verify their OSPF router-id. In this example setup, they are R6 and R3.
    
3>show ip ospf
    

    
 Routing Process "ospf 1" with ID 50.0.0.6
    
Supports only single TOS(TOS0) routes

    
r6#show ip ospf
    
Supports opaque LSA


    

    
Routing Process "ospf 1" with ID 50.0.0.6

    
 Supports opaque LSA
    

    
 Supports only single TOS(TOS0) routes
  6. Issue the show run | beg router ospf command in order to check the configuration that starts at the OSPF configuration.
    
R6#show run | include router ospf
    

    
router ospf 1
    

    
 router-id 50.0.0.6
    

    
 log-adjacency-changes
    

    
 network 50.0.0.0 0.0.0.255 area 0
    

    
 network 192.168.0.0 0.0.0.255 area 0
    
network 192.168.2.0 0.0.0.255 area 0

    
r3#show run | begin router ospf 
    

    

    
router ospf 1

    
 log-adjacency-changes
    
network 50.0.0.0 0.0.0.255 area 0

    
 network 192.168.0.0 0.0.0.255 area 0
    

    
 network 192.168.3.0 0.0.0.255 area 0
    In the previous example, the router-id command was removed and the OSPF process was not restarted. The same problem can also result from a loopback interface that is removed and configured somewhere else.
  7. Issue the clear ip ospf 1 process command and the show ip ospf command in order to clear the process.
    
r3#clear ip ospf 1 process 
    

    
Reset OSPF process? [no]: y
    

    
r3#show ip ospf
    

    
 Routing Process "ospf 1" with ID 50.0.0.6
    
Supports only single TOS(TOS0) routes

    
 Supports opaque LSA
    As shown in the previous example, the wrong IP address still appears.
  8. Issue the show ip int brie command in order to check the interface.
    
r3#show ip int brie
    

    
Interface                  IP-Address      OK? Method Status                Protocol
    
Ethernet0/0                192.168.3.1     YES NVRAM  up                    up      

    
Serial2/0                  192.168.0.9     YES NVRAM  up                    up      
    
Serial1/0                  192.168.0.2     YES NVRAM  up                    up      
Loopback0                  unassigned      YES NVRAM  up                    up      

    


    
Loopback1                  50.0.0.6        YES NVRAM  up                    up     


    

    
!--- The highest Loopback IP address
    In order to correct the problem, make sure that either the highest loopback configured on the router is unique in your OSPF network, or configure statically the router-id with the router-id <ip address> command under the OSPF router configuration mode.

Multiple Areas with ASBR

The symptoms of these problems are that the external route, which is learned through the redistribution from static into OSPF process by R6, ASBR router flaps from the routing table on all routers within OSPF Area 0. The external route is 120.0.0.0/16 and the problem is noticed on Router 5 in Area 0. Start to troubleshoot from there.
duplicate_router_id_ospf2.gif
  1. Issue the show ip route command a few times consecutively in order to see the symptom.
    
r5#show ip route 120.0.0.0
    

    
Routing entry for 120.0.0.0/16, 1 known subnets
    

    
O E2    120.0.0.0 [110/20] via 192.168.0.9, 00:00:03, Serial2/0
    

    
r5#show ip route 120.0.0.0
    

    
% Network not in table
    

    
r5#
  2. Take a look at the OSPF database in order to check whether the LSA is received. If you issue the show ip ospf databasecommand several times in a row, you notice that the LSA is received by two routers, 50.0.0.6 and 50.0.0.7. If you look at the age of the second entry, if present, you notice that its value changes dramatically.
    
r5#show ip ospf database | begin Type-5 
    

    
                Type-5 AS External Link States
    

    
Link ID         ADV Router      Age         Seq#       Checksum Tag
    
120.0.0.0       50.0.0.6        2598        0x80000001 0xE10E   0

    
r5#show ip ospf database | begin Type-5 
    
120.0.0.0       50.0.0.7        13          0x80000105 0xD019   0

    

    
Type-5 AS External Link States

    
Link ID         ADV Router      Age         Seq#       Checksum Tag
    

    
120.0.0.0       50.0.0.6        2599        0x80000001 0xE10E   0
    
120.0.0.0       50.0.0.7        14          0x80000105 0xD019   0

    
r5#show ip ospf database | begin Type-5 
    

    
                Type-5 AS External Link States
    

    
Link ID         ADV Router      Age         Seq#       Checksum Tag
    
120.0.0.0       50.0.0.6        2600        0x80000001 0xE10E   0

    
r5#show ip ospf database | begin Type-5 
    
120.0.0.0       50.0.0.7        3601        0x80000106 0x6F6    0

    

    
Type-5 AS External Link States

    
Link ID         ADV Router      Age         Seq#       Checksum Tag
    

    
120.0.0.0       50.0.0.6        2602        0x80000001 0xE10E   0
    

    
r5#show ip ospf database | begin Type-5
    

    
                Type-5 AS External Link States
    

    
Link ID         ADV Router      Age         Seq#       Checksum Tag
    
120.0.0.0       50.0.0.6        2603        0x80000001 0xE10E   0

    
r5#
  3. You also notice strange behavior if you look at the sequence number for the LSAs that are received from 50.0.07, which is the advertising router. Review what other LSAs are received from 50.0.0.7. If you issue the show ip ospf database adv-router 50.0.0.7 command several times in a row, the entries vary quickly, as shown in this example.
    
r5#show ip ospf database adv-router 50.0.0.7
    


    
            OSPF Router with ID (50.0.0.5) (Process ID 1)
    
Router Link States (Area 0)


    
Link ID         ADV Router      Age         Seq#       Checksum Link count
    
50.0.0.7        50.0.0.7        307         0x8000000D 0xDF45   6

                Type-5 AS External Link States


    
120.0.0.0       50.0.0.7        9           0x8000011B 0xA42F   0
    
Link ID         ADV Router      Age         Seq#       Checksum Tag




    
r5#show ip ospf database network adv-router 50.0.0.7
    

    
            OSPF Router with ID (50.0.0.5) (Process ID 1)
    

    
r5#show ip ospf database network adv-router 50.0.0.7
    

    
            OSPF Router with ID (50.0.0.5) (Process ID 1)
    This last output does not show anything. Either the route is flapping or there is a problem of another kind, most probably a duplicate router ID within the OSPF domain.
  4. Issue the show ip ospf database command in order to view the external LSAs advertised by 50.0.0.7.
    
r5#show ip ospf database external adv-router 50.0.0.7
    


    
            OSPF Router with ID (50.0.0.5) (Process ID 1)
    
Type-5 AS External Link States


    
  Options: (No TOS-capability, DC)
    
Delete flag is set for this LSA
  LS age: MAXAGE(3600)

    
  Advertising Router: 50.0.0.7
    
LS Type: AS External Link
  Link State ID: 120.0.0.0 (External Network Number )

    
        Metric Type: 2 (Larger than any link state path)
    
LS Seq Number: 80000136
  Checksum: 0xA527
  Length: 36
  Network Mask: /16
        TOS: 0 

    
r5#show ip ospf database  external adv-router 50.0.0.7
    
Metric: 16777215 
        Forward Address: 0.0.0.0
        External Route Tag: 0


    

    

    
            OSPF Router with ID (50.0.0.5) (Process ID 1)
    

    
r5#
  5. Look at the SPF calculation reasons in order to verify this. X means that SPF runs every 10 seconds because of an External LSA (type 5) flap and indeed, you see that SPF runs.
    
r5#show ip ospf statistic
    

    
  Area 0: SPF algorithm executed 2 times
    
SPF calculation time

    
Delta T   Intra D-Intra Summ    D-Summ  Ext     D-Ext   Total   Reason
    
00:47:23   0    0       0       0       0       0       0       X

    
00:33:21   0    0       0       0       0       0       0       X
    
00:46:33   0    0       0       0       0       0       0       X
00:32:05   0    0       0       0       0       0       0       X

    
00:10:03   0    0       0       0       0       0       0       R, SN, X
    
00:10:13   0    0       0       0       0       0       0       R, SN, X
00:09:53   0    0       0       0       0       0       0       R, 

    
00:09:23   0    0       0       0       0       0       0       X
    
00:09:43   0    0       0       0       0       0       0       R, SN, X

    
00:09:33   0    0       0       0       0       0       0       X
  6. It is known that the problem is outside the current area. Turn your focus on the ABR. Telnet to the ABR Router 2 in order to have more visibility on other areas than OSPF area 0. Issue the show ip ospf border-routers and show ip ospf database network adv-router commands.
    
r2#show ip ospf border-routers 
    


    
OSPF Process 1 internal Routing Table
    

    
Codes: i - Intra-area route, I - Inter-area route
    

    
i 50.0.0.7 [20] via 192.168.2.1, Ethernet0/0, ASBR, Area 1, SPF 25
    

    
r2#show ip ospf database network adv-router 50.0.0.7 
    

    
            OSPF Router with ID (50.0.0.2) (Process ID 1)
    
Net Link States (Area 1)


    
  Options: (No TOS-capability, DC)
    
Routing Bit Set on this LSA
  LS age: 701
  LS Type: Network Links

    
  Advertising Router: 50.0.0.7
    
Link State ID: 192.168.1.2 (address of Designated Router)
  LS Seq Number: 80000001
  Checksum: 0xBC6B
  Length: 32

    
        Attached Router: 50.0.0.1
    
Network Mask: /24

    
        Attached Router: 50.0.0.7
  7. The faulty router is on the same LAN as 50.0.0.1. It must be Router 6. Issue the show ip ospf command.
    
r6#show ip ospf
    

    
 Routing Process "ospf 1" with ID 50.0.0.7
    
Supports only single TOS(TOS0) routes

    
 It is an autonomous system boundary router.
    

    
 Supports opaque LSA
  8. Once the faulty router is found, refer to the Single Area Network section of this document to correct the problem.

Error Message: %OSPF-4-FLOOD_WAR: Process 60500 flushes LSA ID 10.x.x.0 type-5 adv-rtr 10.40.x.x in area 10.40.0.0

The %OSPF-4-FLOOD_WAR: Process 60500 flushes LSA ID 10.35.70.4 type-5 adv-rtr 10.40.0.105 in area 10.40.0.0error message is received.
This error message states that the router originates or flushes LSA at a high rate. A typical scenario in a network may be where one router in the network originates LSA and the second router flushes that LSA. A detailed description of this error message is provided here:
  • Process 60500 - The OSPF process that reports the error. In this example, the process ID is 60500.
  • re-originates or flushes (keyword) - Indicates if the router originates LSA or flushes. In this error message, the routerflushes LSA.
  • LSA ID 10.35.70.4 - Link state ID for which a flood war is detected. In this example, it is 10.35.70.4.
  • type -5 - LSA type. This example has a Type 5 LSA.
    Note: A flood war has a different root cause for every LSA.
  • adv-rtr - Router which originates LSA (that is, 10.40.0.105).
  • Area - Area to which the LSA belongs. In this example, the LSA belongs to 10.40.0.0.
Posted by at
Location: Patiala, Punjab, India

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)