Common Port and Interface Problems
Port or Interface Status is Disable or Shutdown
An obvious but sometimes overlooked cause of port connectivity failure is an incorrect configuration on the switch. If a port has a solid orange light, this means the software inside the switch shut down the port, either by way of the user interface or by internal processes.
Some port LEDs of the platform work differently in regard to STP. For example, the Catalyst 1900/2820 turns ports orange when they are in STP blocking mode. In this case, an orange light can indicate the normal functions of the STP. The Catalyst 6000/5000/4000 does not turn the port light orange when it blocks for STP.
Make sure the port or module has not been disabled or powered down for some reason. If a port or module is manually shut down on one side of the link or the other, the link does not come up until you re-enable the port. Check the port status on both sides.
For CatOS, check show port and, if the port is disabled, re-enable it:
Port Name Status Vlan Duplex Speed Type----- -------------------- ---------- ---------- ------ ----- ------------3/1 disabled 1 auto auto 10/100BaseTX
!--- Use the set port enable mod/port command to re-enable this port.
Use the show module command to determine if the module is disabled. If it is, re-enable it:
Mod Slot Ports Module-Type Model Sub Status--- ---- ----- ------------------------- ------------------- ------ --------2 2 2 1000BaseX Supervisor WS-X6K-SUP1A-2GE Yes ok3 3 48 10/100BaseTX Ethernet WS-X6348-RJ-45 no disable16 2 1 Multilayer Switch Feature WS-F6K-MSFC no ok
!--- Use the set module enable mod/port command to re-enable this port.
For Cisco IOS, use the show run interface command and check to see if the interface is in a shutdown state:
Switch#sh run interface fastEthernet 4/2!interface FastEthernet4/2switchport trunk encapsulation dot1qswitchport mode trunk shutdown!--- Use the no shut command in config-if mode to re-enable this interface.duplex full speed 100end
If the port goes into shutdown mode immediately after a reboot of the switch, the probable cause is the port security setting. If unicast flooding is enabled on that port, it can cause the port to shut down after a reboot. Cisco recommends that you disable the unicast flooding because it also ensure that no flooding occurs on the port once the MAC address limit is reached.
Port or Interface Status is errDisable
By default, software processes inside the switch can shut down a port or interface if certain errors are detected.
When you look at show port command for CatOS the status can read errdisable:
switch>(enable) sh port 4/3Port Name Status Vlan Duplex Speed Type----- -------------------- ---------- ---------- ------ ----- ------------4/3 errdisable 150 auto auto 10/100BaseTX!--- The show port command displays a status of errdisable.
Or use the show interface card-type {slot/port} status command for Cisco IOS:
Router#show int fasteth 2/4 statusPort Name Status Vlan Duplex Speed TypeGi2/4 err-disabled 1 full 1000 1000BaseSX
!--- The show interfaces card-type {slot/port} status command for Cisco IOS
!--- displays a status of errdisabled.
!--- The show interfaces status errdisabled command shows all the interfaces
!--- in this status.
The show logging buffer command for CatOS and the show logging command for Cisco IOS also display error messages (exact message format varies) that relate to the errdisable state.
Ports or interlaces being shut down as a result of errdisable are referred to as reasons in CatOS and causes in Cisco IOS. The reasons or causes for this happening range from EtherChannel misconfiguration that causes a PAgP flap, duplex mismatch, BPDU port-guard and portfast configured at the same time, UDLD that detects a one-way link, etc.
You have to manually re-enable the port or interface to take it out the errdisable state unless you configure an errdisable recovery option. In CatOS software 5.4(1) and later you have the ability to automatically re-enable a port after a configurable amount of time spent in the errdisable state. Cisco IOS on most switches also has this functionality. The bottom line is that even if you configure the interface to recover from errdisable the problem reoccurs until the root cause is determined.
For more information on the causes of and recovery from the errdisable status for switches that run CatOS, refer to Recovering From errDisable Port State on the CatOS Platforms.
Note: Use this link as a reference for errdisable status on switches that run Cisco IOS, as well since the root causes are the same no matter which operating system you run.
This table shows a comparison of the commands used to configure verify and troubleshoot the errdisable status on switches that run CatOS and Cisco IOS. Choose a command to go to the command documentation.
| CatOS errdisable Commands | Action | Cisco IOS errdisable Commands |
|---|---|---|
| set errdisable-timeout {enable | disable} {reason} | set or configure | errdisable detect causeerrdisable recovery cause |
| set errdisable-timeout interval {interval | set or configure | errdisable recovery {interval |
| show errdisable-timeout | verify & troubleshoot | show errdisable detect show interfaces status err-disabled |
Port or Interface Status is Inactive
One common cause of inactive ports on switches that run CatOS is when the VLAN they belong to disappears. The same problem can occur on switches that run Cisco IOS when interfaces are configured as layer 2 switchports that use the switchport command.
Every port in a Layer 2 switch belongs to a VLAN. Every port on a Layer 3 switch configured to be a L2 switchport must also belong to a VLAN. If that VLAN is deleted, then the port or interface becomes inactive.
Note: Some switches show a steady orange (amber) light on each port when this happens.
For CatOS, use the show port or show port status command along with the show vlan command to verify:
Switch> (enable) sh port status 2/2Port Name Status Vlan Duplex Speed Type----- -------------------- ---------- ---------- ------ ----- ------------2/2 inactive 2 full 1000 1000BaseSX!--- Port 2/2 is inactive for VLAN 2.Switch> (enable) sh vlanVLAN Name Status IfIndex Mod/Ports, Vlans---- -------------------------------- --------- ------- ------------------------1 default active 5 2/1!--- VLANs are displayed in order and VLAN 2 is missing.
For Cisco IOS, use the show interfaces card-type {slot/port} switchport command along with show vlan to verify.
Router#sh interfaces fastEthernet 4/47 switchportName: Fa4/47Switchport: EnabledAdministrative Mode: static accessOperational Mode: static accessAdministrative Trunking Encapsulation: negotiateOperational Trunking Encapsulation: native Negotiation of Trunking: OffAccess Mode VLAN: 11 ((Inactive))!--- FastEth 4/47 is inactive.Router#sh vlanVLAN Name Status Ports---- -------------------------------- --------- -------------------------------1 default active Gi1/1, Gi2/1, Fa6/6!--- VLANs are displayed in order and VLAN 11 is missing.10 UplinkToGSR's active Gi1/2, Gi2/2 30 SDTsw-1ToSDTsw-2Link active Fa6/45
If the switch that deleted the VLAN is a VTP server for the VTP domain, every server and client switch in the domain has the VLAN removed from their VLAN table as well. When you add the VLAN back into the VLAN table from a VTP server switch, the ports of the switches in the domain that belong to that restored VLAN become active again. A port remembers what VLAN it is assigned to, even if the VLAN itself is deleted.
Uplink Port or Interface Status is Inactive
On a Catalyst 4510R series switch, in order to enable both the 10-Gigabit Ethernet and the Gigabit Ethernet SFP uplink ports, there is an optional configuration. In order to enable the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces, issue the hw-module uplink select all command. After you issue the command, re-boot the switch or else the output of the show interface status module module number command shows the uplink port as inactive.
Cisco IOS Software Release 12.2(25)SG supports the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces on Catalyst 4500 switches.
Note: On the Catalyst 4503, 4506, and 4507R series switches, this capability is automatically enabled.
Deferred Counter on the Catalyst Switch Interface Starts Incrementing
The issue is because the traffic load destined for the switch is excessive and causes the frames to be discarded. Normally the deferred frames are the number of frames that have been transmitted successfully after waiting for the media, because the media was busy. This is usually seen in half duplex environments where the carrier is already in use when it tries to transmit a frame. But in full duplex environments the issue occurs when the excessive load is destined for the switch.
This is the workaround:
- Hardcode both ends of the link to full duplex so that the negotiation mismatch can be avoided.
- Change the cable and patch panel cord to ensure that the cable and patch cords are not defective.
Note: If the Deferred Counter error increments on a GigabitEthernet of a Supervisor 720, turn on speed negotiation on the interface as a workaround.
Intermittent Failure to set timer [value] from vlan [vlan no]
The issue occurs when Encoded Address Recognition Logic (EARL) is unable to set the CAM aging time for the VLAN to the required number of seconds. Here, the VLAN aging time is already set to fast aging.
When the VLAN is already in fast aging, EARL cannot set the VLAN to fast aging, and aging timer set process is blocked. The default CAM aging time is five minutes, which means that the switch flushes the table of learned MAC addresses every five minutes. This ensures that the MAC address table (the CAM table) contains the most recent entries.
Fast aging temporarily sets the CAM aging time to the number of seconds that the user specifies, and is used in conjunction with the Topology Change Notification (TCN) process. The idea is that when a topology change occurs, this value is necessary to flush the CAM table faster, to compensate for the topology change.
Issue the show cam aging command to check the CAM aging time on the switch. TCNs and fast aging are fairly rare. As a result, the message has a severity level of 3. If the VLANs are frequently in fast aging, check the reason for fast aging.
The most common reason for TCNs is client PCs connected directly to a switch. When you power up or down the PC, the switch port changes state, and the switch starts the TCN process. This is because the switch does not know that the connected device is a PC; the switch only knows that the port has changed the state.
In order to resolve this issue, Cisco has developed the PortFast feature for host ports. An advantage of PortFast is that this feature suppresses TCNs for a host port.
Note: PortFast also bypasses spanning-tree calculations on the port, and is therefore only suitable for use on a host port.
In order to enable PortFast on the port, configure one of these commands:
set spantree portfast mod/port enable | disable
or
set port host mod/port Cisco recommends this command if the switch runs CatOS5.4 or higher versions.
Trunking Mode Mismatch
Check the trunking mode on each side of the link. Make sure both sides are in the same mode (both trunking with the same method: ISL or 802.1q, or both not trunking). If you turn the trunking mode to on (as opposed to auto or desirable) for one port and the other port has the trunking mode set to off, they are not able to communicate. Trunking changes the formatting of the packet. The ports need to be in agreement as to what format they use on the link or they do not understand each other.
For CatOS, use the show trunk {mod/port}command to verify the trunk status and Native VLAN (for dot1q) matches on both sides.
Switch> (enable) sh trunk 3/1* - indicates vtp domain mismatchPort Mode Encapsulation Status Native vlan-------- ----------- ------------- ------------ -----------3/1 desirable dot1q trunking 1Port Vlans allowed on trunk-------- ---------------------------------------------------------------------3/1 1-1005,1025-4094!--- Output truncated.
For Cisco IOS, use the show interfaces card-type {mod/port} trunk command to verify the trunking configuration and Native VLAN.
Router#sh interfaces fastEthernet 6/1 trunkPort Mode Encapsulation Status Native vlanFa6/1 desirable 802.1q trunking 1Port Vlans allowed on trunkFa6/1 1-4094!--- Output truncated.
Jumbos, Giants, and Baby Giants
The Maximum Transmission Unit (MTU) of the data portion of an ethernet frame is 1500 bytes by default. If the transmitted traffic MTU exceeds the supported MTU the switch does not forward the packet. Also, dependent upon the hardware and software, some switch platforms increment port and interface error counters as a result.
- Jumbo frames are not defined as part of the IEEE Ethernet standard and are vendor-dependent. They can be defined as any frame bigger than the standard ethernet frame of 1518 bytes (which includes the L2 header and Cyclic Redundancy Check (CRC)). Jumbos have larger frame sizes, typically > 9000 bytes.
- Giant frames are defined as any frame over the maximum size of an ethernet frame (larger than 1518 bytes) that has a bad FCS.
- Baby Giant frames are just slightly larger than the maximum size of an ethernet frame. Typically this means frames up to 1600 bytes in size.
Support for jumbo and baby giants on Catalyst switches varies by switch platform, sometimes even by modules within the switch. The software version is also a factor.
Refer to Configuring Jumbo/Giant Frame Support on Catalyst Switches for more information on system requirements, configuring and troubleshooting for jumbo and baby giant issues.
Cannot Ping End Device
Check the end device by pinging from the directly connected switch first, then work your way back port by port, interface by interface, trunk by trunk until you find the source of the connectivity issue. Make sure each switch can see the end device's MAC address in its Content-Addressable Memory (CAM) table.
For CatOS, use the show cam dynamic {mod/port} command.
Switch> (enable) sh cam dynamic 3/1* = Static Entry. + = Permanent Entry. # = System Entry. R = Router Entry.X = Port Security Entry $ = Dot1x Security EntryVLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]---- ------------------ ----- -------------------------------------------2 00-40-ca-14-0a-b1 3/1 [ALL]!--- A workstation on VLAN 2 with MAC address 00-40-ca-14-0a-b1 is seen in the CAM table!--- on the trunk port of a switch running CatOS.Total Matching CAM Entries Displayed =1Console> (enable)
For Cisco IOS, use the show mac address-table dynamic command, or substitute the interface keyword.
Router# sh mac-address-table int fas 6/3Codes: * - primary entryvlan mac address type learn qos ports------+----------------+--------+-----+---+--------------------------* 2 0040.ca14.0ab1 dynamic No -- Fa6/3!--- A workstation on VLAN 2 with MAC address 0040.ca14.0ab1 is directly connected!--- to interface fastEthernet 6/3 on a switch running Cisco IOS.
Once you know the switch actually has the MAC address of the device in it's CAM table, determine whether this device is on the same or different VLAN from where you are trying to ping.
If the end device is on a different VLAN from where you are trying to ping, a L3 switch or router must be configured to allow the devices to communicate. Make sure your L3 addressing on the end device and on the router/ L3 switch is correctly configured. Check the IP address, subnet mask, default gateway, dynamic routing protocol configuration, static routes, etc.
Using Set Port Host or Switchport Host to Fix Startup Delays
If stations are not able to talk to their primary servers when connecting through a switch, the problem can involve delays on the switch port becoming active after the physical layer link comes up. In some cases, these delays can be up to 50 seconds.
Some workstations simply cannot wait this long before finding their server without giving up. These delays are caused by STP, trunking negotiations (DTP), and EtherChannel negotiations (PAgP). All of these protocols can be disabled for access ports where they are not needed, so the switch port or interface starts forwarding packets a few seconds after it establishes a link with its neighbor device.
The set port host command was introduced in CatOS Version 5.4. This command sets the trunking and channel modes to off and puts the port in a STP forwarding state.
Switch> (enable) set port host 3/5-10Port(s) 3/5-10 channel mode set to off.!--- The set port host command also automatically turns off etherchannel on the ports.Warning: Spantree port fast start should only be enabled on ports connectedto a single host. Connecting hubs, concentrators, switches, bridges, etc. toa fast start port can cause temporary spanning tree loops. Use with caution.!--- Notice the switch warns you to only enable port host on access ports.Spantree ports 3/5-10 fast start enabled.Dot1q tunnel feature disabled on port(s) 3/5-10.Port(s) 3/5-10 trunk mode set to off.!--- The set port host command also automatically turns off trunking on the ports.
Note: For CatOS versions earlier than version 5.4, the set spantree portfast {mod/port} enable command was used. In current versions of CatOS, you still have the option to use only this command, but this requires that you turn off trunking and etherchannel separately to help fix workstation startup delays. The additional commands to do this are: set port channel {mod/port} off and set trunk {mod/port} off .
For Cisco IOS, you can use the switchport host command to disable channeling and to enable spanning-tree portfast and theswitchport nonegotiate command to turn off DTP negotiation packets. Use the interface-range command to do this on multiple interfaces at once.
Router6k-1(config)#int range fastEthernet 6/13 - 18Router6k-1(config-if-range)#switchportRouter6k-1(config-if-range)#switchport hostswitchport mode will be set to accessspanning-tree portfast will be enabledchannel group will be disabled!--- Etherchannel is disabled and portfast is enabled on interfaces 6/13 - 6/18.Router6k-1(config-if-range)#switchport nonegotiate!--- Trunking negotiation is disabled on interfaces 6/13 - 6/18.Router6k-1(config-if-range)#endRouter6k-1#
Cisco IOS has the option to use the global spanning-tree portfast default command to automatically apply portfast to any interface configured as a layer 2 access switchport. Check the Command Reference for your release of software to verify the availability of this command. You can also use the spanning-tree portfast command per interface, but this requires that you turn off trunking and etherchannel separately to help fix workstation startup delays.
Refer to Using Portfast and Other Commands to Fix Workstation Startup Connectivity Delays for more information how to fix startup delays.
Speed/Duplex, Autonegotiation, or NIC Issues
If you have a large amount of alignment errors, FCS errors, or late collisions, this can indicate one of these:
- Duplex Mismatch
- Bad or Damaged Cable
- NIC Card Issues
Duplex Mismatch
A common issue with speed/duplex is when the duplex settings are mismatched between two switches, between a switch and a router or between the switch and a workstation or server. This can occur when manually hardcoding the speed and duplex or from autonegotiation issues between the two devices.
If the mismatch occurs between two Cisco devices with the Cisco Discovery Protocol (CDP) enabled, you see the CDP error messages on the console or in the logging buffer of both devices. CDP is useful to detect errors, as well as port and system statistics on nearby Cisco devices. CDP is Cisco proprietary and works by sending packets to a well-known mac address 01-00-0C-CC-CC-CC.
The example shows the log messages that result from a duplex mismatch between two Catalyst 6000 series switches: one that runs CatOS, and the other that runs Cisco IOS. These messages generally tell you what the mismatch is and where it occurs.
2003 Jun 02 11:16:02 %CDP-4-DUPLEXMISMATCH:Full/half duplex mismatch detected on port 3/2!--- CatOS switch sees duplex mismatch.Jun 2 11:16:45 %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet6/2(not half duplex), with TBA04251336 3/2 (half duplex).!--- Cisco IOS switch sees duplex mismatch.
For CatOS, use the show cdp neighbor [mod/port] detail command to display CDP information for Cisco neighbor devices.
Switch> (enable) sh cdp neighbor 3/1 detailPort (Our Port): 3/1Device-ID: RouterIP Address: 10.1.1.2Device Addresses: Holdtime: 133 secCapabilities: ROUTER SWITCH IGMPVersion:Cisco Internetwork Operating System SoftwareIOS (tm) c6sup2_rp Software (c6sup2_rp-PK2S-M), Version 12.1(13)E6, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) TAC Support: http://www.cisco.com/tacCompiled Fri 18-Apr-03 15:35 by hqluongCopyright (c) 1986-2003 by cisco Systems, Inc. Platform: cisco Catalyst 6000Port-ID (Port on Neighbors's Device): FastEthernet6/1!--- Neighbor device to port 3/1 is a Cisco Catalyst 6000 Switch on!--- FastEth 6/1 running Cisco IOS.VTP Management Domain: test1Native VLAN: 1Duplex: full!--- Duplex is full.System Name: unknownSystem Object ID: unknownManagement Addresses: unknownPhysical Location: unknownSwitch> (enable)
For Cisco IOS, use the show cdp neighbors card-type {slot/port} detail command to display CDP information for Cisco neighbor devices.
Router#sh cdp neighbors fastEthernet 6/1 detail-------------------------Device ID: TBA04251336IP address: 10.1.1.1Entry address(es):Platform: WS-C6006, Capabilities: Trans-Bridge Switch IGMPInterface: FastEthernet6/1, Port ID (outgoing port): 3/1Holdtime : 152 secVersion :WS-C6006 Software, Version McpSW: 6.3(3) NmpSW: 6.3(3)Copyright (c) 1995-2001 by Cisco Systems!--- Neighbor device to FastEth 6/1 is a Cisco Catalyst 6000 Switchadvertisement version: 2!--- on port 3/1 running CatOS.VTP Management Domain: 'test1'Native VLAN: 1Duplex: full!--- Duplex is full.Router#
Setting auto speed/duplex on one side and 100/Full-duplex on the other side is also a misconfiguration, and can result in a duplex mismatch. If the switch port receives a lot of late collisions, this usually indicates a duplex mismatch problem and can result in the port being placed in an errdisable status. The half duplex side only expects packets at certain times, not at any time, and therefore counts packets received at the wrong time as collisions. There are other causes for late collisions besides duplex mismatch but this is one of the most common reasons. Always set both sides of the connection to auto-negotiate speed/duplex, or set the speed/duplex manually on both sides.
For CatOS, use the show port status [mod/port] command to display the speed and duplex status as well as other information. Use the set port speed and set port duplex commands to hardcode both sides to 10 or 100 and half or full as necessary.
Switch> (enable) sh port status 3/1Port Name Status Vlan Duplex Speed Type----- -------------------- ---------- ---------- ------ ----- ------------3/1 connected 1 a-full a-100 10/100BaseTXSwitch> (enable)
For Cisco IOS, use the show interfaces card-type {slot/port} status command to display speed and duplex settings as well as other information. Use the speed and duplex commands from interface configuration mode to hardcode both sides to 10 or 100 and half or full as necessary.
Router#sh interfaces fas 6/1 statusPort Name Status Vlan Duplex Speed TypeFa6/1 connected 1 a-full a-100 10/100BaseTX
If you use the show interfaces command without the status option, you see a setting for speed and duplex, but you do not know whether this speed and duplex was achieved through autonegotiation or not.
Router#sh int fas 6/1FastEthernet6/1 is up, line protocol is up (connected)Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,Encapsulation ARPA, loopback not setreliability 255/255, txload 1/255, rxload 1/255Full-duplex, 100Mb/s!--- Full-duplex and 100Mbps does not tell you whether autoneg was used to achieve this.!--- Use the sh interfaces fas 6/1 status command to display this.
My spouse and I absolutely love your blog and find many of your post's to be exactly I'm looking for. Would you offer guest writers to write content for yourself? I wouldn't mind composing a post or elaborating on most of the subjects you write regarding here. Again, awesome weblog!
ReplyDelete