It is currently Mon Dec 11, 2017 2:08 am


All times are UTC




Post new topic Reply to topic  [ 26 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: [PATCH] Fix for packet looping on platforms with Gt96k FE
PostPosted: Sat Feb 22, 2014 10:17 pm 
Offline

Joined: Sat Feb 22, 2014 9:47 pm
Posts: 11
Greetings,

If running emulated chassis such as 2691 that uses the Gt96k FastEthernet controller, activating PIM on the FastEthernet interface will cause IOS to put it into the promiscuous mode. This is, broadly speaking, necessary because a router must be prepared to handle multicast packets in the entire D-class multicast range.

The Gt96k controller emulation in common/dev_gt.c is currently written to accept all frames in the promisc mode. However, it turns out this is not what we really want for a router: with an FastEthernet in promisc mode, the router will process any received frame including unicast frames, and being a router, it will try to route packets contained in the received frames. On emulated LAN segments with three or more routers, sending an unicast packet while simultaneously having PIM activated on the routers' interfaces will result in every router receiving the packet and routing it, resulting into packet multiplication, looping and ultimate network meltdown. It was essentially impossible to use the C2691 emulated platform in Dynamips/GNS3 to experiment with multicast routing if using shared Ethernet segments.

I have modified the dev_gt.c to have the Gt96k's promisc mode apply only to multicast frames. For a router, I do not see any reason to process all unicast frames even if in promisc mode. It would be nice to know if this is exactly what true Cisco 2691 platforms do - perhaps the behavior of Gt96k is slightly modified in them - but for now, this change appears to fix the issue with looping packets whlle not limiting unicast connectivity.

I would like to ask you to verify the attached patch, and if deemed acceptable, integrate it into dynamips codebase. Thank you!

Best regards,
Peter




Attachments:
File comment: Patch correcting dev_gt.c to make promisc mode apply only to multicast frames
dynamips.patch [1.21 KiB]
Downloaded 150 times
Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Sun Feb 23, 2014 12:56 pm 
Offline

Joined: Wed May 22, 2013 7:48 am
Posts: 93
Location: Portugal
Can you provide a simple project that demonstrates this? (instructions appreciated)

Filtering packets in the hardware is probably the wrong thing to do since in promiscuous mode the OS is supposed to decide what to do. (there are probably settings that control what is done)
However, in doing so the OS might be relying on unimplemented features from the network cards. More investigation is needed.

_________________
Dynamips maintainer and sporadic developer
https://github.com/GNS3/dynamips


Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Sun Feb 23, 2014 2:28 pm 
Offline

Joined: Sat Feb 22, 2014 9:47 pm
Posts: 11
Flavio,

Perhaps we are stumbling across two different issues. I have tested the patch to perfectly solve my issues. Let me reiterate - perhaps we'll find a place where we start diverging. To the testing environment: I am using Dynamips 0.2.11 from the GNS3 project, plus Dynagen, but I am not running the GNS3. The IOS I am using is c2691-advipservicesk9-mz.124-15.T13.bin.

Assume a topology of three routers, R1, R2, and R3, connected with a common "backbone" segment. The dynagen config would be similar to this:

Code:
        [[ROUTER R1]]
                model = 2691
                f0/0 = LAN B1
       
        [[ROUTER R2]]
                model = 2691
                f0/0 = LAN B1

        [[ROUTER R3]]
                model = 2691
                f0/0 = LAN B1


Each router is configured very plainly at the beginning (substitute x with the router number):

Code:
hostname Rx
!
interface FastEthernet0/0
ip address 10.0.0.x 255.255.255.0
no shutdown


At this point, the connectivity between the three routers is working flawlessly:

Code:
R1(config-if)# do ping 10.0.0.2 rep 100 size 36

Type escape sequence to abort.
Sending 100, 36-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/18/24 ms
R1(config-if)# do ping 10.0.0.2 rep 100 size 1500

Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/19/28 ms
R1(config-if)# do ping 10.0.0.3 rep 100 size 36

Type escape sequence to abort.
Sending 100, 36-byte ICMP Echos to 10.0.0.3, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/19/24 ms
R1(config-if)# do ping 10.0.0.3 rep 100 size 1500

Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 10.0.0.3, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/19/24 ms


On R1 (R2 and R3 would produce the same result), the show controllers fa0/0 informs us that the card is currently in a non-promisc mode:

Code:
R1(config-if)# do show controllers fa0/0
Interface FastEthernet0/0
Hardware is GT96K FE ADDR: 65F8ED20, FASTSEND: 6071A02C, MCI_INDEX: 0
DIST ROUTE ENABLED: 0Route Cache Flag: 11
GPIO 2 CONF= 0 GPIO 2 IO= 0  CIU arbit = 80000000
PHY add register = 0x0  PHY data register = 0x8000000
Port Conf Reg= 0x80 ENABLE HT8K HMOD0
Port Conf Ex Reg= 0x4CD00


The Port Conf Reg is currently set to 0x80; if promisc mode was active then the 0x1 flag would be set, resulting into the PCR value of 0x81.

Next, the PIM-DM is activated on all three routers:

Code:
ip multicast-routing
!
interface FastEthernet0/0
ip pim dense-mode


The show controllers fa0/0 on R1 changes a little, indicating that the card has been put to promisc mode:

Code:
R1(config-if)# do show controllers fa0/0
Interface FastEthernet0/0
Hardware is GT96K FE ADDR: 65F8ED20, FASTSEND: 6071A02C, MCI_INDEX: 0
DIST ROUTE ENABLED: 0Route Cache Flag: 11
GPIO 2 CONF= 0 GPIO 2 IO= 0  CIU arbit = 80000000
PHY add register = 0x0  PHY data register = 0x8000000
Port Conf Reg= 0x81 PMODE ENABLE HT8K HMOD0
Port Conf Ex Reg= 0x4CD00


Now, a single ping from R1 to R2 will start wreaking havoc - this ping will be also received by R3 and hairpinned back to R2, along with an ICMP redirect message to R1 to use a direct path. However, the hairpinned ping will be also received by R1 which will forward it again to R2 (and unknowingly to R3), and the ICMP redirect will be received by R2 which will forward it to R1 (and unknowingly to R3), etc. Just before the ping, I am starting a packet capture on R1 fa0/0 interface to record the sequence of events.

Code:
R1(config-if)#do ping 10.0.0.2 rep 1

Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 16/16/16 ms
R1(config-if)#


The attached PCAP file (vanilla.cap) records the communication as seen on R1 fa0/0 exactly after this single ping. Trying to run a continuous ping from R1 even minutes later produces the well-known symptom of packets progressively getting delayed:

Code:
R1(config-if)#do ping 10.0.0.2 rep 100         

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 424/517/732 ms
R1(config-if)#


If there are more continuous pings running, the packets will start getting lost (in this scenario, I am pinging R3 from R2, R2 from R1 and later R1 from R3):

Code:
R1(config-if)#do ping 10.0.0.2 rep 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!.!.......................
Success rate is 76 percent (76/100), round-trip min/avg/max = 148/931/1932 ms
R1(config-if)#


These packet losses are most probably caused by tail drops on the interface's egress queues:

Code:
R1#show int fa0/0
FastEthernet0/0 is up, line protocol is up
  Hardware is Gt96k FE, address is c000.10f6.0000 (bia c000.10f6.0000)
  Internet address is 10.0.0.1/24
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 5/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex, 10Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1507
  Queueing strategy: fifo
  Output queue: 38/40 (size/max)


Running the very same topology and configuration in the patched Dynamips produces no ill results (again, before starting pinging R2 from R1, I am capturing R1 fa0/0 interface to monitor the communication):

Code:
R1#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 12/18/24 ms


The first packet got lost for the usual ARP reasons (I am performing this test right after reloading the entire configured topology). The second PCAP file (patched.cap) shows the usual communication on the segment, without the packets being looped, even though PIM-DM is running.

Quote:
Filtering packets in the hardware is probably the wrong thing to do since in promiscuous mode the OS is supposed to decide what to do.


I agree; however, more advanced NIC controllers have a separate promisc mode for unicast and for multicast frames, so on these controllers, activating the multicast routing requires only the multicast promisc mode to become active. My patch mimics that behavior.

Apart from that, I believe that if the NIC itself does not pass at least some kind of additional hint to the IOS as to why the frame was accepted, the IOS is not capable of distinguishing between frames received because we wanted them to be received anyway (our own MAC, vMAC for FHRP protocols, etc.), and frames received because the promisc mode is on. Now the only possible hint I could find is the M-bit. Following the datasheet file DSA-655899.pdf Figure 46, the M-bit would be set for frames received because their destination was found in the hash table, and would be be cleared for frames received because of promisc mode and not present in the hash table. The dev_gt.c appears to follow this chart precisely and I conclude that the M-bit setting is done correctly, yet the IOS does not appear to take it into account. So either the M-bit is not passed to the IOS in Dynamips, or the IOS simply ignores it. So far, my only solution was to limit the promisc mode to multicast frames.

Would any of this be helpful? I apologize for the overly long post but I wanted to remove as much ambiguosity as possible.

Best regards,
Peter


Attachments:
File comment: vanilla.cap: Snapshot of the looping traffic on the segment after R2 is pinged with a single packet from R1 (vanilla Dynamips)

patched.cap: Snapshot of the (normal) traffic after pinging R2 from R1 (patched Dynamips)

captures.zip [6.54 KiB]
Downloaded 144 times
Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Sun Feb 23, 2014 8:37 pm 
Offline

Joined: Sat Feb 22, 2014 9:47 pm
Posts: 11
Flavio,

Let's hold my patch for a moment. If bridging is configured on a FastEthernet interface, the interface will also be put into promisc mode. In this case, accepting unicast frames in promisc mode is necessary. My patch would cause them to be ignored which is obviously wrong.

I will try to investigate more and will keep you updated. Sorry for the confusion so far.

Best regards,
Peter


Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Sun Feb 23, 2014 11:23 pm 
Offline

Joined: Wed May 22, 2013 7:48 am
Posts: 93
Location: Portugal
Loved the detailed instructions, thanks! =)
Forgot how useful those show commands were... I don't use cisco routers or dynamips, just happened to need GNS3 for a class about protocols last year. =~~

Unfortunately, I wasn't able to reproduce the ping timeouts.
Tried 1000 pings with the uncompressed c2691-adventerprisek9_sna-mz.124-13b image I had already setup.
Tried again with c2691-advipservicesk9-mz.124-15.T6.bin, the closest version to yours I could find.

However, I did notice my dual-core cpu going from ~35% to >90% when the pings started so maybe the problem you experienced is because your cpu is spiking even more?

_________________
Dynamips maintainer and sporadic developer
https://github.com/GNS3/dynamips


Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Wed Feb 26, 2014 6:47 am 
Offline

Joined: Sat Feb 22, 2014 9:47 pm
Posts: 11
Hi Flavio,

It's been a pleasure. So after a couple of days, this is my second - and hopefully now correct - attempt at the solution.

Consider the vanilla Dynamips 0.2.11 with the topology as described in my previous post. R1 is configured with 10.0.0.1, R2 is configured with 10.0.0.2, no multicast routing activated yet. What caught my eye was this "Software MAC address filter" in show controllers fa0/0:

Code:
R1# show controll fa0/0 | b filter
Software MAC address filter(hash:length/addr/mask/hits):
  0x00:  0  ffff.ffff.ffff  0000.0000.0000         0
  0x54:  0  c000.548d.0000  0000.0000.0000         0
  0xC0:  0  0100.0ccc.cccc  0000.0000.0000         0


It seemed to correspond to the hash table downloaded to the Gt96k adapter but there were many things unclear: first, why is this called the software MAC address filter if the hash table is implemented in hardware; second, what is the mask field since the hash table in Gt96k does not store any masks; third, where does the hit field come from since Gt96k does not maintain a hash table hit counter. I started to realize that this must be an additional, truly IOS-based (thus software) MAC address filter that may be additionaly used to filter incoming frames based on more complex criteria than the Gt96k is able to process.

An interesting fact is that if the card is in non-promisc mode, this filter is skipped - you can see that the hit counters do not increment even though frames are received and processed. This is logical - in non-promisc mode, you accept frames destined only to very particular destinations (broadcast, your own MAC address, vMAC for FHRP protocols and that's about it).

When multicast routing and PIM is activated and the interface is put into promisc mode by IOS, things change:

Code:
R1# show controll fa0/0 | b filter
Software MAC address filter(hash:length/addr/mask/hits):
  0x00:  0  ffff.ffff.ffff  0000.0000.0000         0
  0x53:  0  0100.5e00.000d  0000.0000.0000         0
  0x54:  0  c000.548d.0000  0000.0000.0000         0
  0x76:  0  0100.5e00.0128  0000.0000.0000         0
  0xC0:  0  0100.0ccc.cccc  0000.0000.0000         0
  0x00:  0  0100.5e7f.ffff  0000.007f.ffff         0


One of the very important lines that has been added is the last one that obviously matched the entire MAC range used for IPv4 multicast (the mask here is obviously bitwise-ORed with the destination MAC address to produce the matched address). When, however, pinging 239.1.2.3 from R2, notice that the hit counter on this last entry does not increment:

Code:
R1# show controll fa0/0 | b filter
Software MAC address filter(hash:length/addr/mask/hits):
  0x00:  0  ffff.ffff.ffff  0000.0000.0000         0
  0x53:  0  0100.5e00.000d  0000.0000.0000         0
  0x54:  0  c000.548d.0000  0000.0000.0000         0
  0x76:  0  0100.5e00.0128  0000.0000.0000         0
  0xC0:  0  0100.0ccc.cccc  0000.0000.0000         3
  0x00:  0  0100.5e7f.ffff  0000.007f.ffff         0


This was strange to me - why would IOS add an entry to this software MAC filter if it does not get any hits? Even stranger, when pinging R1 (unicast) from R2, the entry corresponding to R1's MAC address is incremented (the one labeled with 0x54):

Code:
R1# show controll fa0/0 | b filter
Software MAC address filter(hash:length/addr/mask/hits):
  0x00:  0  ffff.ffff.ffff  0000.0000.0000         0
  0x53:  0  0100.5e00.000d  0000.0000.0000         0
  0x54:  0  c000.548d.0000  0000.0000.0000         5
  0x76:  0  0100.5e00.0128  0000.0000.0000         0
  0xC0:  0  0100.0ccc.cccc  0000.0000.0000         5
  0x00:  0  0100.5e7f.ffff  0000.007f.ffff         0


Why would a MAC address that has an entry already present in Gt96k hash table get hits in the software MAC filter?

I started feeling that somehow, the logic of processing the frames is reversed: those frames whose destination was already found in Gt96k hash table were additionally subjected to this software MAC filter although they should have been accepted unconditionally, and vice versa, frames whose destination was not found in the Gt96k hash table were accepted without being matched towards this software MAC filter. This was quite the opposite to what I would expect.

So I focused on the M-bit in the Rx descriptor. The common/dev_gt.c sources contain a comment that the meaning of this bit is unclear as the data sheet calls it a "Match" or a "Miss" bit in different places. Currently, the common/dev_gt.c treats it as the Match bit, i.e. if the destination MAC is found in the Gt96k hash table, the M bit is set. However, considering the behavior demonstrated with the software MAC filter above, it occurred to me that IOS treats this bit as the Miss bit, and suddenly all started making sense. Because the M bit was erroneously set on received frames that were found in Gt96k hash table, IOS considered them a Miss and subjected them to the software MAC filter - but because the router's own MAC address is in the software filter as well, unicast frames for this router were accepted. All other frames - unicast frames for different destinations, multicast frames, etc. - were not found in the Gt96k hash table, so the M bit was cleared, causing IOS to consider them a Match and these frames were accepted unconditionally. Hence the unicast frame/packet looping. Simply put, the meaning of the M bit in the data sheet and in the common/dev_gt.c is reversed. It is not a Match bit - it is a Miss bit.

After modifying the common/dev_gt.c to set the M bit on frames not found in the Gt96k hash table, everything started working properly. No unicast frame/packet looping anymore; the multicast and the software bridging work correctly. The following output is from R1 with configured multicast routing, running in patched dynamips:

Code:
R1# show controll fa0/0 | b filter
Software MAC address filter(hash:length/addr/mask/hits):
  0x00:  0  ffff.ffff.ffff  0000.0000.0000         0
  0x53:  0  0100.5e00.000d  0000.0000.0000         0
  0x55:  0  c000.552d.0000  0000.0000.0000         0
  0x76:  0  0100.5e00.0128  0000.0000.0000         0
  0xC0:  0  0100.0ccc.cccc  0000.0000.0000         0
  0x00:  0  0100.5e7f.ffff  0000.007f.ffff         3


The entries already present in the Gt96k hash table do not get a hit. The "wildcard" multicast entry is incremented after pinging 239.1.2.3 from a different router.

I am attaching a new patch - and once again, I would like to ask you to have a look and if considered appropriate, integrate it into dynamips sources.

Quote:
Unfortunately, I wasn't able to reproduce the ping timeouts.


It requires a bit of effort to achieve that - if you are reproducing my topology, try pinging simultaneously R2 from R1, R3 from R2, R1 from R3 (make each router send and receive a high number of pings). Obviously, depending on the power of your CPU, dynamips may be able to keep up with the packet flood to a certain amount. Nevertheless, I believe we both agree that the looping of packets we've seen is an ill effect, and the provided patch corrects that. You have said yourself your CPU went up to over 90% just with these three routers - now imagine a server running 64 such routers instead of 3 (which is what happened to me during the multicast seminar ;) ).

Thank you!

Best regards,
Peter


Attachments:
File comment: Corrected patch for Gt96k.
dynamips-gt.patch [2.16 KiB]
Downloaded 194 times
Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Wed Feb 26, 2014 11:52 am 
Offline

Joined: Wed May 22, 2013 7:48 am
Posts: 93
Location: Portugal
Awesome!!! Just look at those lightning fast pings (1/1/8 ms) and hardly any cpu load. =D

The patch is very good, to the point and with documentation. I'll commit it. ^^
What do you want me to put as your author string?
Ex: my commits have "Flávio J. Saraiva <[email protected]>"

_________________
Dynamips maintainer and sporadic developer
https://github.com/GNS3/dynamips


Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Wed Feb 26, 2014 3:35 pm 
Offline

Joined: Sat Feb 22, 2014 9:47 pm
Posts: 11
Hi Flavio,

Well, I am not sure if changing a single line in the source code is worth mentioning me as an author - but I would be extremely honored.

Let's use "Peter Palúch <[email protected]>"

Thank you!

Best regards,
Peter


Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Wed Feb 26, 2014 4:34 pm 
Offline
Site Admin

Joined: Sat Oct 11, 2008 1:41 pm
Posts: 2668
Location: Canada
Indeed impressive work! :) Thanks a lot Peter.

_________________
Jeremy, GNS3 Programmer & Benevolent Dictator for Life.


Top
 Profile  
 
 Post subject: Re: [PATCH] Fix for packet looping on platforms with Gt96k F
PostPosted: Wed Feb 26, 2014 9:20 pm 
Offline

Joined: Wed May 22, 2013 7:48 am
Posts: 93
Location: Portugal
paluchpeter wrote:
Well, I am not sure if changing a single line in the source code is worth mentioning me as an author - but I would be extremely honored.
Well, that single line of code has impact. ;D

Fixed in this commit: 16f7c39.
Will be a part of dynamips 0.2.12.

Thanks. ^^
Feel free to report other issues, even if you don't have a patch for them.



_________________
Dynamips maintainer and sporadic developer
https://github.com/GNS3/dynamips


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 26 posts ]  Go to page 1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group

phpBB SEO