* Socket Buffers and Memory Managment @ 2007-07-17 17:20 vinay ravuri 2007-07-17 19:41 ` Stephen Hemminger 0 siblings, 1 reply; 10+ messages in thread From: vinay ravuri @ 2007-07-17 17:20 UTC (permalink / raw) To: netdev Hi, I am fairly new to linux socket buffers and have the following questions! I am working with a custom ethernet MAC that does not allow me to specify a particular memory location for the h/w to DMA the packet into (Rx side). Instead, it has a pool of fixed size buffers with some h/w specific headers around each buffer that are managed by h/w and will pick a free buffer and DMA the packet. It appears dev_alloc_skb() actually allocates the physical memory and doesn't allow the user to specify the skb.data to something specific to what I want which is a problem for me. First is my assumption correct that I am cannot pick an arbitrary skb.data location in struct sk_buff? I want to avoid copying the dma'ed data into a new socket buffer as it is expense. Is there any ways around this problem? Also, if the h/w gives me a single packet in multiple locations (i.e. non-contiguous chunks of memory), can socket buffers handle chains of buffers? I am looking for a facility like mbuf's in netbsd where one can chain multiple buffers together to make construct a single packet. Please e-mail me responses to vinay_nyc@yahoo•com Thanks, Vinay ____________________________________________________________________________________ Bored stiff? Loosen up... Download and play hundreds of games for free on Yahoo! Games. http://games.yahoo.com/games/front ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-17 17:20 Socket Buffers and Memory Managment vinay ravuri @ 2007-07-17 19:41 ` Stephen Hemminger 2007-07-17 19:44 ` David Miller ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Stephen Hemminger @ 2007-07-17 19:41 UTC (permalink / raw) To: vinay ravuri; +Cc: netdev On Tue, 17 Jul 2007 10:20:58 -0700 (PDT) vinay ravuri <vinaynyc@yahoo•com> wrote: > Hi, > > I am fairly new to linux socket buffers and have the > following questions! > > I am working with a custom ethernet MAC that does not > allow me to specify a particular memory location for > the h/w to DMA the packet into (Rx side). Instead, it > has a pool of fixed size buffers with some h/w > specific headers around each buffer that are managed > by h/w and will pick a free buffer and DMA the packet. Sounds like sucky hardware... You need to copy to a newly allocated skb, see 8139too.c > It appears dev_alloc_skb() actually allocates the > physical memory and doesn't allow the user to specify > the skb.data to something specific to what I want > which is a problem for me. First is my assumption > correct that I am cannot pick an arbitrary skb.data > location in struct sk_buff? I want to avoid copying > the dma'ed data into a new socket buffer as it is > expense. Is there any ways around this problem? You could play tricks with skb frags but it would be fragile and not worth the trouble. The problem is that the receive skb can stay in the system for a really long time (until the application reads the data) so your fixed size buffer pool in hardware would get exhausted. > Also, if the h/w gives me a single packet in multiple > locations (i.e. non-contiguous chunks of memory), can > socket buffers handle chains of buffers? I am looking > for a facility like mbuf's in netbsd where one can > chain multiple buffers together to make construct a > single packet. Yes, skb frag list could be used for that but you don't want to do that. See above. Copy the data into an new skb and reserve any necessary bytes so IP header is aligned. I.e. if using ethernet header (14 bytes), do skb_reserve(skb, 2) before copying the data. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-17 19:41 ` Stephen Hemminger @ 2007-07-17 19:44 ` David Miller 2007-07-18 17:13 ` Roy Pledge 2007-07-19 6:51 ` vinay ravuri 2 siblings, 0 replies; 10+ messages in thread From: David Miller @ 2007-07-17 19:44 UTC (permalink / raw) To: shemminger; +Cc: vinaynyc, netdev From: Stephen Hemminger <shemminger@linux-foundation•org> Date: Tue, 17 Jul 2007 20:41:29 +0100 > Sounds like sucky hardware... Although expect more of the same in the future, not less. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-17 19:41 ` Stephen Hemminger 2007-07-17 19:44 ` David Miller @ 2007-07-18 17:13 ` Roy Pledge 2007-07-18 21:22 ` Stephen Hemminger 2007-07-19 6:51 ` vinay ravuri 2 siblings, 1 reply; 10+ messages in thread From: Roy Pledge @ 2007-07-18 17:13 UTC (permalink / raw) To: Stephen Hemminger; +Cc: vinay ravuri, netdev Stephen Hemminger wrote: > > You could play tricks with skb frags but it would be fragile > and not worth the trouble. The problem is that the receive > skb can stay in the system for a really long time (until the application > reads the data) so your fixed size buffer pool in hardware > would get exhausted. > Perhaps you could elaborate on why this is considered fragile? It seems to me that as long as proper page management is performed, this mechanism should be stable for processing non-contiguous receive buffers. I agree that replenishment needs to be addressed, but I see that as an independent issue from using frag lists on receive. >> Also, if the h/w gives me a single packet in multiple >> locations (i.e. non-contiguous chunks of memory), can >> socket buffers handle chains of buffers? I am looking >> for a facility like mbuf's in netbsd where one can >> chain multiple buffers together to make construct a >> single packet. > > Yes, skb frag list could be used for that but you don't > want to do that. See above. Copy the data into an new > skb and reserve any necessary bytes so IP header is > aligned. I.e. if using ethernet header (14 bytes), do > skb_reserve(skb, 2) before copying the data. But isn't avoiding the copy the point of all this? With some hardware models using skbs as receive buffers is inconvenient and can be wasteful if jumbo frames are required. Wouldn't a mechanism where it is possible to create (or populate) an skb from data in a pre existing buffer without copying be more general and address more hardware models? It seems to me that more and more hardware is supporting scatter/gather output for received data, but the page* model in the frag list can be restrictive in how data is placed before being passed into the stack. Thanks Roy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-18 17:13 ` Roy Pledge @ 2007-07-18 21:22 ` Stephen Hemminger 0 siblings, 0 replies; 10+ messages in thread From: Stephen Hemminger @ 2007-07-18 21:22 UTC (permalink / raw) To: Roy Pledge; +Cc: vinay ravuri, netdev On Wed, 18 Jul 2007 13:13:21 -0400 Roy Pledge <Roy.Pledge@freescale•com> wrote: > Stephen Hemminger wrote: > > > > You could play tricks with skb frags but it would be fragile > > and not worth the trouble. The problem is that the receive > > skb can stay in the system for a really long time (until the application > > reads the data) so your fixed size buffer pool in hardware > > would get exhausted. > > > Perhaps you could elaborate on why this is considered fragile? It seems to me > that as long as proper page management is performed, this mechanism should be > stable for processing non-contiguous receive buffers. I agree that > replenishment needs to be addressed, but I see that as an independent issue from > using frag lists on receive. Imagine you had a 4MB receive area. If you used fraglist to point to the data area, you would still be at risk of receive lock up when you have all the receive area used up in skb's waiting in the receive queue of the sockets. You might get it to work if the receive are was very large (ie much larger than tcp_rmem), but at that point you might as well as be able to access all of system memory. Using a DMA I/O engine would help, but you probably won't have one available. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-17 19:41 ` Stephen Hemminger 2007-07-17 19:44 ` David Miller 2007-07-18 17:13 ` Roy Pledge @ 2007-07-19 6:51 ` vinay ravuri 2007-07-19 7:04 ` pradeep singh ` (2 more replies) 2 siblings, 3 replies; 10+ messages in thread From: vinay ravuri @ 2007-07-19 6:51 UTC (permalink / raw) To: Stephen Hemminger; +Cc: netdev How about the following approach: I allocate an skb of 0 bytes and replace data element of skb struct (i.e. skb.data = addr_given_by_hw) when the h/w interrupts me with a packet. I register for a destructor for this skb and when the kernel is ready to free the skb, I make sure that my free is invoked - Ofcourse this is assuming that their is a facility in linux socket buffers to be able to do destructors. Is this approach a viable, if so, are any gottcha's? -Vinay --- Stephen Hemminger <shemminger@linux-foundation•org> wrote: > On Tue, 17 Jul 2007 10:20:58 -0700 (PDT) > vinay ravuri <vinaynyc@yahoo•com> wrote: > > > Hi, > > > > I am fairly new to linux socket buffers and have > the > > following questions! > > > > I am working with a custom ethernet MAC that does > not > > allow me to specify a particular memory location > for > > the h/w to DMA the packet into (Rx side). > Instead, it > > has a pool of fixed size buffers with some h/w > > specific headers around each buffer that are > managed > > by h/w and will pick a free buffer and DMA the > packet. > > Sounds like sucky hardware... > You need to copy to a newly allocated skb, see > 8139too.c > > > It appears dev_alloc_skb() actually allocates the > > physical memory and doesn't allow the user to > specify > > the skb.data to something specific to what I want > > which is a problem for me. First is my assumption > > correct that I am cannot pick an arbitrary > skb.data > > location in struct sk_buff? I want to avoid > copying > > the dma'ed data into a new socket buffer as it is > > expense. Is there any ways around this problem? > > You could play tricks with skb frags but it would be > fragile > and not worth the trouble. The problem is that the > receive > skb can stay in the system for a really long time > (until the application > reads the data) so your fixed size buffer pool in > hardware > would get exhausted. > > > Also, if the h/w gives me a single packet in > multiple > > locations (i.e. non-contiguous chunks of memory), > can > > socket buffers handle chains of buffers? I am > looking > > for a facility like mbuf's in netbsd where one can > > chain multiple buffers together to make construct > a > > single packet. > > Yes, skb frag list could be used for that but you > don't > want to do that. See above. Copy the data into an > new > skb and reserve any necessary bytes so IP header is > aligned. I.e. if using ethernet header (14 bytes), > do > skb_reserve(skb, 2) before copying the data. > ____________________________________________________________________________________ Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase. http://farechase.yahoo.com/ ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-19 6:51 ` vinay ravuri @ 2007-07-19 7:04 ` pradeep singh 2007-07-19 8:10 ` Stephen Hemminger 2007-07-19 9:08 ` Evgeniy Polyakov 2 siblings, 0 replies; 10+ messages in thread From: pradeep singh @ 2007-07-19 7:04 UTC (permalink / raw) To: vinay ravuri; +Cc: Stephen Hemminger, netdev On 7/19/07, vinay ravuri <vinaynyc@yahoo•com> wrote: > How about the following approach: > > I allocate an skb of 0 bytes and replace data element > of skb struct (i.e. skb.data = addr_given_by_hw) when > the h/w interrupts me with a packet. I register for a > destructor for this skb and when the kernel is ready > to free the skb, I make sure that my free is invoked - > Ofcourse this is assuming that their is a facility in > linux socket buffers to be able to do destructors. Is > this approach a viable, if so, are any gottcha's? I wonder if passing a zero size will work correctly with alloc_skb/__alloc_skb as you still have just skb frag list to work with in this case. And as Stephen pointed out, if i understand correctly that would be still a problem. yes/no? thanks --pradeep > > -Vinay > > > --- Stephen Hemminger > <shemminger@linux-foundation•org> wrote: > > > On Tue, 17 Jul 2007 10:20:58 -0700 (PDT) > > vinay ravuri <vinaynyc@yahoo•com> wrote: > > > > > Hi, > > > > > > I am fairly new to linux socket buffers and have > > the > > > following questions! > > > > > > I am working with a custom ethernet MAC that does > > not > > > allow me to specify a particular memory location > > for > > > the h/w to DMA the packet into (Rx side). > > Instead, it > > > has a pool of fixed size buffers with some h/w > > > specific headers around each buffer that are > > managed > > > by h/w and will pick a free buffer and DMA the > > packet. > > > > Sounds like sucky hardware... > > You need to copy to a newly allocated skb, see > > 8139too.c > > > > > It appears dev_alloc_skb() actually allocates the > > > physical memory and doesn't allow the user to > > specify > > > the skb.data to something specific to what I want > > > which is a problem for me. First is my assumption > > > correct that I am cannot pick an arbitrary > > skb.data > > > location in struct sk_buff? I want to avoid > > copying > > > the dma'ed data into a new socket buffer as it is > > > expense. Is there any ways around this problem? > > > > You could play tricks with skb frags but it would be > > fragile > > and not worth the trouble. The problem is that the > > receive > > skb can stay in the system for a really long time > > (until the application > > reads the data) so your fixed size buffer pool in > > hardware > > would get exhausted. > > > > > Also, if the h/w gives me a single packet in > > multiple > > > locations (i.e. non-contiguous chunks of memory), > > can > > > socket buffers handle chains of buffers? I am > > looking > > > for a facility like mbuf's in netbsd where one can > > > chain multiple buffers together to make construct > > a > > > single packet. > > > > Yes, skb frag list could be used for that but you > > don't > > want to do that. See above. Copy the data into an > > new > > skb and reserve any necessary bytes so IP header is > > aligned. I.e. if using ethernet header (14 bytes), > > do > > skb_reserve(skb, 2) before copying the data. > > > > > > > ____________________________________________________________________________________ > Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase. > http://farechase.yahoo.com/ > - > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger•kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Pradeep ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-19 6:51 ` vinay ravuri 2007-07-19 7:04 ` pradeep singh @ 2007-07-19 8:10 ` Stephen Hemminger 2007-07-19 9:08 ` Evgeniy Polyakov 2 siblings, 0 replies; 10+ messages in thread From: Stephen Hemminger @ 2007-07-19 8:10 UTC (permalink / raw) To: vinay ravuri; +Cc: netdev On Wed, 18 Jul 2007 23:51:03 -0700 (PDT) vinay ravuri <vinaynyc@yahoo•com> wrote: > How about the following approach: > > I allocate an skb of 0 bytes and replace data element > of skb struct (i.e. skb.data = addr_given_by_hw) when > the h/w interrupts me with a packet. I register for a > destructor for this skb and when the kernel is ready > to free the skb, I make sure that my free is invoked - > Ofcourse this is assuming that their is a facility in > linux socket buffers to be able to do destructors. Is > this approach a viable, if so, are any gottcha's? > > -Vinay You need to use frag list for that since upper layers expect to be able to use that data area for normal use, ie bridging/routing, etc. Also access to data area would be non-cached so you want to make sure it is only accessed once. But how will you handle a slow receiver where all the skb's end up staying queued. Won't you exhaust your packet memory. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-19 6:51 ` vinay ravuri 2007-07-19 7:04 ` pradeep singh 2007-07-19 8:10 ` Stephen Hemminger @ 2007-07-19 9:08 ` Evgeniy Polyakov 2007-07-20 11:50 ` Andi Kleen 2 siblings, 1 reply; 10+ messages in thread From: Evgeniy Polyakov @ 2007-07-19 9:08 UTC (permalink / raw) To: vinay ravuri; +Cc: Stephen Hemminger, netdev Hi. On Wed, Jul 18, 2007 at 11:51:03PM -0700, vinay ravuri (vinaynyc@yahoo•com) wrote: > How about the following approach: > > I allocate an skb of 0 bytes and replace data element > of skb struct (i.e. skb.data = addr_given_by_hw) when > the h/w interrupts me with a packet. I register for a > destructor for this skb and when the kernel is ready > to free the skb, I make sure that my free is invoked - > Ofcourse this is assuming that their is a facility in > linux socket buffers to be able to do destructors. Is > this approach a viable, if so, are any gottcha's? It will not work, since kfree_skb() eventually tries to free skb->head into kmem cache, so you will need to hack kfree_skb() not to try to release that data at all. Likely the best zero-copy approach in your case is to use frag_list, but be ready that your netwrok will regularily stall - when hardware buffers are all in use and not yet freed, you will not be able to send/receive new packets, although amount of memory will allow that. > -Vinay -- Evgeniy Polyakov ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Socket Buffers and Memory Managment 2007-07-19 9:08 ` Evgeniy Polyakov @ 2007-07-20 11:50 ` Andi Kleen 0 siblings, 0 replies; 10+ messages in thread From: Andi Kleen @ 2007-07-20 11:50 UTC (permalink / raw) To: Evgeniy Polyakov; +Cc: vinay ravuri, Stephen Hemminger, netdev Evgeniy Polyakov <johnpol@2ka•mipt.ru> writes: > Hi. > > On Wed, Jul 18, 2007 at 11:51:03PM -0700, vinay ravuri (vinaynyc@yahoo•com) wrote: > > How about the following approach: > > > > I allocate an skb of 0 bytes and replace data element > > of skb struct (i.e. skb.data = addr_given_by_hw) when > > the h/w interrupts me with a packet. I register for a > > destructor for this skb and when the kernel is ready > > to free the skb, I make sure that my free is invoked - > > Ofcourse this is assuming that their is a facility in > > linux socket buffers to be able to do destructors. Is > > this approach a viable, if so, are any gottcha's? > > It will not work, since kfree_skb() eventually tries to free skb->head > into kmem cache, And in addition if the skbuff is ever passed towards the socket layer the destructor will be overwritten -Andi ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2007-07-20 10:55 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2007-07-17 17:20 Socket Buffers and Memory Managment vinay ravuri 2007-07-17 19:41 ` Stephen Hemminger 2007-07-17 19:44 ` David Miller 2007-07-18 17:13 ` Roy Pledge 2007-07-18 21:22 ` Stephen Hemminger 2007-07-19 6:51 ` vinay ravuri 2007-07-19 7:04 ` pradeep singh 2007-07-19 8:10 ` Stephen Hemminger 2007-07-19 9:08 ` Evgeniy Polyakov 2007-07-20 11:50 ` Andi Kleen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox