Subject: Re: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit Proposed Topics

From: Matt Mackall <[email-masked]>
Date: Sat, 26 Mar 2005 21:48:31 -0800

I'm cc:ing this to netdev, where this discussion really ought to be.
There's a separate networking summit and I suspect most of the
networking heavies aren't reading ksummit-discuss or open-iscsi.
It's getting rather far afield for ksummit-discuss so people should
trim that from follow-ups.

On Sun, Mar 27, 2005 at 05:51:49AM +0200, Andrea Arcangeli wrote:
> On Thu, Mar 24, 2005 at 07:43:41PM -0800, Matt Mackall wrote:
> > There may be network multipath. But I think we can have a single
> > socket mempool per logical device and a single skbuff mempool shared
> > among those sockets.
> 
> If we'll have to reserve more than 1 packet per each socket context,
> then the mempool probably can't be shared.

I believe the mempool can be shared among all sockets that represent
the same storage device. Packets out any socket represent progress.

> I wonder if somebody has ever reproduced deadlocks
> by swapping on software-tcp-iscsi.

Yes, done before it was even called iSCSI.

> > And that still leaves us with the lack of buffers to receive ACKs
> > problem, which is perhaps worse.
> 
> The mempooling should take care of the acks too.

The receive buffer is allocated at the time we DMA it from the card.
We have no idea of its contents and we won't know what socket mempool
to pull the receive skbuff from until much higher in the network
stack, which could be quite a while later if we're under OOM load. And
we can't have a mempool big enough to handle all the traffic that
might potentially be deferred for softirq processing when we're OOM,
especially at gigabit rates.

I think this is actually the tricky piece of the problem and solving
the socket and send buffer allocation doesn't help until this gets
figured out.

We could perhaps try to address this with another special receive-side
alloc_skb that fails most of the time on OOM but sometimes pulls from
a special reserve.

> Perhaps the mempooling overhead will be too huge to pay for it even when
> it's not necessary, in such case the iscsid will have to pass a new
> bitflag to the socket syscall, when it creates the socket meant to talk
> with the remote disk.

I think we probably attach a mempool to a socket after the fact. And
no, we can't have a mempool attached to every socket.

-- 
Mathematics is the supreme nostalgia of our time.