From: Hans Schillstrom <hans@schillstrom•com>
To: "Eric W. Biederman" <ebiederm@xmission•com>
Cc: netdev@vger•kernel.org, Daniel Lezcano <daniel.lezcano@free•fr>
Subject: Re: Race condition when creating multiple namespaces?
Date: Tue, 12 Apr 2011 06:56:53 +0200 [thread overview]
Message-ID: <201104120656.54281.hans@schillstrom.com> (raw)
In-Reply-To: <m1ei58co08.fsf@fess.ebiederm.org>
On Tuesday, April 12, 2011 02:27:35 Eric W. Biederman wrote:
> Hans Schillstrom <hans@schillstrom•com> writes:
>
> > Hello
> > I'v been strugling with this for some time now
> >
> > When creating multiple namespaces using lxc-start, un-initialized network namespace parts will be called by the new process in the namespace.
> > ex. when using conntrack or ipvsadm to quickly, (a sleep 2 "solves" the problem).
> > (From what I can see syscall clone() is used in lx-start i.e. do_fork will be called later on.)
> > Actually I was debugging ip_vs when closing multiple ns when I fell into this one.
> >
> > I have a loop that create 33 containers whith lxc-start ... -- test.sh
> > the first thing the new conatiner does in test.sh is
> > #!/bin/bash
> > iptables -t mangle -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark
> > nc -l -p1234
> >
> > This results in NULL ptr in ip_conntrack_net_init(struct *net)
>
> Ouch!
>
> > and in anoither test test.sh looks like this
> > #!/bin/bash
> > ipvsadm --start-daemon=master --mcast-interface=lo
> > nc -l -p1234
> >
> > And this results in an uniitialized spinlock in ip_vs_sync
> >
> > I put a printk in nsproxy: copy_namespaces() and could see a dozens of them
> > before anything appears from ipvs or conntrack.
> >
> > My feeling is that when you start up user processes in a new name space,
> > all kernel related init should have been done (you should not need to add a sleep to get it working)
> >
> > All test made by using todays net-next-2.6 (2.6.39-rc1)
> >
> > Note:
> > That neither conntrack or ip_vs modules where loaded,
> > if modules where loaded before creating new namespaces it all works...
> >
> > Finally the question,
> > Should it really work to load modules within a namespace ,
> > that is a part of netns ?
>
> >From an implementation point of view kernel modules are not in a
> namespace, so there should be no difference between being in a namespace
> and loading a kernel networking module and not being in a namespace and
> loading in a kernel module.
>
> It does sound like you have hit a module loading race, and perhaps
> a race that is confined to network namespaces.
>
> My head is in another problem so I won't be able to look at this for
> a bit. But if you are getting into ip_conntrack_net_init with
> a NULL network namespace something spectacularly bad is happening.
OK I'll continue to dig into this.
>
> In particular it looks like you must be hitting a bug in for_each_net.
> Which would pretty much have to be a race in adding or removing from
> net_namespace_list.
It was further down in proc_net_fops_create()
>
> I took a quick skim through the code and whenever we modify the
> net_namespace we hold but the net_mutex and inside it the rtnl_lock so I
> don't immediate see how you could be getting a NULL net into
> ip_conntrack_net_init.
I do had the same problem in ip_vs a couple of times, but at that time I thought it was my changes...
In the ip_vs case it seems to be more like a race or a missing lock one core reach a "not fully" initialized ipvs struct.
That could be my fault like bad order when calling register_pernet_subsys...
>
> Is there a codepath besides register_pernet_subsys that is calling
> ip_conntrack_net_init?
Not what I can see...
>
> Do you have any local modifications that could be messing up register_pernet_subsys?
Not right now (I took them away, a clean git clone)
>
> Eric
>
I will continue with this today
Thanks a lot
Hans
next prev parent reply other threads:[~2011-04-12 4:56 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-04-11 21:01 Race condition when creating multiple namespaces? Hans Schillstrom
2011-04-12 0:27 ` Eric W. Biederman
2011-04-12 4:56 ` Hans Schillstrom [this message]
2011-04-14 20:46 ` Hans Schillstrom
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=201104120656.54281.hans@schillstrom.com \
--to=hans@schillstrom$(echo .)com \
--cc=daniel.lezcano@free$(echo .)fr \
--cc=ebiederm@xmission$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox