Bug#857295: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership

Discussion:

Stiepan

2017-03-14 16:17:53 UTC

You are welcome. As stated in my reply to Serge H. Hallyn's off-list message, in the meantime I have installed version 2.0.7 from jessie-backports and am unable to reproduce the issue, as I cannot start unprivileged containers anymore (due to a network error). According to Debian's tracker page for lxc, the version that I have installed from backports is 2.0.7-1, which does not include latest upstream fixes. I guess that I have to wait for the 2.0.7-2 package - which includes latest upstream fixes - to land in jessie-backports for these issues (both security and functional) to be fixed.

CC-ing the Debian address for this bug, as they explicitly asked to do this in case there is a need to reopen the Debian bug, which seems to be the case here (at least, for Jessie, since the intermediary 2.0.7-1 .deb apparently breaks unprivileged networking, besides not fixing the security issue).
To the Debian team in charge of this bug:
As unprivileged mode is not activated by default on Debian, I understand that this is not a priority, but it would still be nice to have this fixed quickly.
By the way, not directly related to this specific bug, but I hope that snapd + LXD somehow finds its way into jessie-backports: that would be great!

Stiepan

-------- Original Message --------
Subject: Re: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 14 March 2017 2:06 AM
UTC Time: 14 March 2017 01:07

I don't know whether that is the same bug, or a related one, but on Debian8 using LXC from jessie-backports, setting the default route in a container affects the host - namely, from an unpriv. container, setting the route sets the host's route as well.
lxc-info --version outputs 2.0.6 and no update is currently available (on Debian).

Thanks for the report. I just tried to reproduce the issue on Ubuntu
16.04 with 2.0.7-0ubuntu1~16.04.2, which is the package patched for the
issue that I announced in this thread. I couldn't reproduce it.

I then installed an old 2.0.6 based deb (2.0.6-0ubuntu1~ubuntu16.04.1)
and still couldn't reproduce it.

I'd suggest opening an upstream bug here:

https://github.com/lxc/lxc/issues/new

(Normally, they prefer private security bugs on Launchpad but your
report to this list is already public so I don't see a need.)

Tyler

Stiepan
-------- Original Message --------
Subject: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 9 March 2017 5:54 PM
UTC Time: 9 March 2017 16:55
Jann Horn discovered that the lxc-user-nic program could be tricked into
operating on a network namespace over which the caller did not hold
privilege.
The behavior didn't follow what was documented in the lxc-user-nic(1)
It ensures that the calling user is privileged over the network
namespace to which the interface will be attached.
This issue is CVE-2017-5985.
https://lists.linuxcontainers.org/pipermail/lxc-users/2017-March/012925.html
https://launchpad.net/bugs/1654676
https://github.com/lxc/lxc/commit/16af238036a5464ae8f2420ed3af214f0de875f9
Tyler

Stiepan

2017-03-15 10:56:19 UTC

Permalink

I have found a workaround to start the container in user mode again on Debian 8:
use a "true" br0 bridge instead of lxc-br0 and disable, or stop the lxc-net service.

Under these conditions, using lxc 2.0.7(-1) from jessie-backports, I am not able to reproduce the routing issue I saw running lxc 2.0.6 in user mode using lxc-net. So a safe fallback (for Debian 8), for the time being, seems to be to avoid using user mode lxc networking and use a plain old br0 instead. This should work on all CPU architectures (even on powerpc, known to be recalcitrant to lxc on Debian 8).

Stiepan

-------- Original Message --------
Subject: Re: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 14 March 2017 5:17 PM
UTC Time: 14 March 2017 16:17
From: ***@itk.swiss
To: oss-***@lists.openwall.com <oss-***@lists.openwall.com>, ***@bugs.debian.org <***@bugs.debian.org>
StÃ©phane Graber <***@ubuntu.com>, ***@ubuntu.com

You are welcome. As stated in my reply to Serge H. Hallyn's off-list message, in the meantime I have installed version 2.0.7 from jessie-backports and am unable to reproduce the issue, as I cannot start unprivileged containers anymore (due to a network error). According to Debian's tracker page for lxc, the version that I have installed from backports is 2.0.7-1, which does not include latest upstream fixes. I guess that I have to wait for the 2.0.7-2 package - which includes latest upstream fixes - to land in jessie-backports for these issues (both security and functional) to be fixed.

CC-ing the Debian address for this bug, as they explicitly asked to do this in case there is a need to reopen the Debian bug, which seems to be the case here (at least, for Jessie, since the intermediary 2.0.7-1 .deb apparently breaks unprivileged networking, besides not fixing the security issue).
To the Debian team in charge of this bug:
As unprivileged mode is not activated by default on Debian, I understand that this is not a priority, but it would still be nice to have this fixed quickly.
By the way, not directly related to this specific bug, but I hope that snapd + LXD somehow finds its way into jessie-backports: that would be great!

Stiepan

-------- Original Message --------
Subject: Re: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 14 March 2017 2:06 AM
UTC Time: 14 March 2017 01:07

Stiepan

2017-03-24 09:03:57 UTC

Permalink

Fyi, now that lxc 2.0.7-2 landed in jessie-backports, I am getting a new error when trying to start an lxc instance (running jessie as well) using a virtual br0 rather than "plain old" br0 (all of this in unprivileged mode), namely: lxc_delete_network:3028 - Failed to remove interface "vethXJW6PL" from host: Operation not permitted. With "plain old" br0, it still works as expected.

Stiepan

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message --------
Subject: Bug#857295: Info received ([oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership)
Local Time: 15 March 2017 11:56 AM
UTC Time: 15 March 2017 10:57
From: ***@bugs.debian.org
To: Stiepan <***@itk.swiss>

Thank you for the additional information you have supplied regarding
this Bug report.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
pkg-lxc <pkg-lxc-***@lists.alioth.debian.org>

If you wish to submit further information on this problem, please
send it to ***@bugs.debian.org.

Please do not send mail to ***@bugs.debian.org unless you wish
to report a problem with the Bug-tracking system.

--
857295: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=857295
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems

Evgeni Golov

2017-03-24 09:17:59 UTC

Permalink

Hi,

Post by Stiepan
Fyi, now that lxc 2.0.7-2 landed in jessie-backports, I am getting a new error when trying to start an lxc instance (running jessie as well) using a virtual br0 rather than "plain old" br0 (all of this in unprivileged mode), namely: lxc_delete_network:3028 - Failed to remove interface "vethXJW6PL" from host: Operation not permitted. With "plain old" br0, it still works as expected.

Can you alaborate a bit more on your network setup please?
What is a "virtual br0"? How do you you set this up?

My setup uses brctl to setup the bridge and then unpviliged containers
work fine. I guess that is "plain old" for ya?

Regards
Evgeni

Stiepan

2017-03-24 14:51:24 UTC

Permalink

Hi,

Using a bridge set up with libvirt (as in http://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29) doesn't work.
Neither does using a bridge set up as indicated in https://wiki.debian.org/LXC/SimpleBridge#Using_lxc-net (causes the same errors as with libvirt).
Using a classical / "plain old" / you-name-it bridge, set up as in http://wiki.libvirt.org/page/Networking#Altering_the_interface_config, does work.

By the way, the lxc_delete_network:3028... additional error I was seeing pops up only when /etc/lxc/lxc-usernet is still set to use br0, whilst the LXC container is set to use virbr0 and hence can be ignored, sorry about that. When properly configured (i.e. when both are configured to use virbr0, or lxcbr0), container startup simply fails with a "Failed to create the configured network" error, but still fails, whereas when using classical br0, it works.

So, if your bridge is set up as suggested in https://wiki.debian.org/BridgeNetworkConnections' Manual bridge setup section, using either brctl or /etc/network/interfaces (for a persistent config), we have the same configuration and it works, which is fine. Still, I thought that LXC enabled using lxcbr0 bridges in user mode, as lxc-user-nic's man page suggests is possible. Can you confirm whether this is the case with the current version?

Regards,
Stiepan

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message --------
Subject: Re: [pkg-lxc-devel] Bug#857295: Info received ([oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership)
Local Time: 24 March 2017 10:17 AM
UTC Time: 24 March 2017 09:17
From: ***@debian.org
To: Stiepan <***@itk.swiss>, ***@bugs.debian.org

Hi,

Evgeni Golov

2017-03-26 10:17:54 UTC

Permalink

Hi Stiepan,

Post by Stiepan
Using a bridge set up with libvirt (as in http://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29) doesn't work.

Is that what the libvirt package does on Debian out-of-the-box?
If so it works just fine for me on my laptop where I put the containers on the vibr0 created by libvirt.

Post by Stiepan
Neither does using a bridge set up as indicated in https://wiki.debian.org/LXC/SimpleBridge#Using_lxc-net (causes the same errors as with libvirt).

So I just fired a fresh jessie+backports Vagrant box and it worked fine (incl network in the container):

$ vagrant init debian/jessie64
$ vagrant up
$ vagrant ssh

***@jessie:~$ sudo nano /etc/apt/sources.list
deb http://httpredir.debian.org/debian jessie-backports main

***@jessie:~$ sudo apt update

***@jessie:~$ sudo apt install lxc/jessie-backports lxcfs

***@jessie:~$ sudo nano /etc/default/lxc-net
USE_LXC_BRIDGE="true"

***@jessie:~$ systemctl enable lxc-net
***@jessie:~$ systemctl restart lxc-net

***@jessie:~$ ip a s dev lxcbr0
3: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 scope global lxcbr0
valid_lft forever preferred_lft forever

***@jessie:~$ sudo sysctl -w kernel.unprivileged_userns_clone=1

***@jessie:~$ exit # needed to trigger lxcfs' PAM module

$vagrant ssh

***@jessie:~$ cat /proc/self/cgroup
8:perf_event:/
7:blkio:/
6:net_cls,net_prio:/
5:freezer:/user/vagrant/0
4:devices:/
3:cpu,cpuacct:/
2:cpuset:/
1:name=systemd:/user/vagrant/0

***@jessie:~$ mkdir ~/.config/lxc/ -p

***@jessie:~$ nano ~/.config/lxc/default.conf
xc.include = /etc/lxc/default.conf
lxc.id_map = u 0 624288 65536
lxc.id_map = g 0 624288 65536

***@jessie:~$ sudo nano /etc/lxc/lxc-usernet
vagrant veth lxcbr0 10

***@jessie:~$ lxc-create -n jessie -t download -- -d debian -r jessie -a amd64

***@jessie:~$ nano .local/share/lxc/jessie/config
lxc.network.type=veth
lxc.network.flags=up
lxc.network.link=lxcbr0

***@jessie:~$ lxc-start -n jessie
***@jessie:~$ lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6
jessie RUNNING 0 - - -

Post by Stiepan
Using a classical / "plain old" / you-name-it bridge, set up as in http://wiki.libvirt.org/page/Networking#Altering_the_interface_config, does work.

I don't see any technical difference between the plain br0 setup with this link and the ones created by lxc-net or libvirt.
Can you point them out please?

Post by Stiepan
By the way, the lxc_delete_network:3028... additional error I was seeing pops up only when /etc/lxc/lxc-usernet is still set to use br0, whilst the LXC container is
set to use virbr0 and hence can be ignored, sorry about that. When properly configured (i.e. when both are configured to use virbr0, or lxcbr0), container startup
simply fails with a "Failed to create the configured network" error, but still fails, whereas when using classical br0, it works.

Can you please provide the steps how to setup your setup from a plain jessie or stretch image?

Post by Stiepan
So, if your bridge is set up as suggested in https://wiki.debian.org/BridgeNetworkConnections' Manual bridge setup section, using either brctl or
/etc/network/interfaces (for a persistent config), we have the same configuration and it works, which is fine. Still, I thought that LXC enabled using lxcbr0 bridges
in user mode, as lxc-user-nic's man page suggests is possible. Can you confirm whether this is the case with the current version?

lxc-user-nic is to attach a user-namespace-nic to an existing bridge, you can't create a bridge with it.

Stiepan

2017-03-28 09:41:02 UTC

Permalink

Hi Evgeni,

First of all, thank you for posting the detailed steps you used to reproduce a working config, which is by far the best explanation on how to set up LXC for running Debian 8 on top of Debian 8 I have seen yet.

To answer your questions, yes I've been using Jessie's libvirt package "as is". No idea why it doesn't work. However, by following your explanation, I was able to reproduce a working config, using user-mode networking and lxcbr0, on i386, with only minor adaptations and on ppc64, with one big difference: cgmanager has to be used there (see https://linuxcontainers.org/cgmanager/getting-started/) for LXC to run in unprivileged mode, with or without networking, be it using br0 or lxcbr0. Hence, this, in combination with your explanation, makes it work on both architectures I could test.

About br0 v.s. lxcbr0: the main difference between them is the fact that the latter involves running dnsmasq in the background (and therefore has firewall implications, namely an iptables rule is added) to provide ip masquerading (that is, NAT) to the containers.
Note: containerops.org/2013/11/19/lxc-networking provides in-depth coverage of the subject + also covers linux network namespaces in general.

Regarding how to setup our setups:
I think that the Debian wiki on LXC, https://wiki.debian.org/LXC, should be updated with your simplified method - save for the vagrant part of it - and eventually, my architecture-specific notes as well, as I remember reading in a 2.0.7-1-related post that powerpc compatibility was seen as an issue.

As for your last comment, I agree that bridges, be it br0 or lxcbr0, have to be created by root; I was referring to using them as an unprivileged user.

Last but not least, the security bug I had spotted was most likely equivalent to CVE-2017-5985, as I could not reproduce it now that it works again using a lxcbr0 bridge as well (setting the default route in an unprivileged container using user-mode networking does not affect the host anymore), when running lxc 2.0.7-2 from jessie-backports (which includes the fixes for the aforementioned CVE). I will post an update on oss-security with the security-relevant part of this thread.

Stiepan

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message --------
Subject: Re: [pkg-lxc-devel] Bug#857295: Bug#857295: Info received ([oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership)
Local Time: 26 March 2017 12:17 PM
UTC Time: 26 March 2017 10:17
From: ***@debian.org
To: Stiepan <***@itk.swiss>, ***@bugs.debian.org

Hi Stiepan,

Post by Stiepan
Using a bridge set up with libvirt (as in http://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29) doesn't work.

Is that what the libvirt package does on Debian out-of-the-box?
If so it works just fine for me on my laptop where I put the containers on the vibr0 created by libvirt.

Post by Stiepan
Neither does using a bridge set up as indicated in https://wiki.debian.org/LXC/SimpleBridge#Using_lxc-net (causes the same errors as with libvirt).

Post by Stiepan
Using a classical / "plain old" / you-name-it bridge, set up as in http://wiki.libvirt.org/page/Networking#Altering_the_interface_config, does work.

I don't see any technical difference between the plain br0 setup with this link and the ones created by lxc-net or libvirt.
Can you point them out please?

Can you please provide the steps how to setup your setup from a plain jessie or stretch image?

lxc-user-nic is to attach a user-namespace-nic to an existing bridge, you can't create a bridge with it.

Stiepan

2017-03-28 10:45:34 UTC

Permalink

Thanks to the 2.0.7-2 update by Evgeni Golov and his crystal-clear instructions on how to use lxcbr0 with this version, I could confirm that the issue with the host's routing table being affected by changes in the containers' routing tables is not there anymore when using that version (lxc 2.0.7-2 from jessie-backports), which includes the fixes to CVE-2017-5985 which were brought in LXC 2.0.7 (upstream).

This was thus basically a variation of said CVE, which probably doesn't need to be separately numbered as such, the core problem at stake being the same:
network namespace ownership was not respected by a setuid-root program enabling the user to configure networks as non-root, which is now solved.
This leads me to a suggestion to the upstream developers: couldn't the same be achieved using specific network-related capabilities, instead of setuid-root, thereby further reducing the risk of lxc-user-nic being exploited and hence, reducing overall attack surface (in unprivileged mode)?
I have read in https://wiki.ubuntu.com/UserNamespace that the approach of using "targeted capabilities" was then considered. This is probably the closest to what I am suggesting (specifically for lxc-user-nic - the current approach with 1-1 uid mappings seems fine for network-unrelated things).

Stiepan

-------- Original Message --------
Subject: Re: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 15 March 2017 11:55 AM
UTC Time: 15 March 2017 10:56
From: ***@itk.swiss
To: oss-***@lists.openwall.com
oss-***@lists.openwall.com <oss-***@lists.openwall.com>, ***@bugs.debian.org <***@bugs.debian.org>, StÃ©phane Graber <***@ubuntu.com>, ***@ubuntu.com

I have found a workaround to start the container in user mode again on Debian 8:
use a "true" br0 bridge instead of lxc-br0 and disable, or stop the lxc-net service.

Under these conditions, using lxc 2.0.7(-1) from jessie-backports, I am not able to reproduce the routing issue I saw running lxc 2.0.6 in user mode using lxc-net. So a safe fallback (for Debian 8), for the time being, seems to be to avoid using user mode lxc networking and use a plain old br0 instead. This should work on all CPU architectures (even on powerpc, known to be recalcitrant to lxc on Debian 8).

Stiepan

-------- Original Message --------
Subject: Re: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 14 March 2017 5:17 PM
UTC Time: 14 March 2017 16:17
From: ***@itk.swiss
To: oss-***@lists.openwall.com <oss-***@lists.openwall.com>, ***@bugs.debian.org <***@bugs.debian.org>
StÃ©phane Graber <***@ubuntu.com>, ***@ubuntu.com

You are welcome. As stated in my reply to Serge H. Hallyn's off-list message, in the meantime I have installed version 2.0.7 from jessie-backports and am unable to reproduce the issue, as I cannot start unprivileged containers anymore (due to a network error). According to Debian's tracker page for lxc, the version that I have installed from backports is 2.0.7-1, which does not include latest upstream fixes. I guess that I have to wait for the 2.0.7-2 package - which includes latest upstream fixes - to land in jessie-backports for these issues (both security and functional) to be fixed.

CC-ing the Debian address for this bug, as they explicitly asked to do this in case there is a need to reopen the Debian bug, which seems to be the case here (at least, for Jessie, since the intermediary 2.0.7-1 .deb apparently breaks unprivileged networking, besides not fixing the security issue).
To the Debian team in charge of this bug:
As unprivileged mode is not activated by default on Debian, I understand that this is not a priority, but it would still be nice to have this fixed quickly.
By the way, not directly related to this specific bug, but I hope that snapd + LXD somehow finds its way into jessie-backports: that would be great!

Stiepan

-------- Original Message --------
Subject: Re: [oss-security] LXC: CVE-2017-5985: lxc-user-nic didn't verify network namespace ownership
Local Time: 14 March 2017 2:06 AM
UTC Time: 14 March 2017 01:07

Serge E. Hallyn

2017-03-28 14:49:04 UTC

Permalink

Post by Stiepan
Thanks to the 2.0.7-2 update by Evgeni Golov and his crystal-clear instructions on how to use lxcbr0 with this version, I could confirm that the issue with the host's routing table being affected by changes in the containers' routing tables is not there anymore when using that version (lxc 2.0.7-2 from jessie-backports), which includes the fixes to CVE-2017-5985 which were brought in LXC 2.0.7 (upstream).
network namespace ownership was not respected by a setuid-root program enabling the user to configure networks as non-root, which is now solved.
This leads me to a suggestion to the upstream developers: couldn't the same be achieved using specific network-related capabilities, instead of setuid-root, thereby further reducing the risk of lxc-user-nic being exploited and hence, reducing overall attack surface (in unprivileged mode)?
I have read in https://wiki.ubuntu.com/UserNamespace that the approach of using "targeted capabilities" was then considered. This is probably the closest to what I am suggesting (specifically for lxc-user-nic - the current approach with 1-1 uid mappings seems fine for network-unrelated things).

The targeted capabilities wouldn't help here, because in fact
lxc-user-nic requires privilege against the parent namespace.

-serge