Greeting,
I'm trying to debug a bug under alpine linux when it comes to session based libvirt vms.
when starting a vm that has a virt nic binded to a bridge, I get this error: Unable to create tap device vnet0: Operation not permitted
I've looked into the code of libvirt and narrowed it down to this func: virNetDevTapCreate
I've taken the relevant code to a side file for testing, there is the code I use:
# include <linux/if.h>
# include <linux/if_tun.h> /* IFF_TUN, IFF_NO_PI */
#include <stddef.h>
#include <stdlib.h>
# include <sys/ioctl.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>
enum {
VIR_NETDEV_TAP_CREATE_NONE = 0,
/* Bring the interface up */
VIR_NETDEV_TAP_CREATE_IFUP = 1 << 0,
/* Enable IFF_VNET_HDR on the tap device */
VIR_NETDEV_TAP_CREATE_VNET_HDR = 1 << 1,
/* Set this interface's MAC as the bridge's MAC address */
VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE = 1 << 2,
/* The device will persist after the file descriptor is closed */
VIR_NETDEV_TAP_CREATE_PERSIST = 1 << 3,
/* The device is allowed to exist before creation */
VIR_NETDEV_TAP_CREATE_ALLOW_EXISTING = 1 << 4,
};
int main()
{
int fd;
char *tunpath = "/dev/net/tun";
size_t tapfdSize = 1;
struct ifreq ifr = { 0 };
unsigned int flags = VIR_NETDEV_TAP_CREATE_IFUP;
if (1)
flags |= VIR_NETDEV_TAP_CREATE_VNET_HDR;
if ((fd = open(tunpath, O_RDWR)) < 0) {
perror("Unable to open, is tun module loaded?");
exit(1);
}
snprintf(ifr.ifr_name, 5, "vnet%d", 0);
ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
/* If tapfdSize is greater than one, request multiqueue */
if (tapfdSize > 1)
ifr.ifr_flags |= IFF_MULTI_QUEUE;
if (flags & VIR_NETDEV_TAP_CREATE_VNET_HDR)
ifr.ifr_flags |= IFF_VNET_HDR;
if (ioctl(fd, TUNSETIFF, &ifr) < 0) {
perror("Unable to create tap device");
}
return 0;
}
it compiles fine and works under user root.
I have a user named foo which I use for the sessioned vm, looking at /dev/net/tun's permissions, I see this:
$ ll /dev/net/tun
crw-rw-rw- 1 root netdev 10, 200 Jul 4 15:52 /dev/net/tun
so I added foo to netdev group, now it has the following id output: uid=1002(foo) gid=1002(foo) groups=1002(foo),28(netdev),34(kvm),36(qemu),102(libvirt)
and ran the code again, I'm getting the same error.
I went to the libvirt community and one of the devs tried to help me with it, he concluded that there is something wrong in the alpine because it works in fedora.
in contrast, /dev/null has the same permissions as /dev/net/tun but the group is root and I can write to it as user foo.
any ideas what I am missing?
Thanks,
Dagg
On Thu, 4 Jul 2024 16:15:46 +0200
daggs <daggs@gmx.com> wrote:
> Greeting,> > I'm trying to debug a bug under alpine linux when it comes to session based libvirt vms.> when starting a vm that has a virt nic binded to a bridge, I get this error: Unable to create tap device vnet0: Operation not permitted> I've looked into the code of libvirt and narrowed it down to this func: virNetDevTapCreate> I've taken the relevant code to a side file for testing, there is the code I use:> # include <linux/if.h>> # include <linux/if_tun.h> /* IFF_TUN, IFF_NO_PI */> #include <stddef.h>> #include <stdlib.h>> # include <sys/ioctl.h>> #include <fcntl.h>> #include <errno.h>> #include <stdio.h>> > > enum {> VIR_NETDEV_TAP_CREATE_NONE = 0,> /* Bring the interface up */> VIR_NETDEV_TAP_CREATE_IFUP = 1 << 0,> /* Enable IFF_VNET_HDR on the tap device */> VIR_NETDEV_TAP_CREATE_VNET_HDR = 1 << 1,> /* Set this interface's MAC as the bridge's MAC address */> VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE = 1 << 2,> /* The device will persist after the file descriptor is closed */> VIR_NETDEV_TAP_CREATE_PERSIST = 1 << 3,> /* The device is allowed to exist before creation */> VIR_NETDEV_TAP_CREATE_ALLOW_EXISTING = 1 << 4,> };> > int main()> {> int fd;> char *tunpath = "/dev/net/tun";> size_t tapfdSize = 1;> struct ifreq ifr = { 0 };> unsigned int flags = VIR_NETDEV_TAP_CREATE_IFUP;> if (1)> flags |= VIR_NETDEV_TAP_CREATE_VNET_HDR;> > if ((fd = open(tunpath, O_RDWR)) < 0) {> perror("Unable to open, is tun module loaded?");> exit(1);> }> > snprintf(ifr.ifr_name, 5, "vnet%d", 0);> ifr.ifr_flags = IFF_TAP | IFF_NO_PI;> /* If tapfdSize is greater than one, request multiqueue */> if (tapfdSize > 1)> ifr.ifr_flags |= IFF_MULTI_QUEUE;> > if (flags & VIR_NETDEV_TAP_CREATE_VNET_HDR)> ifr.ifr_flags |= IFF_VNET_HDR;> > if (ioctl(fd, TUNSETIFF, &ifr) < 0) {> perror("Unable to create tap device");> }> > return 0;> }> > it compiles fine and works under user root.> I have a user named foo which I use for the sessioned vm, looking at /dev/net/tun's permissions, I see this:> $ ll /dev/net/tun> crw-rw-rw- 1 root netdev 10, 200 Jul 4 15:52 /dev/net/tun> > so I added foo to netdev group, now it has the following id output: uid=1002(foo) gid=1002(foo) groups=1002(foo),28(netdev),34(kvm),36(qemu),102(libvirt)> and ran the code again, I'm getting the same error.> I went to the libvirt community and one of the devs tried to help me with it, he concluded that there is something wrong in the alpine because it works in fedora.> in contrast, /dev/null has the same permissions as /dev/net/tun but the group is root and I can write to it as user foo.> > any ideas what I am missing?
Can you run your app under strace? To show which syscall that fails.
Do you run this under docker? if so it might be libseccomp that is
causing problems.
-nc
> > Thanks,> > Dagg
> > Greeting,> >> > I'm trying to debug a bug under alpine linux when it comes to session based libvirt vms.> > when starting a vm that has a virt nic binded to a bridge, I get this error: Unable to create tap device vnet0: Operation not permitted> > I've looked into the code of libvirt and narrowed it down to this func: virNetDevTapCreate> > I've taken the relevant code to a side file for testing, there is the code I use:> > # include <linux/if.h>> > # include <linux/if_tun.h> /* IFF_TUN, IFF_NO_PI */> > #include <stddef.h>> > #include <stdlib.h>> > # include <sys/ioctl.h>> > #include <fcntl.h>> > #include <errno.h>> > #include <stdio.h>> >> >> > enum {> > VIR_NETDEV_TAP_CREATE_NONE = 0,> > /* Bring the interface up */> > VIR_NETDEV_TAP_CREATE_IFUP = 1 << 0,> > /* Enable IFF_VNET_HDR on the tap device */> > VIR_NETDEV_TAP_CREATE_VNET_HDR = 1 << 1,> > /* Set this interface's MAC as the bridge's MAC address */> > VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE = 1 << 2,> > /* The device will persist after the file descriptor is closed */> > VIR_NETDEV_TAP_CREATE_PERSIST = 1 << 3,> > /* The device is allowed to exist before creation */> > VIR_NETDEV_TAP_CREATE_ALLOW_EXISTING = 1 << 4,> > };> >> > int main()> > {> > int fd;> > char *tunpath = "/dev/net/tun";> > size_t tapfdSize = 1;> > struct ifreq ifr = { 0 };> > unsigned int flags = VIR_NETDEV_TAP_CREATE_IFUP;> > if (1)> > flags |= VIR_NETDEV_TAP_CREATE_VNET_HDR;> >> > if ((fd = open(tunpath, O_RDWR)) < 0) {> > perror("Unable to open, is tun module loaded?");> > exit(1);> > }> >> > snprintf(ifr.ifr_name, 5, "vnet%d", 0);> > ifr.ifr_flags = IFF_TAP | IFF_NO_PI;> > /* If tapfdSize is greater than one, request multiqueue */> > if (tapfdSize > 1)> > ifr.ifr_flags |= IFF_MULTI_QUEUE;> >> > if (flags & VIR_NETDEV_TAP_CREATE_VNET_HDR)> > ifr.ifr_flags |= IFF_VNET_HDR;> >> > if (ioctl(fd, TUNSETIFF, &ifr) < 0) {> > perror("Unable to create tap device");> > }> >> > return 0;> > }> >> > it compiles fine and works under user root.> > I have a user named foo which I use for the sessioned vm, looking at /dev/net/tun's permissions, I see this:> > $ ll /dev/net/tun> > crw-rw-rw- 1 root netdev 10, 200 Jul 4 15:52 /dev/net/tun> >> > so I added foo to netdev group, now it has the following id output: uid=1002(foo) gid=1002(foo) groups=1002(foo),28(netdev),34(kvm),36(qemu),102(libvirt)> > and ran the code again, I'm getting the same error.> > I went to the libvirt community and one of the devs tried to help me with it, he concluded that there is something wrong in the alpine because it works in fedora.> > in contrast, /dev/null has the same permissions as /dev/net/tun but the group is root and I can write to it as user foo.> >> > any ideas what I am missing?>> Can you run your app under strace? To show which syscall that fails.>
$ strace ./test
execve("./test", ["./test"], 0x7ffd948179e0 /* 17 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7f9eff1a1b28) = 0
set_tid_address(0x7f9eff1a1f90) = 3922
brk(NULL) = 0x559659075000
brk(0x559659077000) = 0x559659077000
mmap(0x559659075000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x559659075000
mprotect(0x7f9eff19e000, 4096, PROT_READ) = 0
mprotect(0x5596579ff000, 4096, PROT_READ) = 0
open("/dev/net/tun", O_RDWR|O_LARGEFILE) = 3
ioctl(3, TUNSETIFF, 0x7ffd72891cf0) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="Unable to create tap device", iov_len=27}], 2Unable to create tap device) = 27
writev(2, [{iov_base="", iov_len=0}, {iov_base=":", iov_len=1}], 2:) = 1
writev(2, [{iov_base="", iov_len=0}, {iov_base=" ", iov_len=1}], 2 ) = 1
writev(2, [{iov_base="", iov_len=0}, {iov_base="Operation not permitted", iov_len=23}], 2Operation not permitted) = 23
writev(2, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
exit_group(0) = ?
+++ exited with 0 +++
> Do you run this under docker? if so it might be libseccomp that is> causing problems.
no, I'm running inside a libvirt qemu vm. e.g. nested vm
>> -nc>> >> > Thanks,> >> > Dagg>>
On Thu, 4 Jul 2024 18:17:55 +0200
daggs <daggs@gmx.com> wrote:
> > > Greeting,> > >> > > I'm trying to debug a bug under alpine linux when it comes to session based libvirt vms.> > > when starting a vm that has a virt nic binded to a bridge, I get this error: Unable to create tap device vnet0: Operation not permitted> > > I've looked into the code of libvirt and narrowed it down to this func: virNetDevTapCreate> > > I've taken the relevant code to a side file for testing, there is the code I use:> > > # include <linux/if.h>> > > # include <linux/if_tun.h> /* IFF_TUN, IFF_NO_PI */> > > #include <stddef.h>> > > #include <stdlib.h>> > > # include <sys/ioctl.h>> > > #include <fcntl.h>> > > #include <errno.h>> > > #include <stdio.h>> > >> > >> > > enum {> > > VIR_NETDEV_TAP_CREATE_NONE = 0,> > > /* Bring the interface up */> > > VIR_NETDEV_TAP_CREATE_IFUP = 1 << 0,> > > /* Enable IFF_VNET_HDR on the tap device */> > > VIR_NETDEV_TAP_CREATE_VNET_HDR = 1 << 1,> > > /* Set this interface's MAC as the bridge's MAC address */> > > VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE = 1 << 2,> > > /* The device will persist after the file descriptor is closed */> > > VIR_NETDEV_TAP_CREATE_PERSIST = 1 << 3,> > > /* The device is allowed to exist before creation */> > > VIR_NETDEV_TAP_CREATE_ALLOW_EXISTING = 1 << 4,> > > };> > >> > > int main()> > > {> > > int fd;> > > char *tunpath = "/dev/net/tun";> > > size_t tapfdSize = 1;> > > struct ifreq ifr = { 0 };> > > unsigned int flags = VIR_NETDEV_TAP_CREATE_IFUP;> > > if (1)> > > flags |= VIR_NETDEV_TAP_CREATE_VNET_HDR;> > >> > > if ((fd = open(tunpath, O_RDWR)) < 0) {> > > perror("Unable to open, is tun module loaded?");> > > exit(1);> > > }> > >> > > snprintf(ifr.ifr_name, 5, "vnet%d", 0);> > > ifr.ifr_flags = IFF_TAP | IFF_NO_PI;> > > /* If tapfdSize is greater than one, request multiqueue */> > > if (tapfdSize > 1)> > > ifr.ifr_flags |= IFF_MULTI_QUEUE;> > >> > > if (flags & VIR_NETDEV_TAP_CREATE_VNET_HDR)> > > ifr.ifr_flags |= IFF_VNET_HDR;> > >> > > if (ioctl(fd, TUNSETIFF, &ifr) < 0) {> > > perror("Unable to create tap device");> > > }> > >> > > return 0;> > > }> > >> > > it compiles fine and works under user root.> > > I have a user named foo which I use for the sessioned vm, looking at /dev/net/tun's permissions, I see this:> > > $ ll /dev/net/tun> > > crw-rw-rw- 1 root netdev 10, 200 Jul 4 15:52 /dev/net/tun> > >> > > so I added foo to netdev group, now it has the following id output: uid=1002(foo) gid=1002(foo) groups=1002(foo),28(netdev),34(kvm),36(qemu),102(libvirt)> > > and ran the code again, I'm getting the same error.> > > I went to the libvirt community and one of the devs tried to help me with it, he concluded that there is something wrong in the alpine because it works in fedora.> > > in contrast, /dev/null has the same permissions as /dev/net/tun but the group is root and I can write to it as user foo.> > >> > > any ideas what I am missing? > >> > Can you run your app under strace? To show which syscall that fails.> > > $ strace ./test> execve("./test", ["./test"], 0x7ffd948179e0 /* 17 vars */) = 0> arch_prctl(ARCH_SET_FS, 0x7f9eff1a1b28) = 0> set_tid_address(0x7f9eff1a1f90) = 3922> brk(NULL) = 0x559659075000> brk(0x559659077000) = 0x559659077000> mmap(0x559659075000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x559659075000> mprotect(0x7f9eff19e000, 4096, PROT_READ) = 0> mprotect(0x5596579ff000, 4096, PROT_READ) = 0> open("/dev/net/tun", O_RDWR|O_LARGEFILE) = 3
This means the permissions to /dev/net/tun and the group etc works.
> ioctl(3, TUNSETIFF, 0x7ffd72891cf0) = -1 EPERM (Operation not permitted)
So it is the ioctl that fails, for some reason.
Is the kernel module `tun` loaded`?
> writev(2, [{iov_base="", iov_len=0}, {iov_base="Unable to create tap device", iov_len=27}], 2Unable to create tap device) = 27> writev(2, [{iov_base="", iov_len=0}, {iov_base=":", iov_len=1}], 2:) = 1> writev(2, [{iov_base="", iov_len=0}, {iov_base=" ", iov_len=1}], 2 ) = 1> writev(2, [{iov_base="", iov_len=0}, {iov_base="Operation not permitted", iov_len=23}], 2Operation not permitted) = 23> writev(2, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2> ) = 1> exit_group(0) = ?> +++ exited with 0 +++> > > > Do you run this under docker? if so it might be libseccomp that is> > causing problems. > no, I'm running inside a libvirt qemu vm. e.g. nested vm> > >> > -nc> > > > >> > > Thanks,> > >> > > Dagg > >> >
> > > > Greeting,> > > >> > > > I'm trying to debug a bug under alpine linux when it comes to session based libvirt vms.> > > > when starting a vm that has a virt nic binded to a bridge, I get this error: Unable to create tap device vnet0: Operation not permitted> > > > I've looked into the code of libvirt and narrowed it down to this func: virNetDevTapCreate> > > > I've taken the relevant code to a side file for testing, there is the code I use:> > > > # include <linux/if.h>> > > > # include <linux/if_tun.h> /* IFF_TUN, IFF_NO_PI */> > > > #include <stddef.h>> > > > #include <stdlib.h>> > > > # include <sys/ioctl.h>> > > > #include <fcntl.h>> > > > #include <errno.h>> > > > #include <stdio.h>> > > >> > > >> > > > enum {> > > > VIR_NETDEV_TAP_CREATE_NONE = 0,> > > > /* Bring the interface up */> > > > VIR_NETDEV_TAP_CREATE_IFUP = 1 << 0,> > > > /* Enable IFF_VNET_HDR on the tap device */> > > > VIR_NETDEV_TAP_CREATE_VNET_HDR = 1 << 1,> > > > /* Set this interface's MAC as the bridge's MAC address */> > > > VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE = 1 << 2,> > > > /* The device will persist after the file descriptor is closed */> > > > VIR_NETDEV_TAP_CREATE_PERSIST = 1 << 3,> > > > /* The device is allowed to exist before creation */> > > > VIR_NETDEV_TAP_CREATE_ALLOW_EXISTING = 1 << 4,> > > > };> > > >> > > > int main()> > > > {> > > > int fd;> > > > char *tunpath = "/dev/net/tun";> > > > size_t tapfdSize = 1;> > > > struct ifreq ifr = { 0 };> > > > unsigned int flags = VIR_NETDEV_TAP_CREATE_IFUP;> > > > if (1)> > > > flags |= VIR_NETDEV_TAP_CREATE_VNET_HDR;> > > >> > > > if ((fd = open(tunpath, O_RDWR)) < 0) {> > > > perror("Unable to open, is tun module loaded?");> > > > exit(1);> > > > }> > > >> > > > snprintf(ifr.ifr_name, 5, "vnet%d", 0);> > > > ifr.ifr_flags = IFF_TAP | IFF_NO_PI;> > > > /* If tapfdSize is greater than one, request multiqueue */> > > > if (tapfdSize > 1)> > > > ifr.ifr_flags |= IFF_MULTI_QUEUE;> > > >> > > > if (flags & VIR_NETDEV_TAP_CREATE_VNET_HDR)> > > > ifr.ifr_flags |= IFF_VNET_HDR;> > > >> > > > if (ioctl(fd, TUNSETIFF, &ifr) < 0) {> > > > perror("Unable to create tap device");> > > > }> > > >> > > > return 0;> > > > }> > > >> > > > it compiles fine and works under user root.> > > > I have a user named foo which I use for the sessioned vm, looking at /dev/net/tun's permissions, I see this:> > > > $ ll /dev/net/tun> > > > crw-rw-rw- 1 root netdev 10, 200 Jul 4 15:52 /dev/net/tun> > > >> > > > so I added foo to netdev group, now it has the following id output: uid=1002(foo) gid=1002(foo) groups=1002(foo),28(netdev),34(kvm),36(qemu),102(libvirt)> > > > and ran the code again, I'm getting the same error.> > > > I went to the libvirt community and one of the devs tried to help me with it, he concluded that there is something wrong in the alpine because it works in fedora.> > > > in contrast, /dev/null has the same permissions as /dev/net/tun but the group is root and I can write to it as user foo.> > > >> > > > any ideas what I am missing?> > >> > > Can you run your app under strace? To show which syscall that fails.> > >> > $ strace ./test> > execve("./test", ["./test"], 0x7ffd948179e0 /* 17 vars */) = 0> > arch_prctl(ARCH_SET_FS, 0x7f9eff1a1b28) = 0> > set_tid_address(0x7f9eff1a1f90) = 3922> > brk(NULL) = 0x559659075000> > brk(0x559659077000) = 0x559659077000> > mmap(0x559659075000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x559659075000> > mprotect(0x7f9eff19e000, 4096, PROT_READ) = 0> > mprotect(0x5596579ff000, 4096, PROT_READ) = 0> > open("/dev/net/tun", O_RDWR|O_LARGEFILE) = 3>> This means the permissions to /dev/net/tun and the group etc works.>> > ioctl(3, TUNSETIFF, 0x7ffd72891cf0) = -1 EPERM (Operation not permitted)>> So it is the ioctl that fails, for some reason.>> Is the kernel module `tun` loaded`?
yes it is.
is there a way to compile the kernel faster so I can debug it from within?
>> > writev(2, [{iov_base="", iov_len=0}, {iov_base="Unable to create tap device", iov_len=27}], 2Unable to create tap device) = 27> > writev(2, [{iov_base="", iov_len=0}, {iov_base=":", iov_len=1}], 2:) = 1> > writev(2, [{iov_base="", iov_len=0}, {iov_base=" ", iov_len=1}], 2 ) = 1> > writev(2, [{iov_base="", iov_len=0}, {iov_base="Operation not permitted", iov_len=23}], 2Operation not permitted) = 23> > writev(2, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2> > ) = 1> > exit_group(0) = ?> > +++ exited with 0 +++> >> >> > > Do you run this under docker? if so it might be libseccomp that is> > > causing problems.> > no, I'm running inside a libvirt qemu vm. e.g. nested vm> >> > >> > > -nc> > >> > > >> > > > Thanks,> > > >> > > > Dagg> > >> > >>>