Subject: unable to write to dev node

From: daggs <[email-masked]>
Date: Thu, 4 Jul 2024 16:15:46 +0200

Greeting,

I'm trying to debug a bug under alpine linux when it comes to session based libvirt vms.
when starting a vm that has a virt nic binded to a bridge, I get this error: Unable to create tap device vnet0: Operation not permitted
I've looked into the code of libvirt and narrowed it down to this func: virNetDevTapCreate
I've taken the relevant code to a side file for testing, there is the code I use:
# include <linux/if.h>
# include <linux/if_tun.h>    /* IFF_TUN, IFF_NO_PI */
#include <stddef.h>
#include <stdlib.h>
# include <sys/ioctl.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

enum {
   VIR_NETDEV_TAP_CREATE_NONE = 0,
   /* Bring the interface up */
   VIR_NETDEV_TAP_CREATE_IFUP               = 1 << 0,
   /* Enable IFF_VNET_HDR on the tap device */
   VIR_NETDEV_TAP_CREATE_VNET_HDR           = 1 << 1,
   /* Set this interface's MAC as the bridge's MAC address */
   VIR_NETDEV_TAP_CREATE_USE_MAC_FOR_BRIDGE = 1 << 2,
   /* The device will persist after the file descriptor is closed */
   VIR_NETDEV_TAP_CREATE_PERSIST            = 1 << 3,
   /* The device is allowed to exist before creation */
   VIR_NETDEV_TAP_CREATE_ALLOW_EXISTING     = 1 << 4,
};

int main()
{
	int fd;
        char *tunpath = "/dev/net/tun";
	size_t tapfdSize = 1;
        struct ifreq ifr = { 0 };
	unsigned int flags = VIR_NETDEV_TAP_CREATE_IFUP;
	if (1)
		flags |= VIR_NETDEV_TAP_CREATE_VNET_HDR;

        if ((fd = open(tunpath, O_RDWR)) < 0) {
            perror("Unable to open, is tun module loaded?");
            exit(1);
        }

        snprintf(ifr.ifr_name, 5, "vnet%d", 0);
        ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
        /* If tapfdSize is greater than one, request multiqueue */
        if (tapfdSize > 1)
            ifr.ifr_flags |= IFF_MULTI_QUEUE;

        if (flags &  VIR_NETDEV_TAP_CREATE_VNET_HDR)
            ifr.ifr_flags |= IFF_VNET_HDR;

        if (ioctl(fd, TUNSETIFF, &ifr) < 0) {
            perror("Unable to create tap device");
        }

        return 0;
}

it compiles fine and works under user root.
I have a user named foo which I use for the sessioned vm, looking at /dev/net/tun's permissions, I see this:
$ ll /dev/net/tun
crw-rw-rw- 1 root netdev 10, 200 Jul  4 15:52 /dev/net/tun

so I added foo to netdev group, now it has the following id output: uid=1002(foo) gid=1002(foo) groups=1002(foo),28(netdev),34(kvm),36(qemu),102(libvirt)
and ran the code again, I'm getting the same error.
I went to the libvirt community and one of the devs tried to help me with it, he concluded that there is something wrong in the alpine because it works in fedora.
in contrast, /dev/null has the same permissions as /dev/net/tun but the group is root and I can write to it as user foo.

any ideas what I am missing?

Thanks,

Dagg

---

From: Natanael Copa <[email-masked]>
Date: Thu, 4 Jul 2024 17:46:37 +0200

On Thu, 4 Jul 2024 16:15:46 +0200

> any ideas what I am missing?

Can you run your app under strace? To show which syscall that fails.

Do you run this under docker? if so it might be libseccomp that is
causing problems.

-nc

---

From: daggs <[email-masked]>
Date: Thu, 4 Jul 2024 18:17:55 +0200

> Can you run your app under strace? To show which syscall that fails.
$ strace ./test
execve("./test", ["./test"], 0x7ffd948179e0 /* 17 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7f9eff1a1b28) = 0
set_tid_address(0x7f9eff1a1f90)         = 3922
brk(NULL)                               = 0x559659075000
brk(0x559659077000)                     = 0x559659077000
mmap(0x559659075000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x559659075000
mprotect(0x7f9eff19e000, 4096, PROT_READ) = 0
mprotect(0x5596579ff000, 4096, PROT_READ) = 0
open("/dev/net/tun", O_RDWR|O_LARGEFILE) = 3
ioctl(3, TUNSETIFF, 0x7ffd72891cf0)     = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="Unable to create tap device", iov_len=27}], 2Unable to create tap device) = 27
writev(2, [{iov_base="", iov_len=0}, {iov_base=":", iov_len=1}], 2:) = 1
writev(2, [{iov_base="", iov_len=0}, {iov_base=" ", iov_len=1}], 2 ) = 1
writev(2, [{iov_base="", iov_len=0}, {iov_base="Operation not permitted", iov_len=23}], 2Operation not permitted) = 23
writev(2, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
exit_group(0)                           = ?
+++ exited with 0 +++

> Do you run this under docker? if so it might be libseccomp that is
no, I'm running inside a libvirt qemu vm. e.g. nested vm

---

From: Natanael Copa <[email-masked]>
Date: Fri, 5 Jul 2024 09:25:44 +0200

On Thu, 4 Jul 2024 18:17:55 +0200

> open("/dev/net/tun", O_RDWR|O_LARGEFILE) = 3

This means the permissions to /dev/net/tun and the group etc works.

> ioctl(3, TUNSETIFF, 0x7ffd72891cf0)     = -1 EPERM (Operation not permitted)

So it is the ioctl that fails, for some reason.

Is the kernel module `tun` loaded`?

---

From: daggs <[email-masked]>
Date: Fri, 5 Jul 2024 15:12:04 +0200

> Is the kernel module `tun` loaded`?
yes it is.
is there a way to compile the kernel faster so I can debug it from within?