1================================== 2VDUSE - "vDPA Device in Userspace" 3================================== 4 5vDPA (virtio data path acceleration) device is a device that uses a 6datapath which complies with the virtio specifications with vendor 7specific control path. vDPA devices can be both physically located on 8the hardware or emulated by software. VDUSE is a framework that makes it 9possible to implement software-emulated vDPA devices in userspace. And 10to make the device emulation more secure, the emulated vDPA device's 11control path is handled in the kernel and only the data path is 12implemented in the userspace. 13 14Note that only virtio block device is supported by VDUSE framework now, 15which can reduce security risks when the userspace process that implements 16the data path is run by an unprivileged user. The support for other device 17types can be added after the security issue of corresponding device driver 18is clarified or fixed in the future. 19 20Create/Destroy VDUSE devices 21---------------------------- 22 23VDUSE devices are created as follows: 24 251. Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on 26 /dev/vduse/control. 27 282. Setup each virtqueue with ioctl(VDUSE_VQ_SETUP) on /dev/vduse/$NAME. 29 303. Begin processing VDUSE messages from /dev/vduse/$NAME. The first 31 messages will arrive while attaching the VDUSE instance to vDPA bus. 32 334. Send the VDPA_CMD_DEV_NEW netlink message to attach the VDUSE 34 instance to vDPA bus. 35 36VDUSE devices are destroyed as follows: 37 381. Send the VDPA_CMD_DEV_DEL netlink message to detach the VDUSE 39 instance from vDPA bus. 40 412. Close the file descriptor referring to /dev/vduse/$NAME. 42 433. Destroy the VDUSE instance with ioctl(VDUSE_DESTROY_DEV) on 44 /dev/vduse/control. 45 46The netlink messages can be sent via vdpa tool in iproute2 or use the 47below sample codes: 48 49.. code-block:: c 50 51 static int netlink_add_vduse(const char *name, enum vdpa_command cmd) 52 { 53 struct nl_sock *nlsock; 54 struct nl_msg *msg; 55 int famid; 56 57 nlsock = nl_socket_alloc(); 58 if (!nlsock) 59 return -ENOMEM; 60 61 if (genl_connect(nlsock)) 62 goto free_sock; 63 64 famid = genl_ctrl_resolve(nlsock, VDPA_GENL_NAME); 65 if (famid < 0) 66 goto close_sock; 67 68 msg = nlmsg_alloc(); 69 if (!msg) 70 goto close_sock; 71 72 if (!genlmsg_put(msg, NL_AUTO_PORT, NL_AUTO_SEQ, famid, 0, 0, cmd, 0)) 73 goto nla_put_failure; 74 75 NLA_PUT_STRING(msg, VDPA_ATTR_DEV_NAME, name); 76 if (cmd == VDPA_CMD_DEV_NEW) 77 NLA_PUT_STRING(msg, VDPA_ATTR_MGMTDEV_DEV_NAME, "vduse"); 78 79 if (nl_send_sync(nlsock, msg)) 80 goto close_sock; 81 82 nl_close(nlsock); 83 nl_socket_free(nlsock); 84 85 return 0; 86 nla_put_failure: 87 nlmsg_free(msg); 88 close_sock: 89 nl_close(nlsock); 90 free_sock: 91 nl_socket_free(nlsock); 92 return -1; 93 } 94 95How VDUSE works 96--------------- 97 98As mentioned above, a VDUSE device is created by ioctl(VDUSE_CREATE_DEV) on 99/dev/vduse/control. With this ioctl, userspace can specify some basic configuration 100such as device name (uniquely identify a VDUSE device), virtio features, virtio 101configuration space, the number of virtqueues and so on for this emulated device. 102Then a char device interface (/dev/vduse/$NAME) is exported to userspace for device 103emulation. Userspace can use the VDUSE_VQ_SETUP ioctl on /dev/vduse/$NAME to 104add per-virtqueue configuration such as the max size of virtqueue to the device. 105 106After the initialization, the VDUSE device can be attached to vDPA bus via 107the VDPA_CMD_DEV_NEW netlink message. Userspace needs to read()/write() on 108/dev/vduse/$NAME to receive/reply some control messages from/to VDUSE kernel 109module as follows: 110 111.. code-block:: c 112 113 static int vduse_message_handler(int dev_fd) 114 { 115 int len; 116 struct vduse_dev_request req; 117 struct vduse_dev_response resp; 118 119 len = read(dev_fd, &req, sizeof(req)); 120 if (len != sizeof(req)) 121 return -1; 122 123 resp.request_id = req.request_id; 124 125 switch (req.type) { 126 127 /* handle different types of messages */ 128 129 } 130 131 len = write(dev_fd, &resp, sizeof(resp)); 132 if (len != sizeof(resp)) 133 return -1; 134 135 return 0; 136 } 137 138There are now three types of messages introduced by VDUSE framework: 139 140- VDUSE_GET_VQ_STATE: Get the state for virtqueue, userspace should return 141 avail index for split virtqueue or the device/driver ring wrap counters and 142 the avail and used index for packed virtqueue. 143 144- VDUSE_SET_STATUS: Set the device status, userspace should follow 145 the virtio spec: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html 146 to process this message. For example, fail to set the FEATURES_OK device 147 status bit if the device can not accept the negotiated virtio features 148 get from the VDUSE_DEV_GET_FEATURES ioctl. 149 150- VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping for specified 151 IOVA range, userspace should firstly remove the old mapping, then setup the new 152 mapping via the VDUSE_IOTLB_GET_FD ioctl. 153 154After DRIVER_OK status bit is set via the VDUSE_SET_STATUS message, userspace is 155able to start the dataplane processing as follows: 156 1571. Get the specified virtqueue's information with the VDUSE_VQ_GET_INFO ioctl, 158 including the size, the IOVAs of descriptor table, available ring and used ring, 159 the state and the ready status. 160 1612. Pass the above IOVAs to the VDUSE_IOTLB_GET_FD ioctl so that those IOVA regions 162 can be mapped into userspace. Some sample codes is shown below: 163 164.. code-block:: c 165 166 static int perm_to_prot(uint8_t perm) 167 { 168 int prot = 0; 169 170 switch (perm) { 171 case VDUSE_ACCESS_WO: 172 prot |= PROT_WRITE; 173 break; 174 case VDUSE_ACCESS_RO: 175 prot |= PROT_READ; 176 break; 177 case VDUSE_ACCESS_RW: 178 prot |= PROT_READ | PROT_WRITE; 179 break; 180 } 181 182 return prot; 183 } 184 185 static void *iova_to_va(int dev_fd, uint64_t iova, uint64_t *len) 186 { 187 int fd; 188 void *addr; 189 size_t size; 190 struct vduse_iotlb_entry entry; 191 192 entry.start = iova; 193 entry.last = iova; 194 195 /* 196 * Find the first IOVA region that overlaps with the specified 197 * range [start, last] and return the corresponding file descriptor. 198 */ 199 fd = ioctl(dev_fd, VDUSE_IOTLB_GET_FD, &entry); 200 if (fd < 0) 201 return NULL; 202 203 size = entry.last - entry.start + 1; 204 *len = entry.last - iova + 1; 205 addr = mmap(0, size, perm_to_prot(entry.perm), MAP_SHARED, 206 fd, entry.offset); 207 close(fd); 208 if (addr == MAP_FAILED) 209 return NULL; 210 211 /* 212 * Using some data structures such as linked list to store 213 * the iotlb mapping. The munmap(2) should be called for the 214 * cached mapping when the corresponding VDUSE_UPDATE_IOTLB 215 * message is received or the device is reset. 216 */ 217 218 return addr + iova - entry.start; 219 } 220 2213. Setup the kick eventfd for the specified virtqueues with the VDUSE_VQ_SETUP_KICKFD 222 ioctl. The kick eventfd is used by VDUSE kernel module to notify userspace to 223 consume the available ring. This is optional since userspace can choose to poll the 224 available ring instead. 225 2264. Listen to the kick eventfd (optional) and consume the available ring. The buffer 227 described by the descriptors in the descriptor table should be also mapped into 228 userspace via the VDUSE_IOTLB_GET_FD ioctl before accessing. 229 2305. Inject an interrupt for specific virtqueue with the VDUSE_INJECT_VQ_IRQ ioctl 231 after the used ring is filled. 232 233For more details on the uAPI, please see include/uapi/linux/vduse.h. 234