vhost-device-vsock 0.1.0

A virtio-vsock device using the vhost-user protocol.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178

# vhost-device-vsock

## Design

The crate introduces a vhost-device-vsock device that enables communication between an
application running in the guest i.e inside a VM and an application running on the
host i.e outside the VM. The application running in the guest communicates over VM
sockets i.e over AF_VSOCK sockets. The application running on the host connects to a
unix socket on the host i.e communicates over AF_UNIX sockets. The main components of
the crate are split into various files as described below:

- [packet.rs](src/packet.rs)
  - Introduces the **VsockPacket** structure that represents a single vsock packet
  processing methods.
- [rxops.rs](src/rxops.rs)
  - Introduces various vsock operations that are enqueued into the rxqueue to be sent to the
  guest. Exposes a **RxOps** structure.
- [rxqueue.rs](src/rxqueue.rs)
  - rxqueue contains the pending rx operations corresponding to that connection. The queue is
  represented as a bitmap as we handle connection-oriented connections. The module contains
  various queue manipulation methods. Exposes a **RxQueue** structure.
- [thread_backend.rs](src/thread_backend.rs)
  - Multiplexes connections between host and guest and calls into per connection methods that
  are responsible for processing data and packets corresponding to the connection. Exposes a
  **VsockThreadBackend** structure.
- [txbuf.rs](src/txbuf.rs)
  - Module to buffer data that is sent from the guest to the host. The module exposes a **LocalTxBuf**
  structure.
- [vhost_user_vsock_thread.rs](src/vhost_user_vsock_thread.rs)
  - Module exposes a **VhostUserVsockThread** structure. It also handles new host initiated
  connections and provides interfaces for registering host connections with the epoll fd. Also
  provides interfaces for iterating through the rx and tx queues.
- [vsock_conn.rs](src/vsock_conn.rs)
  - Module introduces a **VsockConnection** structure that represents a single vsock connection
  between the guest and the host. It also processes packets according to their type.
- [vhu_vsock.rs](src/vhu_vsock.rs)
  - exposes the main vhost user vsock backend interface.

## Usage

Run the vhost-device-vsock device:
```
vhost-device-vsock --guest-cid=<CID assigned to the guest> \
  --socket=<path to the Unix socket to be created to communicate with the VMM via the vhost-user protocol> \
  --uds-path=<path to the Unix socket to communicate with the guest via the virtio-vsock device> \
  [--tx-buffer-size=<size of the buffer used for the TX virtqueue (guest->host packets)>]
```
or
```
vhost-device-vsock --vm guest_cid=<CID assigned to the guest>,socket=<path to the Unix socket to be created to communicate with the VMM via the vhost-user protocol>,uds-path=<path to the Unix socket to communicate with the guest via the virtio-vsock device>[,tx-buffer-size=<size of the buffer used for the TX virtqueue (guest->host packets)>]
```

Specify the `--vm` argument multiple times to specify multiple devices like this:
```
vhost-device-vsock \
--vm guest-cid=3,socket=/tmp/vhost3.socket,uds-path=/tmp/vm3.vsock \
--vm guest-cid=4,socket=/tmp/vhost4.socket,uds-path=/tmp/vm4.vsock,tx-buffer-size=32768
```

Or use a configuration file:
```
vhost-device-vsock --config=<path to the local yaml configuration file>
```

Configuration file example:
```yaml
vms:
    - guest_cid: 3
      socket: /tmp/vhost3.socket
      uds_path: /tmp/vm3.sock
      tx_buffer_size: 65536
    - guest_cid: 4
      socket: /tmp/vhost4.socket
      uds_path: /tmp/vm4.sock
      tx_buffer_size: 32768
```

Run VMM (e.g. QEMU):

```
qemu-system-x86_64 \
  <normal QEMU options> \
  -object memory-backend-file,share=on,id=mem0,size=<Guest RAM size>,mem-path=<Guest RAM file path> \ # size == -m size
  -machine <machine options>,memory-backend=mem0 \
  -chardev socket,id=char0,reconnect=0,path=<vhost-user socket path> \
  -device vhost-user-vsock-pci,chardev=char0
```

## Working example

```sh
shell1$ vhost-device-vsock --vm guest-cid=4,uds-path=/tmp/vm4.vsock,socket=/tmp/vhost4.socket
```
or if you want to configure the TX buffer size
```sh
shell1$ vhost-device-vsock --vm guest-cid=4,uds-path=/tmp/vm4.vsock,socket=/tmp/vhost4.socket,tx-buffer-size=65536
```

```sh
shell2$ qemu-system-x86_64 \
          -drive file=vm.qcow2,format=qcow2,if=virtio -smp 2 -m 512M -mem-prealloc \
          -object memory-backend-file,share=on,id=mem0,size=512M,mem-path="/dev/hugepages" \
          -machine q35,accel=kvm,memory-backend=mem0 \
          -chardev socket,id=char0,reconnect=0,path=/tmp/vhost4.socket \
          -device vhost-user-vsock-pci,chardev=char0
```

### Guest listening

#### iperf

```sh
# https://github.com/stefano-garzarella/iperf-vsock
guest$ iperf3 --vsock -s
host$  iperf3 --vsock -c /tmp/vm4.vsock
```

#### netcat

```sh
guest$ nc --vsock -l 1234

host$  nc -U /tmp/vm4.vsock
CONNECT 1234
```

### Host listening

#### iperf

```sh
# https://github.com/stefano-garzarella/iperf-vsock
host$  iperf3 --vsock -s -B /tmp/vm4.vsock
guest$ iperf3 --vsock -c 2
```

#### netcat

```sh
host$ nc -l -U /tmp/vm4.vsock_1234

guest$ nc --vsock 2 1234
```

### Sibling VM communication

If you add multiple VMs, they can communicate with each other. For example, if you have two VMs with
CID 3 and 4, you can run the following commands to make them communicate:

```sh
shell1$ vhost-device-vsock --vm guest-cid=3,uds-path=/tmp/vm3.vsock,socket=/tmp/vhost3.socket \
          --vm guest-cid=4,uds-path=/tmp/vm4.vsock,socket=/tmp/vhost4.socket
shell2$ qemu-system-x86_64 \
          -drive file=vm1.qcow2,format=qcow2,if=virtio -smp 2 -m 512M -mem-prealloc \
          -object memory-backend-file,share=on,id=mem0,size=512M,mem-path="/dev/hugepages" \
          -machine q35,accel=kvm,memory-backend=mem0 \
          -chardev socket,id=char0,reconnect=0,path=/tmp/vhost3.socket \
          -device vhost-user-vsock-pci,chardev=char0
shell3$ qemu-system-x86_64 \
          -drive file=vm2.qcow2,format=qcow2,if=virtio -smp 2 -m 512M -mem-prealloc \
          -object memory-backend-file,share=on,id=mem0,size=512M,mem-path="/dev/hugepages2" \
          -machine q35,accel=kvm,memory-backend=mem0 \
          -chardev socket,id=char0,reconnect=0,path=/tmp/vhost4.socket \
          -device vhost-user-vsock-pci,chardev=char0
```

```sh
# nc-vsock patched to set `.svm_flags = VMADDR_FLAG_TO_HOST`
guest_cid3$ nc-vsock -l 1234
guest_cid4$ nc-vsock 3 1234
```

## License

This project is licensed under either of

- [Apache License](http://www.apache.org/licenses/LICENSE-2.0), Version 2.0
- [BSD-3-Clause License](https://opensource.org/licenses/BSD-3-Clause)