1LTP KVM Test API 2================ 3 4Testing KVM is more complex than other Linux features. Some KVM bugs allow 5userspace code running inside the virtual machine to bypass (emulated) hardware 6access restrictions and elevate its privileges inside the guest operating 7system. The worst types of KVM bugs may even allow the guest code to crash or 8compromise the physical host. KVM tests therefore need to be split into two 9components – a KVM controller program running on the physical host and a guest 10payload program running inside the VM. The cooperation of these two components 11allows testing even of bugs that somehow cross the virtualization boundary. 12 13NOTE: See also 14 https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[Test Writing Guidelines], 15 https://github.com/linux-test-project/ltp/wiki/C-Test-Case-Tutorial[C Test Case Tutorial], 16 https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API]. 17 181. Basic KVM test structure 19--------------------------- 20 21KVM tests are simple C source files containing both the KVM controller code 22and the guest payload code separated by `#ifdef COMPILE_PAYLOAD` preprocessor 23condition. The file will be compiled twice. Once to compile the payload part, 24once to compile the KVM controller part and embed the payload binary inside. 25The result is a single self-contained binary that'll execute the embedded 26payload inside a KVM virtual machine and print results in the same format as 27a normal LTP test. 28 29A KVM test source should start with `#include "kvm_test.h"` instead of the 30usual `tst_test.h`. The `kvm_test.h` header file will include the other basic 31headers appropriate for the current compilation pass. Everything else in the 32source file should be enclosed in `#ifdef COMPILE_PAYLOAD ... #else ... #endif` 33condition, including any other header file includes. Note that the standard 34LTP headers are not available in the payload compilation pass, only the KVM 35guest library headers can be included. 36 37.Example KVM test 38[source,c] 39------------------------------------------------------------------------------- 40#include "kvm_test.h" 41 42#ifdef COMPILE_PAYLOAD 43 44/* Guest payload code */ 45 46void main(void) 47{ 48 tst_res(TPASS, "Hello, world!"); 49} 50 51#else /* COMPILE_PAYLOAD */ 52 53/* KVM controller code */ 54 55static struct tst_test test = { 56 .test_all = tst_kvm_run, 57 .setup = tst_kvm_setup, 58 .cleanup = tst_kvm_cleanup, 59}; 60 61#endif /* COMPILE_PAYLOAD */ 62------------------------------------------------------------------------------- 63 64The KVM controller code is a normal LTP test and needs to define an instance 65of `struct tst_test` with metadata and the usual setup, cleanup, and test 66functions. Basic implementation of all three functions is provided by the KVM 67host library. 68 69On the other hand, the payload is essentially a tiny kernel that'll run 70on bare virtual hardware. It cannot access any files, Linux syscalls, standard 71library functions, etc. except for the small subset provided by the KVM guest 72library. The payload code must define a `void main(void)` function which will 73be the VM entry point of the test. 74 752. KVM host library 76------------------- 77 78The KVM host library provides helper functions for creating and running 79a minimal KVM virtual machine. 80 812.1 Data structures 82~~~~~~~~~~~~~~~~~~~ 83 84[source,c] 85------------------------------------------------------------------------------- 86struct tst_kvm_instance { 87 int vm_fd, vcpu_fd; 88 struct kvm_run *vcpu_info; 89 size_t vcpu_info_size; 90 struct kvm_userspace_memory_region ram[MAX_KVM_MEMSLOTS]; 91 struct tst_kvm_result *result; 92}; 93------------------------------------------------------------------------------- 94 95`struct tst_kvm_instance` holds the file descriptors and memory buffers 96of a single KVM virtual machine: 97 98- `int vm_fd` is the main VM file descriptor created by `ioctl(KVM_CREATE_VM)` 99- `int vcpu_fd` is the virtual CPU filedescriptor created by 100 `ioctl(KVM_CREATE_VCPU)` 101- `struct kvm_run *vcpu_info` is the VCPU state structure created by 102 `mmap(vcpu_fd)` 103- `size_t vcpu_info_size` is the size of `vcpu_info` buffer 104- `struct kvm_userspace_memory_region ram[MAX_KVM_MEMSLOTS]` is the list 105 of memory slots defined in this VM. Unused memory slots have zero 106 in the `userspace_addr` field. 107- `struct tst_kvm_result *result` is a buffer for passing test result data 108 from the VM to the controller program, mainly `tst_res()`/`tst_brk()` flags 109 and messages. 110 111[source,c] 112------------------------------------------------------------------------------- 113struct tst_kvm_result { 114 int32_t result; 115 int32_t lineno; 116 uint64_t file_addr; 117 char message[0]; 118}; 119------------------------------------------------------------------------------- 120 121`struct tst_kvm_result` is used to pass test results and synchronization data 122between the KVM guest and the controller program. Most often, it is used 123to pass `tst_res()` and `tst_brk()` messages from the VM, but special values 124can also be used to send control flow requests both ways. 125 126- `int32_t result` is the message type, either one of the `TPASS`, `TFAIL`, 127 `TWARN`, `TBROK`, `TINFO` flags or a special control flow value. Errno flags 128 are not supported. 129- `int32_t lineno` is the line number for `tst_res()`/`tst_brk()` messages. 130- `uint64_t file_addr` is the VM address of the filename string for 131 `tst_res()`/`tst_brk()` messages. 132- `char message[0]` is the buffer for arbitrary message data, most often used 133 to pass `tst_res()`/`tst_brk()` message strings. 134 1352.2 Working with virtual machines 136~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 137 138The KVM host library provides default implementation of the setup, cleanup 139and test functions for `struct tst_test` in cases where you do not need 140to customize the VM configuration. You can either assign these functions 141to the `struct tst_test` instance directly or call them from your own function 142that does some additional steps. All three function must be used together. 143 144- `void tst_kvm_setup(void)` 145- `void tst_kvm_run(void)` 146- `void tst_kvm_cleanup(void)` 147 148Note: `tst_kvm_run()` calls `tst_free_all()`. Calling it will free all 149previously allocated guarded buffers. 150 151- `void tst_kvm_validate_result(int value)` – Validate whether the value 152 returned in `struct tst_kvm_result.result` can be safely passed 153 to `tst_res()` or `tst_brk()`. If the value is not valid, the controller 154 program will be terminated with error. 155 156- `uint64_t tst_kvm_get_phys_address(const struct tst_kvm_instance *inst, 157 uint64_t addr)` – Converts pointer value (virtual address) from KVM virtual 158 machine `inst` to the corresponding physical address. Returns 0 if 159 the virtual address is not mapped to any physical address. If virtual memory 160 mapping is not enabled in the VM or not available on the arch at all, this 161 function simply returns `addr` as is. 162 163- `int tst_kvm_find_phys_memslot(const struct tst_kvm_instance *inst, 164 uint64_t paddr)` – Returns index of the memory slot in KVM virtual machine 165 `inst` which contains the physical address `paddr`. If the address is not 166 backed by a memory buffer, returns -1. 167 168- `int tst_kvm_find_memslot(const struct tst_kvm_instance *inst, 169 uint64_t addr)` – Returns index of the memory slot in KVM virtual machine 170 `inst` which contains the virtual address `addr`. If the virtual address 171 is not mapped to a valid physical address backed by a memory buffer, 172 returns -1. 173 174- `void *tst_kvm_get_memptr(const struct tst_kvm_instance *inst, 175 uint64_t addr)` – Converts pointer value (virtual address) from KVM virtual 176 machine `inst` to host-side pointer. 177 178- `void *tst_kvm_alloc_memory(struct tst_kvm_instance *inst, unsigned int slot, 179 uint64_t baseaddr, size_t size, unsigned int flags)` – Allocates a guarded 180 buffer of given `size` in bytes and installs it into specified memory `slot` 181 of the KVM virtual machine `inst` at base address `baseaddr`. The buffer 182 will be automatically page aligned at both ends. See the kernel 183 documentation of `KVM_SET_USER_MEMORY_REGION` ioctl for list of valid 184 `flags`. Returns pointer to page-aligned beginning of the allocated buffer. 185 The actual requested `baseaddr` will be located at 186 `ret + baseaddr % pagesize`. 187 188- `struct kvm_cpuid2 *tst_kvm_get_cpuid(int sysfd)` – Get a list of supported 189 virtual CPU features returned by `ioctl(KVM_GET_SUPPORTED_CPUID)`. 190 The argument must be an open file descriptor returned by `open("/dev/kvm")`. 191 192- `void tst_kvm_create_instance(struct tst_kvm_instance *inst, 193 size_t ram_size)` – Creates and fully initializes a new KVM virtual machine 194 with at least `ram_size` bytes of memory. The VM instance info will be 195 stored in `inst`. 196 197- `int tst_kvm_run_instance(struct tst_kvm_instance *inst, int exp_errno)` – 198 Executes the program installed in KVM virtual machine `inst`. Any result 199 messages returned by the VM will be automatically printed to controller 200 program output. Returns zero. If `exp_errno` is non-zero, the VM execution 201 syscall is allowed to fail with the `exp_errno` error code and 202 `tst_kvm_run_instance()` will return -1 instead of terminating the test. 203 204- `void tst_kvm_destroy_instance(struct tst_kvm_instance *inst)` – Deletes 205 the KVM virtual machine `inst`. Note that the guarded buffers assigned 206 to the VM by `tst_kvm_create_instance()` or `tst_kvm_alloc_memory()` will 207 not be freed. 208 209The KVM host library does not provide any way to reset a VM instance back 210to initial state. Running multiple iterations of the test requires destroying 211the old VM instance and creating a new one. Otherwise the VM will exit 212without reporting any results on the second iteration and the test will fail. 213The `tst_kvm_run()` function handles this issue correctly. 214 2153. KVM guest library 216-------------------- 217 218The KVM guest library provides a minimal implementation of both the LTP 219test library and the standard C library functions. Do not try to include 220the usual LTP or C headers in guest payload code, it will not work. 221 2223.1 Standard C functions 223~~~~~~~~~~~~~~~~~~~~~~~~ 224 225`#include "kvm_test.h"` 226 227The functions listed below are implemented according to the C standard: 228 229- `void *memset(void *dest, int val, size_t size)` 230- `void *memzero(void *dest, size_t size)` 231- `void *memcpy(void *dest, const void *src, size_t size)` 232- `char *strcpy(char *dest, const char *src)` 233- `char *strcat(char *dest, const char *src)` 234- `size_t strlen(const char *str)` 235 2363.2 LTP library functions 237~~~~~~~~~~~~~~~~~~~~~~~~~ 238 239`#include "kvm_test.h"` 240 241The KVM guest library currently provides the LTP functions for reporting test 242results. All standard result flags except for `T*ERRNO` are supported 243with the same rules as usual. However, the printf-like formatting is not 244implemented yet. 245 246- `void tst_res(int result, const char *message)` 247- `void tst_brk(int result, const char *message) __attribute__((noreturn))` 248 249A handful of useful macros is also available: 250 251- `TST_TEST_TCONF(message)` – Generates a test program that will simply print 252 a `TCONF` message and exit. This is useful when the real test cannot be 253 built due to missing dependencies or arch limitations. 254 255- `ARRAY_SIZE(arr)` – Returns the number of items in statically allocated 256 array `arr`. 257 258- `LTP_ALIGN(x, a)` – Aligns integer `x` to be a multiple of `a`, which 259 must be a power of 2. 260 2613.3 Arch independent functions 262~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 263 264`#include "kvm_test.h"` 265 266Memory management in KVM guest library currently uses only primitive linear 267buffer for memory allocation. There are no checks whether the VM can allocate 268more memory and the already allocated memory cannot be freed. 269 270- `void *tst_heap_alloc(size_t size)` – Allocates a block of memory on the heap. 271 272- `void *tst_heap_alloc_aligned(size_t size, size_t align)` – Allocates 273 a block of memory on the heap with the starting address aligned to given 274 value. 275 2763.4 x86 specific functions 277~~~~~~~~~~~~~~~~~~~~~~~~~~ 278 279`#include "kvm_test.h"` + 280`#include "kvm_x86.h"` 281 282- `struct kvm_interrupt_frame` – Opaque arch-dependent structure which holds 283 interrupt frame information. Use the functions below to get individual values: 284 285- `uintptr_t kvm_get_interrupt_ip(const struct kvm_interrupt_frame *ifrm)` – 286 Get instruction pointer value from interrupt frame structure. This may be 287 the instruction which caused an interrupt or the one immediately after, 288 depending on the interrupt vector semantics. 289 290- `int (*tst_interrupt_callback)(void *userdata, 291 struct kvm_interrupt_frame *ifrm, unsigned long errcode)` – Interrupt handler 292 callback prototype. When an interrupt occurs, the assigned callback function 293 will be passed the `userdata` pointer that was given 294 to `tst_set_interrupt_callback()`, interrupt frame `ifrm` and the error 295 code `errcode` defined by the interrupt vector semantics. If the interrupt 296 vector does not generate an error code, `errcode` will be set to zero. 297 The callback function must return 0 if the interrupt was successfully 298 handled and test execution should resume. Non-zero return value means that 299 the interrupt could not be handled and the test will terminate with error. 300 301- `void tst_set_interrupt_callback(unsigned int vector, 302 tst_interrupt_callback func, void *userdata)` – Register new interrupt 303 handler callback function `func` for interrupt `vector`. The `userdata` 304 argument is an arbitrary pointer that will be passed to `func()` every time 305 it gets called. The previous interrupt handler callback will be removed. 306 Setting `func` to `NULL` will remove any existing interrupt handler 307 from `vector` and the interrupt will become fatal error. 308 309[source,c] 310------------------------------------------------------------------------------- 311struct page_table_entry_pae { 312 unsigned int present: 1; 313 unsigned int writable: 1; 314 unsigned int user_access: 1; 315 unsigned int write_through: 1; 316 unsigned int disable_cache: 1; 317 unsigned int accessed: 1; 318 unsigned int dirty: 1; 319 unsigned int page_type: 1; 320 unsigned int global: 1; 321 unsigned int padding: 3; 322 uint64_t address: 40; 323 unsigned int padding2: 7; 324 unsigned int prot_key: 4; 325 unsigned int noexec: 1; 326} __attribute__((__packed__)); 327 328struct kvm_cpuid { 329 unsigned int eax, ebx, ecx, edx; 330}; 331 332struct kvm_cregs { 333 unsigned long cr0, cr2, cr3, cr4; 334}; 335 336struct kvm_sregs { 337 uint16_t cs, ds, es, fs, gs, ss; 338}; 339------------------------------------------------------------------------------- 340 341`struct page_table_entry_pae` is the page table entry structure for PAE and 34264bit paging modes. See Intel(R) 64 and IA-32 Architectures Software 343Developer's Manual, Volume 3, Chapter 4 for explanation of the fields. 344 345- `uintptr_t kvm_get_page_address_pae(const struct page_table_entry_pae *entry)` 346 – Returns the physical address of the memory page referenced by the given 347 page table `entry`. Depending on memory mapping changes done by the test, 348 the physical address may not be a valid pointer. The caller must determine 349 whether the address points to another page table entry or a data page, using 350 the known position in page table hierarchy and `entry->page_type`. Returns 351 zero if the `entry` does not reference any memory page. 352 353- `void kvm_set_segment_descriptor(struct segment_descriptor *dst, uint64_t baseaddr, uint32_t limit, unsigned int flags)` - 354 Fill the `dst` segment descriptor with given values. The maximum value 355 of `limit` is `0xfffff` (inclusive) regardless of `flags`. 356 357- `void kvm_parse_segment_descriptor(struct segment_descriptor *src, uint64_t *baseaddr, uint32_t *limit, unsigned int *flags)` - 358 Parse data in the `src` segment descriptor and copy them to variables 359 pointed to by the other arguments. Any parameter except the first one can 360 be `NULL`. 361 362- `int kvm_find_free_descriptor(const struct segment_descriptor *table, size_t size)` - 363 Find the first segment descriptor in `table` which does not have 364 the `SEGFLAG_PRESENT` bit set. The function handles double-size descriptors 365 correctly. Returns index of the first available descriptor or -1 if all 366 `size` descriptors are taken. 367 368- `unsigned int kvm_create_stack_descriptor(struct segment_descriptor *table, size_t tabsize, void *stack_base)` - 369 Convenience function for registering a stack segment descriptor. It'll 370 automatically find a free slot in `table` and fill the necessary flags. 371 The `stack_base` pointer must point to the bottom of the stack. 372 373- `void kvm_get_cpuid(unsigned int eax, unsigned int ecx, 374 struct kvm_cpuid *buf)` – Executes the CPUID instruction with the given 375 `eax` and `ecx` arguments and stores the results in `buf`. 376 377- `void kvm_read_cregs(struct kvm_cregs *buf)` – Copies the current values 378 of control registers to `buf`. 379 380- `void kvm_read_sregs(struct kvm_sregs *buf)` - Copies the current values 381 of segment registers to `buf`. 382 383- `uint64_t kvm_rdmsr(unsigned int msr)` – Returns the current value 384 of model-specific register `msr`. 385 386- `void kvm_wrmsr(unsigned int msr, uint64_t value)` – Stores `value` 387 into model-specific register `msr`. 388 389- `void kvm_exit(void) __attribute__((noreturn))` – Terminate the test. 390 Similar to calling `exit(0)` in a regular LTP test, although `kvm_exit()` 391 will terminate only one iteration of the test, not the whole host process. 392 393See Intel(R) 64 and IA-32 Architectures Software Developer's Manual 394for documentation of standard and model-specific x86 registers. 395 3963.5 AMD SVM helper functions 397~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 398 399`#include "kvm_test.h"` + 400`#include "kvm_x86.h"` + 401`#include "kvm_x86_svm.h"` 402 403The KVM guest library provides basic helper functions for creating and running 404nested virtual machines using the AMD SVM technology. 405 406.Example code to execute nested VM 407[source,c] 408------------------------------------------------------------------------------- 409int guest_main(void) 410{ 411 ... 412 return 0; 413} 414 415void main(void) 416{ 417 struct kvm_svm_vcpu *vm; 418 419 kvm_init_svm(); 420 vm = kvm_create_svm_vcpu(guest_main, 1); 421 kvm_svm_vmrun(vm); 422} 423------------------------------------------------------------------------------- 424 425- `int kvm_is_svm_supported(void)` - Returns non-zero value if the CPU 426 supports AMD SVM, otherwise returns 0. 427 428- `int kvm_get_svm_state(void)` - Returns non-zero value if SVM is currently 429 enabled, otherwise returns 0. 430 431- `void kvm_set_svm_state(int enabled)` - Enable or disable SVM according 432 to argument. If SVM is disabled by host or not supported, the test will exit 433 with `TCONF`. 434 435- `void kvm_init_svm(void)` - Enable and fully initialize SVM, including 436 allocating and setting up host save area VMCB. If SVM is disabled by host or 437 not supported, the test will exit with `TCONF`. 438 439- `struct kvm_vmcb *kvm_alloc_vmcb(void)` - Allocate new VMCB structure 440 with correct memory alignment and fill it with zeroes. 441 442- `void kvm_vmcb_set_intercept(struct kvm_vmcb *vmcb, unsigned int id, unsigned int state)` - 443 Set SVM intercept bit `id` to given `state`. 444 445- `void kvm_init_guest_vmcb(struct kvm_vmcb *vmcb, uint32_t asid, uint16_t ss, void *rsp, int (*guest_main)(void))` - 446 Initialize new SVM virtual machine. The `asid` parameter is the nested 447 page table ID. The `ss` and `rsp` parameters set the stack segment and stack 448 pointer values, respectively. The `guest_main` parameter sets the code entry 449 point of the virtual machine. All control registers, segment registers 450 (except stack segment register), GDTR and IDTR will be copied 451 from the current CPU state. 452 453- `struct kvm_svm_vcpu *kvm_create_svm_vcpu(int (*guest_main)(void), int alloc_stack)` - 454 Convenience function for allocating and initializing new SVM virtual CPU. 455 The `guest_main` parameter is passed to `kvm_init_guest_vmcb()`, 456 the `alloc_stack` parameter controls whether a new 8KB stack will be 457 allocated and registered in GDT. Interception will be enabled for `VMSAVE` 458 and `HLT` instructions. If you set `alloc_stack` to zero, you must configure 459 the stack segment register and stack pointer manually. 460 461- `void kvm_svm_vmrun(struct kvm_svm_vcpu *cpu)` - Start or continue execution 462 of a nested virtual machine. Beware that FPU state is not saved. Do not use 463 floating point types or values in nested guest code. Also do not use 464 `tst_res()` or `tst_brk()` functions in nested guest code. 465 466See AMD64 Architecture Programmer's Manual Volume 2 for documentation 467of the Secure Virtual Machine (SVM) technology. 468 4694. KVM guest environment 470------------------------ 471 472KVM guest payload execution begins with bootstrap code which will perform 473the minimal guest environment setup required for running C code: 474 475- Activate the appropriate CPU execution mode (IA-32 protected mode 476 on 32-bit x86 or the 64-bit mode on x86_64). 477- Create indentity mapping (virtual address = physical address) of the lower 478 2GB memory region, even if parts of the region are not backed by any host 479 memory buffers. The memory region above 2GB threshold is left unmapped 480 except for one memory page reserved for the `struct tst_kvm_result` buffer. 481- Initialize 8KB stack. 482- Install default interrupt handlers for standard CPU exception vectors. 483 484When the environment setup is complete, bootstrap will call `void main(void)` 485function implemented by the test program. To finish execution of guest payload, 486the test can either return from the `main()` function or call `kvm_exit()` 487at any point. 488