1# Debugging protected VMs
2
3AVF is largely about protected VMs. This in turn means that anything that is
4happening inside the VM cannot be observed from outside of the VM. But as a
5developer, you need to be able to look into it when there’s an error in your
6VM. To satisfy such contradictory needs, AVF allows you to start a protected VM
7in a debuggable mode and provides a bunch of debugging mechanisms you can use
8to better understand the behavior of the VM and diagnose issues.
9
10Note: running a protected VM in a debuggable mode introduces many loopholes
11which can be used to nullify the protection provided by the hypervisor.
12Therefore, the debugable mode should never be used in production.
13
14## Enable debugging
15
16The following sections describe the two ways debugging can be enabled.
17
18### Debug level
19
20Debug level is a per-VM property which indicates how debuggable the VM is.
21There currently are two levels defined: NONE and FULL. NONE means that the VM
22is not debuggable at all, and FULL means that [all the debugging
23features](#debugging-features) are supported.
24
25Debug level is by default NONE. You can set it to FULL either via a Java API
26call in your app or via a command line argument `--debug` as follows:
27
28```java
29VirtualMachineConfig.Builder.setDebugLevel(DEBUG_LEVEL_FULL);
30```
31
32or
33
34```shell
35adb shell /apex/com.android.virt/bin/vm run-microdroid --debug full
36```
37
38or
39
40```shell
41m vm_shell
42vm_shell start-microdroid --auto-connect -- --protected --debug full
43```
44
45Note: `--debug full` is the default option when omitted. You need to explicitly
46use `--debug none` to set the debug level to NONE.
47
48### Dump device tree
49
50The VMs device tree can be dumped on creation by adding the `--dump_device_tree`
51argument and passing a path where the device tree gets dumped to, as follows:
52
53```shell
54adb shell /apex/com.android.virt/bin/vm run-microdroid --dump-device-tree PATH
55```
56
57Note: you can set the system property
58`hypervisor.virtualizationmanager.dump_device_tree` to true to always dump the
59device tree to `/data/misc/virtualizationservice/$CID/device_tree.dtb` where
60$CID is the CID of the VM. To set the property, run:
61
62```shell
63adb root
64adb shell setprop hypervisor.virtualizationmanager.dump_device_tree true
65```
66
67### Debug policy
68
69Debug policy is a per-device property which forcibly enables selected debugging
70features, even for the VMs with debug level NONE.
71
72The main purpose of debug policy is in-field debugging by the platform
73developers (device makers, SoC vendors, etc.) To understand it, let’s imagine
74that you have an application of pVM. It’s configured as debug level NONE
75because you finished the development and the team-level testing. However, you
76get a bug report from your QA team or from beta testers. To fix the bug, you
77should be able to look into the pVM but you do not want to change the source
78code to make the VM debuggable and rebuild the entire software, because that
79may hurt the reproducibility of the bug.
80
81Note: Not every devices is guaranteed to support debug policy. It is up to the
82device manufacturer to implement this in their bootloader. Google Pixel
83devices for example support this after Pixel 7 and 7 Pro. Pixel 6 and 6 Pro
84don't support debug policy.
85
86In the Pixel phones supporting debug policy, it is provisioned by installing a
87device tree overlay like below to the Pixel-specific partition `dpm`.
88
89```
90/ {
91    fragment@avf {
92        target-path = "/";
93
94        __overlay__ {
95            avf {
96                guest {
97                    common {
98                        log = <1>; // Enable kernel log and logcat
99                        ramdump = <1>; // Enable ramdump
100                    }
101                    microdroid {
102                        adb = <1>; // Enable ADB connection
103                    }
104                }
105            };
106        };
107    };
108}; /* end of avf */
109```
110
111To not enable a specific debugging feature, set the corresponding property
112value to other than `<1>`, or delete the property.
113
114As a reference, in Pixel phones, debug policy is loaded as below:
115
1161. Bootloader loads it from the `dpm` partition and verifies it.
1171. Bootloader appends the loaded debug policy as the [configuration
118   data](../../guest/pvmfw/README.md#configuration-data) of the pvmfw.
1191. When a pVM is started, pvmfw [overlays][apply_debug_policy] the debug policy to the baseline
120   device tree from crosvm.
1211. OS payload (e.g. Microdroid) [reads][read_debug_policy] the device tree and enables specific
122   debugging feature accordingly.
123
124**Note**: Bootloader MUST NOT load debug policy when the bootloader is in LOCKED state.
125
126[apply_debug_policy]: https://cs.android.com/android/platform/superproject/main/+/main:packages/modules/Virtualization/pvmfw/src/fdt.rs;drc=0d52747770baa14d44c0779b5505095b4251f2e9;l=790
127[read_debug_policy]: https://cs.android.com/android/platform/superproject/main/+/main:packages/modules/Virtualization/microdroid_manager/src/main.rs;drc=65c9f1f0eee4375535f2025584646a0dbb0ea25c;l=834
128
129## Debugging features
130
131AVF currently supports the following debugging features:
132
133* ADB connection (only for Microdroid)
134* Capturing console output
135* Capturing logcat output (only for Microdroid)
136* [Capturing kernel ramdump](ramdump.md) (only for Microdroid)
137* Capturing userspace crash dump (only for Microdroid)
138* [Attaching GDB to the kernel](gdb_kernel.md)
139* [Attaching GDB to the userspace process](gdb_userspace.md) (only for Microdroid)
140* [Tracing hypervisor events](tracing.md)
141