1# Dittosuite 2 3Dittosuite is a collection of tools that simplifies writing and running complex 4workloads. 5 6These workloads are defined by a set of instructions including, but not limited 7to, CPU load, memory allocations, file system operations, Binder IPC, that can 8be combined to run sequentially or in parallel. 9 10A new high-level language, Dittolang, defines the workload in the form of a 11textual representation of Protocol Buffers (aka .ditto files). 12 13The operations defined in the .ditto files are interpreted and executed by 14Dittobench on the target device. 15 16## Benchmarking 17 18During the Dittobench execution of the instructions defined in the .ditto file, 19the tool tracks performance metrics such as execution time and I/O bandwidth 20(more in the [Sampling](#sampling) section). 21 22# Doxygen documentation 23 24In addition to this readme, the project includes a Doxygen configuration, which 25can be generated with the following command: 26 27``` 28$ doxygen 29``` 30 31# How to run 32 33``` 34$ ./dittobench [options] [.ditto file] 35``` 36 37To run a benchmark, a well formed .ditto file must be provided, see section 38[How to write .ditto files](#writing-ditto-files) 39In addition, these options can be set: 40 41- `--results-output=<string>` (default: report). Select the results output 42 format. Options: report, csv. 43- `--log-stream=<string>` (default: stdout). Select the output stream for 44 the log messages. Options: stdout, logcat. 45- `--log-level=<string>` (default: INFO). Select to output messages which are 46 at or below the set level. Options: VERBOSE, DEBUG, INFO , WARNING, ERROR, 47 FATAL. 48- `--parameters=string`. If the benchmark takes parameters, they can be passed 49 through this option, separated by commas. 50 51# Writing .ditto files {#writing-ditto-files} 52 53## Sections 54 55Every .ditto file should begin with this skeleton: 56 57``` 58main: {}, 59global {} 60``` 61 62Optionally, it can contain `init` and `clean_up` sections: 63 64``` 65main: {}, 66init: {}, 67clean_up: {}, 68global {} 69``` 70 71### `global` 72 73The `global` section contains general benchmark parameters. 74Available options: 75 76- (optional) `string absolute_path` (`default = ""`). Specifies the absolute 77 path for files created by benchmarks. This parameter simplifies the 78 definition of paths among different file system `Instruction`s, for example, 79 when different benchmarks should be run in different paths with different 80 file systems. 81 82### `main` 83 84`main` is the entry point for the benchmark. 85 86It contains a single `instruction`. 87 88Having a single instruction does not mean that the tool is limited to one 89instruction. In fact, as will be explained later, multiple instances of the 90same instruction can be executed with the parameter `repeat`, or a series of 91different instructions can be execute using the special instruction 92`instruction_set`. 93 94### `init` (optional) 95 96`init` initializes the benchmarking environment. 97 98It executes instructions similar to `main`, but the results are not collected 99at the end of the execution. 100 101### `clean_up` (optional) 102 103`clean_up` is optional and can be used to reset the benchmarking environment to 104the initial state, e.g, delete benchmark files. Similar to `init`, it executes 105instructions like `main`, but results are not collected in the end. 106 107## Instructions 108 109Every workload is composed of one or more instructions. 110 111Almost everything in Ditto `main` itself is an instruction! 112 113Here is an example of a .ditto file. 114 115``` 116main: { 117 instruction_set: { 118 instructions: [ 119 { 120 open_file: { 121 path_name: "newfile2.txt", 122 output_fd: "test_file" 123 } 124 }, 125 { 126 close_file: { 127 input_fd: "test_file" 128 } 129 } 130 ] 131 }, 132 repeat: 10 133}, 134global { 135 absolute_path: "/data/local/tmp/"; 136} 137``` 138 139See more examples in `example/`. 140 141### `instruction` 142 143``` 144{ 145 <name of the instruction>: { 146 <first argument>, 147 <second argument>, 148 ... 149 }, 150 <general instruction options> 151} 152``` 153 154Currently available options: 155 156- (optional) `int repeat` (`default = 1`). Specifies how many times the 157 instruction should be repeated. 158 159### `instruction_set` 160 161``` 162{ 163 instruction_set: { 164 instructions: { 165 { 166 <name of the first instruction>: { 167 <first argument>, 168 <second argument>, 169 ... 170 }, 171 <general instruction options> 172 }, 173 { 174 <name of the second instruction>: { 175 <first argument>, 176 <second argument>, 177 ... 178 }, 179 <general instruction options> 180 }, 181 ... 182 }, 183 iterate_options: {...} 184 }, 185 <general instruction options> 186} 187``` 188 189Instruction set is an Instruction container that executes the contained 190instructions sequentially. 191 192Instruction set can optionally iterate over a list and execute the provided set 193of instructions on each item from the list. To use it, `iterate_options` should 194be set with these options: 195 196- `string list_name` - Shared variable name pointing to a list of values. 197- `string item_name` - Shared variable name to which a selected value should be 198 stored. 199- (optional) `Order order` (`default = SEQUENTIAL`) - Specifies if 200 the elements of the list should be accessed sequentially or randomly. Options: 201 `SEQUENTIAL`, `RANDOM`. 202- (optional) `Reseeding reseeding` (`default = ONCE`) - Specifies how often the 203 random number generator should be reseeded with the same provided (or 204 generated) seed. Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 205- (optional) `uint32 seed` - Seed for the random number generator. If the seed 206 is not provided, 207 current system time is used as the seed. 208 209### `multithreading` 210 211``` 212multithreading: { 213 threads: [ 214 { 215 instruction: {...}, 216 spawn: <number of threads to spawn with the provided instruction> 217 }, 218 ... 219 ] 220} 221``` 222 223Multithreading is another instruction container that executes the specified 224instructions (or instruction sets) in different threads. If the optional 225`spawn` option for a specific instruction (or instruction set) is provided, 226then the provided number of threads will be created for it. 227 228Arguments: 229 230- `Thread threads`: an array of `Thread`s that will be executed in parallel. 231 Each `Thread` specifies: 232 - the `Instruction` to run. 233 - (optional, default 1) `int32 spawn`: number of threads/processes to be 234 created for this instruction. 235 - (optional) `string name`: alias name to be assigned to the thread/process. 236 - (optional) `SchedAttr sched_attr`: scheduling parameters. 237 - (optional, default -1) `int64 sched_affinity`: bitmask that defines what 238 CPUs the thread/process can run on. 239- (optional, default false) `bool fork`: if true, creates processes, otherwise 240 creates threads. 241 242### `open_file` 243 244Opens the file specified by the given path or by a shared variable name 245pointing to a file path. If neither of those are provided, a random 9-digit 246name is generated. Optionally saves the file descriptor which can then be used 247by subsequent instructions. Also, can optionally create the file if it does not 248already exist. 249 250Arguments: 251 252- (optional) `string path_name` - Specifies the file path. 253- (OR, optional) `string input` - Shared variable name pointing to a file path. 254- (optional) `string output_fd` - Shared variable name to which output file 255 descriptor should be saved. 256- (optional) `bool create` (`default = true`) - Specifies if the file should be 257 created if it does not already exist. If the file exists, nothing happens. 258 259### `delete_file` 260 261Deletes the file with a file path or a shared variable name pointing to a file 262path. Uses `unlink(2)`. 263 264Arguments: 265 266- `string path_name` - Specifies the file path.<br/> 267 OR<br/> 268 `string input` - Shared variable name pointing to a file path. 269 270### `close_file` 271 272Closes the file with the provided file descriptor. 273Uses `close(2)`. 274 275Arguments: 276 277- `string input_fd` - Shared variable name pointing to a file descriptor. 278 279### `resize_file` 280 281Resizes the file with the provided file descriptor and new size. If the 282provided size is greater than the current file size, `fallocate(2)` is used, 283otherwise `ftruncate(2)` is used. 284 285Arguments: 286 287- `string input_fd` - Shared variable name pointing to a file descriptor. 288- `int64 size` - New file size (in bytes). 289 290### `resize_file_random` 291 292Resizes the file with the provided file descriptor and a range for the new 293size. The new file size is randomly generated in the provided range and if the 294generated size is greater than the current file size, `fallocate(2)` is used, 295otherwise `ftruncate(2)` is used. 296 297Arguments: 298 299- `string input_fd` - Shared variable name pointing to a file descriptor. 300- `int64 min` - Minimum value (in bytes) 301- `int64 max` - Maximum value (in bytes) 302- (optional) `uint32 seed` - Seed for the random number generator. If the seed 303 is not provided, current system time is used as the seed. 304- (optional) `Reseeding reseeding` (`default = ONCE`). How often the random 305 number generator should be reseeded with the provided (or generated) seed. 306 Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 307 308### `write_file` 309 310Writes to file with the provided file descriptor. For `SEQUENTIAL` access, the 311blocks of data will be written sequentially and if the end of the file is 312reached, new blocks will start from the beginning of the file. For `RANDOM` 313access, the block offset, to which data should be written, will be randomly 314chosen with uniform distribution. `10101010` byte is used for the write 315operation to fill the memory with alternating ones and zeroes. Uses 316`pwrite64(2)`. 317 318Arguments: 319 320- `string input_fd` - Shared variable name pointing to a file descriptor. 321- (optional) `int64 size` (`default = -1`) - How much data (in bytes) should be 322 written in total. If it is set to `-1`, then file size is used. 323- (optional) `int64 block_size` (`default = 4096`) - How much data (in bytes) 324 should be written at once. If it is set to `-1`, then file size is used. 325- (optional) `int64 starting_offset` (`default = 0`) - If `access_order` is set 326 to `SEQUENTIAL`, then the blocks, to which the data should be written, will 327 start from this starting offset (in bytes). 328- (optional) `Order access_order` (`default = SEQUENTIAL`) - Order of the 329 write. Options: `SEQUENTIAL` and `RANDOM`. 330- (optional) `uint32 seed` - Seed for the random number generator. If the seed 331 is not provided, current system time is used as the seed. 332- (optional) `bool fsync` (`default = false`) - If set, `fsync(2)` will be 333 called after the execution of all write operations. 334- (optional) `Reseeding reseeding` (`default = ONCE`) - How often the random 335 number generator should be reseeded with the provided (or generated) seed. 336 Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 337 338### `read_file` 339 340Reads from file with the provided file descriptor. For `SEQUENTIAL` 341access, the blocks of data will be read sequentially and if the end of 342the file is reached, new blocks will start from the beginning of the file. For 343`RANDOM` access, the block offset, from which data should be read, will 344be randomly chosen with uniform distribution. Calls `posix_fadvise(2)` before 345the read operations. Uses `pread64(2)`. 346 347Arguments: 348 349- `string input_fd` - Shared variable name pointing to a file descriptor. 350- (optional) `int64 size` (`default = -1`) - How much data (in bytes) should be 351 read in total. If it is set to `-1`, then file size is used. 352- (optional) `int64 block_size` (`default = 4096`) - How much data (in bytes) 353 should be read at once. If it is set to `-1`, then file size is used. 354- (optional) `int64 starting_offset` (`default = 0`) - If `access_order` is set 355 to `SEQUENTIAL`, then the blocks, from which the data should be read, will 356 start from this starting offset (in bytes). 357- (optional) `Order access_order` (`default = SEQUENTIAL`) - Order of the read. 358 Options: `SEQUENTIAL` and `RANDOM`. 359- (optional) `uint32 seed` - Seed for the random number generator. If the seed 360 is not provided, current system time is used as the seed. 361- (optional) `ReadFAdvise fadvise` (`default = AUTOMATIC`) - Sets the argument 362 for the `posix_fadvise(2)` operation. Options: `AUTOMATIC`, `NORMAL`, 363 `SEQUENTIAL` and `RANDOM`. If `AUTOMATIC` is set, then 364 `POSIX_FADV_SEQUENTIAL` or `POSIX_FADV_RANDOM` will be used for `SEQUENTIAL` 365 and `RANDOM` access order respectively. 366- (optional) `Reseeding reseeding` (`default = ONCE`) - How often the random 367 number generator should be reseeded with the provided (or generated) seed. 368 Options: `ONCE`, `EACH_ROUND_OF_CYCLES`, `EACH_CYCLE`. 369 370### `read_directory` 371 372Reads file names from a directory and stores them as a list in a shared 373variable. Uses `readdir(3)`. 374 375Arguments: 376 377- `string directory_name` - Name of the directory 378- `string output` - Shared variable name to which files names should be saved. 379 380### `invalidate_cache` 381 382Drops kernel caches, including dentry, inode and page caches. This is done by 383calling `sync()` and then writing `3` to `/proc/sys/vm/drop_caches`. 384 385# Sampling {#sampling} 386 387TODO 388 389# Dependencies 390 391## Android 392 393The project is currently being developed as part of the Android Open Source 394Project (AOSP) and is supposed to run out-of-the-box. 395 396## Linux 397 398The following utilities are required to build the project on Linux: 399 400``` 401sudo apt install cmake protobuf-compiler 402 403``` 404 405# Testing 406 407## Linux 408 409A suite of unit tests is provided in the test/ directory. 410In Linux these tests can be run with the following commands: 411 412``` 413mkdir build 414cd build 415make 416cd test 417ctest 418``` 419 420### Coverage 421 422As an extension to testing, coverage is not a metric that guarantees good 423quality testing, but at least shows what is not been tested yet. 424 425One way of getting coverage data is to rely on `llvm` to build the code with 426extra flags to generate coverage information, `llvm-cov` to extract coverage 427data, and `lcov` to aggregate and export all the coverage information into a 428human-readable format. 429 430``` 431mkdir build 432cd build 433CC=/usr/bin/clang CXX=/usr/bin/clang++ cmake -DCMAKE_BUILD_TYPE=Debug .. 434make 435ctest --test-dir test 436lcov -d ./CMakeFiles/ -b . --gcov-tool $PWD/../test/llvm-gcov.sh --capture -o cov.info 437genhtml cov.info -o coverage_html 438``` 439 440> **_NOTE:_**: `lcov` version `2.0-1` has issues such as `geninfo: ERROR: "XXX: 441> function YYY found on line but no corresponding 'line' coverage data point. 442> Cannot derive function end line.` This can be solved by downgrading to 443> version 1.6. The lcov repository already has a binary, so `PATH` can be 444> updated with its `bin` folder. 445 446