1# How to write tests using FileCheck 2 3## What is FileCheck 4 5FileCheck can be seen as an advanced version of grep. We use it for writing 6small annotated unit tests for optimization passes. FileCheck used in PyTorch is 7inspired by [LLVM FileCheck 8Tool](https://llvm.org/docs/CommandGuide/FileCheck.html), but is not the same. 9FileCheck is available for writing both C++ and python tests. 10 11## How does it work 12 13Let's look at a test written with FileCheck. The following test verifies that 14CSE pass removes one out of two similar `aten::mul` nodes. Here is how the test 15looks like: 16 17```python 18def test_cse(): 19 input_str = """graph(%a : Tensor, %b : Tensor): 20 # CHECK: aten::mul 21 %x : Tensor = aten::mul(%a, %b) 22 # Check that the second aten::mul is removed by CSE. 23 # CHECK-NOT: aten::mul 24 %y : Tensor = aten::mul(%a, %b) 25 # CHECK: return 26 return (%x, %y) 27 """ 28 parsed = parse_ir(input_str) 29 optimized = run_cse(parsed) 30 FileCheck().run(input_str, optimized) 31``` 32 33Let's look in detail at how it works. First, the input string is parsed by 34`parse_ir`. At that stage all annotations are ignored since they are written in 35comments, so this is what parser essentially sees: 36 37``` 38graph(%a : Tensor, %b : Tensor): 39 %x : Tensor = aten::mul(%a, %b) 40 %y : Tensor = aten::mul(%a, %b) 41 return (%x, %y) 42``` 43 44We then run CSE on the parsed IR and expect it to remove the second `aten::mul`, 45which is redundant. After CSE our IR looks like this: 46 47``` 48graph(%a : Tensor, %b : Tensor): 49 %x : Tensor = aten::mul(%a, %b) 50 return (%x, %x) 51``` 52 53And now we run `FileCheck` passing to it both original input string and the 54optimized IR. From the input string `FileCheck` ignores everything except `# 55CHECK` pragmas and essentially it sees the input string like this: 56 57``` 58 # CHECK: aten::mul (1) 59 # CHECK-NOT: aten::mul (2) 60 # CHECK: return (3) 61``` 62 63It then checks that the optimized IR satisfies the specified annotations. It 64first finds string `%x : Tensor = aten::mul(%a, %b)` matching the annotation (1), 65then it finds string `return (%x, %x)` matching the annotation (3), and since 66there were no lines matching `aten::mul` after the match (1) and before the 67match (3), the annotation (2) is also satisfied. 68 69One could also register FileCheck annotations using a builder API. To generate 70annotations from the example above one would write: 71```python 72 FileCheck().check("aten::mul") \ 73 .check_not("aten::mul") \ 74 .check("return") \ 75 .run(optimized) 76``` 77 78## Supported pragmas 79 80* `CHECK: <pattern>` 81 Scans the input until `PATTERN` is found. Fails if the pattern is not found. 82* `CHECK-NEXT: <pattern>` 83 Scans the input on the line immediately following the previous CHECK until 84 `PATTERN` is found. Fails if the pattern is not found on that line. 85* `CHECK-NOT: <pattern>` 86 Scans the input and fails if `PATTERN` is found on any line. The scan stops when 87 a match for a next `CHECK` is found. 88* `CHECK-SAME: <pattern>` 89 Checks that PATTERN is found in the line of the last match. 90* `CHECK-COUNT-<num>: <pattern>` 91 Scans the input and succeeds when a line containing at least `NUM` entries of 92 `PATTERN` is found. 93* `CHECK-COUNT-EXACTLY-<num>: <pattern>` 94 Scans the input and succeeds when a line containing exactly `NUM` entries of 95 `PATTERN` is found. 96* `CHECK-DAG: <pattern>` 97 Works similar to the usual `CHECK` pragma, but also matches if there exists a 98 way to reorder the CHECK-DAG pragmas to satisfy all patterns. 99 For example the following pattern: 100 ``` 101 # CHECK: foo 102 # CHECK-DAG: bar 103 # CHECK-DAG: ham 104 # CHECK: end 105 ``` 106 would match the following input (note that `ham` and `bar` are swapped): 107 ``` 108 foo 109 ham 110 bar 111 end 112 ``` 113* `CHECK-SOURCE-HIGHLIGHTED: <pattern>` 114 Check for highlighted source ranges. This is useful when writing tests regarding generated error messages that require source code highlighting. 115 For example the following pattern: 116 ``` 117 # CHECK-SOURCE-HIGHLIGHTED: raise Exception("raised exception 118 ``` 119 would match the following input: 120 ``` 121 def method_that_raises() -> torch.Tensor: 122 raise Exception("raised exception") # noqa: TRY002 123 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE 124 builtins.Exception: raised exception 125 ``` 126* `CHECK-REGEX: <pattern>` 127 Scans the input until `PATTERN` is matched, accepts RE syntax for std::regex. 128