1*62c56f98SSadaf Ebrahimi# Mbed TLS test framework 2*62c56f98SSadaf Ebrahimi 3*62c56f98SSadaf EbrahimiThis document is an overview of the Mbed TLS test framework and test tools. 4*62c56f98SSadaf Ebrahimi 5*62c56f98SSadaf EbrahimiThis document is incomplete. You can help by expanding it. 6*62c56f98SSadaf Ebrahimi 7*62c56f98SSadaf Ebrahimi## Unit tests 8*62c56f98SSadaf Ebrahimi 9*62c56f98SSadaf EbrahimiSee <https://mbed-tls.readthedocs.io/en/latest/kb/development/test_suites> 10*62c56f98SSadaf Ebrahimi 11*62c56f98SSadaf Ebrahimi### Unit test descriptions 12*62c56f98SSadaf Ebrahimi 13*62c56f98SSadaf EbrahimiEach test case has a description which succinctly describes for a human audience what the test does. The first non-comment line of each paragraph in a `.data` file is the test description. The following rules and guidelines apply: 14*62c56f98SSadaf Ebrahimi 15*62c56f98SSadaf Ebrahimi* Test descriptions may not contain semicolons, line breaks and other control characters, or non-ASCII characters. <br> 16*62c56f98SSadaf Ebrahimi Rationale: keep the tools that process test descriptions (`generate_test_code.py`, [outcome file](#outcome-file) tools) simple. 17*62c56f98SSadaf Ebrahimi* Test descriptions must be unique within a `.data` file. If you can't think of a better description, the convention is to append `#1`, `#2`, etc. <br> 18*62c56f98SSadaf Ebrahimi Rationale: make it easy to relate a failure log to the test data. Avoid confusion between cases in the [outcome file](#outcome-file). 19*62c56f98SSadaf Ebrahimi* Test descriptions should be a maximum of **66 characters**. <br> 20*62c56f98SSadaf Ebrahimi Rationale: 66 characters is what our various tools assume (leaving room for 14 more characters on an 80-column line). Longer descriptions may be truncated or may break a visual alignment. <br> 21*62c56f98SSadaf Ebrahimi We have a lot of test cases with longer descriptions, but they should be avoided. At least please make sure that the first 66 characters describe the test uniquely. 22*62c56f98SSadaf Ebrahimi* Make the description descriptive. “foo: x=2, y=4” is more descriptive than “foo #2”. “foo: 0<x<y, both even” is even better if these inequalities and parities are why this particular test data was chosen. 23*62c56f98SSadaf Ebrahimi* Avoid changing the description of an existing test case without a good reason. This breaks the tracking of failures across CI runs, since this tracking is based on the descriptions. 24*62c56f98SSadaf Ebrahimi 25*62c56f98SSadaf Ebrahimi`tests/scripts/check_test_cases.py` enforces some rules and warns if some guidelines are violated. 26*62c56f98SSadaf Ebrahimi 27*62c56f98SSadaf Ebrahimi## TLS tests 28*62c56f98SSadaf Ebrahimi 29*62c56f98SSadaf Ebrahimi### SSL extension tests 30*62c56f98SSadaf Ebrahimi 31*62c56f98SSadaf Ebrahimi#### SSL test case descriptions 32*62c56f98SSadaf Ebrahimi 33*62c56f98SSadaf EbrahimiEach test case in `ssl-opt.sh` has a description which succinctly describes for a human audience what the test does. The test description is the first parameter to `run_test`. 34*62c56f98SSadaf Ebrahimi 35*62c56f98SSadaf EbrahimiThe same rules and guidelines apply as for [unit test descriptions](#unit-test-descriptions). In addition, the description must be written on the same line as `run_test`, in double quotes, for the sake of `check_test_cases.py`. 36*62c56f98SSadaf Ebrahimi 37*62c56f98SSadaf Ebrahimi### SSL cipher suite tests 38*62c56f98SSadaf Ebrahimi 39*62c56f98SSadaf EbrahimiEach test case in `compat.sh` has a description which succinctly describes for a human audience what the test does. The test description is `$TITLE` defined in `run_client`. 40*62c56f98SSadaf Ebrahimi 41*62c56f98SSadaf EbrahimiThe same rules and guidelines apply as for [unit test descriptions](#unit-test-descriptions). In addition, failure cause in `compat.sh` is not classified as `ssl-opt.sh`, so the information of failed log files are followed as prompt. 42*62c56f98SSadaf Ebrahimi 43*62c56f98SSadaf Ebrahimi## Running tests 44*62c56f98SSadaf Ebrahimi 45*62c56f98SSadaf Ebrahimi### Outcome file 46*62c56f98SSadaf Ebrahimi 47*62c56f98SSadaf Ebrahimi#### Generating an outcome file 48*62c56f98SSadaf Ebrahimi 49*62c56f98SSadaf EbrahimiUnit tests, `ssl-opt.sh` and `compat.sh` record the outcome of each test case in a **test outcome file**. This feature is enabled if the environment variable `MBEDTLS_TEST_OUTCOME_FILE` is set. Set it to the path of the desired file. 50*62c56f98SSadaf Ebrahimi 51*62c56f98SSadaf EbrahimiIf you run `all.sh --outcome-file test-outcome.csv`, this collects the outcome of all the test cases in `test-outcome.csv`. 52*62c56f98SSadaf Ebrahimi 53*62c56f98SSadaf Ebrahimi#### Outcome file format 54*62c56f98SSadaf Ebrahimi 55*62c56f98SSadaf EbrahimiThe outcome file is in a CSV format using `;` (semicolon) as the delimiter and no quoting. This means that fields may not contain newlines or semicolons. There is no title line. 56*62c56f98SSadaf Ebrahimi 57*62c56f98SSadaf EbrahimiThe outcome file has 6 fields: 58*62c56f98SSadaf Ebrahimi 59*62c56f98SSadaf Ebrahimi* **Platform**: a description of the platform, e.g. `Linux-x86_64` or `Linux-x86_64-gcc7-msan`. 60*62c56f98SSadaf Ebrahimi* **Configuration**: a unique description of the configuration (`mbedtls_config.h`). 61*62c56f98SSadaf Ebrahimi* **Test suite**: `test_suite_xxx`, `ssl-opt` or `compat`. 62*62c56f98SSadaf Ebrahimi* **Test case**: the description of the test case. 63*62c56f98SSadaf Ebrahimi* **Result**: one of `PASS`, `SKIP` or `FAIL`. 64*62c56f98SSadaf Ebrahimi* **Cause**: more information explaining the result. 65