1Upreving Linux Kernel 2===================== 3 4Occasionally, the GitLab CI needs a Linux Kernel update to enable new kernel 5features, device drivers, bug fixes etc to CI jobs. 6Kernel uprevs in GitLab CI are relatively simple, but prone to lots of 7side-effects since many devices from different platforms are involved in the 8pipeline. 9 10Kernel repository 11----------------- 12 13The Linux Kernel used in the GitLab CI is stored at the following repository: 14https://gitlab.freedesktop.org/gfx-ci/linux 15 16It is common that Mesa kernel brings some patches that were not merged on the 17Linux mainline, that is why Mesa has its own kernel version which should be used 18as the base for newer kernels. 19 20So, one should base the kernel uprev from the last tag used in the Mesa CI, 21please refer to ``.gitlab-ci/image-tags.yml`` ``KERNEL_TAG`` variable. 22Every tag has a standard naming: ``vX.YZ-for-mesa-ci-<commit_short_SHA>``, which 23can be created via the command: 24 25:code:`git tag vX.YZ-for-mesa-ci-$(git rev-parse --short HEAD)` 26 27Building Kernel 28--------------- 29 30The kernel files are loaded from the artifacts uploaded to S3 from gfx-ci/linux. 31 32Updating Kconfigs 33^^^^^^^^^^^^^^^^^ 34 35When a Kernel uprev happens, it is worth compiling and cross-compiling the 36Kernel locally, in order to update the Kconfigs accordingly. Remember that the 37resulting Kconfig is a merge between *Mesa CI Kconfig* and *Linux tree 38defconfig* made via ``merge_config.sh`` script located at Linux Kernel tree. 39 40Kconfigs location 41""""""""""""""""" 42 43+------------+------------------------------------------------------+-------------------------------------+ 44| Platform | Mesa CI Kconfig location | Linux tree defconfig | 45+============+======================================================+=====================================+ 46| arm | kernel/configs/mesa3d-ci_arm.config\@gfx-ci/linux | arch/arm/configs/multi_v7_defconfig | 47+------------+------------------------------------------------------+-------------------------------------+ 48| arm64 | kernel/configs/mesa3d-ci_arm64.config\@gfx-ci/linux | arch/arm64/configs/defconfig | 49+------------+------------------------------------------------------+-------------------------------------+ 50| x86-64 | kernel/configs/mesa3d-ci_x86_64.config\@gfx-ci/linux | arch/x86/configs/x86_64_defconfig | 51+------------+------------------------------------------------------+-------------------------------------+ 52 53Updating image tags 54------------------- 55 56Every kernel uprev should update 3 image tags, located at two files. 57 58:code:`.gitlab-ci/container/gitlab-ci.yml` tag 59^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 60- **KERNEL_URL** for the location of the new kernel 61 62:code:`.gitlab-ci/image-tags.yml` tags 63^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 64- **KERNEL_ROOTFS_TAG** to rebuild rootfs with the new kernel 65- **DEBIAN_X86_TEST_GL_TAG** to ensure that the new rootfs is being used by the GitLab x86 jobs 66 67Development routine 68------------------- 69 701. Compile the newer kernel locally for each platform. 712. Compile device trees for ARM platforms 723. Update Kconfigs. Are new Kconfigs necessary? Is CONFIG_XYZ_BLA deprecated? Does the ``merge_config.sh`` override an important config? 734. Push a new development branch to `Kernel repository`_ based on the latest kernel tag used in GitLab CI 745. Hack ``build-kernel.sh`` script to clone kernel from your development branch 756. Update image tags. See `Updating image tags`_ 767. Run the entire CI pipeline, all the automatic jobs should be green. If some job is red or taking too long, you will need to investigate it and probably ask for help. 77 78When the Kernel uprev is stable 79^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 80 811. Push a new tag to Mesa CI `Kernel repository`_ 822. Update KERNEL_URL ``debian/x86_test-gl`` job definition 833. Open a merge request, if it is not opened yet 84 85Tips and Tricks 86--------------- 87 88Compare pipelines 89^^^^^^^^^^^^^^^^^ 90 91To have the most confidence that a kernel uprev does not break anything in Mesa, 92it is suggested that one runs the entire CI pipeline to check if the update affected the manual CI jobs. 93 94Step-by-step 95"""""""""""" 96 971. Create a local branch in the same git ref (should be the main branch) before branching to the kernel uprev kernel. 982. Push this test branch 993. Run the entire pipeline against the test branch, even the manual jobs 1004. Now do the same for the kernel uprev branch 1015. Compare the job results. If a CI job turned red on your uprev branch, it means that the kernel update broke the test. Otherwise, it should be fine. 102 103Bare-metal custom kernels 104^^^^^^^^^^^^^^^^^^^^^^^^^ 105 106Some CI jobs have support to plug in a custom kernel by simply changing a variable. 107This is great, since rebuilding the kernel and rootfs may takes dozens of minutes. 108 109For example, Freedreno jobs ``gitlab.yml`` manifest support a variable named 110``BM_KERNEL``. If one puts a gz-compressed kernel URL there, the job will use that 111kernel to boot the Freedreno bare-metal devices. The same works for ``BM_DTB`` in 112the case of device tree binaries. 113 114Careful reading of the job logs 115^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 116 117Sometimes a job may turn to red for reasons unrelated to the kernel update, e.g. 118LAVA ``tftp`` timeout, problems with the freedesktop servers etc. 119So it is important to see the reason why the job turned red, and retry it if an 120infrastructure error has happened. 121