I’m a big supporter of finding problems before they get into the code base. The earlier you catch issues, the easier they are to fix. One of the main tools that helps with this is a Continuous Integration (CI) farm. A CI farm allows you to run extensive tests like deqp or piglit on a merge request or even on a private git branch before any code is merged, which significantly helps catch problems early.

I’m not the first one at Igalia to think this is really important. We already have a large Raspberry Pi board farm available on freedesktop’s GitLab instance that serves as a powerful tool for validating changes before they hit the main branch.

For a while, however, the etnaviv board farm has been offline. The main reason? I needed to clean up the setup: re-house it in a proper rack, redo all the wiring, and add more devices. What initially seemed like a few days’ worth of work spiraled into months of delay, mostly because I wanted to transition to using ci-tron.

Getting Familiar with the Ci-Tron Setup

Before diving into my journey, let’s quickly cover what makes up a ci-tron board farm.

  • Ci-Tron Gateway: This component is the central hub that manages devices.
  • PDU (Power Delivery Unit): A PDU is a device that manages the electrical power distribution to all the components in the CI farm. It allows you to remotely control the power, including power cycling devices, which is crucial for automating device management.
  • DUT (Device Under Test): The heart of the CI farm—these are the devices where the actual testing happens.

The Long Road to a Working Farm

Over the past few months, I’ve been slowly preparing for the big ci-tron transition. The first step was ensuring my PDU was compatible. It wasn’t initially supported, but after some hacking, I got it working and submitted a merge request (MR). After a few rounds of revisions, it was merged, expanding ci-tron’s PDU support significantly.

The next and most critical step was getting a DUT to boot up correctly. Initially, ci-tron only supported iPXE as a boot method, but my devices are using U-Boot. I tried to make it work anyway, but the network initialization failed too often, and I found myself sinking hours into debugging.

Thankfully, rudimentary support for a U-Boot based boot flow was eventually added. After some tweaks, I managed to get my DUTs booting — but not without complications. A major problem was getting the correct Device Tree Blob (DTB) to load, which was needed for ci-tron’s training rounds. A Device Tree Blob (DTB) is a binary representation of the hardware layout of a device. The DTB is used by the Linux kernel to understand the hardware configuration, including components like the CPU, memory, and peripherals. In my case, ensuring that the correct DTB was provided was crucial for the DUT to boot and be correctly managed by ci-tron. While integrating the DTB into U-Boot was suggested, it wasn’t ideal. Updating the bootloader just to change a DTB is cumbersome, especially with multiple devices in the farm.

With the booting issue taking up too much time, I decided to put it on hold and focus on something else: gfxinfo.

Gfxinfo Integration Challenges

gfxinfo is a neat feature that automatically tags a DUT based on the GPU model in the system, avoiding the need for manually assigning tags like gc2000. In theory, it’s very convenient—but in practice, there were hurdles.

gfxinfo tags Vivante GPUs using the device tree node information. However, since Vivante GPUs are quite generic, they don’t have a specific model property that uniquely identifies them. The plan was to pull this information using ioctl() calls to the etnaviv kernel driver. It took a lot of back and forth in review due to the internal gfxinfo API being under-documented, but after a lot of effort, I finally got the necessary code merged. You can find all of it in this MR.

Final Push: Getting Everything to Boot

There was still one major obstacle — getting the DUT to boot reliably. Luckily, mupuf was already working on it and made a significant MR with over 80 patches to address the boot issues. Introducing “boots db,” a feature designed to decouple the boot process, granting full control over DHCP, TFTP, and HTTP servers to each job. This is paired with YAML configurations to flexibly define the boot specifics for each board.

As of a few days ago, the latest official ci-tron gateway image contains everything needed to get an etnaviv DUT up and running successfully.

tags

I have to say, I’m very impressed with the end result. It took a lot longer than I had anticipated, but we finally have a plug-and-play CI farm solution for etnaviv. There are still a few missing features—like Network Block Device (NBD) support and some advanced statistics—but the ci-tron team is doing an excellent job, and I’m optimistic about what’s coming next.

Conclusion: A Long Road, but Worth It

The journey to get the etnaviv board farm back online was longer than expected, full of unexpected challenges and technical hurdles. But it was worth it. The result is a robust, automated solution that makes CI testing easier and more reliable for everyone. With ci-tron, it’s easier to find and fix problems before they ever make it into the code base, which is exactly what a good CI setup should be all about. There is still some work to be done on the GitLab side to switch all etnaviv jobs to the new board farm.

If you’re thinking about setting up your own CI farm or migrating to ci-tron, I hope my experience helps smooth the road for you a bit. It might be a long journey, but the end results are absolutely worth it.