Written by Karl Ding
on 

Continuous Integration and Solar Cars

At the University of Waterloo, the Midnight Sun Solar Rayce Car Team makes use of Travis CI to test our embedded software that powers our open source solar vehicle.

Writing code for embedded systems is rather different compared to the usual software development that most students enter University with. Besides the fact that C concepts like memory management and pointers pose a seemingly higher barrier of entry, you're often working against hardware constraints, lack of specific documentation, and end up debugging hardware issues like bad solder joints. Inevitably, as systems get more complicated and interdependent, bugs will occur that need to be ironed out before a race (like the American Solar Challenge, or the World Solar Challenge). Pins will need to be probed, continuity needs to be checked, oscilliscopes may need to be brought out, and you can't always just printf or dump things over CAN. Often you'll find hours wasted chasing bugs through the system, only to suddenly realize that an integer was overflowing—something you committed to the repository during Midterm Week while sleep-deprived.

Of course, being primarly an undergraduate team (with many of the members in the co-op program), we all bring best practices from places we've worked at, and try to avoid doing things that we see haven't worked well.

The Past

MSXI was our first car in the Cruiser Class, which is a new vehicle class pioneered by the World Solar Challenge, aimed at designing more practical cars that could potentially see everyday use. It represented a large shift in the design between the Challenger class we used to race in.

For MSXI, all our boards and firmware ran on Texas Instrument's MSP430F5xxx family of micrprocessors. And for the sake of convenience, most people on the Software Team used TI's Code Composer Studio to code, compile, and debug issues in the firmware. This decision was made before my time on the team, but I believe the rationale was that many Engineering programs taught courses involving the MSP430, so team members would have been familiar with the microcontroller.

The Future

In our new vehicle, MSXII (pronounced MS12), one of the things we began was a complete electrical overhaul, with the goal of standardizing hardware across all our boards, and creating a foundation that could be iterated upon in future cars—the "redesign to end all redesigns", if you will. This involved creating an architecture consisting of what we coined a "controller board", with "carrier boards" that plug into each controller. These "carrier boards" would be simple, and have one job, and do it well—sort of applying software principles to a hardware project. We selected the STM ARM Cortex M0 microprocessor, which gives us hardware multiplication and a few other nice things. As a side effect, this would allow us to use the GCC ARM embedded toolchain.

Part of the electrical redesign has involved writing a Hardware Abstraction Layer—essentially, a common API that can be shared across various device platforms. Currently, with this HAL, we are aiming to support x86 (for unit testing) and stm32f0xx devices (and possibly the MSP430 family). Having this HAL allows the implementation for each driver to change across architectures, but the code for each board remains the same. This means that once code is written, it can be quickly ported to another microcontroller without changing any of the logic, simply by changing the driver implementations.

As a result, we've started down the long path of writing test-driven code, and this has meant that it is now possible to start using great tools like Travis CI, and begin putting things like continuous integration in practice.

Development tools

The tools we use for development include

Of course, for other projects like telemetry and tooling, we choose languages that make solving the problems easier (like Golang and Python).

Workflow

At Midnight Sun, we use a simplified version of the Git Flow model for software development.

  1. A JIRA ticket is created under the project
  2. A feature branch is created and developed against
  3. When code is ready, a Pull Request is created
  4. A Software Lead will work with the contributor and review their code
  5. The Pull Request is squashed and merged into master

In addition, we use GitHub's branch protection functionality, so people can't push directly to master. This means that our master branch contains code that is tested and has been reviewed, and theoretically should work when flashed onto a board. As an added bonus, this also allows us to git bisect and find which commit caused a particular issue (or whoever broke the build).

Being at the University of Waterloo, the co-op program results in different "streams" for the team—by documenting progress on projects and having a centralized location where tasks can be tracked, it allows those on co-op to be kept in the loop. Of course, we also encourage those on co-op to contribute, but it isn't always feasible. Atlassian has generously provided us with JIRA, HipChat and Confluence licenses, which we use to track issues, talk about projects (while maintaining separation between our personal lives and the team), and document knowledge.

Firmware

We use a "monorepo" that houses all our individual projects, which each live in its own directory in projects/.

Initially, we tried having a ms-common submodule that contained drivers shared by all our boards (things like CAN, SPI, GPIO). We quickly realized that with the way our build system was set up, combined with our workflow, it just didn't make sense, having to frequently update the submodule in the main repo.

Build System

We use GNU Make, with a custom Makefile written for Make 4.0+.

This means that to build a project, we can just use make

make project PROJECT=queue

Each project or library defines a rules.mk file, where the developer define the dependencies the build unit has.

On the side, we've been evaluating whether or not to switch over to a modern build system (like Pants or Bazel)—Pants looks nice, but seems to currently lack support for custom toolchains, and Bazel's vendor dependency management for Golang isn't the greatest.

Setting up Travis CI

In order to get GCC ARM embedded toolchain working on Travis CI's container-based infrastructure, we simply add the GCC ARM Embedded PPA.

Here's how our .travis.yml file looks like at the time of writing.

notifications:
  email: false

language: c

# enable "Build pushes" and "Build pull requests" in Travis CI
# - pull requests will always be built
# - only build pushes to master
branches:
  only:
    - master

env:
  # add target platforms to build matrix
  # lint is a target so we don't need to lint multiple times
  - TARGET='build_all PLATFORM=x86'
  - TARGET='build_all PLATFORM=stm32f0xx'
  - TARGET='test_all PLATFORM=x86'
  - TARGET=lint

before_install:
  # update the libraries (ms-common, etc.)
  - git submodule update --init --recursive

before_script:
  # install GCC ARM from GNU ARM Embedded Toolchain PPA
  - sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa -y
  - sudo apt-get -qq update
  - sudo apt-get -qq install gcc-arm-embedded

  # build GNU Make 4.1 from source
  - wget http://ftp.gnu.org/gnu/make/make-4.1.tar.gz
  - tar xvf make-4.1.tar.gz
  - cd make-4.1
  - ./configure
  - make
  - sudo make install
  - cd ..
  - rm -rf make-4.1

  # add Make to path
  - export PATH=/usr/local/bin/:$PATH

script:
  - make $TARGET

Travis CI's C build matrix allows us to specify that workers should spawn and process all of our targets in the env we defined at the top of the .travis.yml file.

We should be wrapping up the standard drivers for the HAL sometime this term, and following that, we'll be making strides next on implementing device-specific drivers for each of our boards.

We're doing some cool things in E5-1002! Our software team has grown from 1 member to being composed of primarily Computer Science students (with a few ECE and Nano students) over the past year, and we're always recruiting. If you'd like to support Midnight Sun, you can drop us a line here, or come by one of our weekly General Meetings.