Continuous Integration

Owned by Avtar Gill
Dec 07, 2017
2 min read


Introduction

The Fluid Project uses Buildkite and Jenkins for hosted continuous integration. Jenkins has been in use for a few years but search for alternatives has been underway due to issues documented FLUID-6220. Buildkite is a new option currently being tested with the Infusion repository. The rest of this page focusses solely on Buildkite-related details.

Buildkite Details

Buildkite provides free accounts to open source projects. They offer a dashboard, API, and agents. The agents, hosted on user-provided infrastructure, poll the API for jobs. Jobs are triggered via webhooks payloads emitted when GitHub pull requests or branch merges take place. As jobs are run on agents results are sent back to the API and reflected on the dashboard.

It is useful to subscribe to Buildkite status emails as reliability issues they experience impact Fluid CI.

A fluid-bot GitHub account is used to post job status updates in PR threads.

Known Issues

  • Anonymous access - currently a Buildkite account is required to access pipelines and build results. Their support team said anonymous access is a feature that is being actively developed
  • Potential security issues via PRs - changes submitted via PRs will trigger CI jobs. This is mainly an issue if a contributor is not known. An issue has been filed requesting a more strict default that would prevent unrecognized contributors from triggering jobs. Until then a workaround is to block jobs for PR branches using the dashboard and use the unblock-buildkite-pr-job service to unblock jobs
  • Triggering builds for existing PRs - two ways of triggering jobs for existing PRs is to either push new commits or trigger them manually using the dashboard

Set Up a New Pipeline

To set up a new pipeline visit the Fluid Project's Buildkite dashboard and follow these steps:

  1. Make sure the fluid-bot account has push access to the project's repository
  2. Click the new pipeline plus button
  3. Fill in the required Name and Git Repository fields
  4. Make sure the first step is a Wait and block pipeline one with a Run this CI job label and : branch filter
  5. The second step should be a Read steps from repository type
  6. Check the Everyone box under Teams

Then visit the GitHub settings for the new pipeline and:

  1. Enter master for the Branch filter pattern in the Branch Limiting section
  2. Make sure that Build pull requests, Skip pull request builds for existing commits, Build pull requests from third-party forked repositories, Prefix third-party fork branch names, and Update pull request commit statuses boxes are checked
  3. A webhook URL will be provided along with instructions on how to set it up using the GitHub web UI