Note: This repo is in maintenance mode as we assess whether Serverless GitHub Actions might provide a better experience going forward

LambCI

*Serverless continuous integration*


---

Automate your testing and deployments with:

• 1000 concurrent builds out of the box (can request more)
• No maintenance of web servers, build servers or databases
• Zero cost when not in use (ie, 100% utilization)
• Easy to integrate with the rest of your AWS resources

---

Contents

• [Overview](#what-is-it)
• [Installation](#installation)
• [Configuration](#configuration)
• [Updating](#updating)
• [Security](#security)
• [Language Recipes](#language-recipes)
• [Extending with ECS](#extending-with-ecs)
• [Questions](#questions)

---

What is it?

LambCI is a package you can upload to AWS Lambda that
gets triggered when you push new code or open pull requests on GitHub and runs your tests (in the Lambda environment itself) – in the same vein as Jenkins, Travis or CircleCI.

It integrates with Slack, and updates your Pull Request and other commit statuses on GitHub to let you know if you can merge safely.

!LambCI in action

It can be easily launched and kept up-to-date as a CloudFormation
Stack, or you can manually create the
different resources yourself.

Installed languages

• Node.js 12.x (including npm/npx)
• Python 3.6 (including pip)
• Gcc 7.2 (including c++)

Supported languages

• Check the [Recipes](#language-recipes) list below on how to configure these:
• Node.js (any version via nave)
• Python ([3.8.0, 3.7.4, 3.6.9](#python))
• Java (OpenJDK [1.8.0](#java))
• Go ([any version](#go))
• Ruby ([2.7.0, 2.6.5, 2.5.7, 2.4.9, 2.3.8, 2.2.10, 2.1.10, 2.0.0-p648](#ruby))
• PHP ([7.3.13, 7.2.26, 7.1.33, 7.0.32, 5.6.38](#php))

Prerequisites

• An Amazon AWS account
• A GitHub OAuth token ([see below](#1-create-a-github-token))
• (optional) A Slack API token ([see below](#2-create-a-slack-token-optional))

Current Limitations (due to the Lambda environment itself)

• No root access
• 500MB disk space
• 15 min max build time
• Bring-your-own-binaries – Lambda has a limited selection of installed software
• 3.0GB max memory
• Linux only

You can get around many of these limitations by [configuring LambCI to send tasks to an ECS cluster](#extending-with-ecs) where you can run your builds in Docker.

Installation

You don't need to clone this repository – the easiest way to install LambCI is to deploy it from the Serverless Application Repository or https://lambci.s3.amazonaws.com/templates/template.yaml" rel="noopener nofollow" target="_blank">directly spin up a CloudFormation stack. This will create a collection of related AWS resources, including the main LambCI Lambda function and DynamoDB tables, that you can update or remove together – it should take around 3 minutes to spin up.

You can use multiple repositories from the one stack, and you can run multiple stacks with different names side-by-side too (eg, lambci-private and lambci-public).

If you'd prefer to run your stack after cloning this repository, you can use npm run deploy – this depends on AWS SAM CLI being installed.

1. Create a GitHub token

You can create a token in the Personal access tokens section of your GitHub settings. If you're setting up LambCI for an organization, it might be a good idea to create a separate GitHub user dedicated to running automated builds (GitHub calls these "machine users") – that way you have more control over which repositories this user has access to.

Click the Generate new token button and then select the appropriate access levels.

LambCI only needs read access to your code, but unfortunately GitHub webhooks have rather crude access mechanisms and don't have a readonly scope for private repositories – the only options is to choose repo ("Full control").

If you're only using LambCI for public repositories, then you just need access to commit statuses:

Then click the "Generate token" button and GitHub will generate a 40 character hex OAuth token.

2. Create a Slack token (optional)

You can obtain a Slack API token by creating a bot user (or you can use the token from an existing bot user if you have one) – this direct link should take you there, but you can navigate from the App Directory via Browse Apps > Custom Integrations > Bots.

Pick any name, and when you click "Add integration" Slack will generate an API token that looks something like xoxb--

3. Launch the LambCI CloudFormation stack

You can either deploy it from the Serverless Application Repository or https://lambci.s3.amazonaws.com/templates/template.yaml" rel="noopener nofollow" target="_blank">use this direct CloudFormation link or navigate in your AWS Console to Services > CloudFormation, choose "Create Stack" and use the S3 link:

Then click Next where you can enter a stack name (lambci is a good default),
API tokens and a Slack channel – you'll also need to
make up a secret to secure your webhook and enter it as the
GithubSecret – any randomly generated value is good here, but make sure you
still have it handy to enter when you setup your webhooks in GitHub later on.

Click Next, and then Next again on the Options step (leaving the default options selected), to get to the final Review step:

Check the acknowledgments, click Create Change Set and then Execute to start the resource creation process:

Once your stack is created (should be done in a few minutes) you're ready to add the webhook to any repository you like!

You can get the WebhookUrl from the Outputs of the CloudFormation stack:

Then create a new Webhook in any GitHub repo you want to trigger under
Settings > Webhooks (https://github.com///settings/hooks/new)
and enter the WebhookUrl from above as the Payload URL, ensure Content type
is application/json and enter the GithubSecret you generated in the first step as the Secret:

Assuming you want to respond to Pull Requests as well as Pushes, you'll need to choose
"Let me select individual events", and check Pushes and Pull requests.

Then "Add webhook" and you're good to go!

By default LambCI only responds to pushes on the master branch and pull
requests ([you can configure this](#configuration)), so try either of those –
if nothing happens, then check Services > CloudWatch > Logs in the AWS
Console and see the [Questions](#questions) section below.

Installing as a nested stack in another CloudFormation stack

You can also embed LambCI in your own stack, using a AWS::Serverless::Application resource:

yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  LambCI:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: arn:aws:serverlessrepo:us-east-1:553035198032:applications/lambci
        SemanticVersion: 0.11.2
      Parameters:
        GithubToken: '123456789abcdef123456789abcdef123456789'
        GithubSecret: 'my-web-secret'
        SlackChannel: '#general'
        SlackToken: 'xoxb-123456789-abcdefABCDEFabcdef'

Outputs:
  S3Bucket:
    Description: Name of the build results S3 bucket
    Value: !GetAtt LambCI.Outputs.S3Bucket
  WebhookUrl:
    Description: GitHub webhook URL
    Value: !GetAtt LambCI.Outputs.WebhookUrl

If you save the above as template.yml, then you can use the
AWS SAM CLI to deploy from the same directory:

console
sam deploy --stack-name lambci --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

Configuration

Many configuration values can be specified in a .lambci.js, .lambci.json or package.json file in the root of your repository – and all values can be set in the DynamoDB configuration table (named -config, eg, lambci-config)

For example, the default command that LambCI will try to run is npm ci && npm test, but let's say you have a python project – you could put the following in .lambci.json in your repository root:

json
{
  "cmd": "pip install --user tox && tox"
}

(LambCI bundles pip and adds $HOME/.local/bin to PATH)

If you have a more complicated build setup, then you could specify make or create a bash script in your repository root:

json
{
  "cmd": "./lambci-test.sh"
}

Overriding default properties

LambCI resolves configuration by overriding properties in a cascading manner in the following order:

1. Default config ([see below](#default-configuration))
2. global project key in lambci-config DynamoDB table
3. gh// project key in lambci-config DynamoDB table
4. lambci property in package.json file in repository root
5. .lambci.js or .lambci.json file in repository root

You can use the command line to edit the DynamoDB config values:

console
lambci config secretEnv.GITHUB_TOKEN abcdef01234
lambci config --project gh/mhart/kinesalite secretEnv.SLACK_TOKEN abcdef01234

Or the AWS console:

!Global config in DynamoDB

So if you wanted to use a different Slack token and channel for a particular project, you could create an item in the config table with the project key gh// that looks similar to the global config above, but with different values:

js
{
  project: 'gh/mhart/kinesalite',
  secretEnv: {
    SLACK_TOKEN: 'xoxb-1234243432-vnjcnioeiurn'
  },
  notifications: {
    slack: {
      channel: '#someotherchannel'
    }
  }
}

Using the command line:

console
lambci config --project gh/mhart/kinesalite secretEnv.SLACK_TOKEN xoxb-1234243432-vnjcnioeiurn
lambci config --project gh/mhart/kinesalite notifications.slack.channel '#someotherchannel'

Config file overrides

Here's an example package.json overriding the cmd property:

json
{
  "name": "some-project",
  "scripts": {
    "lambci-build": "eslint . && mocha"
  },
  "lambci": {
    "cmd": "npm ci && npm run lambci-build"
  }
}

And the same example using .lambci.js:

js
module.exports = {
  cmd: 'npm ci && npm run lambci-build'
}

The ability to override config properties using repository files depends on the allowConfigOverrides property ([see the default config below](#default-configuration)).

Branch and pull request properties

Depending on whether LambCI is building a branch from a push or a pull request, config properties can also be specified to override in these cases.

For example, to determine whether a build should even take place, LambCI looks at the top-level build property of the configuration. By default this is actually false, but if the branch is master, then LambCI checks for a branches.master property and if it's set, uses that instead:

js
{
  build: false,
  branches: {
    master: true
  }
}

If a branch just has a true value, this is the equivalent of {build: true}, so you can override other properties too – ie, the above snippet is just shorthand for:

js
{
  build: false,
  branches: {
    master: {
      build: true
    }
  }
}

So if you wanted Slack notifications to go to a different channel to the default for the develop branch, you could specify:

js
{
  branches: {
    master: true,
    develop: {
      build: true,
      notifications: {
        slack: {
          channel: '#dev'
        }
      }
    }
  }
}

You can also use regular expression syntax to specify config for branches that
match, or don't match (if there is a leading !). Exact branch names are
checked first, then the first matching regex (or negative regex) will be used:

js
// 1. Don't build gh-pages branch
// 2. Don't build branches starting with 'dev'
// 3. Build any branch that doesn't start with 'test-'
{
  build: false,
  branches: {
    '/^dev/': false,
    '!/^test-/': true,
    'gh-pages': false,
  }
}

Default configuration

This configuration is hardcoded in utils/config.js and overridden by any config from the DB (and config files)

```js
{
cmd: 'npm ci && npm test',
env: { // env values exposed to build commands
},
secretEnv: { // secret env values, exposure depends on inheritSecrets config below
GITHUB_TOKEN: '',
GITHUB_SECRET: '',
SLACK_TOKEN: '',
},
s3Bucket: '', // bucket to store build artifacts
notifications: {
slack: {
channel: '#general',
username: 'LambCI',
iconUrl: 'https://lambci.s3.amazonaws.com/assets/logo-48x48.png',
asUser: false,
},
},
build: false, // Build nothing by default except master and PRs
branches: {
master: true,
},
pullRequests: {
fromSelfPublicRepo: true, // Pull requests from same (private) repo will build
fromSelfPrivate

…

Collapse