Automagic Deployment with Github Actions

Jon Johnson (he/him)

jrjohnson
jrjohnson_
jrjohnson-ucsf

Plain Text and Slides At:

UCTech Slack: https://uctech.slack.com

UCSF Library Educational Technology

Team Supports, Hosts, and Develops Moodle, Kaltura and Ilios

The Problem

Building Something People Enjoy Using is Stressful

They want them to work

and be fast

and be accurate

and have new useful features

It's Friday the 13th in 2020, I double dog dare you to deploy to production!
— Molly Struve 🦄 (@molly_struve) November 13, 2020

Some Things That Can Help

Some Things That Can HelpAgile Process 

Some Things That Can HelpAgile Process 
Tests 

Some Things That Can HelpAgile Process 
Tests 
Continuous Integration 

Some Things That Can HelpAgile Process 
Tests 
Continuous Integration 
Continuous Delivery
(or at least easy to start deploys)

The UCTech Connection

2009 Agile all the things!

2011 Test all the things!
Merced

2012 Automate all the things!

2013 CI is all the things!
Irvine

I'm constantly building on things I've learned at UCTech
UCTech is the first place I realized that some of this fancy stuff could work in our structure

Old shady tree at Uc Irvine where I sat stunned after learning about CI

Leaned about CI at Irvine. Afterwards I was so blown away I had to take a walk and ended up sitting under a tree thinking about how amazing it would be to test every change. How freeing that would be (I was pretty bogged down in an endless cycle of fixing bugs and causing other bugs)

Let's Talk About Agile

Let's Talk About AgileMake It Whatever You Want

Let's Talk About AgileMake It Whatever You WantAs Long as You Can Get Feedback Fast

There are a bunch of ways to do agile well
One of the only ways to do it wrong is to get too caught up in trying to do it by someones else's definition
I lead two "agile" teams with 80% of the same people and they run totally different processes
- Both of them can get code into production in under an hour

Let's Talk About Tests

The first rule of magic: have a clear intent.

This is why TDD works. You express your intent by writing a test. That way the fae running around the CPU can’t twist your spell into a curse.
— Beth (@bethcodes) August 10, 2022

I'm incredibly lazy and super forgetful so I need the computer to handle checks.
I'm sloppy and I can't spell
Tests also include code style rules
You can do Test Driven Development or Test Reminder Development or Write Tests When Done Development, just write the tests

Let's Talk About TestsIt will take at least a week to write your first test

Let's Talk About TestsIt will take at least a week to write your first testIt will take another two weeks to get it running automatically

Let's Talk About Tests

It will take at least a week to write your first test

It will take another two weeks to get it running automatically

Great First Tests

1!=2

visit(url) get 200 code

Output name of favorite dog

Jayden a brown and black chihuahua smashing his face in next to my laptop screen

Let's Talk About Tests

Bad Tests are Still Great Tests

Ilios Had Bad Tests

Getting them to run was a perfect first day.
Getting them to fail was a great second day.
In my second week the "terrible" tests saved me from a horrible mistake

Jayden a small chihuahua is sitting with his elbow out looking relaxed

Let's Talk About Tests

Bad Tests are Still Great Tests

Ilios Had Bad Tests

Getting them to run was a perfect first day.
Getting them to fail was a great second day.
In my second week the "terrible" tests saved me from a horrible mistake

Jayden a small chihuahua is sitting with his elbow out looking relaxed

Write bad tests!!!

Easier to Test with a Solid Foundation

ember testing docs screenshot symfony testing docs screenshot

We built on top of Ember.js and Symfony
Prominent testing documentation
Up to date and community maintained documentation
We knew we wouldn't have to document our tools
The more integrated the framework the easier it is to hire productive people
STAY UP TO DATE (semver, release plans, BC promises)

Tests are a Speed Multiplier

I know it seems counterintuitive but just hear me out: It gets faster to do things correctly if you keep doing them correctly. It might be slow at first if you’ve just been chucking things in the bin- change can be hard but don’t give up!! You deserve good code too.
— Melanie Sumner 💥 🐹 (@melaniersumner) August 7, 2018

Writing tests can be painful
Writing tests can be slow
Running tests takes time
Still worth it
4144 Merged PRs on Frontend since September 2014
2396 Merged PRs in common Since July 2017 over 264 weeks
Something like 20 changes / week (just in the UI)
Test unlock staying up to date (and Secure!!!!)

Show Us The Github Actions Already!!!

--everyone here today

Continuous Integration With Github Actions

Choose When it Runs

on:
  push:
    branches:
      - master
  pull_request:
  schedule:
    - cron: '33 2 * * 1' # weekly, on Monday morning

https://github.com/ilios/ilios/blob/master/.github/workflows/ci.yml

Continuous Integration With Github Actions

Jobs and Steps

...
jobs:
  code_style:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: shivammathur/setup-php@v2
      with:
        coverage: none
        php-version: 8.1
        extensions: apcu
    - run: vendor/bin/phpunit

https://github.com/ilios/ilios/blob/master/.github/workflows/ci.yml

Checkout

Actions (or workflows) are a blank slate
You even have to tell them to checkout your code
Github Runners:
Ubuntu
Windows Server
OSX
You can also host your own bespoke runner

We're using a community action here to install PHP

Github Maintains a Marketplace of Such Actions
You can also write your own

Last part is "run"

Run a CLI command to do whatever you want

Continuous Integration With Github Actions

Jobs (Code Style)

...
code_style:
  steps:
  - run: composer install --no-interaction --prefer-dist
  - run: vendor/bin/phpcs
  - run: bin/console lint:twig templates custom
  - run: |
      bin/console cache:warmup --env=dev
      vendor/bin/phpstan analyse --no-progress
  - run: |
      bin/console cache:warmup --env=test
      vendor/bin/phpstan analyse -c tests/phpstan.neon.dist tests --no-progress

https://github.com/ilios/ilios/blob/master/.github/workflows/ci.yml

It's a confusing time in PHP land

Lots of ways to analyze your code
Here we have two linters which look at code style
And a static analysis tool which can catch all kinds of other interesting bugs

Continuous Integration With Github Actions

Tests (PHPUnit)

tests:
  strategy:
    matrix:
      php-version: [8.1,8.2]
  steps:
  ...
  - name: Use PHP ${{ matrix.php-version }}
    uses: shivammathur/setup-php@v2
    with:
      php-version: ${{ matrix.php-version }}
  - name: install dependencies
    run: composer install --no-interaction --prefer-dist
  - name: Run Tests
    run: vendor/bin/phpunit

https://github.com/ilios/ilios/blob/master/.github/workflows/ci.yml

Matrix of Versions

Our official policy is to support the most recently released version of PHP
But for at least 3 months after that release we support the previous version
We run our tests against both
We also run regular weekly tests against the upcoming beta releases so we can report any issues

Continuous Integration With Github Actions

MySQL Migrations

test_migrations_against_mysql:
  name: Test Migrations Against MySQL
  steps:
  ...
  - name: Drop, Create, Migrate, and Validate DB
    env:
      ILIOS_DATABASE_URL: mysql://root:root@127.0.0.1:3306/ilios?serverVersion=8.0
    run: |
      sudo systemctl start mysql.service
      bin/console doctrine:database:drop --if-exists --force
      bin/console doctrine:database:create
      bin/console doctrine:migrations:migrate  --no-interaction
      bin/console doctrine:schema:validate

https://github.com/ilios/ilios/blob/master/.github/workflows/ci.yml

MySQL is installed already

The Github runner has mysql on it, we just need to start it

Continuous Integration With Github Actions

Combine Jobs

...
build_containers:
  steps:
  ...
  - name: Build fpm
    uses: docker/build-push-action@v3
    with:
      target: fpm
run_containers:
  steps:
  ...
  - run: docker load --input /tmp/fpm-image/fpm.tar
  - run: |
      docker run -d --name ilios-fpm fpm:testing
      docker exec ilios-fpm php bin/console monitor:health
  - if: failure()
    run: docker logs ilios-fpm

https://github.com/ilios/ilios/blob/master/.github/workflows/ci.yml

Let's Talk About Deployment

Jackson, a brown and black chihuahua min-pin mix waiting with his toy ball

The Many Deploys of Ilios

 The Many Deploys of IliosPreview UI Changes

 The Many Deploys of IliosPreview UI Changes
Publish NPM Packages

 The Many Deploys of IliosPreview UI Changes
Publish NPM Packages
Push Code to AWS

 The Many Deploys of IliosPreview UI Changes
Publish NPM Packages
Push Code to AWS
Push Containers to Registry 

Preview UI Changes

Publish Packages

Publish  PackagesTag Version

Publish  PackagesTag Version
Create Release 

Publish  PackagesTag Version
Create Release 
Publish Package

Tag Version

on:
workflow_dispatch:
  inputs:
    releaseType:
      description: 'Semver Release Type (major,minor,patch)'
jobs:
  tag:
    steps:
    - uses: actions/checkout@v3
      with:
          token: ${{ secrets.ZORGBORT_TOKEN }}
    - name: Increment Version
      run: npm version ${{ github.event.inputs.releaseType }}
    - name: Push Changes
      run: git push --follow-tags

https://github.com/ilios/common/blob/master/.github/workflows/tag_version.yaml

WTF Is Zorgbort

It's our robot name, we use it for our chatbot and for resource accounts

Tag Version

Screenshot of the Github workflow UI showing a dropdown where user can select the release type

Improving the Tagging Experience with a Chatbot

Slack conversation with a chatbot starting the deploy

Zorgbort Details

Talked about it at UCTech a few years ago (slides on my site)

Changes Since Then

Slack Bolt Framework
Deployed to Lambda with serverless Framework
Provides a friendly chatbot experience on top of actions

Create Release


on:
  push:
    tags:
    - '*'
jobs:
  notes:
    steps:
    - name: Generate Release Notes
      id: notes
      run: |
        NOTES=$(npx generate-github-release-notes ilios common ${{ steps.previousTag.outputs.tag }} ${{steps.nextTag.outputs.tag}})
        echo ${NOTES}
        # remove line breaks from notes so they can be passed around
        NOTES="${NOTES//$'\n'/'%0A'}"
        echo "::set-output name=releaseNotes::$NOTES"
    - uses: ncipollo/release-action@v1
      with:
        body: ${{steps.notes.outputs.releaseNotes}}
        token: ${{ secrets.ZORGBORT_TOKEN }}

https://github.com/ilios/common/blob/master/.github/workflows/create_release.yaml

Create Release

Sample release notes showing bugs fixed, pull requests merged and who worked on code

https://github.com/ilios/common/releases/tag/v69.2.0

Publish Package

name: Publish NPM Package
on:
release:
  types: [published]
jobs:
  publish-npm:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: install dependencies
        run: npm ci --ignore-scripts
      - name: publish
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{secrets.npm_token}}

https://github.com/ilios/common/blob/master/.github/workflows/release.yaml

Publish Package

List of 8 Ilios common NPM releases in the past 3 months

Push Code to

Push Code to AWSTag Version

Push Code to AWSTag Version
Create Release 

Push Code to AWSTag Version
Create Release 
Pack and Push Files 

Push Code to AWSTag Version
Create Release 
Pack and Push Files 
Send Release Data to Sentry

Pack and Push Files

 Pack and Push FilesCompile Styles

 Pack and Push FilesCompile Styles
Transpile Javascript

 Pack and Push FilesCompile Styles
Transpile Javascript
Minify and Compress Files 
(brotli, gzip, plain) 

 Pack and Push FilesCompile Styles
Transpile Javascript
Minify and Compress Files 
(brotli, gzip, plain) 
Send it to S3 
AWS

Pack and Push Files

on:
  push:
    tags:
      - '*'
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
    - name: install dependencies
      run: npm ci
    - name: Ember CLI Deploy
      run: npm run deploy:production
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

https://github.com/ilios/frontend/blob/master/.github/workflows/deploy-production.yml

Framework FTW!

Zero Config or Convention over Configuration Frameworks can often do really nice builds
As long as it will run on Ubuntu, Windows, or OSX you can run it in a workflow

Send Release Data to Sentry

  deploy:
    steps:
    ...
    - name: Install Sentry CLI
      run: npm install -g @sentry/cli
    - name: Create a Sentry.io release
      run: |
        # Create new Sentry release
        export SENTRY_RELEASE=$(sentry-cli releases propose-version)
        sentry-cli releases new $SENTRY_RELEASE
        sentry-cli releases set-commits --auto $SENTRY_RELEASE
        sentry-cli releases files $SENTRY_RELEASE upload-sourcemaps tmp/deploy-dist/
        sentry-cli releases finalize $SENTRY_RELEASE
      env:
        SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}

https://github.com/ilios/frontend/blob/master/.github/workflows/deploy-production.yml

Let's talk about containers!

Docker

Docker is environment as code

FROM php:8.1-fpm
RUN composer install --prefer-dist --no-dev --no-progress --no-scripts --no-interaction; \
  composer dump-autoload --classmap-authoritative --no-dev; \
  composer symfony:dump-env prod; \
  composer run-script --no-dev post-install-cmd; \
  chmod +x bin/console; \
  bin/console cache:warmup;
VOLUME /srv/app/var
COPY docker/fpm/ilios.ini $PHP_INI_DIR/conf.d/ilios.ini
COPY docker/fpm/zz-docker.conf /usr/local/etc/php-fpm.d/zz-docker.conf
COPY docker/fpm/docker-entrypoint.sh /usr/local/bin/docker-entrypoint
RUN chmod +x /usr/local/bin/docker-entrypoint
ENTRYPOINT ["docker-entrypoint"]
CMD ["php-fpm"]

Our Dockerfile: https://github.com/ilios/ilios/blob/master/Dockerfile

What does this mean

We're relying on a base provided by PHP the language
It relies on Debian in turn
Each of this organizations can release new versions at any time
We then layer on anything special we need to run our app

Updates

From distro to php to us every layer is constantly updating
All we need to do is listen and re-build our own image when these changes happen

New PHP Version?

FROM php:8.1-fpm

New PHP Version?

FROM php:8.1-fpm

FROM php:8.1-fpm

New PHP Version?

FROM php:8.1-fpm

FROM php:8.1-fpm

FROM php:8.2-fpm

Installation Optimizations

RUN composer install --prefer-dist;
composer dump-autoload --classmap-authoritative;
composer symfony:dump-env prod;

Push Containers to Registry

 Push Containers to RegistryTag Version

 Push Containers to RegistryTag Version
Create Release 

 Push Containers to RegistryTag Version
Create Release 
Build Multi Platform Containers 

 Push Containers to RegistryTag Version
Create Release 
Build Multi Platform Containers 
Push To Docker Hub

Build Multi Platform Containers

...
steps:
- uses: docker/setup-qemu-action@v2
  with:
    image: tonistiigi/binfmt:latest
    platforms: linux/amd64,linux/arm64
- uses: docker/build-push-action@v3
  with:
    platforms: linux/amd64,linux/arm64

https://github.com/ilios/ilios/blob/master/.github/workflows/deploy-tag.yml

Using emulation we can build for different architectures
- Which turns out to be really important when your team starts getting apple silicon

Push To Docker Hub

jobs:
  tags:
    steps:
    - id: tag
      run: |
        MAJOR_VERSION=$(echo $GITHUB_REF | sed -rn 's#^refs/tags/(v3)\.([0-9]+)\.([0-9]+)$#\1#p')
        MINOR_VERSION=$(echo $GITHUB_REF | sed -rn 's#^refs/tags/(v3)\.([0-9]+)\.([0-9]+)$#\1.\2#p')
        PATCH_VERSION=$(echo $GITHUB_REF | sed -rn 's#^refs/tags/(v3)\.([0-9]+)\.([0-9]+)$#\1.\2.\3#p')
  deploy-docker-containers:
    needs: tags
    runs-on: ubuntu-latest
    - uses: docker/build-push-action@v3
      with:
        tags: ilios/fpm:${{major}},ilios/fpm:${{minor}},ilios/fpm:${{patch}}
        push: true

Push To Docker Hub

Listing of Ilios build tags showing v3.102.2, v3.102, and v3

Nightly Re-Tag

(the latest debian,php,mysql,etc...)

on:
  workflow_dispatch:
  schedule:
    - cron: '21 10 * * *' # daily, in the morning (UTC)
jobs:
  tags:
    steps:
    - run: |
        LATEST_TAG=$(git describe --tags --abbrev=0)
        MAJOR_VERSION=$(echo $LATEST_TAG | sed -rn 's#^(v3)\.([0-9]+)\.([0-9]+)$#\1#p')
        ...
  deploy-docker-containers:
    steps:
    - uses: actions/checkout@v3
      with:
          ref: ${{needs.tags.outputs.latestTag}}
    ...
    - uses: docker/build-push-action@v3
      with:
        tags: ilios/${{ matrix.image }}:${{major}},ilios/${{ matrix.image }}:${{minor}}...
        platforms: linux/amd64,linux/arm64

https://github.com/ilios/ilios/blob/master/.github/workflows/deploy-nightly.yml

Old shady tree at Uc Irvine where I sat stunned after learning about CI

Going back to that mind-blowing talk at UCI that sent me to this tree for a timeout

We're really close now

Automation         You   

Tag Code

Tag Code as v65.4.3
Go To Lunch
v65.4.3 is now in Production
Repeat

Explore The Limits

 Explore The LimitsWeekly Dependency Reset

 Explore The LimitsWeekly Dependency Reset
Special Preview for Testing 

 Explore The LimitsWeekly Dependency Reset
Special Preview for Testing 
Test Updates for Asset Bloat 

 Explore The LimitsWeekly Dependency Reset
Special Preview for Testing 
Test Updates for Asset Bloat 
Run Cron Jobs 

Thanks!

Hope to see you on UCTech Slack

Jon Johnson (he/him)

jrjohnson
jrjohnson_
jrjohnson-ucsf

Plain Text and Slides At:

UCTech Slack: https://uctech.slack.com

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help

UCSF Library Educational Technology

Automagic Deployment with Github Actions

Jon Johnson (he/him)

Plain Text and Slides At:

UCTech Slack: https://uctech.slack.com

UCSF Library Educational Technology

The Problem

Building Something People Enjoy Using is Stressful

They want them to work

and be fast

and be accurate

and have new useful features

Some Things That Can Help

Some Things That Can Help

Agile Process

Some Things That Can Help

Agile Process

Tests

Some Things That Can Help

Agile Process

Tests

Continuous Integration

Some Things That Can Help

Agile Process

Tests

Continuous Integration

Continuous Delivery(or at least easy to start deploys)

The UCTech Connection

2009 Agile all the things!

2011 Test all the things! Merced Merced's logo is behind a firewall, so this is a picture of a squirrel in a trench coat

2012 Automate all the things!

2013 CI is all the things! Irvine Irvine's logo is behind a firewall, so this is a picture of a squirrel in a trench coat

Let's Talk About Agile

Let's Talk About Agile

Make It Whatever You Want

Let's Talk About Agile

Make It Whatever You Want

As Long as You Can Get Feedback Fast

Let's Talk About Tests

Let's Talk About Tests

It will take at least a week to write your first test

Let's Talk About Tests

It will take at least a week to write your first test

It will take another two weeks to get it running automatically

Let's Talk About Tests

It will take at least a week to write your first test

It will take another two weeks to get it running automatically

Great First Tests

Let's Talk About Tests

Bad Tests are Still Great Tests

Ilios Had Bad Tests

Let's Talk About Tests

Bad Tests are Still Great Tests

Ilios Had Bad Tests

Write bad tests!!!

Easier to Test with a Solid Foundation

Tests are a Speed Multiplier

Show Us The Github Actions Already!!!

Continuous Integration With Github Actions

Choose When it Runs

Continuous Integration With Github Actions

Jobs and Steps

Checkout

Github Runners:

We're using a community action here to install PHP

Last part is "run"

Continuous Integration With Github Actions

Jobs (Code Style)

It's a confusing time in PHP land

Continuous Integration With Github Actions

Tests (PHPUnit)

Matrix of Versions

Continuous Integration With Github Actions

MySQL Migrations

MySQL is installed already

Continuous Integration With Github Actions

Combine Jobs

Let's Talk About Deployment

The Many Deploys of Ilios

The Many Deploys of Ilios

Continuous Delivery
(or at least easy to start deploys)

2011 Test all the things!
Merced

2013 CI is all the things!
Irvine

Push Code to

Push Code to

Push Code to

Push Code to

Push Code to

Minify and Compress Files
(brotli, gzip, plain)

Minify and Compress Files
(brotli, gzip, plain)

Send it to S3