jon's avatar Jonathan Johnson

Always Be Updating

This is a text summary of slides for a talk.

Who am I?

  • Web Platform Engineer and Manager @ucsf_library (PHP, Javascript)
  • Technical lead for the Ilios Project open source curriculum management system for health science education.
  • Making my second trip to ISS (first since the oughts)

Our Story

  • Ilios is a 20 year old project which started as an MS Access Database
  • Built by UCSF, but available for free (and supported)
  • Installed at medical, dental, pharmacy, nursing, around the world including here at the UC Davis veterinary school
  • Been through several revisions since then, but in 2014 we decided to rewrite it

Stack

LAMP + JAM

  • LAMP: Linux, Apache, MySQL, PHP
  • PHP API written in Symfony
  • JAM: Javascript and Markup
  • Javascript single page app using Ember.js
  • Hosted on AWS

Small Team

  • Three Engineers (code + operations)
  • Two Product Folks (ownership + management + support)

Sensitive Data

  • Account data (username, email, password)
  • Email and other contact details
  • Class schedules
  • FERPA protected education data

If you want to know how we handle access to data I'd recommend tomorrows Robot Permission Overlord Talk

Our Biggest Risks

Same as everyone else:

  • OWASP Top 10

Huge dependency graph

  • No time for keeping OS packages up to date

OWASP Top 10 2017

Three Items Address Dependencies Directly

  • A4:2017 XML External Entities (XXE)
  • A8:2017 Insecure Deserialization
  • A9:2017 Using Components with Known Vulnerabilities


So how many components do we use?

A Lot!

Like, A Lot, A Lot!

Dependencies

Direct (Production)

  • 39 PHP
  • 88 Javascript

Transitive

  • 153 PHP

  • 7668 Javascript

  • PHP app another 8 just for tests

  • Our javascript is missing a standard library, thus, CHAOS!

Not the Whole Story

Another three similar javascript apps to maintain

No way to count the number of things needed to run PHP, Apache, and Linux. It's HUGE!

[infinity gif goes here]

[if I had ever found one that didn't make dizzy]

Are we building on a house of cards?

image of worlds largest playing card stack

https://www.guinnessworldrecords.com/world-records/tallest-house-of-cards

Maybe

But We Still Need to Keep it Updated

Hopefully that's what you're all here for

But How?

1) Don't Panic!

2) Split updates into small testable chunks

3) Write tests until you're confident in them

4) Just put it in the cloud

5) Use the time you're saving to automate everything

(BONUS) Give this talk: 😱 goto Step One

Pull Request Model

screen capture of status checks

  • Fork an pull
  • PRs
  • Code Review
  • Greenkeeper
  • Code Climate
  • Waffle IO
  • Comfortable standard process for new contributors

Continuous Integration / Delivery

  • CI / CD have buzzword aspirations for sure
  • Loosely Integration is the process of constantly testing changes
  • Always be merging - we should always keep our master branch deployable; that means at any given moment we're sure we can ship master to production.
  • continuous delivery generally means that every change is immediately sent to production
  • We don't do continuous delivery at all. What we do instead is ensure that we could deliver at any time very rapidly. We're all setup for automated delivery, but we hold off on shipping releases until we accumulate enough meaningful changes. Usually this happens at the end of the week, but it could happen any time.

TravisCI

screenshot of Travis Building many pull requests

Azure

screenshot of azure Build Split Into Chunks

  • Our build can be split into chunks to run faster

Functional Review

functional review comment says click tests look fine

  • Every change needs a testable build
  • We automate this with Netlify

Let's Talk about tests

  • I've never met anyone who didn't want to ship well tested quality software. So why don't we all? It's a lot of work. Start small. Make writing tests easy. Spend a lot if time on your test harness. You're the user. Focus on your experience.
  • I'm incredibly lazy and super forgetful so I need the computer to handle checks.
  • I'm sloppy and I can't spell
  • Tests also include code style rules
  • You can do Test Driven Development or Test Reminder Development or Write Tests When Done Development, just write the tests

Magic?

Harry potter holding a wand for the first time

No. Just Good Test Coverage.

  • When we set out to rewrite I wanted to focus hard on testing
  • Lots of reason to do this
  • Save time on hunting down regressions
  • on board new team members more effectively

What I didn't know was how much more secure they would make us

Easier to Test with a Solid Foundation

ember testing docs screenshot symfony testing docs screenshot

  • We built on top of Ember.js and Symfony
  • Prominent testing documentation
  • Up to date and community maintained documentation
  • We knew we wouldn't have to document our tools
  • The more integrated the framework the easier it is to hire productive people
  • STAY UP TO DATE (semver, release plans, BC promises)

Tests are a Speed Multiplier

  • Writing tests can be painful
  • Writing tests can be slow
  • Running tests takes time
  • Still worth it
  • 2760 Merged PRs on Frontend in 4 years (something like 13 changes / week)
  • Test unlock staying up to date

Dependabot Handles Updates

Dependabot Pull Request

  • Every weekday our team wakes up to pull requests from dependabot
  • If tests pass we usually merge them
  • Some libraries are more visual and we hand test those changes
  • Almost everything else can be merged automatically

Our Dependabot Config

version: 1

update_configs:
  - package_manager: php:composer
    directory: "/"
    update_schedule: daily
    ignored_updates:
      - match:
          dependency_name: "symfony/*"
    automerged_updates:
      - match:
          dependency_type: all
          update_type: all

https://github.com/ilios/ilios/blob/master/.dependabot/config.yaml

More than just convenience

Dependabot PR which indicates a new person did the release

In the Cloud

.boxes[

sentry.io logo ]

  • Error reporting as a service
  • Let someone else handle infrastructure wherever possible
  • Save this time! (you're going to need it in a minute)
  • AWS RDS database is slightly more expensive that EC2, but you don't have to manage it

- Aurora even better and cheaper when load is variable and still less to manage

On that subject …

Let's talk about containers!

Docker

Docker is environment as code

FROM php:7.2-apache-stretch

RUN \
    /usr/bin/composer install \
    --prefer-dist \
    --no-dev \
    --no-progress \
    --no-interaction \
    --no-suggest \
    --classmap-authoritative

RUN /usr/bin/composer dump-env $APP_ENV
RUN /usr/bin/composer clear-cache
RUN bin/console assets:install

EXPOSE 80

Our Dockerfile: https://github.com/ilios/ilios/blob/master/Dockerfile

What does this mean

  • We're relying on a base provided by PHP the language
  • It relies on debian stretch and apache in turn
  • Each of this organizations can release new versions at any time
  • We then layer on anything special we need to run our app

Updates

  • From distro to apache to php to us every layer is constantly updating
  • All we need to do is listen and re-build our own image when these changes happen
  • Docker hub does this for us

New PHP Version?

FROM php:7.2-apache-stretch FROM php:7.2-apache-stretch FROM php:7.3-apache-stretch

Installation Optimizations

RUN composer install \
--prefer-dist \
--no-dev \
--no-progress \
--no-interaction \
--no-suggest \
--classmap-authoritative

Automation

You

Tag Code

  1. Tag Code as v65.4.3
  2. Go To Lunch
  3. v65.4.3 is now in Production
  4. Repeat

How does this help us stay secure?

  1. We update the underlying OS as often as we deploy (~ 2 weeks)
  2. Every version is completely atomic and can be tested top to bottom (we're not good at this yet)
  3. The more often we update the less each update impacts us
  4. More time to write tests.

Automate Everything

Automation Brings Speed

  • Jason is always 'Dialing it in'
  • This is a task which may never be completely done, but like CI setup it's always worth getting right
  • Good, reliable, automation truly pays off over the long term

and sometimes money!

@Zorgbort

Or whatever stupid name you want to give your bot

Automate Everything

https://github.com/ucsf-ckm/zorgbort

### Interested? I'll be talking about our chatbot at UCTech this year. Love to see you there!

Discussion / Questions?

UCTech Slack: https://uctech.slack.com

Slides at:

https://jrjohnson.dev/talks/2019-06-always-be-updating.html

Thanks

  • ISS for the invitation
  • UCSF Library
  • My Team

This is the first time I've given this talk so please ask me

at least three questions so I know what needs improvement!