# Rethinking Moodle Hosting

## Who we are - Liip AG - Moodle team since 2005 - Provides - Consulting - Development - SaaS Hosting

## OdyX - Liiper since mid-2012 - Always been involved in Moodle team since - Doing other things too

# What we do… ## & what we had

## Hosting * ~ 50 different clients * always 2-3 different Moodle major releases * client-specific code

## Hosting methods Historically various strategies * Client-specific VMs * Shared hosting with `.deb` packages

## Some years ago ### Focus on our core competence Move away from needing `root`-level access (= SLA)

## Since 2014 * Two `web` servers * One shared `db` server

## Non-root deployment infra * Asked the hoster for some helpers: * `cron` job to reload apache when watchfile present * Apache `include` to load configuration files from non-`/etc` directory * Passwordless `sudo` to `www-data` * Since `stretch`: * Passwordless `sudo /bin/systemctl reload apache2`

## What it does now * client-specific Moodle configuration (incl. Apache specifics) * download/update language packs * cache handling * Let's Encrypt certificates * Shibboleth customizations * pre-upgrade DB dump * …

## Pros * Cost is shared * "Classic" Linux OPS (deployment by devs) * Non-`root` → not our business

## Cons * Deployment is * procedural * slow * fragile * in old `capistrano` / `ruby` * Client isolation is **weak** * Shared ressources * e.g. disk enlargements cause downtime for everyone * No **live** upscale * No disk downscale

## It works … ## … but it shows its age

# In an ideal world

* Fleet management * Build & test in CI * Deploy automatically in CD * Independent instances * Scalability * Pay-per-use * Hosting in `.ch`

# Obvious candidates

# Sure, just do… * Kubernetes * Openshift * docker-machine * …

# But… * Not enough experience (yet ?) * Not our (webagency) job * Don't want to maintain things _at this layer_ * Often pay for a whole cluster (`!=` pay-per-use)

# Some research later * Jelastic * Pay-per-use * Has an API * Does `docker`! * Not one, but multiple `.ch` hosters! ## All good, then !

# Jelastic ## Why we picked it

# What it really is ## Now that we have experience * Clicky-click web interface * "supports" `docker` containers (more later) * Autoscaling, with min and max * Proprietary, with some parts on GitHub

# Docker is wonderful

## Just take an apache+PHP image, put Moodle inside, done. Right?

# Hosting is a hard problem

# PROD is not DEV

### For example * Sending emails reliably * No `php-xdebug` needed * Moodle can (and should) use a local ClamAV daemon * Default settings can hit back * `timeout 10` in haproxy * too low memory usage in `PostgreSQL` * `…`

# Jelastic is not a docker daemon

## It's more `virtuozzo` than `docker` * So we have `VM`s! * `docker` container extracted on top of a `VM` template * Can't run a `buster`-based `docker` container (yet), because their template is not ready.

## Imperfect templates / process * e.g. `exim4` is broken :facepalm:

## Use `docker` containers as `VM` images 1/2 * Run multiple daemons * exim4 * ClamAV * `systemd` timers

## Use `docker` containers as `VM` images 2/2 * Debian stable + security * upgraded in CI * Moodle code * `systemd-timer` cronjobs launcher

## Layered approach * Debian `stretch-slim` * Jelastic-base * `nginx` * `munin` * `nginx-proxy` * `moodle-builder` * `moodle-webserver/php7.2` * `mdl35-client1` image * `mdl35-client2` image * `moodle-webserver/php7.3` * `mdl36-client3` image * `mdl37-client4` image

## Measure what matters * `munin` is easy enough, and has alerts * Complete email loop * Cloudlets and disk usage per node * HTTP response time * Time since * last cron * last backup * Usage * Specific quota usage * disk * users

## Pay-per-use has drawbacks * Pay the memory usage, including cache * ClamAV takes ~750Mb for "being there" * PostgreSQL can take lots too *Uncertainty principle*

## Pay-per-use countermeasure * Patch stable's ClamAV * socket-activation * ping watchdog every scan * killed after one hour * Reduce measuring interval to hourly (from 5 minutes)

# Jelastic API * API documentation is too often **bad** * Some things not implemented (initial setup) But we can: * clone * trigger redeploy * launch commands * measure

# Conclusions

# It's more work than you initially think * Gitlab CI * Building docker images (non-privileged) * Discover `Jelastic` specialties as-we-go * Take on more responsibility * Backup * Monitoring

* Building docker images * privileged runners (:evil:) * migrate to Google's `kaniko` * `crane` for tagging * `docker-compose` still needs privileged runners to test in-fleet

# Jelastic is not perfect * Instances independance * Scalability (up to?) * Pay-per-use really matters (small clients)

# If it looks like `docker` * Looks like * Smells like * It's still not `docker` But we (ab)use this; actual VMs.

# Different semantics * From "scripts to synchronize PROD and STAGING" * To "API call to clone PROD to temporary clone"

# Pay-per-use has incentives Put more things in the same container/VM to fly under the usage radar.

# Taking more responsibilities * Handle app-specific backups * Monitor instances more closely * …

## Questions?

Thank you for your attention!

© 2019 - Didier 'OdyX' Raboud

odyx@liip.ch

odyx@debian.org

https://reveal.odyx.org/Debian/201910-Rethinking-Moodle-hosting.html