Infrastructure Experiments

A very common problem you face when running infrastructure for mission-critical software is making changes to the infrastructure or platform without crashing its clients — those microservices, cron jobs and integrations that users don't even know are there but are responsible for making the investors happy.

Some time ago, a work colleague asked me how our team was deploying infrastructure without affecting our internal and external end-users. The short answer is that you just have to focus on getting used to deploying to prod all the time. The long answer, however, requires that we discuss failure as a property of chaotic systems.

Failure guarantees that, throughout the years and decades that system is in prod, it will eventually fail at least once. And the sooner you find out something's gonna fail, the sooner you can course correct and prevent the system from failing.

Think of the systems you work on at your company: what happens when they're down? How do you prevent them from crashing? How do you remediate service downtime? If you're working on a mission-critical system, thinking about these questions might give you anxiety as the consequences to failure are not light.

To minimize service downtime, SREs tend to perform these changes during low demand periods like off working hours or during weekends (if the product allows it) to avoid a high-severity incident (or worsen the current open incidents). However, this has a downside similar to "no friday deploys" policies:

"If you block devs from merging on Fridays, then you are sacrificing a fifth of your velocity and overall output. That's a lot of fucking output." —— Charity Majors

And they're right: you're sacrificing a lot of puppies because of a lack of reliability in your system. Going beyond this means treating changes to infrastructure as experiments, learning from them and using that to do a better job next time.

Reliability is at the core of systems engineering and distributed systems, which concerns both Software Engineers (SWE) and Site Reliability Engineers (SRE). For instance, consider the following problem:

You own a set of VMs and use them to run a system of distributed microservices, where 50% of VMs run mission-critical services. Say you need to do a maintenance task that requires downtime, such as upgrading the OS version of the VMs. Given that the microservices were designed to run 24x5, how do you maximize the availability of the system?

Most Software Engineers will prefer solving that problem with a mix of "horizontal scaling", "messaging queues" and "write-ahead logs". And that's the answer you'd expect a SWE to immediately think of during a system design session: they should rarely concern themselves with what happens outside of the application domain, but they should have a general notion of how it works and understand that it'll eventually fail, then design their systems based on these assumptions.

SREs, on the other hand, tend to see an application as a black box and work to ensure the production environment is operational and healthy. To achieve this at scale requires heavy adoption of software engineering to automate the required operations, such as ensuring are enough machines to run the containers (like AWS Karpenter does).

Using software to automate this process is the main difference between System Administration and Site Reliability. Sysadmin requires that you spend your budget in toil, which has the common side-effect of create knowledge silos. Site Reliability will instead create distributed systems to automate toil and ensure engineer hours are wisely spent.

Adopting SRE will put you in a better position to attempt an automated solution like a mix of "blue-green deployment" and "canary deployment". You upgrade a percentage of VMs and iteratively migrate applications from the old VMs to them. As you increase the reliability of the system and assert it's healthy, you increase the percentage of VMs and in case of error the migration can be halted, rolled back or retried.

Changing infrastructure without software is painful, as it requires manual management of shell scripts, permissions and system state along with undesired service downtime if you're not quick enough. Also, doing this manually isn't reproducible at all, so the second time you run this playbook it might not work as the system might have changed since the previous run.

Site Reliability says that you must tackle this problem from a software engineering standpoint. You can model the problem domain using immutable data structures and write a reproducible program that automates the task at hand. Let's pretend Python classes are immutable for the purpose of demonstration:

# Upgrades the OS using custom logic or a third-party dependency like Ansible.
from lib.infra.vm import upgrade_os

class VirtualMachine:
  def __init__(self, id: int, config: dict):
    # The format of the ID is left up to the reader.
    self.id = id

    # Config has stuff like "OS version", "CPU cores", "instance type" etc.
    self.config = config

    # Some scheduling queues for applications. 
    # Do not confuse applications with OS processes as one application might have multiple processes.
    self.running_applications = []
    self.idle_applications = []
    self.finished_applications = []
  
  def clone(self) -> VirtualMachine:
    return VirtualMachine(random.randint(1, 100), self.config)

  def upgrade_os(self, new_version: str) -> VirtualMachine:
    new_vm = self.clone()

    if not new_version:
      raise Exception("new_version is required.")
    
    curr_version = new_vm.config["os"]["version"]
    if curr_version > new_version:
      raise Exception("current version is newer than requested version.")

    upgrade_os(new_vm, new_version)
    return new_vm
    
  def list_applications(self):
    return self.running_applications + self.idle_applications + self.finished_applications

def run_experiment(vms: list[VirtualMachine]):
  for old_vm in vms:
    # 50% of VMs run mission-critical services.
    pct = random.random()
    if pc <= 0.5:
      # Clone the VM and upgrade the OS
      new_vm = old_vm.upgrade_os()

      # Migrate the applications
      for app in old_vm.list_applications():
        ok = new_vm.schedule(app)
        if not ok:
          # Let's just log and halt on error for now (i.e. panic).
          print(f"failed to schedule app {app.name}")
          sys.exit(1)

      # Delete the old VM once all apps are migrated.
      # Use while loop as a dumb retry in case deletion is not ok.
      deleted = old_vm.delete()
      while not deleted:
        deleted = old_vm.delete()

Testing this with mock values will evaluate if this program works in a controlled environment, where you can tune the variables of the system. But by experimenting with real applications you'll gain real data which you can use to evolve the script and improve reliability, usability, quality and other -ilities.

The key insight here is recognizing that although all applications live in production, they aren't all made equal: some are mission critical while others are not. By using non-mission critical applications in experiments, you're able to learn faster about wrong assumptions while softening the consequences of failure: profit loss, compliance issues and whatnot.

The stance shifts from doing toil tasks to defining a process for OS upgrades, automating it and constantly experimenting towards improvement. As you fail a lot and become aware of the intricacies of the "production environment", upgrading operating systems becomes as simple as running a script or clicking a button on a web page, even when you're dealing with mission critical applications.

The next step is converting this program into a mature service that can be used by application owners. This is a big step that will lead you to Platform Engineering: the modern intersection of Product, Reliability and Infrastructure.