Separate deployment from release

Overview

Agile practice encourages frequent deployments to production. In some environments, this can lead to tension between the engineering team and business stakeholders, who may be uncomfortable with the idea of deploying new code to production frequently.

To reduce the risk of deploying new code, we agree to separate the deployment of code from the release of user-facing features. This allows us to keep the main branch always deployable, while still enabling rapid development and testing of new features. In concept, this is simple - the team agrees to a definition of what constitutes a "release" versus a "deployment", and agrees that deployments can be made with a lower level of scrutiny than releases.

As a simple guideline, you could consider that changes that introduce new user-visible features, or change existing user-visible features, are "releases", while changes that are purely internal (e.g. refactoring code, updating dependencies, fixing bugs that don't affect users) are "deployments".

By separating deployments from releases, we can still give business stakeholders control over when new features become visible to users, while allowing the engineering team to deploy code more frequently and with less risk.

How to do it

There are several techniques we use to achieve this separation:

• Checked item: Clearly define what constitutes a "release" versus a "deployment" for your project.
• Checked item: Implement guardrails to ensure that changes do not introduce new risks to the system.
• Checked item: Build trust with business stakeholders around the deployment process.
• Checked item: Use feature flags to toggle features on and off, ideally without deploying code.
• Checked item: Deploy new features on "hidden" URLs that are not linked from the main site.
• Checked item: Use "dark launches" to run new backend processes alongside existing ones, and switch over when ready.

Details

Feature Flags

Feature flags are a simple way to toggle features on and off. With a feature flagging system, new features can be safely merged into the main branch and deployed to production, but the feature remains disabled until the flag is turned on.

The simplest possible version of a feature flag is a boolean variable in the code that determines whether a feature is enabled or disabled. More advanced feature flagging systems allow for more complex rules, such as enabling a feature for a subset of users, or based on other criteria. There are many ways to implement feature flags, from simple configuration files to dedicated feature flag management systems. Some of them require code changes to toggle features, and others do not.

The best case scenario is to have a feature flagging system that allows non-technical stakeholders to toggle features on and off without requiring code changes or deployments. This allows for the technical team to be in full control of the deployment process, while still giving business stakeholders granular control over when features are released to users.

"Hidden" URLs

When the risk of users discovering the feature is not a concern, we may deploy new features on URLs that are not linked from the main site. This allows us to merge, test and validate new features in production without exposing them to all users. This is a simple way to separate deployments from releases, as the feature can be fully deployed and tested in production, but remains hidden from the majority of users until it is linked from the main site.

One simple way to implement this is to use a URL query parameter to enable the feature. For example, if you have a form at https://my.state.gov/form that is being replaced with a newer, better version, you could use the URL https://my.state.gov/form?new-feature=true to enable the new feature. This allows you to test the feature in production without exposing it to all users, and is particularly useful for conducting user testing sessions.

Dark Launches

As defined by Martin Fowler, a "dark launch" is the practice of deploying new backend processes alongside existing ones, and switching over when ready. This allows for new features to be fully deployed and tested in production, without exposing them to users until they are ready.

This type of deployment is particularly useful for backend processes that do not have a direct user interface, such as data processing pipelines or background jobs. By deploying the new process alongside the existing one, we can validate its performance and correctness in production, and switch over when we are confident it is ready.

Dark launches also allow us to monitor the new process in production, and gather metrics on its performance and correctness alongside the data being processed by the current system. This allows us to make informed decisions about when to switch over, and ensures that we are not introducing new risks to the system.

Getting buy-in

The most important premise of this play is getting buy-in from all stakeholders on the fact that deployments should require less scrutiny than releases. In particular, business stakeholders often have a hard time understanding the difference between deployments and releases, and may be hesitant to agree to a process that allows for deployments without their explicit approval. To address this, it is important to educate the entire team, be explicit and transparent about the guardrails in place to ensure quality, and build trust around the deployment process over time.

Guardrails

It's important to have guardrails in place to ensure that deployments do not introduce new risks to the system. Some examples of guardrails include:

• Visual regression testing to ensure the user interface doesn't change unexpectedly.
• Automated end-to-end tests to ensure that critical user flows are not broken.
• Performance monitoring to ensure that changes do not degrade system performance.
• Security scanning to ensure that changes do not introduce new vulnerabilities.
• Code reviews to ensure that changes both meet quality standards, and don't accidentally introduce changes that would require a release.
• Monitoring and alerting to quickly identify and address any issues that arise after deployment.

Many of these guardrails are the same ones we would want to already have in place to ensure the overall quality of the system, but it's important to explicitly communicate that they are in place to ensure that deployments can be made with a lower level of scrutiny than releases.