Build reproducible environments

Overview

Engineers have a pithy saying to describe a bug that only appears in certain environments: "Works on my machine." These kinds of bugs are a drain on productivity and morale, as they require significant effort to diagnose and fix, and fixing them doesn't actually improve the software itself.

To minimize these issues, we need to ensure that all environments, including developer local environments, are as homogenous as possible.

What causes environment drift?

Environment inconsistencies can be caused by lots of things. For hosted (non-local) environments, the most common source of environment drift is manual changes. As an example, imagine you are hosting your application on AWS cloud. You need to enable debugging on the dev environment to troubleshoot an issue, so you log into the AWS console, make the change, and complete your testing. When you're done, you forget to revert the change, and suddenly you have one environment that's different from the others.

Other inconsistencies can come from engineers setting the project up in different ways on their local machines and unintentionally ending up with different versions of dependencies, configurations, or tools.

Still other inconsistencies can arise from data-level differences. While most data-level discrepancies won't introduce bugs, if your application keeps configuration in a database, it's easy to end up with different configurations in different environments, probably as a result of manual changes.

How to ensure environment consistency

• Checked item: Use Infrastructure as Code (IaC) to deploy and manage all cloud environments.
• Checked item: Use containerization (e.g., Docker) to bundle applications alongide their dependencies in a consistent way.
• Checked item: Automate the deployment process, and follow the same general sequence of steps for updating local environments.
• Checked item: Use a database migration tool to manage database schema changes across environments.
• Checked item: Automate data-level changes that need to be made across environments.
• Checked item: Disallow write access to hosted environments except through automated processes or by those following a "break glass" procedure.
• Checked item: Document the process for setting up local development environments, and use automated scripts where possible.
• Checked item: Use tools to manage versions of critical dependencies for local development (eg., pyenv, nvm, rbenv).