Why We Use Containers

Not long ago, the tech world was split into two camps: development (Dev) and operations (Ops). Developers would create applications, then hand them over to the Ops team, who would be responsible for deployment and maintenance. This handoff often led to significant issues, as illustrated in the book The Phoenix Project by Kim, Behr, and Spafford. The core problems included:

• Inconsistent Environments: Developers test their applications in environments that don’t match production, leading to failures that no one knows how to resolve. The classic response from Dev is, “It works on my machine.”
• Deployment Complexity: Launching a complex application can involve numerous steps, which vary from application to application, making automation challenging.

These factors contribute to fragile and complicated deployments, resulting in long release cycles.

Containerization has emerged as a pivotal advancement that bridges the gap between Dev and Ops, establishing the foundation for DevOps practices. Containers address several issues by offering:

• Build Once, Run Anywhere: A container image encapsulates an application along with all its dependencies. The only requirement on the host is a container runtime, ensuring that the behavior remains consistent whether the container runs on a developer's laptop or in production.
• Standardized Deployment Process: The container build process is defined in a file (like a Dockerfile), which is unique to each application. All deployments follow a routine of building a container image, pushing it to a registry, pulling it into the production environment, and launching it.

By simplifying and standardizing deployment, containers significantly reduce the complexities that often burden Ops teams, thus enabling faster iteration and shorter release cycles.

Despite their benefits, the widespread adoption of containers for various use cases has led to some issues that make them ill-suited for certain tasks. This article will explore these challenges and propose alternatives.

Challenge 1: Resource Inefficiency

With phrases like “storage is cheap” and “vertical scaling is always an option,” many in the tech industry overlook the efficient use of resources. This lack of consideration can lead to excessive consumption of disk space and bandwidth when using containers.

The fundamental idea behind a container is to bundle all dependencies with the application into one package. The only shared component between the host and container is the Linux kernel, distinguishing containers from virtual machines. Consequently, container images can occupy an excessive amount of disk space and require significant bandwidth for transfer. This issue is compounded by the notion that each application should reside in its own container, leading to increased image sizes.

For large and complex applications, shipping a bulky artifact may seem justifiable for the sake of consistency and ease of deployment. However, is it reasonable to deploy a 1-2 GB container image containing a minimal OS, a Python interpreter, and numerous packages just to run a simple Python script? This situation becomes particularly frustrating when uploading such images over a limited home internet connection.

Several strategies can help mitigate the challenges posed by container bloat: enhancing infrastructure to meet demand, optimizing images, and leveraging cached layers.

Infrastructure enhancement involves scaling up resources to accommodate container requirements, such as using high-spec hardware or building containers in the cloud for better upload speeds. Image optimization, which involves minimizing container images, is often neglected. Cached layers, a feature of technologies like Docker, allow for incremental builds and faster uploads by reusing previously built layers.

However, these approaches do not fully resolve the underlying issues. Many applications, particularly those written in languages like Python and JavaScript, exacerbate container bloat due to their numerous dependencies. Adding new dependencies often necessitates rebuilding the entire container from scratch, leading to inefficiencies akin to discarding an entire shipping container simply to make a minor modification.

As container bloat becomes more prevalent, the industry will need to find smarter solutions, especially as applications built on advanced technologies like CUDA and PyTorch continue to grow in complexity.

Challenge 2: Reproducibility Issues

Once a container image is created, it behaves predictably when executed. However, rebuilding the image from its definition file does not guarantee an identical output.

Non-reproducible builds can create significant challenges. For instance, if a new team member's build breaks production, it may take hours to identify the problem, which could stem from a non-reproducible build step that was masked by a working cached layer.

Additionally, container images are not stored indefinitely, leading to assumptions of a direct correlation between a container image and its definition file in version control. This relationship holds true only if the build process is reproducible, which is often not the case.

While using lock files for dependency management can alleviate some issues, many definition files still include commands that are not reproducible, such as apt-get update && apt-get install ..., and base image versions may not always be pinned. This lack of precision can lead to unexpected breakages.

Challenge 3: Development Friction

Developers prefer a smooth workflow, but deploying applications via containers often introduces a level of complexity that can be described as "awkward."

The ideal scenario is to develop and test code directly within the deployment container. However, since container images are immutable, any changes made during development aren't retained. To circumvent this, developers often create containers that mount their project directories from the host, allowing for live code updates without rebuilding.

Unfortunately, this approach typically requires maintaining multiple definition files—one for development and another for production—which can lead to inconsistencies over time. Non-reproducible builds can complicate matters further, especially when system-level packages need updating.

Consequently, many developers opt to work in local environments and later package their code into containers, which can create gaps between development and production environments. This discrepancy may go unnoticed until deployment, potentially resulting in missing system packages.

On Dev Containers

A recent innovation is the concept of "dev containers," which provide a complete development environment within a container. Often referred to as "cloud IDEs," these environments offer several advantages, including consistent developer setups and reduced contamination from host configurations.

However, dev containers also introduce their own challenges, such as increased systemic bloat, as each project may require a distinct development environment. Additionally, developers face the dilemma of whether to persist data within these containers, which can lead to drifts over time, or risk losing changes when rebuilding the environment.

Challenge 4: Portability and Consistency Limitations

While containers are often likened to cargo containers, this analogy is not entirely accurate. Cargo containers are standardized, allowing for seamless global shipping. In contrast, although software containers can run across various environments, they are not as universally portable as claimed.

Most containers are designed for the Linux kernel, and while they can operate on Mac OS, this requires a built-in Linux VM. As the popularity of ARM64 architecture rises, container portability is further challenged. Solutions like cross-platform image building introduce additional complexities, as all compiled artifacts must be compatible with the target architecture.

Moreover, software containers are not closed systems. While a container runtime is the only dependency required to launch a container, applications often depend on external resources, such as databases or network access, which can limit their portability and consistency.

Challenge 5: Complexity Shift

While containers have simplified many aspects of application deployment, they have shifted complexity from managing application releases to overseeing intricate systems and platforms. The introduction of Kubernetes, a powerful open-source container orchestration tool, exemplifies this shift. While it alleviates some operational burdens, it also introduces a steep learning curve and a sprawling ecosystem of tools.

This complexity creates opportunities for vendors who offer managed services, positioning themselves as necessary resources for organizations navigating the intricacies of a cloud-native platform. However, this reliance can lead to escalating costs, as vendors control the underlying infrastructure.

The growing complexity of container deployments raises questions about whether they truly simplify operations. While containers aim to streamline deployments, the reality is that the systems supporting them can become convoluted.

Alternative 1: Statically Linked Binaries

For those deeply invested in cloud-native technologies, the suggestion to use statically linked binaries may seem unconventional. However, this approach offers a portable, self-contained executable that does not rely on external dependencies other than the operating system kernel.

Operating systems have supported statically linked executables for years. Applications written in languages like Go and Rust can compile into single binaries that encapsulate all runtime dependencies. This method drastically reduces bloat compared to container images, particularly for simple applications.

Statically linked binaries simplify reproducibility, enhance the development experience, and require less complex supporting infrastructure. As long as dependencies are well-managed and compiler settings are consistent, builds can directly correspond to version-controlled commits, eliminating the subtle complexities often found in container definitions.

Alternative 2: Nix and NixOS

Nix is a distinct package manager for Unix systems, designed to ensure fully reproducible builds and environments. It operates differently from conventional package managers, using a functional language to define configurations declaratively.

Nix creates isolated environments for different packages, allowing them to coexist on the same machine, similar to containers. Each application can have its unique set of dependencies while running as native processes on the host.

Nix addresses several challenges associated with containers:

• Reduced Bloat: Nix only introduces as much bloat as necessary for reproducibility, contrasting with the often excessive sizes of container images.
• Inherent Reproducibility: Nix was designed to guarantee reproducible builds, unlike Docker containers where this is an afterthought.
• Enhanced Developer Experience: Nix allows developers to work in isolated, reproducible environments without the need for complex container setups.
• Portability: While containers require a runtime to function, Nix requires the Nix interpreter, allowing for similar portability characteristics.
• Simpler Systems: Nix enables desirable container characteristics, such as isolation and consistency, while eliminating unnecessary layers of abstraction.

While Nix presents a compelling alternative to containers, it requires a cultural shift and a commitment to adopting "the Nix way" of managing projects, which can be a significant hurdle.

Summary

Containers have become integral to modern software development, accompanied by a complex ecosystem of infrastructure and tools. They provide numerous advantages but also present challenges, including:

• Excessive bloat relative to functionality
• Lack of consideration for reproducibility
• Development friction
• Questionable portability
• Escalating complexity in management

These drawbacks can impact the security, cost, and sustainability of systems. This article explored alternatives, including statically linked binaries and Nix(OS), which can mitigate some of these issues. While containers and Kubernetes remain relevant, it's crucial to consider whether they are the best solutions for every situation.

Thank you for engaging with this discussion. I welcome your thoughts in the comments!

The views expressed here are my own. For more insights, feel free to visit my personal blog where I explore various topics.