Webpack 4 - Resolving globally installed dependencies

When using NPM (or yarn) to manage JavaScript dependencies for a project, it is best practice to install the dependencies locally in the project so that multiple NPM projects on the system do not have clashing dependencies or dependency versions. However, it is best to break away from this pattern in favor of using a globally installed version of the dependency if the following cases hold true:

1. The dependency is massive or takes an extraordinarily long time to install.
2. The project is the only NPM project (or one of very few closely related NPM projects) on the system (e.g. running inside a docker container).

One instance where both of these criteria hold true is building a docker container for a BuckleScript (or ReasonML) project.

BuckleScript is primarily a compiler that compiles either OCaml or ReasonML code into JavaScript. Therefore, it would seem that bs-platform (the NPM BuckleScript dependency) should only be required as a devDependency and used during the build process. However, bs-platform also contains some code that must be included in the project during runtime. Therefore, bs-platform must be included in the project as either a dependency or a peerDependency.

The simplest option to include bs-platform in the project is to add it to the dependencies field of package.json and allow NPM (or yarn) to install it locally into the project's node_modules directory. This method works great on a development machine where npm dependencies are cached and only need to be installed again if the package is deleted from the local node_modules directory. However, when installing npm dependencies in a docker container, any change to the package.json file would trigger a full install of all of the npm packages. This typically isn’t a problem for smaller npm dependencies that take less than a few seconds to install, but bs-platform installs and compiles an OCaml compiler from scratch. On my fast machine, this process takes over 6 minutes. On my slower machines, this process takes nearly half an hour. Waiting for over 6 minutes to build the docker container whenever anything changes in the package.json file is unacceptable. Especially during development when package.json changes nearly all the time.

Waiting for docker to build the image with bs-platform installed every time as a local dependency....

So now the situation is this: we have a dependency that takes an extraordinarily long time to install AND this project is the only NPM project running on the system (i.e. docker container). Looks like we have a perfect candidate for breaking away from best practice of installing the dependency locally, and, instead, install the dependency globally ahead of time. This will allow us to install bs-platform a single time and cache it as a docker layer. Then, any changes to package.json will happen on a subsequent docker layer without requiring the reinstall of bs-platform.

Next, since we need to resolve the globally installed dependency at run time, we include it as a peerDependency in package.json. This will inform our build tool (Webpack 4) that the project requires bs-platform, BUT it should already be installed on the system.

Finally, we configure webpack to resolve bs-platform as a globally installed dependency instead of a locally installed dependency by adding the following lines to webpack.config.js:

  resolve: {
    alias: {
      'bs-platform': path.resolve(execSync('npm root -g').toString().trim(),
                                  'bs-platform')
    }
  },

When we are ready to build the ReasonML project, we will run bsb -make-world and webpack --mode production which will output our final JavaScript file to send to the client’s browser.

Building Packages with Docker and Hosting Them in a Private APT Repository on AWS S3

Recently, I’ve needed to automate the installation of a specific package on an Ubuntu machine. Normally, this task would be as easy as apt-get install <pkgname>, but the officially hosted package on the Ubuntu APT repository was not built with the required configure options. The most maintainable solution to this problem then was to download the source, compile it with the right configure options on the target operating system distribution, and upload the resulting package to a privately hosted APT repository so that it could later be downloaded with apt-get install.

The challenge of compiling the source code on the target operating system distribution turned out to be an annoying issue since the target operating system was a completely different operating system than my development environment. One potential solution would be to spin up a virtual machine of the target operating system distribution, run the build script on that virtual machine, and then copy the newly built package to my development machine where I could upload it to the private APT repository (I did not want to upload the resulting package to the APT repository directly from the VM since I did not want to copy my secret credentials of the APT repository onto another machine). Although this solution would work, it would be very annoying and time-consuming to spin up a new VM, run a script on that VM, and then copy files to a different machine every time there was a change in the build script, build process, or source code itself. Annoying and time-consuming tasks are the enemy of a fully automated system, so I wanted to find a way to run the entire process from my development machine with a single command. The answer was containerization. Specifically, Docker.

The more convenient solution is basically the same as the annoying solution except the VM is replaced with a Docker container. A single script on the development machine creates a new Docker container of the target operating system distribution, copies build scripts into the container, runs the build scripts inside the container, and copies the resulting packages back to the development machine where they can be uploaded to the private APT repository.

As an example, I will walk through the specific solution of building Squid 3.5 with SSL enabled for Ubuntu 16.04 and upload the resulting packages to a private APT repository hosted on AWS S3.

The first action in this script is to create a Docker container. On creation, the Docker container runs the build scripts and places the resulting packages into the $CONTAINER_ARTIFACT_PATH directory in the container. The script then copies each package from the $CONTAINER_ARTIFACT_PATH directory on the container to the $LOCAL_ARTIFACT_PATH directory on the development machine. Once all of the packages are on the development machine, the development machine pushes them to the APT repository hosted on AWS S3 using deb-s3 with the AWS credentials configured on the development machine.

Get the rest of the code here on Github.

Automate Everything with the Right Tools for Security and Profit

The security of a system is only as strong as the metaphorical “weakest link” in that system. In the case of our product, the weakest link tends to be the deployment of our infrastructure. Although the engineering skills at our company are spectacular, our combined experience in DevOps is less substantial. That lack of experience, plus the startup mantra guiding us to “move fast and break things” has resulted in an infrastructure deployment strategy that is a manual process twisted up in a mishmash of infrastructure deployment technologies. Sometimes the delicate ecosystem of deployment technologies miraculously manages to deploy a running system into the production environment, but, more often than not, mistakes are made during the deployment process that cause the maintenance of the running system to be a major headache.

In an attempt to make the deployment of the infrastructure more reliable and secure, I have targeted two rules that must be followed to improve the infrastructure deployment strategy:  automate everything and use tools only as they are intended.

Automate everything

To automate everything is one of the core requirements of effective DevOps. Fully automating a process is the only way to scale that process faster than linearly with respect to active human operators. In addition to the DevOps advantage of making possible the unprecedented scaling of system, automating everything also results in several benefits directly related to security. In this section, I describe some of these benefits.

Reviewable infrastructure

At the company, we have a very strict and well-defined process of merging code into the main application. One of the steps in that process includes a mandatory code review from certain owners of the specific component that is being modified. This code review not only ensures quality of the submitted code changes (and, therefore, of the entire codebase itself), it also ensures security of critical components of the application (such as the authentication mechanism) by requiring the security team to review the code before it is submitted.

Similarly, the automated parts of our infrastructure require code reviews from owners of the specific sections of infrastructure that are being modified. However, since part of our infrastructure deployment is manual, those changes require a ticket to be submitted through a change management system which eventually end up in the inbox of a member of the operations team. That member of the operations team, who likely has complete administrator access to the system, then manually implements the change. Although we can log the changes that are manually made to the infrastructure using an auditing service like AWS CloudTrail or AWS Config, discrepancies or mistakes in the implementation can only be noticed after they have already occurred (if they are even noticed at all). Fully automating all of the deployed infrastructure allows us to apply the same rigor of review, for both quality and security, of deployed infrastructure changes as we do for code changes of the main application before the changes are even applied to the system.

Auditable infrastructure

Where reviewability of infrastructure changes is useful before the change occurs, auditability of the infrastructure is useful after the changes have already been applied to the system. Fully automating the infrastructure means that all of the infrastructure is recorded in updated documentation in the form of the automated code. In the case where you, or an external auditor, needs to review how the infrastructure is designed, you can simply refer to the latest version of your automation code in your version control system as the most up to date documentation of the infrastructure.

If any of the infrastructure is deployed manually, then the documentation of that part of the system must be manually updated. The tedious, and typically less prioritized, process of manually keeping documentation in sync with an ever changing project inevitably results in an outdated, and incorrect, description of the infrastructure. However, if the infrastructure is constructed based on the documentation itself, as in a DevOps fully automated system, then the documentation will always inherently be in sync with the real system.

Disaster recovery

In addition to being able to audit, review, and version control a fully automated infrastructure, it is also significantly easier to relaunch a fully automated infrastructure in its entirety. Even though an entire system may not typically need to be completely redeployed under normal conditions, a disaster may require the entire system to be completely redeployed into a different region. The recovery time objective (RTO) of a critical system is usually very short, which requires the mean time to recovery (MTTR) to be as quick as possible. With a fully automated infrastructure, the MTTR can be reduced to the time it takes to press one, or maybe several, buttons (or maybe less if disaster failover is also automated!). Not only is the MTTR of a fully automated infrastructure quicker than a partially manually deployed infrastructure, it is also more reliable and significantly less stressful to deploy it.

Automatic rollback

One advantage of using a version control system is that each iteration of the system is recorded. Not only can you review previous versions of the system, you can also deploy previous versions of the system. This feature is especially useful if a mistake is made in the infrastructure that needs to be immediately rolled back to a previous state. In a manually deployed infrastructure, it can be difficult to even remember what changes were made, and even more difficult to figure out how to reverse them.

No snowflake systems

Another security challenge associated with infrastructure deployment is updating security patches and configuration settings on each of the resources. For example, if a piece of software running throughout your environment requires a security patch, then that update only needs to occur once in the automation code. Similarly, if the configuration of all load balancers needs to be updated to use a more stronger security policy, then that update only needs to occur once in the automation code. If these changes were made manually to each system, then, depending on the complexity of the change, the human operator is likely to unintentionally configure each one slightly differently. Slight differences of configuration settings of the systems can lead to security vulnerabilities that may go unnoticed for a very long time.

Use tools as they are intended

In addition to automating the entire infrastructure, choosing the right tool for the job is very important from a security point of view. Using a tool that is designed for a specific task helps to ensure readability and reliability of the deployed infrastructure defined by that tool.

More specifically, when defining an infrastructure, use an infrastructure definition tool (CloudFormation or Terraform). When configuring servers, use a server configuration tool (Chef, Puppet, or Ansible). When defining a Docker container, use a Dockerfile. When packaging and distributing a piece of software, use a package management system (yum, apt, etc.). Although it seems obvious to use the right tool for the job, each tool requires time and effort of the human operator to learn to use effectively. Couple the extra effort required to learn to use the tool with the fact that many of the tools also offer features to half-heartedly accomplish other tasks, many human operators are tempted to use a single tool outside of its intended domain. Although learning and using a single tool while ignoring other potentially more logical options may seem like a time-saving temptation, the added complexity of using a tool outside of its intended domain results in layers of technical debt that inevitably take more time to resolve in the future.

One example that I have seen of using a tool outside of its intended domain is using Red Hat’s Ansible, a server configuration tool, to define an infrastructure. The main difference between the configuration of an infrastructure and the configuration of a specific server is the number of configuration points in each of these types of systems. An infrastructure has a relatively limited number of configuration points (ELB options, route tables, etc.), whereas a server has an intractably large number of configuration points (installed software, configuration of that software, environment variables, etc.). Because of this difference, infrastructure tool templates are easier to read and understand than server configuration tool templates since infrastructure tool templates are able to explicitly define all configuration points of the system. On the reverse side, server configuration tools are only able to explicitly define the desired configuration points (make sure package A is installed with a specific configuration) while ignoring any part of the system that has not been mentioned (do not uninstall package B if it was not mentioned in the server configuration template). The additional complexity requiring the understanding both the server configuration and the initial state that it applies to is necessary for server configuration, but that additional complexity is unnecessary for infrastructure definition. Therefore, the additional unnecessary complexity of using a server configuration tool to define an infrastructure introduces unnecessary risk into the deployment of the infrastructure.

Another example that I have seen of using a tool outside of its intended domain is using Ansible (again) combined with preprocessing shell scripts to define Docker containers. In this instance, several bash scripts would generate a Dockerfile by replacing variables in a Dockerfile.tpl file (using a combination of environment variables and variables defined in the bash scripts themselves), build the container by running the newly created Dockerfile artifact that would run an Ansible playbook on itself, and then upload the resulting container to a remote container repository. Later, several shell scripts from another repository would pull and run that container with variables defined from the new environment. Needless to say, following the variables through this process or recreating a simple local environment of this tightly coupled system to test these containers proved exceedingly difficult. Understanding that most of this process could have been defined in a single Dockerfile (without Ansible or the complicated preprocessing scripts), accepting this high-level of unnecessary complexity equates to accepting a high-level of unnecessary risk in deploying this system. (In fairness to the writer of this process, that system was initially created to deploy a system directly onto a VM. The containerization of the system was later added as a constraint, and insufficient resources were granted to properly rewrite the process to address the new constraint.)

Solution

Although automating all of the infrastructure and choosing the right tools for the job is a difficult and time-consuming task, it is a necessary task to create a resilient and secure infrastructure. In this section, I describe a workable solution using several specific tools. These tools may not work for your specific system, but they may provide a good place to start.

• Infrastructure definition - Use CloudFormation to define resources in the cloud such as VPCs, route tables, SQS queues, and EC2 instances. Include the installation of a pull-based server configuration agent on each EC2 instance defined so that it will be able to configure itself when it boots.
• Server configuration - Use a pull-based server configuration tool, such as Chef, that can define the configuration of each server in the infrastructure based on the “role” of that server (secure transparent proxies have configuration X, bastion hosts have configuration Y, etc.). When the machines boot up from the infrastructure definition tool, they automatically pull their own configuration from the server configuration tool.
• Container building tool - Use a Dockerfile to define how a container should be built. The additional complexity of requiring preprocessing with bash scripts or self-configuration with Ansible is likely to be a warning sign that the system is not designed properly. Reassess the design and try to follow Docker’s best practices.
• FaaS deployment tools - I am a fan of running small services as FaaS since most of the infrastructure responsibilities are delegated to the cloud service provider. Launch these services with FaaS deployment tool such as Serverless.

Although developing a resilient and secure infrastructure is a difficult and complicated task, following these two rules will immediately take you a long way. Also, as an added benefit, your security team and auditor will thank you.