Table of Contents
This article explores the most frequent issues related to Continuous Integration and Deployment (CI/CD). It was written by our developer, Dmitry, and reflects his unique insights inspired by his development experience in JetRuby. We are glad to share our proven best practices applied in the CI/CD process.
These methods help to reduce “time to market” and streamline the entire development process. If you’re looking for ways to build (we know you are) modern, cloud-native software applications, then the 12-Factor Methodology is not unfamiliar to you. These principles were introduced by Heroku co-founder Adam Wiggins in 2011 and have since become widely adopted in the software development industry.
As one may expect, a mere adherence to the 12 rules will be enough for troubleshooting. However, you can’t continuously streamline the app development process per those 12 rules. I aim to examine the CI/CD issues I most frequently encounter.
Many Deployments with One Codebase Tracked in Revision Control
For sure, the most frequent issue is precisely the first problem listed in the 12-factor Methodology’s flow. This often manifests in the fact that “dependencies” of the environment are hardcoded, which does not let the app work in the new environments different from the one it was built in. It’s pretty simple to stop using configuration files and entirely rely on Environment Variables — they are cool, indeed. By the way, the third rule of the 12-factors methodology describes using configuration files.
Almost every programming language has a library implementation that reads the file variables before launching the application. It prevents issues with the local development of this application. The next problem we’ll discuss relates to another rule of the mentioned methodology.
Build, Release, and Run
Sometimes, you’re strained for time to create a Continuous Delivery (CD) process to build artifacts (application image, bundle, etc.) and store them in some place with the thought of further deployment. In this case, you just get to “extract” the code from the repository, gather all dependencies, and send everything needed to the server in a single action.
Some services (hosting platforms) are configured (designed?) for this process by default. For example, the “build” step appears to be skipped in AWS Elastic Beanstalk, although these processes are separated. However, the practice proves that it’s essential to keep each step in the build -> release -> run chain atomic (meaning each step is complete, separate, and fulfills its task) and repeatable. After all, the project needs to change the branching model sooner or later, be deployed/not deployed more frequently, and set up development/QA environments from specific branches.
We separate these stages as follows:
1) Build: This is when the application is “assembled,” pulling all necessary dependencies together. All required commands are executed (compilation, transpilation, dependency configuration, etc.); we pack the application and then send it to a storage location.
2) Release: We configure the application to operate in a specific environment. Often, web applications are configured during runtime. This step somewhat “spans” across the other two.
3) Run: This is the actual deployment of the application into a specific environment. Ideally, you can build this step to launch the application from any branch in any environment (yes, “let’s deploy the development build to production and test it” should not be extraordinary for you).
In the end, we have three (ok, two and a half) clear-cut steps.
Build
We can launch the first step on any branch or at any commit, assemble a “build,” “image,” “bundle,” or executable code from any code version for any use case. It works for debugging, hypothesis testing, feature testing, early access for internal testing, vulnerability assessment, etc. There are the following best practices that help to streamline these processes through Continuous Delivery (CD):
1) You receive the same artifact regardless of the code version it was launched with. It means Idempotence in the process of building dependencies (hereafter, dependencies refer to all libraries, repositories, and system utilities needed for proper compilation and execution of the application). After launching the build step N with the same branch, you obtain the same artifact. For instance, this process may involve installing gems or Node.js packages.
2) If possible (for example, this is almost unattainable for Single Page Applications (SPAs), but still possible), strive to exclude environment-specific dependencies during the build preparation process (such as backend URLs, database credentials, or launch configurations).
3) Exclude checks and conditions from the build process. You don’t need to run code linters or vulnerability checks at this step — shift these tasks to separate stages.
4) Only install/package dependencies are needed to run the app. Avoid including unnecessary things in the final artifact, such as a reverse proxy with configurations, tests, Kubernetes configurations, linters, and testing frameworks.
These principles help ensure the build process is consistent, efficient and focused on producing artifacts relevant to the intended use case.
Release
As pointed out above, the lines between the first and second steps are often blurred. In the case of the mentioned SPA (Single Page Application), we add some configurations during the build process to closely tie the artifact to the deployment environment. The same applies to the app deployment. This process often includes app configuring for the work in a specific environment. Everything depends on the application, the deployment environment, and the approach. Therefore, it’s impossible to shape specific recommendations for every situation.
Run (Launch)
The final step often determines the further app delivery. What we do here directly impacts how easy the delivery process will be. At this stage, the application code should no longer be a dependency. Ideally, we should deploy the application build from the previous step, with all necessary settings, into a specific environment. The most frequent problem at this step is when deployment scripts are tied to the application code. I once felt tempted to deploy scripts using application libraries working as dependencies. This partially brings us back to the first step, when we must install dependencies to deploy the application. It leads to troubles, not to mention it is not convenient at all.
Deployment should be performed with the most convenient tools, which the application should not dictate.
Problems of using cache in CI/CD
Another equally common yet unpleasant issue is cache usage in CI/CD processes. This is perhaps one of the first improvements on a project, as a well-configured cache can save a lot of time. However, improperly configured caching sooner or later can consume all the time you save by introducing subtle bugs or even failing work pipelines. This issue often arises because the entire project uses a shared cache repository. For instance, the cache is used between different branches and steps without considering the current language version. Usually, this fails the project pipeline built with an older version when its upgrade to the next version occurs after a cache update.
For example, you built it with Ruby version 2.5 in the CI/CD system. If someone updates dependencies, upgrades the version to 3.1, and updates the cache, it can result in the pipeline’s failure with the old version due to non-obvious reasons. In extreme cases, this cache is used for application builds, and problems emerge in the final environment.
What can I do to avoid these problems?
To prevent the issues we described above, we suggest the following practices in “building” and writing the dependency cache for CI&CD:
1) Use the version of the main dependencies (language or framework) explicitly specified in the CI & CD and dependency configurations. This approach allows you to painlessly update and experiment without the risk of breaking all things around.
2) Use a lock file (often with a hash function to get a short identifier for the entire file).
3) In extreme cases, use the names of the main Git branches — master/production/qa/dev, etc.
Using Docker
Misuse of Docker is closely related to the issues I touched upon above. Suppose the project is deployed in Kubernetes, and we have a docker file that describes the production environment. However, the Docket Image may start receiving files built or prepared outside the docker at the build stage. As the most common example, this involves throwing the dependencies cached in the CI & CD system into the docker image. Therefore, if the project uses Docker for assembly, you should apply Docker to prepare dependencies. If there is a problem using any system utilities, do not be lazy and make a “base” image, which can later be used to run tests in CI&CD and build the application. Don’t forget to version this image. As a bonus, patching system dependencies and fixing vulnerabilities will be much easier.
The need for additional services
Some applications cannot be deployed without additional services. Suppose React SPA needs to be delivered to the user. Then, we remember that React does not have tools for processing HTTP requests (if we don’t consider the Next.js framework). Therefore, a web application and some web servers are sometimes packaged in Docker. This is how to create a heavy monster from a simple static site. The final build should contain only the application and the dependencies needed to run it. If the application is launched by the user on the computer, then the build should not contain the means of delivering the application to users. This consideration is a matter of infrastructure configured and deployed separately. If the application runs on a server, the build should not contain configuration files, deployment, configuration scripts, or utilities for managing this server.
Following this rule will keep you from problems with changing the hosting or app deployment in the local environment.
CI&CD runs only in certain conditions
Finally, the problem was quite nerve-wracking for me personally, as well as for the manager and client. This is CI & CD, which runs only on a specific system and under certain conditions. There are several unrelated reasons for this. They include incorrectly configured conditions and CI&CD closely tied to the system on which it runs. If we discuss the conditions in which we need to take specific steps, the most frequent problem is the failure of deployment due to the failure of tests. It happens even more often than a viral meme “Karen, we dropped the prod”.
Theoretically, an adequately configured CI should not even lead to situations where tests fail, or the code linter does not allow changes from the hotfix. Therefore, all the most stringent code review rules should be set up at those stages of development when testers or the client has not yet accepted the code. Set up the automatic launch of tests and linters to open a pull request and run them only on a merge commit. This way, you ensure that people don’t forget to fix conflicts and that the checks consider code changes already in the branch where this pull request is open.
Always leave the option to run the build -> release -> run part manually, allowing developers to upload their code even if they used tabs instead of spaces. Have you ever faced the situation when you need to deploy a hotfix, and GitLab is down because someone is running ten parallel feature tests? Or has Bitbucket fall off on your project? Or have you encountered multi-day “moves” from one service to another? Often, this occurs because the entire CI&CD process is tied to a specific implementation and does not imply the need to run, for example, local deployment. This is a widespread practice. Nobody wants to spend time writing Makefiles or even Rakefiles, not to mention writing the CLI in JavaScript.
I recommend spending some extra time and setting up the project to be able to run all steps from CI&CD locally. Utilize any convenient tools so that any team member with the right keys can build and deploy your application to production faster than Atlassian support fixes BitBucket.
Conclusion
I hope my article has provided some ideas on running the fast CI/CD process and avoiding a few tricky pitfalls. Subscribe to our blog to keep up with other reviews of our experts and improve your development practice!
Other articles that may attract your attention:
