Table of Contents
In our company work the developers that do their best to develop qualitative software. We use the test-driven development approach and write the tests before the actual code to make it as reliable and effective as possible.
Additionally, we use automated builds for running the tests on every project when making a branch on Gitlab.
However, with the growth of our company, the infrastructure couldn’t handle the number our developers uploaded into DVCS. We came up with several solutions for this challenge.
Thus, we tried to run our tests faster than it was on the personal stations of our developers. Read this article to find out how we implemented this solution.
We also tried to resolve an issue with our infrastructure with an affordable price for its support.
Before moving all the infrastructure to Autoscale runners, we used self-hosting hardware located in our office. Among the expenditures were power costs. One machine cost was about 2-3$ per month. They functioned around-the-clock even if they were not loaded.
Moreover, because of loading, we couldn’t run the necessary number of builds concurrently. As a result, the average amount of build’s timeout could take up to 10 minutes. All the builds were terribly slow and unstable. All of these significantly slowed down the development process.
Solution
Our DevOps department reached a solution to start using Autoscale EC2 instances. This provided us with a possibility of using the necessary “power” as well as keeping the builds on in the needed moments.
So we got rid of our self-hosted solutions and start using AWS EC2. This was the beginning of our new infrastructure optimization.
We had 4 machines that were used as Gitlab Runners. The cost of every machine was around $500. All other vacant machines were given to our new developers so that we could save much money on buying the new ones.
We used the following runner’s settings.
Based on the settings, we could concurrently run 20 builds on 10 machines with 1 GB memory on each one.
We used packer for creating our personal AMI . Here’s how a config looks like.
This config allows creating AMI that is suitable for our work. There can be found the settings for ElasticSearch, SWAP as well as our keys for connecting with GitLab.
The Result
Using this configuration we significantly increased the effectiveness of our runners. Moreover, we solved an issue with builds lagging.
Thanks to Knapsack, all the builds have been parallelized to the great amount of machines.
Using Knapsack and fail-fast strategies, we were able to reduce the average amount of build on 25-30%. And now let the numbers talk for themselves.
The total infrastructure’s cost: $23
In contrast:
4 machines:
Monthly ~$12-14 + time spent waiting for the builds
1-time payment: $2000 cost of unused machines
For 1 Year: $2156
Autoscale Runners:
Monthly: ~$23 with no time for waiting
For 1 year: $276
As you can see, we successfully reduced both the company’s expenditures and time for the code delivery.
