10 signs of a mature development pipeline
How to deliver software in a reliable and resilient way
We all know the speed of developing features is crucial for online businesses. Times when software was created in big chunks delivered once a year or at most once a month are gone. We now live in continuous delivery times, where new components of our system must be available for the end user immediately.
In this blog post I present 10 signs of a mature delivery pipeline. They will let your developers be as much productive as possible and meet clients needs rapidly.
1. You use Git for version control
First things first. You must be sure you use the best tools out there. There were a number of version control systems in the past: CVS, Mercurial or SVN. But the current undeniable leader is Git. It is fast, distributed and lets developers be productive even offline. If your organization still uses an old-fashioned version control system you should migrate to Git as soon as possible.
It lets your developers work on a lightweight feature branches, incorporate git-flow style and easily fix conflicts. It allows them to easily review each others code and, most importantly Git has gazillion integrations that can be easily plugged into a code delivery pipeline.
2. You have a Continuous Integration server
If you want developers to deliver changes quickly, you must ensure they do not break stuff that is already implemented in the application. You can verify that with a Continuous Integration server.
Tests are sometimes time-consuming and it is much better to run them automatically instead of relying on developers to execute them on local machines. Developers’ main concern is to deliver features as soon as possible. It may lead to breaking stuff from time to time.
On the other hand your main concern is to build a platform that is resistant to developers mistakes and does not allow them to introduce new bugs to the system. With Continuous Integration you can rest peacefully, because it verifies the code right after commits are pushed to the remote repository. With this tool you are on a good way to build a reliable software delivery pipeline.
3. You have multiple environments
If you can verify the correctness of application on the CI server, you should be able to deploy it somewhere as well. The most common pattern is to have 3 environments: sandbox, staging and production. With them developers can verify how the app is behaving among the others. You may have services, queues, databases that are hard to recreate on a local environment so it is better to do it in a single one.
First versions of the application are deployed to sandbox where you can verify if the app fits to your stack and how it behaves. A lot of bugs can appear here and it’s totally safe, because this is still a development environment. Developers can fix them on the local machine and deploy to the sandbox again.
The next environment should be staging. When you use microservices architecture it is worth to start multiple instances of the application in this environment. While on sandbox you verified the app integrates with other parts of the system correctly, now it is worth to verify how it behaves in the environment that is more similar to production.
The rate of changes in staging is much lower than in sandbox. This allows you to also run automated tests in this environment and verify the most common paths in the whole system, check how your application fits into them. If they pass, the possibility of having bugs in production is much lowered.
The last environment is the production which is the place where your system lives and is available for your clients. If the app went smoothly through tests on the local machine, was built by the continuous integration server, passed sandbox and staging tests it can be served to clients. Number of bugs has been lowered as much as possible, so it is highly possible that everything will work on the clients side.
4. Developers can deploy the newest version of application to production with a single click
Next step is the ease of deployment. Do you remember the times when you were building jar files locally and sending them to your application server like Tomcat or Glassfish? They are totally gone.
With Git as a source code repository and build in continuous integration pipeline it should be as easy as possible for developers to deploy the newest version of application to production with a single click. It can even be a terminal command or chatbot message as well. There should be no place for building packages on the developers’ side anymore.
The project pipeline should do this for them, so they do not lose time for deploying the application, but do what they are best at — which is designing and developing code.
5. You can monitor the behavior of your application in production
In the old days, developers were crafting the code for couple of months, delivering a package to another team, and starting a new project. They were not aware of what was happening in the production. That is not the case anymore. Nowadays developers should be responsible for monitoring application behavior in production.
To enable this you must build a set of tools that makes it possible. Metrics collection (e.g. Prometheus or Dropwizard) and Grafana are one of the most popular right now.
Why is it important that developers are in charge of monitoring the application? They have the best knowledge of how the stuff works underneath and therefore they can spot any errors in a matter of minutes and fix them in a short period. Thanks to that you ensure your clients receive the best user experience possible, and in case of failures bugs are fixed.
Instead of errors being reported by clients and reaching developers through a couple of layers of your organization, they can be spotted immediately. To make it possible the monitoring is a must.
6. You have alerts set up
It would be better to know about failures before your clients, wouldn’t it?
Or at least, before the majority of them become aware. If you have set up monitoring already, the next step is to set up the alerting.
By measuring stuff you can define circumstances when your application is not usable by clients. That can be slow response times, glitches on databases, or an unexpected big load that makes your application throttle.
If your team is being alerted immediately, they can recognize the issue, estimate the resolution time and you can communicate to your clients what is happening.
In such scenario you are in charge of failure, taking care of your clients and telling them that everything is under control. You are perceived much better in comparison to a scenario when your clients inform you that something is going wrong, and you do not even know what actually happens.
7. You have a centralized logs server
Failures happen. Monitoring is set up, alerting has worked, you fixed the issue. But the next day you want to dig deeper to find the root cause of the failure.
If you work in the microservices world, it would be a hard work to log into every single machine to checkout the application logs.
Instead of that, you should collect them to one single place. You can use ELK stack or Graylog. It doesn’t matter. What is important is to have durable, searchable and aggregated logs from all the places, so developers can easily spot on them and dig into details of the failures. Checking each single node, looking for log files and digging through them is a Sisyphean work. Do not lose money you pay developers on something like this.
8. You can rollback changes in a matter of seconds
Even though your team uses all best quality assurance practices, sometimes the failure happens only in production. All tests passed in the local environment, continuous integration server, sandbox and even on staging.
But production is a different one. It has real traffic, real data, etc. It can be some stuff related to configuration, infrastructure, or communication with other services.
If application is unusable right after the deploy, you must be able to rollback changes to the last working version. It should not require going through all pipeline stages once again, resetting the Git master version, etc.
It must be possible to simply choose the version to which application should be restored and replace the current one immediately. The software was already tested, it went through the whole pipeline. You should not waste time for building the package again and running all the tests. You need to revert quickly.
9. You can deploy hot fix in a matter of seconds
Some changes are impossible to be rolled backed. You have introduced changes that requires a quick fix to make your application works stable. You must be able to commit changes and deploy them to production passing by your whole anti-bug pipeline. It is a risky stuff, but situations requires it couples of times.
You must be sure you can deploy hot fix without waiting for your CI to finish running tests. You know what you do, so you must be able to deploy a hot fix in a matter of seconds. It’s a risky thing, but the only possible solution to minimize the time when system is affected by a fail change introduced with the last deployment.
Make sure your CD pipeline has this feature if you do not want to lose trust of clients. And of course, you should not use it on a daily basis. You should pursue to eliminate such deploys in your organisation.
10. You can configure runtime properties without redeployment
And the last (but not the least) sign of a mature development pipeline. Possibility of changing the runtime properties without redeployment.
If you make your software open for reconfiguring without restart or redeploy you make it possible to react on stuff happening in production. Whether it’s blocking an offending client, short cutting the broken downstream service or increasing the number of IO threads. If you can change those settings without a need of redeploy you can slip from a failure without even a notice ;)
Wrap up
It is crucial to build resilient development pipeline to make sure your application is serving well for your clients. It is worth to invest time in the infrastructure, ease of development and deployment. It costs some time and money, but it pays off in the long run in happiness of developers and trust of clients.
You do not have to deliver all points of this article from the very beginning, but make sure you are reaching the goal of having a mature development pipeline.
BTW, we are always looking for outstanding professionals to join our team!
