Developers know the pain of manually handling deployments, testing and code versioning. It’s time-consuming, prone to errors and downright frustrating when things break at the worst possible moment. But when CI/CD is done right, you achieve results faster and with higher quality. Just ensure you follow the CI/CD best practices established by industry experts and open source communities, making your process smooth and reliable.
Having built countless CI/CD pipelines over the years, I understand the frustration of not finding such web development information in one place. So, in this guide, I’ll share my tried-and-true strategies for setting up CI/CD environments from scratch. I’ll also share some of my best-kept secrets on improving your setup over time.
Key highlights:
- CI/CD automates the integration, testing and deployment of code changes, keeping updates reliable and fast.
- Best practices for CI/CD pipeline, such as trunk-based development and streamlined environment management, accelerate development cycles and improve deployment efficiency.
- CI/CD tools like GitHub Actions, Jenkins and hosting-based platforms simplify secure CI/CD pipeline management and promote team collaboration.
What is CI/CD?
CI/CD stands for continuous integration and continuous delivery (or deployment). It is a reliable process to integrate code or configuration changes into existing software and deliver them to users. This automated pipeline ensures that new updates are properly tested, combined with the current codebase and safely deployed to users without manual intervention.
Now, every word of CI/CD has a deep meaning. Let’s quickly go over each one of the key terms to better understand why they’re all combined into a single process.
Software development stage | Purpose of the stage |
---|---|
Continuous integration (CI) | In software development, everything changes dynamically and requires constant updates, bug fixes, security patches and other improvements. Long gone are the times when a new application version was delivered every year. Now, it’s a weekly update or even quicker. Continuous integration helps support this schedule where the software must accept changes incrementally and nonstop. |
Continuous delivery (CD) | Once development changes are made, it’s time to make them available for all users. But anything can go wrong when bringing changes to multiple devices and backend systems. Continuous delivery fully automates this process to ensure reliable updates, including the ability to quickly roll back changes in case of emergencies. |
Why is CI/CD important?
CI/CD tool users perform better across all the DORA metrics, according to the 2024 State of CI/CD Report. While 40% of developers not using CI/CD platforms reported a code lead time for changes performance of more than one month, this figure dropped to just 19% for those using self-hosted and managed CI/CD platforms. The takeaway: CI/CD is clearly associated with improved delivery performance.
CI/CD benefits include:
- Ensuring the deployment process is reliable
- Saving team time on every project build or deployment
- Speeding up delivery time
- Reducing the number of errors on production
- Improving security at every step of the deployment
- Providing a notification system for critical steps of the delivery process
Key CI/CD repository approaches
One of the core requirements for a successful CI/CD is a shared place to store all code. Most of the time, it’s a Git repository hosted on GitHub, Bitbucket or GitLab. Once the repository is created, here are the common approaches to working with it — and all affect CI/CD significantly.
CI/CD repository approach | How it works | Pros | Cons |
---|---|---|---|
Trunk-based approach | There’s only one main branch in the repository. Every team member can create their own branches, but these are short-lived, lasting no longer than one day. After the changes are made, a ready solution is merged back into the main branch. |
|
|
Limited environment-based approach | This is a classic approach when there are several environments (usually two or three) that the feature has to pass through until it’s released. At a minimum, there are two environments: development and production. All the functionality is first developed in the development environment. After testing and approval, it gets merged into production. |
|
|
Unlimited environment-based approach | This approach simplifies integration by allowing developers to verify their functionality in their own branch rather than having it merged into a single environment. |
|
|
How to choose a reliable and scalable CI/CD platform
There are two major categories of CI/CD platforms:
- Repository-based platform
- Hosting-based platform
There is a third approach through special services that does only CI/CD. But now, it has mostly been integrated into either of the other two approaches and is no longer mainstream.
I prefer keeping everything in the repository, so that’s my favorite approach, which I would go with most of the time.
Repository-based CI/CD platform
Every popular Git repository hosting offers its own system that can complete all CI/CD work.
- GitHub offers Actions
- Bitbucket and Gitlab call it CI/CD Pipelines
According to a study published on arXiv, GitHub Actions is the most popular tool, being used in 57.8% of CI/CD repositories.
Here are the steps to enable CI/CD in GitHub Actions:
- Create a .github/workflows folder in your repository.
- Include at least one .yml file with the necessary configuration.
- Push your code to the repository. GitHub automatically detects the configuration and executes it when the specified conditions are met.
- Define the rules and operations to be performed in the .yml file. The basic rules of GitHub Actions workflow include a branch name that GitHub checks to define if it needs to run your workflow.
- Split your process into multiple jobs — for example, “test,” “build,” “deploy,” “backup” and so on.
- List all steps required to get the job done for each job. For example, a “build” job may contain dependencies installation, building of your project code and running optimizations.
Here is a sample main.yml file to illustrate a possible workflow structure. It’s triggered when someone pushes into the “main” branch.
name: Build main on: push: branches: - main jobs: build: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Set up Node.js uses: actions/setup-node@v4 with: node-version: '20' - name: Install dependencies run: npm install - name: Run tests run: npm test - name: Build the project run: npm run build - name: Upload build artifacts (optional) if: success() uses: actions/upload-artifact@v4 with: name: build path: build/
Hosting-based CI/CD platform
A lot of well-established CMS or frameworks already have CI/CD support from hosting providers. So, you don’t have to configure almost anything in your repository. Just connect it with your hosting provider by granting it basic access to your repository. The hosting provider’s CI/CD will automatically link your branches to the corresponding environments and handle all deployment tasks for you.
In case your hosting provider doesn’t have built-in CI/CD support, it is always possible to configure it with your repository and basic FTP/SFTP access.
Dedicated CI/CD services
The most popular tools that are not repository-based include:
- Jenkins
- CircleCI
- Cycode
- Travis CI
- Azure DevOps
- Bamboo
- TeamCity
Despite not being inherently tied to repositories, these tools can be easily integrated with your code repository. One notable advantage of Jenkins, for instance, is its self-hosting capability, which provides full control over deployment security and the potential for significantly improved deployment performance. Additionally, Jenkins and similar tools offer exceptional flexibility, enabling virtually unlimited integration options and extensive ways to visualize the deployment process.
Another benefit that dedicated services can offer is automatic dependency control, detailed monitoring, deep security risks analysis and vulnerability prevention. Cycode, for instance, has built-in source code leakage detection, secrets scanning and real-time threat detection. These features are in high demand in security-sensitive industries such as banking, finance, communications, the military and high-tech. Having a dedicated service can save significant resources compared to building a similar solution in-house.
6 best practices for successful CI/CD
Here are the best practices that you should follow when building a standard CI/CD pipeline. The exact tools will vary depending on your technology stack, but there is a common logic behind any CI/CD.
1. Choose one deployment method
It doesn’t matter which CI/CD platform you have chosen — it must be the only way you deploy changes. Sometimes, it’s tempting to log into your server and modify several files manually. But there are several disadvantages, as it can:
- Cause multiple other issues related to caching, bugs and unexpected behavior.
- Make it hard to track those changes and then understand how to fix them.
- Create a potential security loophole in your CI/CD system.
2. Keep in sync with production
In case you have several CI/CD environments, keep them periodically synchronized with production. It helps to:
- Reduce the probability of a new bug going unnoticed during a new release, just because a dev environment lacks data.
- Address the bugs discovered in production. These bugs are often hard to reproduce in other environments because the new environments don’t have the same information.
Of course, not everything should be copied from production. Sensitive user data should be kept only in production or in your backup system.
3. Run tests locally
A lot of tasks can be checked locally during development. It reduces the time that developers spend waiting to verify their work. Also running verifications in your CI/CD system may be charged per run, so having local verification would save some costs.
To ensure that a new code is not breaking any guidelines and basic coding standards, an automated code verification should run every time changes are committed to the repository.
The most common tasks to run locally are:
- Unit testing
- Code linting
- Code formatting
4. Ensure deployment rollback capability
Always have a quick way to revert any changes caused by your CI/CD. Automatically make a backup of your system before deployment. Then, have a workflow that you can easily run to restore the changes. Create backups not only for your code but also for your database and any other systems affected during deployment.
5. Leverage manual deployment triggers
Include all frequently used operations into your CI/CD, which could help you automate the deployment process or validation of your code. You can even have handy tools like database backup, synchronization between environments, running custom commands on your servers and many other useful automation. You can then use your CI/CD as a panel for all your operations and environment management.
6. Configure notifications
It is good to know when the CI/CD process starts, but it’s even more important to know when it ends or if there are any issues during the process preventing it from completion. Set up email notifications for every important event in your CI/CD process that you want to know immediately about. Sometimes, services allow you to have text messages as a way of notifying. You can also connect the service with your Slack or Discord.
How to set up a CI/CD system for your team
Adding a new technology to your development process is always a challenge. Someone from your software development team may already be doing everything manually and be happy about the process.
And then there’s the biggest risk of having someone in charge of deployment or a part of it: what if they are not available at some point? You would not know how to deploy the changes.
So, for smooth CI/CD management, be ready to dedicate some time, up to one or two working days, for its configuration. Even if it’s 30 minutes of manual work, you will get rewarded quite soon. I recommend the following steps to go over:
- Write down the existing deployment process — all the steps that you’re doing.
- Split the process into key steps: testing, building, deployment and cache clearing.
- List down all your environments.
- Create a separate workflow file in GitHub Actions or other repository vendors like GitLab continuous integration for each environment. Every step of your process is a job in your workflow file. Search for ready job templates first. There are plenty of them.
- Complete any credentials requirement (if any part of your process requires that). It’s part of CI/CD security best practices to always store this sensitive information in repository secrets and never have it directly in your configuration file. This could be passwords, API keys, tokens and so on.
- Test every job on your development pipeline.
- Configure notifications for critical parts of your CI/CD pipeline.
- Create another workflow that would revert the changes done by the initial pipeline.
- Proceed with other environments after your first workflow is complete
- Integrate your CI/CD with a task tracking system, so you can always see what was deployed and in which environment.
Over time, you will notice that the integration and deployment process has become stable, and you will see fewer issues during deployment and fewer bugs related to it on production.
Let’s help you follow the best practices for CI/CD pipeline
CI/CD is a must-have process in any modern software development. Configure it early in your development cycle and gradually evolve while your development scales.
If you need help, get in touch.
Our team at Productive Shop has CI/CD processes pre-configured, backed by the CI/CD pipeline best practices.
We set them up for our clients — in case they don’t have CI/CD — or enhance the existing CI/CD pipelines making them more reliable, fast and error-prone. We not only configure the repository but also connect CI/CD pipelines to task management systems, such as Wrike, Asana or JIRA. This allows us to track task progress closely and improve end-to-end speed of delivery.
Contact us and let’s discuss how we can help you streamline your development projects.
Frequently asked questions
What is YML?
YML (or YAML) is a human-readable file format, often used for configuration files in CI/CD pipelines and for other configurations. There are several rules that must be followed when writing a YAML configuration:
- White spaces define the structure. So, the rules should be properly indented according to the hierarchy.
- A hyphen “-” is used for a list of items.
- Comments begin with “#”
Here’s an example of a YAML file:
# this is a configuration file for the development deployment name: Deploy dev on: push: branches: - develop
What are pipeline observability tools?
Pipeline observability tools are features integrated into CI/CD systems that provide real-time insights into the operation and performance of your pipelines. They allow you to monitor the status, progress and outcomes of your CI/CD workflows, making it easier to identify issues.
These tools function much like a car’s dashboard, which displays vital information such as speed and fuel levels. They provide a clear view of your pipeline’s health and performance, helping you stay in control, make informed decisions and ensure smooth deployment processes.
For example, GitHub Actions provides a real-time dashboard to track the progress of CI/CD pipelines. You can see previously run pipelines and analyze steps that were run. If a pipeline run is not successful, there is a way to restart it from the beginning or from a failed step. Similar functionality is available in Bitbucket and GitLab CI/CD.
Other good examples of popular tools include:
- New Relic
- Cycode
- Datadog
- Vector.dev
- Dynatrace
What is CI/CD security?
CI/CD security is the integration of security protocols and measures throughout the CI/CD pipeline. During the deployment process, connections to remote servers are established. In order to provide a secure connection, SSH or SFTP protocols must be used along with SSH keys.
Every modern CI/CD platform allows us to keep credentials in a secure place to make sure they are not visible during the deployment process in the pipeline real-time dashboard. Also, the credentials will not be visible in the pipeline configuration after they are saved. In GitHub Actions, the secured storage is located on the Secrets and Variables page.
What is an artifact repository?
An artifact repository is a special repository that keeps only the results of a build process. These artifacts may include compiled binaries, libraries, container images, configuration files, documentation and other deliverables that result from the development lifecycle. For example, it may keep archives of a ready-to-download version of software.
How can you automate a CI/CD process?
To automate a CI/CD process, follow these steps:
- Write down all of the required CI/CD steps, just in plain text.
- Configure the corresponding steps in GitHub Actions or any other alternative tool.
- Verify that every step works properly and produces the required artifacts or actions.
- Make sure that your CI/CD process starts automatically after the code is pushed into your repository.
What is the point of staging in CI/CD?
Staging usually is a client-facing environment that allows to verification of new features independently from the development environment. Most of the time the development environment is filled with testing information that, on one hand, would confuse real users or clients and, on the other hand, would not allow to properly verify if the new functionality is bug-free and works according to the requirements.
Contrary to the development environment, staging is a full or slightly outdated, but still valid copy of the production environment. So it is quite accurate and what the clients will see is likely to appear in the same way on production.
Normally, it is a CI/CD best practice to have a separate pipeline that would synchronize development with staging and production with staging environments.
Is SLSA the best standard for CI/CD pipelines?
Supply-chain levels for software artifacts (SLSA) is one of the best standards for addressing security concerns, as it’s designed to enhance the integrity and security of software supply chains. It’s also a reliable and well-defined framework to protect artifacts of CI/CD pipelines from being tampered with.
The main goal is to prevent unauthorized dependencies from being introduced into the system, such as open source software or third-party vendor code.