Continuous Integration (CI) Jobs for Maven and Node.js Projects
Introduction:
Continuous Integration (CI) is a software development practice that involves automatically building, testing, and deploying code changes whenever they are pushed to a version control repository. Two fundamental concepts in CI/CD pipelines are "stages" and "jobs." This document provides a comprehensive explanation of both stages and jobs, their significance, and how they work together to optimize software delivery.
Stages:
- Definition: Stages are logical divisions within a CI/CD pipeline that group related tasks or jobs together.
- Purpose: Each stage represents a specific phase in the software development lifecycle, defining the flow of tasks from code changes to deployment.
- Advantages:
- Modularity: Stages break down the CI/CD pipeline into manageable units, allowing teams to focus on specific aspects of the development process.
- Clarity: They create a clear workflow, enabling developers and stakeholders to understand the sequence of steps in the pipeline easily.
- Parallel Execution: Jobs within a stage run in parallel, optimizing resource utilization and reducing overall build and deployment times.
- Dependency Management: Stages can have dependencies on each other, ensuring that certain stages execute only after successful completion of their dependencies.
Jobs:
- Definition: A job represents a single unit of work or a task that needs to be performed within a stage.
- Purpose: Each job performs a specific action, such as building the application, running tests, scanning for vulnerabilities, or deploying to a server.
- Advantages:
- Granularity: Jobs allow for fine-grained control over tasks, enabling teams to create specialized and reusable components.
- Parallel Execution: Jobs within a stage run concurrently, maximizing the use of available resources and reducing the time required for the entire pipeline to complete.
CI helps ensure that code changes are integrated smoothly and that potential issues are detected early in the development process. In this document, we will explain how to enable CI jobs for Maven and Node.js projects using the GitLab CI/CD platform as an example.
Prerequisites:
- Version control repository (e.g., GitLab repository) with the Maven and Node.js projects.
- Basic knowledge of Maven and Node.js build tools.
- Access to a GitLab CI/CD platform (can be self-hosted or cloud-based).
Step 1: Set Up GitLab CI/CD Configuration (Create .gitlab-ci.yml File):
The .gitlab-ci.yml
file is a configuration file used by GitLab CI/CD to define the CI/CD pipeline for your project. It is written in YAML format and is typically placed in the root directory of your GitLab repository. The file defines the stages, jobs, and scripts to execute for each job.
- Create a file named .gitlab-ci.yml in the root directory of your Maven and Node.js projects.
- Open the .gitlab-ci.yml file and define the CI/CD pipeline stages and jobs.
Step 2: Configuring Maven and Node.js Projects:
Maven Project:
- Ensure that your Maven project has a pom.xml, checkstyle-checker.xml file defining the project's dependencies and configurations.
- You must be aware of the Maven version that is being utilized.
Node.js Project:
- Ensure that your Node.js project has a package.json file defining the project's dependencies and scripts (e.g., "test").
- You must be aware of the Node.js version that is being utilized i.e
18.17.0-alpine3.18
.
Step 3: Define Variables (Maven/Node.js)
Node.js:
Please follow these steps for node.js project
To create CI/CD variables through GitLab, you can follow these steps:
Log In to Your GitLab Account: If you don't have a GitLab account, you'll need to create one. If you already have an account, log in.
Navigate to Your Profile Settings:
- From the left side dropdown menu, select "Settings."
Access Tokens:
- In the left sidebar, under "CI/CD," click on it then here you can view and create variables."
Variables store information, like passwords and secret keys, that you can use in job scripts.
Variables can be:
Protected:
Only exposed to protected branches or tags.Masked:
Hidden in job logs. Must match masking requirements.
You have to define variables in setting→ CI/CD
Key | Value |
---|---|
BUILD_NUMBER | 0 |
BUILD_TIME | 0 |
CI_PIPELINE_IID_TOKEN | *********** |
MERGE_BRANCH | 0 |
SONAR_PASS | ************ |
Maven:
Please follow these steps for Maven project
You have to define variables in setting→ CI/CD
Key | Value |
---|---|
BUILD_NUMBER | 0 |
BUILD_TIME | 0 |
MERGE_BRANCH | 0 |
SONAR_PASS | ************(copy it from any previous project ) |
PROJECT_ACCESS_TOKEN | ************ |
To create an access token through GitLab, you can follow these steps:
Log In to Your GitLab Account: If you don't have a GitLab account, you'll need to create one. If you already have an account, log in.
Navigate to Your Profile Settings:
- From the left side dropdown menu, select "Settings."
Access Tokens:
- In the left sidebar, under "Access Tokens," click on "Access Tokens."
Create a New Access Token:
- Click on the "Create a token" button.
- Provide a name for the token. This name is for your reference to identify the purpose of the token (e.g., "Documentation Access").
- Choose the scopes or permissions that this token should have. Scopes define what actions the token is allowed to perform (e.g., read user information, access repositories).
- Set an expiration date for the token if desired.
Generate Token:
- Click on the "Create token" button.
- The token will be generated, and you'll be shown the token string only once. Make sure to copy and save this token securely, as you won't be able to see it again.
Pre defined Variables: This section defines some variables used throughout the pipeline:
IMAGE_NAME
: A variable that holds the image name for merge requests. It includes the CI_REGISTRY_IMAGE (GitLab Container Registry URL) along with the source branch name and a TIME_TOKEN.IMAGE_NAME_BRANCH
: A variable that holds the image name for branch builds. It includes the CI_REGISTRY_IMAGE along with the commit branch and commit SHA.MAVEN_OPTS
: A variable that sets Maven options to specify a custom Maven repository location.
Include: This section allows you to include external YAML templates from another project repository ($CI_PROJECT_ROOT_NAMESPACE/ci-templates
). It includes multiple YAML templates for different stages in the pipeline. These templates are reusable configurations for specific job types.
Stages: This section defines the order in which the jobs will be executed. Each job belongs to a specific stage, and GitLab CI/CD will run the jobs in the defined stage order.
Step 4: Setup Jobs (MAVEN CI JOB):
If you want by pass the test cases which is causing build failure add the following line in maven build and maven artifact CI jobs.
#- mvn clean
#- mvn package
- mvn clean package -DskipTests
Maven:
Now, let's go through each job and its explanation. A Maven CI job typically involves the following steps:
Checking out the source code from the version control system (e.g., Git, SVN).
- Running Maven commands to compile the Java code and create artifacts (e.g., JAR files, WAR files).
- Running unit tests to ensure code quality and identifying potential issues.
- Packaging the application and generating deployable artifacts.
- Optionally, deploying the built artifacts to a repository or a server.
maven-validate: This job runs the validation tasks for Maven projects. It extends the .maven_validate
template from the included maven-validate.yaml
file. It runs on every commit and merge request, but only for commits with names matching the patterns /_f-.+$/
and /_b-.+$/
.
maven-compile: This job compiles the Maven project. It extends the .maven_compile
template from the included maven-compile.yaml
file. It runs only on successful commits and merge requests with names matching the patterns /_f-.+$/
and /_b-.+$/
.
maven-test: This job runs the tests for the Maven project. It extends the .maven_test
template from the included maven-test.yaml
file. It runs only on successful commits and merge requests with names matching the patterns /_f-.+$/
and /_b-.+$/
.
maven-format: This job formats the code for the Maven project. It extends the .maven_format
template from the included maven-format.yaml
file. It runs only on successful commits and merge requests with names matching the patterns /_f-.+$/
and /_b-.+$/
.
mvn_artifacts: This job creates artifacts (JAR files) for the Maven project. It uses the Maven image version 3.6.0 and runs only on successful commits and merge requests with names matching the patterns /_f-.+$/
and /_b-.+$/
. It caches the Maven repository for faster subsequent builds.
sonarqube-check: This job performs SonarQube analysis on the Maven project. It extends the .sonarqube_scan
template from the included sonarqube.yaml
file. It runs only on successful merge requests.
time_build: This job is used to set build-time environment variables. It extends the .build_time
template from the included build_time.yml
file. It runs only on successful merge requests.
mvn_build: This job builds the Maven project and generates artifacts. It extends the .mvn_tag
template from the included mvn_build_tag.yaml
file. It runs only on successful merge requests and depends on the time_build
job.
gitlab_build_branch: This job builds a Docker image for branch commits. It runs on a manual trigger, using the Docker image version 'latest' and Docker-in-Docker (DinD) service. It runs after the mvn_artifacts
job.
gitlab_build_merge: This job builds a Docker image for merge requests. It uses the Docker image version 'latest' and Docker-in-Docker (DinD) service. It runs only on successful merge requests and depends on the mvn_build
and time_build
jobs.
trivy_scanning_branch: This job performs security scanning using Trivy for the Docker image built from branch commits. It runs only on successful branch builds and depends on the gitlab_build_branch
job.
trivy_scanning_merge: This job performs security scanning using Trivy for the Docker image built from merge requests. It runs only on successful merge requests and depends on the gitlab_build_merge
and time_build
jobs.
grype_scan_branch: This job performs vulnerability scanning using Grype for the Docker image built from branch commits. It runs only on successful branch builds and depends on the gitlab_build_branch
job.
grype_scan_merge: This job performs vulnerability scanning using Grype for the Docker image built from merge requests. It runs only on successful merge requests and depends on the gitlab_build_merge
and time_build
jobs.
mvn_build_tag: This job tags the successful build version for a new tag. It extends the .mvn_tag
template from the included mvn_build_tag.yaml
file. It runs only on new tags.
gitlab_publish: This job publishes the Docker image to the GitLab Container Registry for a new tag. It extends the .gitlab_publish
template from the included gitlab-publish.yaml
file. It runs only on new tags.
Node.js :
- Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine. It allows developers to write server-side applications using JavaScript.
- A Node.js CI job typically involves the following steps:
- Checking out the source code from the version control system (e.g., Git, SVN).
- Installing Node.js and its dependencies on the CI server.
- Installing the project's dependencies using npm (Node Package Manager).
- Running linting tools to enforce code style and identify potential issues.
- Running unit tests using testing frameworks like Mocha, Jest, or Jasmine to ensure code quality and functionality.
- Generating code coverage reports to track test coverage.
- Optionally, building and bundling the application for production deployment.
Now, let's break down the individual stages and jobs:
node-test:(Stage: test): This job is responsible for running Node.js tests. It executes npm install to install dependencies and npm run coverage to run the test suite. The job is set to: run only on merge requests (merge_requests) and branch commits (refs with regex patterns matching branch names).
node-format: (Stage: code_format): This job checks the code formatting of the Node.js project. It executes npm run format:check to perform the formatting check. Like
node-test:This job is set to run only on merge requests and branch commits.
time_build:(Stage: init): This job sets the build time for merge requests and tags. It is set to run on success only for merge requests and tags.
node_artifacts:(Stage: artifacts): This job builds Node.js artifacts, including the dist directory, and exports the ${TIME_TOKEN} variable using a file named .ci_status/ci_time: It runs on success for merge requests, branch commits, and tags.
sonarqube-test:(Stage: sonarqube-scan): This job runs the SonarQube scan to analyze the code quality of the project. It uses the SonarScanner CLI and is set to run only on merge requests and branch commits.
gitlab_build_branch:(Stage: build-gitlab): This job is a manual action that builds a Docker image for regular branch commits. It tags the image with the branch name and commit SHA and pushes it to the GitLab Container Registry.
gitlab_build_merge:(Stage: build-gitlab): This job builds a Docker image for merge requests. It tags the image with the merge request source branch name and the ${TIME_TOKEN} The job also sets the ${TIME_TOKEN} and ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME} variables using GitLab API calls.
trivy_scanning_branch:(Stage: scan): This job performs a Trivy vulnerability scan on the Docker image built from regular branch commits. It uses the .trivy_scan_template: which is likely defined in the included templates.
trivy_scanning_merge:(Stage: scan): This job performs a Trivy vulnerability scan on the Docker image built from merge requests. It uses the .trivy_scan_template and sets the ${TIME_TOKEN}
grype_scan_branch:(Stage: scan): This job performs a Grype vulnerability scan on the Docker image built from regular branch commits. It uses the .grype_template, which is likely defined in the included templates.
grype_scan_merge:(Stage: scan): This job performs a Grype vulnerability scan on the Docker image built from merge requests. It uses the .grype_template and sets the ${TIME_TOKEN}
These jobs collectively form a CI/CD pipeline for Node.js projects. They handle various tasks, including testing, code formatting, artifact creation, Docker image building, and security scanning to ensure high code quality and security for the Node.js application.
Step 4: Trigger the CI Job:
- Make changes to your Maven and Node.js projects.
- Push the changes to your GitLab repository.
- The GitLab CI/CD platform will automatically detect the changes and trigger the CI pipeline defined in the .gitlab-ci.yml file.
- Monitor the CI/CD pipeline progress on the GitLab platform. The pipeline will execute the defined stages and jobs for building and testing the Maven and Node.js projects.
Step 5: Review the CI Job Results:
- If the CI pipeline execution is successful, you will see a green checkmark next to each job in the pipeline.
- If any job fails, you will see a red "X" next to the failed job, and details about the failure will be displayed.
Optional: Automated Deployment (Node.js):
To automate the deployment of your Node.js application to a staging or production environment, you can add an additional deployment stage to your .gitlab-ci.yml file and configure the necessary deployment steps, such as copying files to the server or triggering a deployment script.
Workflow of CI
Feature Branch: Following stages will be included in the CI JOB in a feature branch.
- Code is validated, compiled, tested, format checking, Sonar scanning ( will not break if quality gate fails), and artifacts are generated. These steps are run automatically
- The Docker image is created manually and will only be pushed to Gitlab.
- If the Docker image is created, scanning will be done by Grype and Trivy.
- Scanning will only generate reports but will not break anything.
- We need to retain the last 3 images on a feature branch.
Feature merge to Develop:
- Code is validated, compiled, tested, format checking, Sonar scanning ( will break the CI job if quality gate fails i.e. code quality is less than 30%), and artifacts are generated.
- The Docker image will undoubtedly be created automatically (will include the source branch name in the docker image and the time of the build, this will indicate the docker image is created by the merge request).
- Scanning will be done by Grype and Trivy. If critical vulnerabilities are found, the CI job will break the flow and will not succeed.
- We will retain every Docker image.
Tag created from Develop:
- If we are creating a git tag from the develop branch, it means we are ready for SR. release, hence only the Docker image will be tagged with the name of the git tag.
- No other scanning will be performed here.
- Will retain every Docker image.
Develop merge to master/main:
- Code is validated, compiled, tested, format checking, Sonar scanning ( will break the CI job if quality gate fails i.e. code quality is less than 30%), and artifacts are generated.
- The Docker image will undoubtedly be created automatically (will include the source branch name in the docker image and the time of the build, this will indicate the docker image is created by the merge request).
- Scanning will be done by Grype and Trivy. If critical vulnerabilities are found, the CI job will break the flow and will not succeed.
- We will retain every Docker image.
Tag created from master/main:
- If we are creating a git tag from the main branch, it means we are ready for production release, hence only the Docker image will be tagged with the name of the git tag.
- No other scanning will be performed here.
- Will retain every Docker image.
Conclusion:
By enabling CI jobs for your Maven and Node.js projects, you can ensure that code changes are automatically built, tested, and deployed with confidence, reducing the risk of introducing bugs or issues into your applications. The GitLab CI/CD platform makes this process seamless and straightforward, allowing you to focus on development while maintaining a reliable and efficient development workflow.