Overview
In modern DevOps practices, maintaining seamless version control for integration projects is crucial, especially when working with complex middleware platforms like Software AG’s webMethods. This article explores a streamlined approach to committing webMethods packages to Git repositories using Jenkins automation. By integrating Jenkins and Git, you can automate the process of tracking changes, enhancing collaboration, and ensuring code quality across development cycles. This guide will walk through setting up a Jenkins pipeline specifically configured to commit webMethods package changes to Git, simplifying the process and reducing manual tasks.
Prerequisite
- A Jenkins server that is up and running
- A GitLab account for repository management
- GitLab installed on the same machine where Jenkins is running
Generate GitLab access token
A GitLab access token is necessary, as it will be used in the Jenkins process to interact with Git.
- Login to your Gitlab account.
- Click on the profile and then edit the profile.
- Click on the ‘Access Token’ tab and then the ‘Add new token’ button.
- Give your token a name and set the token expiry date.
- Check all available options and create the token.
- copy your token and save it for further use.
Architecture
We will write a Jenkins pipeline with parameters such as GROUP_NAME, BRANCH, TYPE, PACKAGE_NAME, and GROUP_ID. Once the Jenkins job is triggered, Jenkins will follow the steps below to commit the package.
- Jenkins will download the package from an SFTP server. This step is optional; however, since our Jenkins server is running in a container, we need to download it from SFTP. In your setup, you can skip this step and adjust the code to move the package from a local directory to the Jenkins directory. The remaining steps will stay the same.
- TYPE will have either New or Existing option, if New is selected then a new repository will be created in GitLab and the package will be committed, else the existing package will be committed to GitLab.
- Email will be triggered to support email address based on the Jenkins job status.
Credential setup
- Navigate to Manage Jenkins->Credentials.
- Create a credentials entry for SFTP with SFTP user and password.
- Create another entry using the ‘Secret text’ type and store the GitLab token.
Email configuration
Navigate to Manage Jenkins->System and fill ‘Extended E-mail Notification’ with SMTP server and user details which will be used to send email notification after the pipeline execution.
Create Jenkins pipeline
- Create a new pipeline job in Jenkins and provide a description to the job.
- Check the ‘?‘ option and configure the below fields.
- Variable Name GROUP_NAME
- Type – Choice Parameter.
- Choices- provide available group names separated by new lines.
- Description – provide a description.
- Variable Name BRANCH
- Type – String.
- Default Value – master.
- Description – provide a description.
- Variable Name TYPE
- Type – Choice Parameter.
- Choices- put New and Existing separated by new lines.
- Description – provide a description.
- Variable Name PACKAGE_NAME
- Description – provide a description.
- Variable Name GROUP_ID
- Description – provide a description.
- Variable Name GROUP_NAME
Write Pipeline Script
Insert the following script into the pipeline, and update the GitLab and SFTP credential IDs accordingly. Additionally, update the SFTP username, SFTP server host, and package path. Finally, modify the email address to receive notifications as needed.
pipeline {
agent any
parameters {
choice(name: 'GROUP_NAME', choices: ['HarmoniGate-Personal', 'HarmoniGate', 'Webmethods','WmPackages'], description: 'GitLab Group Name')
string(name: 'BRANCH', defaultValue: 'master', description: 'Git Branch Name')
choice(name: 'TYPE', choices: ['Existing', 'New'], description: 'Select the commit type.')
string(name: 'PACKAGE_NAME', defaultValue: '', description: 'Package Name')
string(name: 'GROUP_ID', defaultValue: '', description: 'GroupID')
}
environment {
SFTP_CREDENTIALS = credentials('94c514a4-d289-43b2-8d32-678718440046') // SFTP credentials
GITLAB_TOKEN = credentials('gitlab-token-id') // GitLab token
}
stages {
stage('Preparation') {
steps {
script {
deleteDir() // Clear workspace
echo "**********Workspace is cleaned***************"
}
}
}
stage('Download from SFTP') {
steps {
script {
sh '''
sshpass -p ${SFTP_CREDENTIALS_PSW} scp -r -o StrictHostKeyChecking=no rdpuser@88.000.000.000:/home/rdpuser/cicd_pipeline/gitlab-commits/${PACKAGE_NAME} .
'''
echo "*****************${PACKAGE_NAME} downloaded ****************"
}
}
}
stage('Handle New GitLab Project') {
when {
expression { return params.TYPE == 'New' }
}
steps {
script {
def branchName = params.BRANCH
def packageName = params.PACKAGE_NAME
// Create new GitLab project
def createProjectResponse = sh(script: """
curl --silent --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
--data "name=${packageName}" \
--data "namespace_id=${GROUP_ID}" \
"https://gitlab.com/api/v4/projects"
""", returnStdout: true).trim()
def project = readJSON(text: createProjectResponse)
def projectUrl = project.web_url
// Initialize git and set up remote repository
dir("${env.WORKSPACE}/${packageName}") {
sh """
git init
git config user.email "gitlab@gmail.com"
git config user.name "Jenkins"
git remote add origin https://oauth2:${GITLAB_TOKEN}@gitlab.com/${GROUP_NAME}/${packageName}.git
git add .
git commit -m "Initial commit"
git branch -M ${branchName}
git push -u origin ${branchName}
"""
echo "*****************Package commited successfully ****************"
}
}
}
}
stage('Handle Existing GitLab Project') {
when {
expression { return params.TYPE == 'Existing' }
}
steps {
script {
def groupName = params.GROUP_NAME
def branchName = params.BRANCH
def packageName = params.PACKAGE_NAME
def gitUrl = "https://oauth2:${GITLAB_TOKEN}@gitlab.com/${groupName}/${packageName}.git"
dir("${env.WORKSPACE}/${packageName}") {
// Clone the existing project only if it doesn't exist
if (!fileExists('.git')) {
sh """
git clone -b ${branchName} ${gitUrl} .
"""
}
// Set remote URL to use the token for authentication
sh """
git remote set-url origin ${gitUrl}
"""
// Add new changes, commit, and push
sh """
git config user.email "gitlab@gmail.com"
git config user.name "Jenkins"
git add .
git commit -m "Updated with new package"
git push origin ${branchName}
"""
echo "*****************Package commited successfully ****************"
}
}
}
}
stage('Remove the Package') {
steps {
script {
deleteDir() // Clear workspace after build
echo "*****************Workspace is cleaned ****************"
}
}
}
stage('Clear Workspace') {
steps {
script {
script {
sh '''
sshpass -p ${SFTP_CREDENTIALS_PSW} ssh -o StrictHostKeyChecking=no rdpuser@88.000.000.000 "rm -rf /home/rdpuser/cicd_pipeline/gitlab-commits/${PACKAGE_NAME}"
'''
}
echo "****************${PACKAGE_NAME} is now deleted from build server***************"
}
}
}
}
post {
success {
mail to: 'support@harmonigate.com',
subject: "${PACKAGE_NAME} successfully commited to Git",
body: "Build ${env.JOB_NAME} ${env.BUILD_NUMBER} was successful.\n\n" +
"Details: ${env.BUILD_URL}"
}
failure {
mail to: 'support@harmonigate.com',
subject: "${PACKAGE_NAME} commited to Git Failed",
body: "Build ${env.JOB_NAME} ${env.BUILD_NUMBER} failed.\n\n" +
"Details: ${env.BUILD_URL}"
}
}
}
Code breakdown :
Pipeline and Parameter Definitions
- Agent
agent any: Specifies that the pipeline can run on any available Jenkins agent. - Parameters
Defines input parameters that allow you to customize the Jenkins job:- GROUP_NAME: GitLab group name for project storage.
- BRANCH: Git branch to which changes are pushed.
- TYPE: Specifies if this is a new or existing project (New or Existing).
- PACKAGE_NAME: Name of the package to commit.
- GROUP_ID: GitLab group ID (used when creating new projects).
- Environment
Sets environment variables for secure credentials:- SFTP_CREDENTIALS: SFTP credentials to download packages.
- GITLAB_TOKEN: GitLab token for authentication.
Stages
- Preparation
- Purpose: Cleans the workspace by deleting all files and directories before the build starts.
- Steps:
- deleteDir(): Clears the workspace.
- echo “Workspace is cleaned”: Confirms the workspace is ready.
- Download from SFTP
- Purpose: Downloads the package from an SFTP server.
- Steps:
- Uses sshpass with SCP to securely copy the package from the SFTP server to the Jenkins workspace.
- Displays a message once the download is complete.
- Handle New GitLab Project
- Purpose: Creates a new GitLab project and initializes it if TYPE is set to New.
- Steps:
- Create GitLab Project:
- Sends an HTTP request to GitLab’s API to create a new project in the specified group, using GROUP_ID.
- Stores the project URL for reference.
- Initialize Git and Commit Package:
- Initializes a new Git repository in the package directory.
- Sets user configuration for Git.
- Adds the GitLab remote URL using the token for authentication.
- Adds all files, commits with a message, creates the branch, and pushes to GitLab.
- Create GitLab Project:
- Handle Existing GitLab Project
- Purpose: Commits changes to an existing GitLab project if TYPE is Existing.
- Steps:
- Defines the Git URL with the GitLab token for authentication.
- Clone Project:
- If the .git directory doesn’t exist, clones the existing project from GitLab.
- Sets the remote URL to use the GitLab token.
- Commit Changes:
- Adds all files, commits with a message, and pushes to the specified branch.
- Remove the Package
- Purpose: Cleans up the workspace by deleting all files and directories after the build completes.
- Steps:
- deleteDir(): Clears the workspace.
- Displays a message confirming workspace cleanup.
- Clear Workspace on SFTP Server
- Purpose: Removes the package from the SFTP server to keep the build server clean.
- Steps:
- Connects to the SFTP server using sshpass and removes the package directory.
- Displays a message confirming deletion.
Post-Build Actions
- Success Notification
- Sends an email to support@harmonigate.com if the build completes successfully.
- Email includes:
- Subject: ${PACKAGE_NAME} successfully committed to Git
- Body: Build details, including job name and build URL.
- Failure Notification
- Send an email to support@harmonigate.com if the build fails.
- Email includes:
- Subject: ${PACKAGE_NAME} committed to Git Failed
- Body: Build details, including job name and build URL.
Collect GitLab GroupID :
- Navigate to GitLab and open the group.
- Click on the three dots, you will see the group id.
Conclusion
We have successfully completed the development for the GitLab commit. Now, when you run the Jenkins job, it will commit the package to GitLab, and a notification will be sent via email.