Hands-on Jenkins CI Project (Intermediate) – Failure-Driven Debugging on AWS
Building a Jenkins CI pipeline on AWS EC2 by intentionally breaking things and fixing them like in real production environments.
1️⃣ Introduction
In this article, I document a hands-on Jenkins CI project built from scratch on a fresh AWS EC2 instance.
Instead of focusing only on a successful pipeline, I intentionally introduced real-world failure scenarios such as network misconfiguration, service access issues, webhook errors, and permission problems.
The objective of this project is to understand:
How Jenkins behaves under failure
How to debug CI issues systematically
How real DevOps engineers troubleshoot production-like problems
This project is written in a failure-driven learning approach, which closely reflects real industry environments.
2️⃣ Project Level & Scope
Project Level: Intermediate
Category: DevOps / CI / Jenkins
This project goes beyond beginner tutorials by focusing on:
Infrastructure-level issues (ports & security groups)
Jenkins service setup and debugging
GitHub webhook failures
Docker permission and integration issues
Root cause analysis and fixes
3️⃣ Tools & Technologies Used
AWS EC2 – Infrastructure
Ubuntu 22.04 LTS – Operating System
Jenkins – CI automation
GitHub – Source code & webhook trigger
Docker – Build and containerization
Linux – System administration
4️⃣ Project Architecture (High-Level)
The CI flow in this project looks like this:
Developer pushes code to GitHub
GitHub webhook triggers Jenkins automatically
Jenkins pulls the code
Jenkins builds the application using Docker
Build results are visible in the Jenkins dashboard
This simple architecture is intentionally kept minimal to focus on debugging and learning, not complexity
5️⃣ Environment Setup – AWS EC2 Instance
I started by launching a new EC2 instance to ensure a clean environment.
EC2 Configuration:
AMI: Ubuntu 22.04 LTS
Instance Type: t3.micro
SSH Access: Enabled (port 22)
⚠️ Important (Intentional Failure):
At this stage, port 8080 was NOT opened in the security group.
This was done intentionally to simulate a common real-world mistake.
6️⃣ Phase 1 – Jenkins Installation
After connecting to the EC2 instance via SSH, I first updated the system packages and installed Java, which is a mandatory prerequisite for running Jenkins.
☕ Installing Java (OpenJDK 17)
sudo apt update
sudo apt install openjdk-17-jdk -y
Once installed, I verified the Java installation by checking the version:
java -version
Expected output:
openjdk version "17.x.x"
This confirmed that Java was installed successfully and ready for Jenkins.
🧩 Installing Jenkins
Next, I installed Jenkins using the official Jenkins repository to ensure a stable and supported installation.
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
/usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
After adding the repository, I installed Jenkins and started the service:
sudo apt update
sudo apt install jenkins -y
sudo systemctl start jenkins
sudo systemctl enable jenkins
🔍 Verifying Jenkins Service Status
To confirm that Jenkins was running correctly at the service level, I checked its status:
sudo systemctl status jenkins
At this point, Jenkins was successfully installed and running on the EC2 instance.
⚠️ Important Note (Intentional Failure Setup)
Although the Jenkins service was running, the Jenkins web interface was not yet accessible because port 8080 was not allowed in the EC2 security group.
This was an intentional configuration choice to simulate a real-world failure scenario, which will be addressed in the next phase.
7️⃣ Failure Scenario 1 – Jenkins UI Not Accessible
After installing Jenkins and confirming that the service was running, I attempted to access the Jenkins web interface using the browser:
http://<EC2-PUBLIC-IP>:8080
❌ However, the Jenkins UI did not load and resulted in a connection timeout.
🔍 Root Cause Analysis
At this point, Jenkins was:
Installed correctly
Running as a service
Listening on port 8080
The issue was not with Jenkins itself.
On closer inspection, I found that:
Port 8080 was not allowed in the EC2 security group
AWS blocks inbound traffic by default unless explicitly permitted
This confirmed that the problem was at the network (infrastructure) level, not the application level.
🔧 Fix – Allowing Port 8080 in Security Group
To resolve this issue, I updated the EC2 security group inbound rules to allow Jenkins traffic.
I added a new inbound rule with the following configuration:
Type: Custom TCP
Port Range: 8080
Source: My IP / Anywhere (for learning purposes)
Click on Save rules.
✅ Verification
After updating the security group, I refreshed the browser and accessed Jenkins again:
http://<EC2-PUBLIC-IP>:8080
This time, the Jenkins Unlock Jenkins screen loaded successfully.
🧠 Key Learning
This issue highlights a common real-world scenario where:
Services are running correctly
But remain inaccessible due to missing network permissions
Understanding where to debug (application vs infrastructure) is a critical DevOps skill.
You can paste this directly after Phase 1 failure section.
8️⃣ Phase 2 – Jenkins Initial Setup & Unlocking Jenkins
Once the Jenkins web interface became accessible, the next step was to complete the initial Jenkins setup.
🔐 Retrieving the Initial Admin Password
When Jenkins is accessed for the first time, it requires an initial admin password, which is stored on the server.
I retrieved the password using the following command:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
This command printed a one-time password used to unlock Jenkins.
🔓 Unlocking Jenkins
I pasted the retrieved password into the Unlock Jenkins screen in the browser and proceeded to the next step.
🔌 Installing Recommended Plugins
Jenkins then prompted for plugin installation.
I selected:
👉 Install suggested plugins
This installs commonly used plugins required for:
Git integration
Build execution
Jenkins UI stability
The installation took a few minutes to complete.
👤 Creating the First Admin User
After plugin installation, Jenkins prompted for admin user creation.
I created a dedicated admin user instead of continuing with the default setup.
This is a recommended best practice for:
Better access control
Auditability
Production-like setup
🎉 Jenkins Dashboard Access
Once the setup is completed, click on “Start using Jenkins”.
At this stage:
Jenkins was fully installed
Plugins were configured
Admin user was created
The Jenkins environment was ready for CI/CD configuration
9️⃣ Phase 3 – CI Pipeline Validation Using Jenkins Pipeline
With Jenkins fully set up, the next step was to validate a CI pipeline by creating a Pipeline job and running it manually before introducing any automation.
This phase focuses on establishing a baseline for the pipeline and understanding how Jenkins executes pipeline code from a repository.
🎯 Objective
Create a Jenkins Pipeline job
Load the pipeline from a GitHub repository
Execute the pipeline manually
Observe and validate pipeline behavior
🔧 9.1 Creating a Jenkins Pipeline Job
From the Jenkins dashboard:
Click New Item
Enter a job name:
Jenkins-CI-CD-DemoSelect Pipeline
Click OK
🔗 9.2 Configuring Pipeline from SCM
In the job configuration, the pipeline was configured to load from GitHub:
Definition: Pipeline script from SCM
SCM: Git
Repository URL:
https://github.com/Kunja-Ravikiran/Jenkins-CI-CD-DemoBranch:
mainScript Path:
Jenkinsfile
This configuration ensures that Jenkins reads the pipeline definition directly from the repository, enabling version-controlled CI pipelines.
▶️ 9.3 Manual Pipeline Execution
After saving the configuration, the pipeline was triggered manually using:
👉 Build Now
This manual execution step is important to:
Validate repository connectivity
Verify Jenkinsfile detection
Establish a working baseline before automation
🔍 9.4 Observing Pipeline Output
The pipeline execution was monitored via the Console Output, which displays:
Stage-wise execution
Command outputs
Any errors encountered during the run
At this stage, the pipeline either:
Completed successfully, or
Failed with a clear error message (to be analyzed in subsequent steps)
🔴 9.5 Debugging the Initial Pipeline Failure
After completing the manual pipeline execution in Step 9.4, the build failed, which was clearly indicated by the red status in the Jenkins dashboard.
🔍 Locating the Error
To identify the cause of the failure, I opened the failed build and navigated to the Console Output section.
The console output provides a step-by-step execution log of each pipeline stage and is the primary source for debugging Jenkins failures.
🧠 Understanding the Failure
From the console logs, the following observations were made:
Jenkins successfully checked out the source code from GitHub
The
Jenkinsfilewas detected and executed correctlyThe pipeline entered the Docker build stage
The build failed with the message indicating that the Docker command was not found
This confirmed that:
Jenkins installation was correct
Pipeline configuration was valid
The failure occurred due to a missing dependency on the Jenkins host
🔍 Root Cause Analysis
The pipeline failed because:
The
Jenkinsfileincluded a stage that builds a Docker imageJenkins attempted to execute Docker commands on the EC2 instance
Docker was not installed on the Jenkins execution environment
Since Jenkins executes pipeline steps on the underlying operating system, any external tools referenced in the pipeline must already be available on that system.
This type of failure is common in CI/CD pipelines and highlights the importance of preparing the execution environment before running builds.
✅ 9.6 Fix – Installing Docker on the Jenkins Host
Once the root cause was identified, the next step was to prepare the Jenkins host to support Docker-based builds.
🔧 Installing Docker on EC2
I connected to the EC2 instance where Jenkins was running and installed Docker using the system package manager.
sudo apt update
sudo apt install docker.io -y
▶️ Starting and Enabling Docker Service
After installation, Docker was started and enabled to run automatically on system boot.
sudo systemctl start docker
sudo systemctl enable docker
🔎 Verifying Docker Installation
To confirm that Docker was installed successfully, I verified the Docker version.
docker --version
This confirmed that Docker was now available on the Jenkins host.
🔴 Phase 4 – Docker Permission Denied Error in Jenkins Pipeline
After installing Docker on the Jenkins host and re-running the pipeline, the build failed again, even though Docker was now available on the system.
🔍 Observing the New Failure
As with the previous failure, I navigated to the failed build and reviewed the Console Output to understand the cause.
This time, the error message clearly indicated a permission-related issue while Jenkins was trying to interact with the Docker daemon.
🔍 Root Cause Analysis
The failure occurred because:
Jenkins runs as a dedicated
jenkinsuserDocker commands require access to the Docker daemon
By default, only users with appropriate permissions can access Docker
The
jenkinsuser was not authorized to communicate with Docker
As a result, even though Docker was installed, Jenkins did not have the required permissions to execute Docker commands.
This is a very common real-world CI/CD issue when integrating Jenkins with Docker.
✅ Fix – Granting Docker Access to Jenkins User
To resolve this issue, the Jenkins user was granted permission to access Docker.
🔧 Updating User Permissions
I added the jenkins user to the Docker group on the EC2 instance.
sudo usermod -aG docker jenkins
This allows Jenkins to execute Docker commands without requiring root access.
🔄 Restarting Jenkins
After updating user permissions, Jenkins was restarted to ensure the new group membership took effect.
sudo systemctl restart jenkins
🔎 Verifying the Fix
Once Jenkins was restarted, I re-ran the pipeline from the Jenkins dashboard.
This time:
Jenkins successfully executed Docker commands
The Docker image was built without permission issues
The pipeline completed all stages successfully
🚀 Phase 5 – Automating the Pipeline using GitHub Webhooks
With the Jenkins pipeline now executing successfully through manual triggers, the next step was to automate the build process so that the pipeline runs automatically whenever code is pushed to the repository.
This eliminates the need for manual intervention and completes the CI workflow.
🔗 Configuring Jenkins for Webhook Triggers
In the Jenkins pipeline configuration, webhook-based triggering was enabled by selecting the option to trigger builds when GitHub sends push events.
This allows Jenkins to automatically start the pipeline whenever new code is pushed to the repository.
The pipeline was configured to load the Jenkinsfile directly from the GitHub repository, ensuring that both pipeline logic and source code remain version-controlled.
🔔 Adding a Webhook in GitHub
Next, a webhook was configured in the GitHub repository settings.
The webhook was set up to:
Notify Jenkins on every push event
Send payloads in JSON format
Target the Jenkins webhook endpoint.
GitHub webhook configuration
🔁 Verifying Automated Builds via GitHub Webhooks
After configuring the webhook in GitHub, the integration was verified by pushing a new commit to the repository.
Upon pushing the change:
GitHub sent a webhook event to Jenkins
Jenkins automatically triggered the pipeline
The build executed without any manual intervention
This confirmed that the CI pipeline was now fully automated and correctly integrated with GitHub.
As an optional verification step, the Docker container built by the Jenkins pipeline was run on the host and accessed via the browser. The application was successfully served through Nginx, confirming end-to-end CI automation.
🧾 Conclusion
In this project, I built a Jenkins-based CI pipeline from scratch and progressively enhanced it to support fully automated builds triggered by GitHub webhooks.
Instead of focusing only on the happy path, the pipeline was intentionally validated through real-world failure scenarios such as missing environment dependencies and permission issues. Each failure was diagnosed using Jenkins console logs and resolved at the appropriate layer, reinforcing the importance of structured debugging in CI/CD workflows.
By the end of the project, the pipeline was able to:
Execute as code using a Jenkinsfile
Build Docker images reliably
Handle environment and permission-related failures
Trigger automatically on every code push using GitHub webhooks
This hands-on approach closely reflects how CI pipelines behave in production environments and highlights the role of DevOps engineers in ensuring not just correct pipeline logic, but also a properly prepared execution environment.
🧠 Key Takeaways
Jenkins pipelines should always be validated manually before enabling automation
Console Output is the primary tool for debugging CI failures
CI pipelines depend on the execution environment as much as the pipeline code
Missing tools and permission issues are common and expected in real-world setups
GitHub webhooks enable reliable, event-driven CI automation without manual triggers
🔗 Final Notes
This project was designed as an intermediate-level CI implementation, focusing on practical debugging, environment readiness, and automation rather than complex application logic.
The complete source code and configuration used in this project are available in the associated GitHub repository.
This project focuses on building a robust Continuous Integration (CI) pipeline using Jenkins, Docker, and GitHub webhooks. The primary goal was to automate build validation and work through real-world CI failure scenarios such as missing dependencies, permission issues, and webhook automation.
Continuous Deployment (CD) was intentionally kept out of scope to maintain a clear focus on CI fundamentals and failure-driven debugging. The pipeline will be extended to support CD in a follow-up project, where the built Docker image will be deployed automatically to a target environment.