We use Jenkins internally to manage project builds and deployments.

One day, when I opened the Jenkins dashboard, all the job settings had disappeared. At first, I thought I was seeing things incorrectly, but even after refreshing several times, it was the same. All Pipeline Scripts were empty, leaving us unable to build or deploy anything.

When we previously used GitHub Actions, such problems naturally didn't occur. Currently, we're using a combination of GitLab and Jenkins (due to internal circumstances, we cannot use GitLab CI), but when Jenkins stopped working, our CI/CD essentially came to a complete halt. It was quite alarming.

---

The Problem

It was immediately clear that something had gone seriously wrong with Jenkins. All job listings had disappeared, and it appeared to be in a freshly installed state. It felt as though everything except user accounts had been wiped.

Root Cause Analysis

Upon investigation, we found the problem was due to plugin conflicts and network policies.

• Some plugins were referencing specific overseas mirrors, but server access to that country was blocked (country blocking rule applied)
• Failed installations caused conflicts with existing versions
• Ultimately, this caused job settings in Jenkins (2.513.1) to disappear Specifically, the problematic plugins were:
  • commons-lang3-api
  • workflow-step-api
  • display-url-api
  • plain-credentials

Some of these failed to install, while others caused conflicts, ultimately resulting in all jobs being reset (or at least appearing that way in the console).

Recovery Process

Once we understood the cause, the solution was straightforward:

1. We modified the server blocking policy (allowing access to the relevant country) to enable plugin installation.

2. We updated all plugins.

3. We upgraded Jenkins itself to the latest version (2.516.3).

   sudo apt update
   apt list --upgradable | grep jenkins
   sudo apt upgrade jenkins

4. Fortunately, the build history remained intact. We could see the Pipeline Scripts from successful builds and copied them back into the job settings.

5. We ran builds in the development environment to verify normal operation, then completed builds in the production environment.

---

Conclusion

This incident reminded me that Jenkins is a tool with high plugin dependency.

• Network blocking alone can cause problems
• Plugin conflicts can lead to job settings being lost
• Job settings must always be backed up (especially Pipeline Scripts, though they can be extracted from previous build records)

I had assumed everything would be fine since it had always worked well, but I was caught off guard by being complacent. This incident reminded me of the fundamental principle that CI/CD should be managed like code.

Of course, if you use GitHub Actions or GitLab CI, you wouldn't need to worry about this as they're managed alongside your repository.