Eric Campbell 6 min read Infrastructure

CI/CD Pipeline: Continuous Integration and Delivery

Modern software teams ship code multiple times per day. This wasn't always possible. Traditional software delivery involved manual builds, lengthy testing cycles, and deployment processes that...

Key Insights

  • CI/CD pipelines eliminate manual deployment errors and reduce release cycles from weeks to hours by automating the entire software delivery process from code commit to production
  • Continuous Integration focuses on automating builds and tests on every commit, while Continuous Delivery automates deployment readiness, and Continuous Deployment takes it further by automatically releasing to production
  • A well-designed pipeline includes automated testing, security scanning, artifact management, and environment-specific deployment strategies like blue-green or canary releases to minimize production risks

Introduction to CI/CD

Modern software teams ship code multiple times per day. This wasn’t always possible. Traditional software delivery involved manual builds, lengthy testing cycles, and deployment processes that required entire teams working weekends. Integration hell—where merging weeks of isolated development work caused catastrophic failures—was the norm.

CI/CD pipelines solve these problems through automation. Continuous Integration ensures code changes integrate smoothly by building and testing every commit. Continuous Delivery extends this by automating deployment preparation, making releases a business decision rather than a technical challenge. Continuous Deployment goes one step further, automatically pushing every validated change to production.

The business value is substantial: faster time-to-market, reduced deployment risk, improved code quality, and developer productivity gains. Teams using mature CI/CD practices deploy 200 times more frequently with 24 times faster recovery from failures compared to low performers, according to the State of DevOps reports.

Core CI/CD Concepts and Workflow

A CI/CD pipeline consists of automated stages that code passes through from commit to deployment. The fundamental stages are:

  • Build: Compile code, resolve dependencies, create artifacts
  • Test: Run automated tests (unit, integration, end-to-end)
  • Deploy: Release artifacts to target environments

Understanding the distinction between Continuous Delivery and Continuous Deployment is critical. Continuous Delivery means your code is always deployment-ready, but requires manual approval for production releases. Continuous Deployment automatically pushes every change that passes all pipeline stages directly to production without human intervention.

Here’s a basic GitHub Actions workflow demonstrating the pipeline flow:

name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Build application
        run: npm run build
        
      - name: Upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build-output
          path: dist/

  test:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'
          
      - run: npm ci
      - run: npm test -- --coverage
      
      - name: Upload coverage reports
        uses: codecov/codecov-action@v3

  deploy:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Download artifacts
        uses: actions/download-artifact@v3
        with:
          name: build-output
          
      - name: Deploy to production
        run: |
          echo "Deploying to production..."
          # Deployment commands here

Setting Up Continuous Integration

Continuous Integration is the foundation of any CI/CD pipeline. Every code commit triggers an automated build and test cycle, providing immediate feedback to developers. This prevents integration issues from accumulating.

A robust CI process includes:

  1. Automated builds on every commit
  2. Comprehensive test execution (unit, integration, contract tests)
  3. Code quality checks (linting, static analysis, security scans)
  4. Artifact generation for deployment

Your branching strategy directly impacts CI effectiveness. Trunk-based development, where developers commit to main frequently with short-lived feature branches, works best with CI. Require all branches to pass CI checks before merging.

Here’s a Jenkins pipeline (Jenkinsfile) showing a comprehensive CI setup:

pipeline {
    agent any
    
    environment {
        DOCKER_REGISTRY = 'registry.company.com'
        IMAGE_NAME = 'myapp'
    }
    
    stages {
        stage('Build') {
            steps {
                script {
                    sh 'mvn clean package -DskipTests'
                }
            }
        }
        
        stage('Unit Tests') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/*.xml'
                    jacoco(
                        execPattern: 'target/jacoco.exec',
                        classPattern: 'target/classes',
                        sourcePattern: 'src/main/java'
                    )
                }
            }
        }
        
        stage('Code Quality') {
            steps {
                sh 'mvn sonar:sonar'
            }
        }
        
        stage('Build Docker Image') {
            steps {
                script {
                    docker.build("${DOCKER_REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER}")
                }
            }
        }
        
        stage('Push to Registry') {
            steps {
                script {
                    docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-credentials') {
                        docker.image("${DOCKER_REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER}").push()
                        docker.image("${DOCKER_REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER}").push('latest')
                    }
                }
            }
        }
    }
    
    post {
        failure {
            mail to: 'team@company.com',
                 subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
                 body: "Build failed: ${env.BUILD_URL}"
        }
    }
}

Implementing Continuous Delivery

Continuous Delivery automates everything required to deploy code, but keeps production releases as manual decisions. This involves environment management, configuration handling, and deployment strategies that minimize risk.

Key components include:

  • Environment parity: Dev, staging, and production should be nearly identical
  • Configuration management: Environment-specific settings externalized from code
  • Approval gates: Manual checkpoints before production deployment
  • Deployment strategies: Blue-green, canary, or rolling updates

Blue-green deployments maintain two identical production environments, switching traffic between them for zero-downtime releases. Canary deployments gradually roll out changes to a subset of users before full deployment.

Here’s a Kubernetes deployment manifest using a rolling update strategy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
        version: "2.1.0"
    spec:
      containers:
      - name: myapp
        image: registry.company.com/myapp:2.1.0
        ports:
        - containerPort: 8080
        env:
        - name: ENVIRONMENT
          value: "production"
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: url
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

For infrastructure provisioning, use Infrastructure as Code. Here’s a Terraform example:

resource "aws_ecs_service" "app" {
  name            = "myapp-${var.environment}"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = var.environment == "production" ? 5 : 2
  
  deployment_configuration {
    maximum_percent         = 200
    minimum_healthy_percent = 100
  }
  
  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = "myapp"
    container_port   = 8080
  }
  
  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
}

Essential Pipeline Components

A production-grade pipeline requires more than build, test, and deploy stages. Essential components include:

Version Control Hooks: Trigger pipelines on commits, pull requests, or tags. Configure branch protection rules requiring CI success before merging.

Comprehensive Testing: Layer unit tests (fast, isolated), integration tests (database, APIs), and end-to-end tests (full user workflows). Run tests in parallel to minimize pipeline duration.

Security Scanning: Integrate SAST (static analysis), DAST (dynamic analysis), dependency scanning, and container image scanning. Fail builds on high-severity vulnerabilities.

Artifact Management: Store build artifacts, Docker images, and deployment packages in dedicated repositories (Artifactory, Nexus, ECR).

Monitoring and Notifications: Alert teams on pipeline failures, deployment completions, and anomalies. Integrate with Slack, PagerDuty, or email.

Here’s a complete GitHub Actions workflow incorporating these components:

name: Production Pipeline

on:
  push:
    branches: [main]

jobs:
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          severity: 'CRITICAL,HIGH'
          
      - name: SAST with Semgrep
        uses: returntocorp/semgrep-action@v1
        with:
          config: auto

  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Build and test
        run: |
          npm ci
          npm run build
          npm test -- --coverage
                    
      - name: SonarCloud Scan
        uses: SonarSource/sonarcloud-github-action@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

  deploy:
    needs: [security-scan, build-and-test]
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy to Kubernetes
        run: |
          kubectl set image deployment/myapp \
            myapp=registry.company.com/myapp:${{ github.sha }} \
            --record
                      
      - name: Verify deployment
        run: |
          kubectl rollout status deployment/myapp
                    
      - name: Notify Slack
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: 'Deployment to production completed'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}
        if: always()

Best Practices and Common Pitfalls

Optimize for Speed: Developers need feedback within minutes, not hours. Parallelize tests, cache dependencies, and use incremental builds. A pipeline taking over 10 minutes needs optimization.

Fail Fast: Run fastest, most likely to fail tests first. Don’t wait 30 minutes to discover a compilation error.

Secrets Management: Never hardcode credentials. Use secret management services (AWS Secrets Manager, HashiCorp Vault) and inject secrets at runtime. Rotate secrets regularly.

Maintain Pipeline as Code: Store pipeline definitions in version control alongside application code. Review pipeline changes like any other code.

Implement Rollback Strategies: Always have a quick rollback mechanism. Keep previous deployment artifacts and maintain database migration reversibility.

Common pitfalls to avoid:

  • Flaky tests: Tests that intermittently fail destroy confidence. Fix or remove them immediately.
  • Overly complex pipelines: If your pipeline requires a manual to understand, simplify it.
  • Ignoring pipeline failures: Broken pipelines must be fixed immediately, not worked around.
  • Testing only in CI: Developers should run tests locally before committing.
  • Skipping stages: Never bypass security scans or tests to “move faster.”

Conclusion and Next Steps

CI/CD pipelines transform software delivery from a manual, error-prone process into a reliable, automated system. Start small: implement basic CI with automated builds and tests, then progressively add deployment automation, security scanning, and advanced deployment strategies.

For teams new to CI/CD, begin with GitHub Actions or GitLab CI—both offer generous free tiers and excellent documentation. Focus on automating your current manual processes one stage at a time. Measure your deployment frequency and lead time, then optimize.

The goal isn’t perfection but continuous improvement. A working pipeline that deploys daily beats a perfect pipeline that never ships. Build, measure, iterate.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.