Eric Campbell 6 min read Engineering

Continuous Testing: Tests in CI/CD Pipeline

Continuous testing means running automated tests at every stage of your CI/CD pipeline, not just before releases. It's the practical implementation of 'shift-left' testing—moving quality verification...

Key Insights

  • Continuous testing shifts quality assurance left in the development cycle, catching bugs when they’re cheapest to fix—at commit time, not production time.
  • The testing pyramid isn’t just theory: run fast unit tests on every commit, integration tests on PR merges, and expensive E2E tests on staging deployments.
  • Flaky tests are pipeline poison—quarantine them aggressively and fix them immediately, or they’ll erode your team’s trust in the entire CI/CD process.

Introduction to Continuous Testing

Continuous testing means running automated tests at every stage of your CI/CD pipeline, not just before releases. It’s the practical implementation of “shift-left” testing—moving quality verification as close to the developer’s keyboard as possible.

The economics are simple: a bug caught during local development costs minutes to fix. The same bug caught in production costs hours of debugging, emergency deployments, and potentially customer trust. Continuous testing creates multiple safety nets throughout your pipeline, each catching issues the previous layer missed.

This isn’t about achieving 100% coverage or testing everything everywhere. It’s about strategic test placement that maximizes confidence while minimizing pipeline duration. Get this balance wrong, and developers will start ignoring test failures or, worse, bypassing the pipeline entirely.

Anatomy of a CI/CD Testing Pipeline

A well-designed testing pipeline follows a funnel pattern: broad, fast checks first, narrowing to slower, more comprehensive tests as code progresses toward production.

The typical flow looks like this:

  1. Commit stage: Lint, format checks, unit tests (seconds to minutes)
  2. Build stage: Compile, bundle, create artifacts (minutes)
  3. Integration stage: API tests, service integration tests (minutes)
  4. Deployment stage: Deploy to staging, run E2E tests (minutes to hours)
  5. Release stage: Smoke tests in production (seconds)

Each stage acts as a gate. Fail early, fail fast. There’s no point running expensive E2E tests if your code doesn’t compile.

# .github/workflows/ci.yml
name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm run lint

  unit-test:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test -- --coverage

  integration-test:
    needs: unit-test
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5          
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm run test:integration
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/test

Notice the needs keyword creating dependencies between jobs. Lint must pass before unit tests run. Unit tests must pass before integration tests spin up a database.

Types of Tests in the Pipeline

The testing pyramid remains the most practical mental model for CI/CD test organization. Wide base of unit tests, narrower middle of integration tests, thin top of E2E tests.

Unit tests verify isolated functions and classes. They’re fast, deterministic, and should comprise 70-80% of your test suite. Run them on every commit.

// user.service.test.js
import { UserService } from './user.service';

describe('UserService', () => {
  describe('validateEmail', () => {
    it('accepts valid email addresses', () => {
      const service = new UserService();
      expect(service.validateEmail('user@example.com')).toBe(true);
    });

    it('rejects emails without @ symbol', () => {
      const service = new UserService();
      expect(service.validateEmail('invalid-email')).toBe(false);
    });

    it('rejects empty strings', () => {
      const service = new UserService();
      expect(service.validateEmail('')).toBe(false);
    });
  });
});

Integration tests verify that components work together correctly—your API with your database, your service with external APIs. They’re slower and require infrastructure.

E2E tests simulate real user journeys through your entire application. They’re the slowest and most brittle, but they catch issues that unit and integration tests miss.

// checkout.e2e.test.js
import { test, expect } from '@playwright/test';

test('user can complete checkout flow', async ({ page }) => {
  await page.goto('/products');
  
  // Add item to cart
  await page.click('[data-testid="product-1"] button');
  await expect(page.locator('[data-testid="cart-count"]')).toHaveText('1');
  
  // Navigate to checkout
  await page.click('[data-testid="checkout-button"]');
  await expect(page).toHaveURL('/checkout');
  
  // Fill shipping details
  await page.fill('[name="address"]', '123 Test Street');
  await page.fill('[name="city"]', 'Test City');
  await page.selectOption('[name="country"]', 'US');
  
  // Complete purchase
  await page.click('[data-testid="place-order"]');
  await expect(page.locator('[data-testid="confirmation"]')).toBeVisible();
});

The scope difference is obvious: the unit test runs in milliseconds with no dependencies. The E2E test requires a running application, browser automation, and potentially test data seeding.

Real-world pipelines need more than sequential job execution. You need caching to avoid reinstalling dependencies, parallelization to reduce total runtime, and artifact storage for test reports.

# .gitlab-ci.yml
stages:
  - test
  - report

variables:
  npm_config_cache: '$CI_PROJECT_DIR/.npm'

.test-template: &test-template
  image: node:20
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - .npm/
      - node_modules/
  before_script:
    - npm ci --cache .npm --prefer-offline

unit-tests:
  <<: *test-template
  stage: test
  parallel: 4
  script:
    - npm run test:unit -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL --coverage
  artifacts:
    paths:
      - coverage/
    reports:
      junit: junit.xml
    expire_in: 1 week

integration-tests:
  <<: *test-template
  stage: test
  services:
    - postgres:15
    - redis:7
  variables:
    POSTGRES_PASSWORD: test
    DATABASE_URL: postgres://postgres:test@postgres:5432/test
    REDIS_URL: redis://redis:6379
  script:
    - npm run test:integration
  artifacts:
    reports:
      junit: junit-integration.xml

coverage-report:
  stage: report
  needs: [unit-tests]
  script:
    - npx nyc merge coverage/ merged-coverage.json
    - npx nyc report --reporter=text --reporter=html
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    paths:
      - coverage-report/

The parallel: 4 directive splits unit tests across four runners using Jest’s built-in sharding. Combined with caching, this can cut a 10-minute test suite down to 3 minutes.

Handling Test Failures and Flaky Tests

Flaky tests—tests that pass and fail intermittently without code changes—are the silent killers of CI/CD trust. When developers see random failures, they start clicking “retry” instead of investigating. Eventually, real failures get ignored.

Handle flakiness systematically:

# .github/workflows/ci.yml
jobs:
  e2e-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run E2E tests with retry
        uses: nick-fields/retry@v2
        with:
          timeout_minutes: 30
          max_attempts: 3
          retry_on: error
          command: npm run test:e2e
      
      - name: Notify on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          channel-id: 'C0123456789'
          payload: |
            {
              "text": "E2E tests failed on ${{ github.ref }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*E2E Test Failure*\nBranch: `${{ github.ref }}`\nCommit: `${{ github.sha }}`\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }            
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Retries are a band-aid, not a solution. Track which tests need retries and fix them. Consider quarantining persistently flaky tests to a separate non-blocking job while you investigate.

Distinguish between blocking and non-blocking test gates. Unit tests should block merges. Experimental E2E tests for new features might warn but allow merges while you stabilize them.

Test Reporting and Metrics

Raw pass/fail isn’t enough. You need trends: Is coverage increasing? Are tests getting slower? Which tests fail most often?

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test -- --coverage --coverageReporters=json-summary
      
      - name: Check coverage thresholds
        run: |
          COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
          echo "Line coverage: $COVERAGE%"
          if (( $(echo "$COVERAGE < 80" | bc -l) )); then
            echo "Coverage below 80% threshold"
            exit 1
          fi          
      
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info
          fail_ci_if_error: true

JUnit XML is the universal format for test results. Most CI tools parse it automatically for dashboard displays. Configure your test runner to output this format alongside its native reporting.

Best Practices and Common Pitfalls

Keep feedback loops under 10 minutes. Developers context-switch after 10 minutes of waiting. If your pipeline takes 30 minutes, parallelize aggressively or move slow tests to post-merge.

Match test environments to production. If production runs on Linux with PostgreSQL 15, your CI should too. Docker makes this straightforward. “Works on my machine” shouldn’t extend to “works in CI but not production.”

Never commit secrets to test configurations. Use CI/CD secret management for API keys, database credentials, and service tokens. Rotate them regularly.

Don’t test external services in CI. Mock third-party APIs. You’re testing your code’s behavior, not whether Stripe’s API is up. External dependencies introduce flakiness you can’t control.

Fail fast, but provide context. When tests fail, developers need to know which test, what assertion, and ideally a screenshot or log snippet. Invest in good error messages and artifact collection.

Review pipeline performance quarterly. Tests accumulate. What was a 5-minute pipeline becomes 20 minutes through gradual addition. Audit regularly, remove redundant tests, and optimize slow ones.

Continuous testing isn’t a checkbox—it’s an ongoing practice. Start with the basics: unit tests on every commit, integration tests on PRs, E2E tests before deployment. Then iterate based on what breaks in production. Every production bug is a missing test waiting to be written.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.