GitOps: Git-Driven Infrastructure Management

Key Insights

GitOps transforms Git repositories into the single source of truth for infrastructure, enabling declarative management where desired state is version-controlled and automatically reconciled with actual state
The pull-based deployment model, where operators running in-cluster continuously sync from Git, provides better security and auditability compared to traditional push-based CI/CD pipelines
Proper secrets management and environment promotion strategies are critical—use tools like Sealed Secrets or External Secrets Operator rather than committing sensitive data to Git

Introduction to GitOps

GitOps represents a fundamental shift in how we manage infrastructure and application deployments. Instead of running imperative scripts that execute commands against your infrastructure, GitOps treats Git as the single source of truth for declarative infrastructure definitions. When you want to change your infrastructure, you commit the desired state to Git—operators running in your cluster detect the difference and automatically reconcile the actual state to match.

This approach brings the benefits of version control—audit trails, rollback capabilities, peer review—to infrastructure management. Every change is traceable, every deployment is reproducible, and your entire infrastructure can be recovered from a Git repository. The shift from “run this command” to “this is what I want” fundamentally changes how teams collaborate on infrastructure.

Core GitOps Principles

GitOps rests on four foundational principles that distinguish it from traditional infrastructure management:

Declarative infrastructure: Your entire system is described declaratively. Rather than scripts that modify state, you define what the end state should look like. Kubernetes manifests, Terraform configurations, and Helm charts are all declarative formats that fit this model.

Git as the source of truth: The desired state of your system lives in Git. Not in a database, not in scripts on someone’s laptop, not in a CI/CD tool’s configuration. Git repositories hold the canonical definition of what should be running.

Automated deployment: Approved changes to the Git repository automatically trigger deployments. No manual kubectl apply commands, no SSH-ing into servers. The system converges toward the declared state automatically.

Continuous reconciliation: Software agents continuously observe the actual state and compare it to the desired state in Git. When drift occurs—whether from manual changes, failures, or other issues—the system automatically corrects it.

The pull versus push model is crucial here. Traditional CI/CD pushes changes from outside the cluster, requiring credentials and network access. GitOps uses operators running inside the cluster that pull changes from Git, providing better security boundaries and audit capabilities.

Here’s a simple declarative Kubernetes deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web-app
        image: myregistry/web-app:v1.2.3
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

This manifest describes the desired state. GitOps operators ensure reality matches this declaration.

GitOps Workflow in Practice

The GitOps workflow creates a continuous loop of observation and reconciliation:

A developer commits infrastructure changes to Git
The commit triggers automated validation and testing
After approval (manual or automated), changes merge to the main branch
The GitOps operator detects the new commit
The operator compares desired state (Git) with actual state (cluster)
The operator applies necessary changes to reconcile the difference
The cycle continues, constantly monitoring for drift

Here’s a GitHub Actions workflow that validates Kubernetes manifests when infrastructure changes:

name: Validate Infrastructure
on:
  pull_request:
    paths:
      - 'kubernetes/**'
      - 'apps/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Validate Kubernetes manifests
        run: |
          kubectl apply --dry-run=client -f kubernetes/
                    
      - name: Run kubeval
        uses: instrumenta/kubeval-action@master
        with:
          files: kubernetes/
          
      - name: Check Kustomize builds
        run: |
          kustomize build kubernetes/overlays/production > /dev/null

An ArgoCD Application manifest configures how the operator syncs your infrastructure:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-app-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/infrastructure
    targetRevision: main
    path: apps/web-app/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Popular GitOps Tools

Three tools dominate the GitOps landscape:

ArgoCD is the most popular choice for Kubernetes-focused GitOps. It provides a powerful UI, supports multiple config management tools (Helm, Kustomize, plain YAML), and offers sophisticated sync strategies. Choose ArgoCD when you want visibility, multi-cluster management, and a mature ecosystem.

Flux is a CNCF graduated project that takes a more Kubernetes-native approach. It uses Custom Resources for all configuration and integrates tightly with the Kubernetes API. Choose Flux when you prefer a fully declarative approach and want to manage the GitOps operator itself through GitOps.

Jenkins X provides an opinionated platform for CI/CD and GitOps combined. It’s best suited for teams wanting a complete solution that includes build pipelines, preview environments, and promotion workflows.

Here’s how to install ArgoCD and deploy an application:

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Create an application
kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/my-app
    targetRevision: HEAD
    path: kubernetes
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Flux uses a different approach with GitRepository and Kustomization resources:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/infrastructure
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: infrastructure

Implementing GitOps for Kubernetes

A well-structured repository is essential for effective GitOps. Organize by environment and application:

infrastructure/
├── apps/
│   ├── web-app/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── dev/
│   │       │   └── kustomization.yaml
│   │       ├── staging/
│   │       │   └── kustomization.yaml
│   │       └── production/
│   │           └── kustomization.yaml
├── infrastructure/
│   ├── base/
│   └── overlays/
└── argocd/
    └── applications/

Kustomize overlays handle environment-specific configurations:

# apps/web-app/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
replicas:
  - name: web-app
    count: 5
images:
  - name: myregistry/web-app
    newTag: v1.2.3
configMapGenerator:
  - name: app-config
    literals:
      - ENVIRONMENT=production
      - LOG_LEVEL=warn

Configure ArgoCD sync policies to control deployment behavior:

syncPolicy:
  automated:
    prune: true      # Delete resources removed from Git
    selfHeal: true   # Revert manual changes
  syncOptions:
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m

Security and Best Practices

Never commit secrets to Git. Use Sealed Secrets or External Secrets Operator to manage sensitive data:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: production/database
        property: password

Implement RBAC for GitOps operators to limit blast radius:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-application-controller
  namespace: production
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-application-controller
  namespace: production
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd-application-controller
subjects:
  - kind: ServiceAccount
    name: argocd-application-controller
    namespace: argocd

Challenges and Solutions

Secret management remains the biggest challenge. Solutions include Sealed Secrets for encryption at rest in Git, External Secrets Operator for dynamic secret injection, or HashiCorp Vault integration. Never compromise on this—leaked credentials in Git history are permanent.

Bootstrapping requires careful planning. You need the GitOps operator to manage itself, creating a chicken-and-egg problem. Solve this by maintaining a minimal bootstrap configuration outside GitOps, then letting the operator manage its own upgrades.

Manual interventions will happen in emergencies. When they do, document them and commit the changes to Git afterward. Configure selfHeal policies carefully—too aggressive and you prevent emergency fixes, too lenient and you lose drift detection benefits.

Monitoring drift is critical. Set up alerts when sync fails, when manual changes are detected, or when reconciliation takes too long. Your GitOps operator should integrate with your observability stack.

GitOps isn’t just a deployment strategy—it’s a operational philosophy that brings discipline and automation to infrastructure management. Start small, perhaps with a single non-critical application, then expand as your team builds confidence in the approach.