Vectorized Execution: SIMD Processing

Key Insights

• SIMD (Single Instruction Multiple Data) processes 4-16 data elements simultaneously, delivering 2-8x speedups for data-parallel workloads without requiring additional CPU cores or threads.
• Modern compilers can auto-vectorize simple loops, but real-world code often needs explicit intrinsics or careful restructuring to unlock SIMD performance—understanding the barriers is essential.
• Portability remains SIMD’s biggest challenge; abstraction libraries like Google Highway and xsimd let you write vectorized code once and target multiple architectures without maintaining separate AVX/NEON implementations.

Introduction to SIMD

Most code you write executes one operation at a time. Load a float, add another float, store the result. Repeat a million times. This scalar processing model is intuitive but leaves significant CPU horsepower unused.

SIMD—Single Instruction Multiple Data—flips this model. Instead of processing one element per instruction, SIMD processes 4, 8, or 16 elements simultaneously. A single add instruction operates on an entire vector of numbers in the same time a scalar add handles one.

This matters enormously for performance-critical applications. Image processing, scientific computing, machine learning inference, database analytics, and even JSON parsing benefit from SIMD. Modern CPUs dedicate substantial silicon to vector execution units, but most application code never touches them.

The performance gains are real: 2-8x speedups on data-parallel workloads without spawning threads or adding complexity. The catch? You need to understand how SIMD works to use it effectively.

How SIMD Works at the Hardware Level

SIMD operates through dedicated vector registers that hold multiple data elements. On x86-64 processors, you’ll encounter three generations:

• SSE: 128-bit registers (XMM0-XMM15), processing 4 floats or 2 doubles
• AVX/AVX2: 256-bit registers (YMM0-YMM15), processing 8 floats or 4 doubles
• AVX-512: 512-bit registers (ZMM0-ZMM31), processing 16 floats or 8 doubles

ARM processors use NEON (128-bit) and SVE (scalable, 128-2048 bits).

Each register is divided into lanes. A 256-bit YMM register has 8 lanes of 32 bits each. When you execute an AVX add instruction, all 8 lanes perform addition in parallel:

Scalar Processing (8 operations):
┌──────┐   ┌──────┐   ┌──────┐
│  A0  │ + │  B0  │ = │  C0  │  ← Cycle 1
└──────┘   └──────┘   └──────┘
┌──────┐   ┌──────┐   ┌──────┐
│  A1  │ + │  B1  │ = │  C1  │  ← Cycle 2
└──────┘   └──────┘   └──────┘
        ... (6 more cycles) ...

AVX2 Vectorized (1 operation):
┌────┬────┬────┬────┬────┬────┬────┬────┐
│ A0 │ A1 │ A2 │ A3 │ A4 │ A5 │ A6 │ A7 │  256-bit YMM register
└────┴────┴────┴────┴────┴────┴────┴────┘
                    +
┌────┬────┬────┬────┬────┬────┬────┬────┐
│ B0 │ B1 │ B2 │ B3 │ B4 │ B5 │ B6 │ B7 │  256-bit YMM register
└────┴────┴────┴────┴────┴────┴────┴────┘
                    =
┌────┬────┬────┬────┬────┬────┬────┬────┐
│ C0 │ C1 │ C2 │ C3 │ C4 │ C5 │ C6 │ C7 │  ← Single Cycle
└────┴────┴────┴────┴────┴────┴────┴────┘

Alignment matters for performance. Aligned loads (_mm256_load_ps) require 32-byte alignment but execute faster on some microarchitectures. Unaligned loads (_mm256_loadu_ps) work with any address but may incur penalties on older CPUs. Modern processors have largely eliminated this penalty, but aligned data still helps cache behavior.

Auto-Vectorization by Compilers

Before reaching for intrinsics, check if your compiler can vectorize automatically. Modern compilers (GCC, Clang, MSVC) analyze loops and generate SIMD instructions when safe.

This loop vectorizes cleanly:

void add_arrays(float* a, float* b, float* c, size_t n) {
    for (size_t i = 0; i < n; i++) {
        c[i] = a[i] + b[i];
    }
}

Compile with gcc -O3 -march=native -fopt-info-vec to see vectorization reports. The compiler recognizes independent iterations and generates AVX instructions.

This loop won’t vectorize:

void prefix_sum(float* a, size_t n) {
    for (size_t i = 1; i < n; i++) {
        a[i] = a[i] + a[i-1];  // Loop-carried dependency
    }
}

Each iteration depends on the previous result—no parallelism exists.

Common vectorization barriers include:

• Loop-carried dependencies: Iteration N depends on iteration N-1
• Pointer aliasing: Compiler can’t prove arrays don’t overlap
• Function calls: Unknown side effects prevent optimization
• Complex control flow: Branches inside loops complicate vectorization

Use restrict to promise no aliasing, and #pragma omp simd or #pragma clang loop vectorize(enable) to encourage vectorization:

void add_arrays(float* restrict a, float* restrict b, 
                float* restrict c, size_t n) {
    #pragma omp simd
    for (size_t i = 0; i < n; i++) {
        c[i] = a[i] + b[i];
    }
}

Explicit SIMD with Intrinsics

When auto-vectorization fails or produces suboptimal code, intrinsics give you direct control. Intel intrinsics map one-to-one with assembly instructions but remain portable C/C++ code.

Here’s a vectorized array sum using AVX2:

#include <immintrin.h>

float sum_avx2(const float* arr, size_t n) {
    __m256 sum_vec = _mm256_setzero_ps();  // 8 zeros
    
    size_t i = 0;
    for (; i + 8 <= n; i += 8) {
        __m256 data = _mm256_loadu_ps(&arr[i]);  // Load 8 floats
        sum_vec = _mm256_add_ps(sum_vec, data);  // Add to accumulator
    }
    
    // Horizontal reduction: sum all 8 lanes
    __m128 hi = _mm256_extractf128_ps(sum_vec, 1);
    __m128 lo = _mm256_castps256_ps128(sum_vec);
    __m128 sum128 = _mm_add_ps(hi, lo);
    sum128 = _mm_hadd_ps(sum128, sum128);
    sum128 = _mm_hadd_ps(sum128, sum128);
    float result = _mm_cvtss_f32(sum128);
    
    // Handle remaining elements
    for (; i < n; i++) {
        result += arr[i];
    }
    
    return result;
}

The horizontal reduction at the end is the awkward part of SIMD programming. You’ve accumulated results across 8 lanes, but now you need a single scalar. This requires extracting and combining lanes—operations that don’t parallelize.

ARM NEON uses different intrinsics but follows the same pattern:

#include <arm_neon.h>

float sum_neon(const float* arr, size_t n) {
    float32x4_t sum_vec = vdupq_n_f32(0.0f);
    
    size_t i = 0;
    for (; i + 4 <= n; i += 4) {
        float32x4_t data = vld1q_f32(&arr[i]);
        sum_vec = vaddq_f32(sum_vec, data);
    }
    
    float result = vaddvq_f32(sum_vec);  // NEON has better horizontal ops
    
    for (; i < n; i++) {
        result += arr[i];
    }
    
    return result;
}

Practical Applications and Benchmarks

SIMD shines in real applications. Here’s vectorized RGB-to-grayscale conversion:

#include <immintrin.h>
#include <stdint.h>

// Weights: 0.299R + 0.587G + 0.114B
void rgb_to_gray_avx2(const uint8_t* rgb, uint8_t* gray, size_t pixels) {
    const __m256 r_weight = _mm256_set1_ps(0.299f);
    const __m256 g_weight = _mm256_set1_ps(0.587f);
    const __m256 b_weight = _mm256_set1_ps(0.114f);
    
    for (size_t i = 0; i < pixels; i += 8) {
        // Load and deinterleave RGB (simplified—real code needs shuffle)
        float r[8], g[8], b[8];
        for (int j = 0; j < 8; j++) {
            r[j] = rgb[(i + j) * 3 + 0];
            g[j] = rgb[(i + j) * 3 + 1];
            b[j] = rgb[(i + j) * 3 + 2];
        }
        
        __m256 r_vec = _mm256_loadu_ps(r);
        __m256 g_vec = _mm256_loadu_ps(g);
        __m256 b_vec = _mm256_loadu_ps(b);
        
        // Weighted sum
        __m256 gray_vec = _mm256_mul_ps(r_vec, r_weight);
        gray_vec = _mm256_fmadd_ps(g_vec, g_weight, gray_vec);
        gray_vec = _mm256_fmadd_ps(b_vec, b_weight, gray_vec);
        
        // Convert back to uint8
        __m256i gray_int = _mm256_cvtps_epi32(gray_vec);
        // ... pack and store (omitted for brevity)
    }
}

Benchmark results on a 4K image (8.3 million pixels), Intel i7-12700K:

Database query engines like DuckDB and ClickHouse use SIMD extensively for filtering, aggregation, and string processing. Column-oriented storage aligns naturally with SIMD—scanning a column of integers is a perfect vectorization target.

Pitfalls and Portability Concerns

SIMD code that works on your development machine may crash on deployment. Not all x86-64 CPUs support AVX2. You need runtime feature detection:

#include <stdbool.h>

#if defined(_MSC_VER)
    #include <intrin.h>
#else
    #include <cpuid.h>
#endif

bool has_avx2(void) {
    #if defined(_MSC_VER)
        int info[4];
        __cpuidex(info, 7, 0);
        return (info[1] & (1 << 5)) != 0;  // EBX bit 5
    #else
        unsigned int eax, ebx, ecx, edx;
        if (__get_cpuid_count(7, 0, &eax, &ebx, &ecx, &edx)) {
            return (ebx & (1 << 5)) != 0;
        }
        return false;
    #endif
}

float sum_array(const float* arr, size_t n) {
    if (has_avx2()) {
        return sum_avx2(arr, n);
    }
    return sum_scalar(arr, n);  // Fallback
}

Maintaining separate implementations for SSE, AVX2, AVX-512, and NEON is painful. Abstraction libraries solve this:

• Google Highway: Header-only C++, excellent performance, wide platform support
• xsimd: C++ expression templates, integrates well with existing code
• std::simd (C++26): Standardized approach, still experimental

Highway example:

#include <hwy/highway.h>

namespace hn = hwy::HWY_NAMESPACE;

float sum_highway(const float* arr, size_t n) {
    const hn::ScalableTag<float> d;
    auto sum = hn::Zero(d);
    
    size_t i = 0;
    for (; i + hn::Lanes(d) <= n; i += hn::Lanes(d)) {
        sum = hn::Add(sum, hn::LoadU(d, arr + i));
    }
    
    float result = hn::ReduceSum(d, sum);
    for (; i < n; i++) result += arr[i];
    return result;
}

This compiles to optimal code on x86, ARM, and RISC-V without modification.

Conclusion

SIMD delivers substantial performance gains for data-parallel workloads—often 3-8x faster than scalar code. The complexity is real: you’re managing vector widths, alignment, horizontal reductions, and platform differences. But for performance-critical paths, the payoff justifies the investment.

Start with auto-vectorization. Check compiler reports, remove aliasing ambiguity, simplify control flow. When that’s insufficient, reach for intrinsics or abstraction libraries.

The future looks promising. C++26’s std::simd will bring portable SIMD to the standard library. Until then, Highway and xsimd provide production-ready abstractions. Write vectorized code once, deploy everywhere—that’s the goal we’re approaching.