Eric Campbell 5 min read Machine Learning

How to Implement SVM for Classification in Python

Support Vector Machines are supervised learning algorithms that find the optimal hyperplane separating different classes in your data. Unlike simpler classifiers that just find any decision boundary,...

Key Insights

  • Support Vector Machines excel at binary classification in high-dimensional spaces by finding the optimal separating hyperplane that maximizes the margin between classes
  • The kernel trick allows SVMs to handle non-linearly separable data without explicitly transforming features, with RBF being the most versatile kernel for real-world problems
  • Proper feature scaling is non-negotiable for SVMs since the algorithm is sensitive to feature magnitudes, and hyperparameter tuning (C and gamma) dramatically impacts performance

Introduction to Support Vector Machines

Support Vector Machines are supervised learning algorithms that find the optimal hyperplane separating different classes in your data. Unlike simpler classifiers that just find any decision boundary, SVMs maximize the margin—the distance between the hyperplane and the nearest data points from each class (called support vectors).

SVMs shine in several scenarios: high-dimensional datasets where features outnumber samples, text classification, image recognition, and bioinformatics. They’re memory-efficient since they only use support vectors for predictions, not the entire training set. The main tradeoff is computational cost on large datasets (100k+ samples) and the need for careful hyperparameter tuning.

The fundamental idea is elegant: in a binary classification problem, SVM finds the hyperplane that creates the widest possible “street” between classes. Data points on the edge of this street are your support vectors, and they’re the only points that matter for defining the decision boundary.

Setting Up the Environment

You’ll need scikit-learn for the SVM implementation, NumPy for numerical operations, pandas for data handling, and matplotlib for visualization.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
from sklearn.metrics import roc_curve, roc_auc_score
import seaborn as sns

# Load the breast cancer dataset
data = datasets.load_breast_cancer()
X = data.data
y = data.target

print(f"Dataset shape: {X.shape}")
print(f"Classes: {data.target_names}")
print(f"Features: {data.feature_names[:5]}...")  # Show first 5 features

The breast cancer dataset contains 569 samples with 30 features each, making it perfect for demonstrating SVM’s effectiveness in higher-dimensional spaces.

Basic SVM Implementation

Start with a linear SVM using default parameters. This establishes a baseline before optimization.

# Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Scale features - critical for SVM
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create and train linear SVM
svm_linear = SVC(kernel='linear', random_state=42)
svm_linear.fit(X_train_scaled, y_train)

# Make predictions
y_pred = svm_linear.predict(X_test_scaled)

# Evaluate
accuracy = accuracy_score(y_test, y_pred)
print(f"Linear SVM Accuracy: {accuracy:.4f}")

Notice the StandardScaler—this isn’t optional. SVMs use distance calculations, so features on different scales will dominate the optimization. A feature ranging from 0-1000 will overwhelm one ranging from 0-1. Always scale your features.

Kernel Functions and Non-Linear Classification

Most real-world data isn’t linearly separable. The kernel trick maps your data into a higher-dimensional space where a linear separator exists, without explicitly computing the transformation.

# Compare different kernels
kernels = ['linear', 'poly', 'rbf', 'sigmoid']
results = {}

for kernel in kernels:
    svm = SVC(kernel=kernel, random_state=42)
    svm.fit(X_train_scaled, y_train)
    y_pred = svm.predict(X_test_scaled)
    accuracy = accuracy_score(y_test, y_pred)
    results[kernel] = accuracy
    print(f"{kernel.upper()} Kernel Accuracy: {accuracy:.4f}")

# Visualize results
plt.figure(figsize=(10, 6))
plt.bar(results.keys(), results.values())
plt.ylabel('Accuracy')
plt.xlabel('Kernel Type')
plt.title('SVM Performance by Kernel Type')
plt.ylim([0.9, 1.0])
plt.show()

The RBF (Radial Basis Function) kernel typically performs best on diverse datasets. It creates circular decision boundaries and can handle complex, non-linear relationships. The polynomial kernel works well when you suspect polynomial relationships, while sigmoid mimics neural networks.

For 2D visualization of decision boundaries:

# Use only 2 features for visualization
X_2d = X[:, :2]
X_train_2d, X_test_2d, y_train_2d, y_test_2d = train_test_split(
    X_2d, y, test_size=0.3, random_state=42
)

scaler_2d = StandardScaler()
X_train_2d_scaled = scaler_2d.fit_transform(X_train_2d)

# Train RBF SVM
svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')
svm_rbf.fit(X_train_2d_scaled, y_train_2d)

# Create mesh for decision boundary
h = 0.02
x_min, x_max = X_train_2d_scaled[:, 0].min() - 1, X_train_2d_scaled[:, 0].max() + 1
y_min, y_max = X_train_2d_scaled[:, 1].min() - 1, X_train_2d_scaled[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

Z = svm_rbf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.contourf(xx, yy, Z, alpha=0.8, cmap=plt.cm.RdYlBu)
plt.scatter(X_train_2d_scaled[:, 0], X_train_2d_scaled[:, 1], c=y_train_2d, 
            edgecolors='k', cmap=plt.cm.RdYlBu)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.title('SVM Decision Boundary (RBF Kernel)')
plt.show()

Hyperparameter Tuning

The C parameter controls the tradeoff between maximizing margin and minimizing classification error. Low C creates a wider margin but allows more misclassifications. High C demands fewer errors but may overfit.

Gamma (for RBF, polynomial, and sigmoid kernels) defines how far the influence of a single training example reaches. Low gamma means far reach (smoother boundaries), high gamma means close reach (more complex boundaries).

# Manual parameter testing
C_values = [0.1, 1, 10, 100]
gamma_values = [0.001, 0.01, 0.1, 1]

best_score = 0
best_params = {}

for C in C_values:
    for gamma in gamma_values:
        svm = SVC(kernel='rbf', C=C, gamma=gamma, random_state=42)
        svm.fit(X_train_scaled, y_train)
        score = svm.score(X_test_scaled, y_test)
        
        if score > best_score:
            best_score = score
            best_params = {'C': C, 'gamma': gamma}

print(f"Best parameters: {best_params}")
print(f"Best score: {best_score:.4f}")

# GridSearchCV - automated and more robust
param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': ['scale', 'auto', 0.001, 0.01, 0.1],
    'kernel': ['rbf', 'poly']
}

grid_search = GridSearchCV(
    SVC(random_state=42), 
    param_grid, 
    cv=5, 
    scoring='accuracy',
    n_jobs=-1,
    verbose=1
)

grid_search.fit(X_train_scaled, y_train)

print(f"\nGridSearchCV Best parameters: {grid_search.best_params_}")
print(f"GridSearchCV Best cross-validation score: {grid_search.best_score_:.4f}")

# Evaluate on test set
best_svm = grid_search.best_estimator_
y_pred_best = best_svm.predict(X_test_scaled)
print(f"Test set accuracy: {accuracy_score(y_test, y_pred_best):.4f}")

GridSearchCV uses cross-validation, giving you more reliable performance estimates than a single train-test split. Always use the test set only for final evaluation, never for hyperparameter selection.

Model Evaluation and Interpretation

Accuracy alone is misleading, especially with imbalanced datasets. Use a comprehensive evaluation strategy.

# Classification report
print("\nClassification Report:")
print(classification_report(y_test, y_pred_best, target_names=data.target_names))

# Confusion matrix
cm = confusion_matrix(y_test, y_pred_best)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', 
            xticklabels=data.target_names, 
            yticklabels=data.target_names)
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.title('Confusion Matrix')
plt.show()

# ROC curve for binary classification
y_scores = best_svm.decision_function(X_test_scaled)
fpr, tpr, thresholds = roc_curve(y_test, y_scores)
roc_auc = roc_auc_score(y_test, y_scores)

plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='darkorange', lw=2, 
         label=f'ROC curve (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend(loc="lower right")
plt.show()

The classification report shows precision (how many predicted positives are actually positive), recall (how many actual positives you caught), and F1-score (harmonic mean of precision and recall). For medical diagnosis or fraud detection, prioritize recall. For spam filtering, prioritize precision.

Real-World Application and Best Practices

Here’s a complete production-ready pipeline incorporating best practices:

from sklearn.pipeline import Pipeline
from sklearn.model_selection import StratifiedKFold

# Create pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf', random_state=42, probability=True))
])

# Stratified K-Fold for imbalanced data
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Parameter grid
param_grid = {
    'svm__C': [0.1, 1, 10],
    'svm__gamma': ['scale', 0.01, 0.1]
}

# Grid search with stratified CV
grid = GridSearchCV(
    pipeline, 
    param_grid, 
    cv=skf, 
    scoring='f1',  # Use F1 for imbalanced data
    n_jobs=-1
)

grid.fit(X_train, y_train)

# Final evaluation
y_pred_final = grid.predict(X_test)
y_proba = grid.predict_proba(X_test)

print(f"Final Test Accuracy: {accuracy_score(y_test, y_pred_final):.4f}")
print(f"\nFinal Classification Report:")
print(classification_report(y_test, y_pred_final))

Critical best practices:

  1. Always scale features before training SVMs
  2. Use pipelines to prevent data leakage between scaling and cross-validation
  3. Choose appropriate metrics based on your problem (F1 for imbalanced data, accuracy for balanced)
  4. Start with RBF kernel unless you have domain knowledge suggesting otherwise
  5. Use cross-validation for hyperparameter tuning, never tune on test data
  6. Consider computational cost - SVMs struggle with datasets over 100k samples; consider SGDClassifier with hinge loss as an alternative
  7. Handle class imbalance with class_weight='balanced' parameter or resampling techniques

SVMs remain powerful tools for classification when you have moderate-sized datasets with complex decision boundaries. Master the fundamentals, tune carefully, and always validate properly.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.