Eric Campbell 5 min read Machine Learning

How to Perform Random Search in Python

Hyperparameter tuning is the process of finding optimal configuration values that govern your model's learning process. Unlike model parameters learned during training, hyperparameters must be set...

Key Insights

  • Random search samples hyperparameters from probability distributions rather than exhaustively testing all combinations, making it exponentially more efficient than grid search in high-dimensional spaces
  • For the same computational budget, random search explores more distinct hyperparameter values and often finds better configurations than grid search, especially when only a few parameters significantly impact model performance
  • Scikit-learn’s RandomizedSearchCV provides production-ready random search with cross-validation, parallel execution, and flexible parameter distributions using scipy.stats

Hyperparameter tuning is the process of finding optimal configuration values that govern your model’s learning process. Unlike model parameters learned during training, hyperparameters must be set before training begins. Poor hyperparameter choices can cripple even the most sophisticated algorithms.

Random search approaches this optimization problem by randomly sampling hyperparameter combinations from predefined distributions. Instead of testing every possible combination like grid search, random search draws samples probabilistically, allowing you to explore the hyperparameter space more efficiently. This becomes critical when dealing with multiple hyperparameters or continuous ranges where exhaustive search is computationally prohibitive.

The key advantage is simple: random search lets you control your computational budget directly through the number of iterations while maintaining broad coverage of the search space.

Grid search creates a Cartesian product of all hyperparameter values, testing every combination. With 3 hyperparameters and 10 values each, you evaluate 1,000 combinations. Add another hyperparameter and you’re at 10,000—exponential growth that quickly becomes unmanageable.

Random search breaks this curse of dimensionality. With the same 4 hyperparameters, you might sample only 100 random combinations, yet you’ve explored 100 distinct values per hyperparameter rather than just 10. Research by Bergstra and Bengio (2012) showed that random search is more efficient when only a few hyperparameters significantly affect performance—a common scenario in practice.

Here’s a visualization comparing search patterns:

import numpy as np
import matplotlib.pyplot as plt

# Simulate grid search
grid_points = np.array([[x, y] for x in np.linspace(0, 1, 5) 
                        for y in np.linspace(0, 1, 5)])

# Simulate random search (same number of evaluations)
np.random.seed(42)
random_points = np.random.uniform(0, 1, (25, 2))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

ax1.scatter(grid_points[:, 0], grid_points[:, 1], s=100, alpha=0.6)
ax1.set_title('Grid Search: 25 evaluations')
ax1.set_xlabel('Hyperparameter 1')
ax1.set_ylabel('Hyperparameter 2')
ax1.grid(True, alpha=0.3)

ax2.scatter(random_points[:, 0], random_points[:, 1], s=100, alpha=0.6, color='orange')
ax2.set_title('Random Search: 25 evaluations')
ax2.set_xlabel('Hyperparameter 1')
ax2.set_ylabel('Hyperparameter 2')
ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Notice how grid search clusters around specific values while random search provides better coverage across the entire space. If the optimal hyperparameter value falls between grid points, grid search will miss it entirely.

Random Search with Scikit-learn

Scikit-learn’s RandomizedSearchCV provides a robust implementation with cross-validation built in. Let’s tune a Random Forest classifier on the digits dataset:

from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
import numpy as np

# Load data
X, y = load_digits(return_X_y=True)

# Define the model
rf = RandomForestClassifier(random_state=42)

# Define parameter distributions
param_distributions = {
    'n_estimators': randint(50, 500),
    'max_depth': randint(3, 20),
    'min_samples_split': randint(2, 20),
    'min_samples_leaf': randint(1, 10),
    'max_features': uniform(0.1, 0.9),
    'bootstrap': [True, False]
}

# Configure random search
random_search = RandomizedSearchCV(
    estimator=rf,
    param_distributions=param_distributions,
    n_iter=100,  # Number of parameter combinations to try
    cv=5,  # 5-fold cross-validation
    scoring='accuracy',
    n_jobs=-1,  # Use all processors
    random_state=42,
    verbose=1
)

# Execute search
random_search.fit(X, y)

# Best parameters and score
print(f"Best parameters: {random_search.best_params_}")
print(f"Best cross-validation score: {random_search.best_score_:.4f}")
print(f"Test score: {random_search.score(X, y):.4f}")

The n_iter parameter controls your computational budget. With 5-fold CV, setting n_iter=100 means 500 total model fits (100 combinations × 5 folds). Adjust this based on your time constraints and model complexity.

Defining Parameter Distributions

Choosing appropriate distributions for your hyperparameters is crucial. Use scipy.stats to define distributions that match the parameter’s nature:

from scipy.stats import randint, uniform, loguniform

# Integer parameters with uniform distribution
param_distributions = {
    # Tree depth: uniform integers from 3 to 19
    'max_depth': randint(3, 20),
    
    # Number of estimators: uniform integers
    'n_estimators': randint(50, 500),
}

# Continuous parameters
param_distributions.update({
    # Learning rate: log-uniform distribution (better for rates)
    # Samples more densely at smaller values
    'learning_rate': loguniform(1e-4, 1e-1),
    
    # Regularization: log-uniform for exponential scale
    'alpha': loguniform(1e-5, 1e0),
    
    # Feature fraction: uniform between 0.5 and 1.0
    'max_features': uniform(0.5, 0.5),  # loc=0.5, scale=0.5
})

# Categorical parameters
param_distributions.update({
    'criterion': ['gini', 'entropy'],
    'bootstrap': [True, False]
})

Use loguniform for parameters that span orders of magnitude (learning rates, regularization). Use uniform for parameters with linear scales (fractions, proportions). Use randint for discrete integer values.

Custom Random Search Implementation

Understanding the mechanics helps you customize the process for specific needs. Here’s a basic implementation from scratch:

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

def random_search_custom(X, y, param_distributions, n_iter=50, cv=5):
    """
    Custom random search implementation.
    """
    best_score = -np.inf
    best_params = None
    results = []
    
    for i in range(n_iter):
        # Sample parameters
        params = {}
        for param_name, distribution in param_distributions.items():
            if isinstance(distribution, list):
                # Categorical parameter
                params[param_name] = np.random.choice(distribution)
            elif hasattr(distribution, 'rvs'):
                # Scipy distribution
                params[param_name] = distribution.rvs()
            else:
                # Assume range tuple (min, max)
                params[param_name] = np.random.uniform(*distribution)
        
        # Train and evaluate
        model = RandomForestClassifier(**params, random_state=42)
        scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')
        mean_score = scores.mean()
        
        # Track results
        results.append({
            'params': params.copy(),
            'mean_score': mean_score,
            'std_score': scores.std()
        })
        
        # Update best
        if mean_score > best_score:
            best_score = mean_score
            best_params = params.copy()
        
        if (i + 1) % 10 == 0:
            print(f"Iteration {i+1}/{n_iter}, Best Score: {best_score:.4f}")
    
    return best_params, best_score, results

# Example usage
param_distributions = {
    'n_estimators': randint(50, 200),
    'max_depth': randint(3, 15),
    'min_samples_split': randint(2, 10)
}

best_params, best_score, results = random_search_custom(
    X, y, param_distributions, n_iter=30, cv=3
)

print(f"\nBest parameters: {best_params}")
print(f"Best score: {best_score:.4f}")

This implementation gives you full control over the sampling process and result tracking, useful when you need custom logic or want to integrate with other optimization frameworks.

Advanced Techniques and Best Practices

Combine random search with preprocessing pipelines to tune both feature engineering and model hyperparameters simultaneously:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC

# Create pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA()),
    ('classifier', SVC())
])

# Define search space across pipeline components
param_distributions = {
    'pca__n_components': randint(10, 50),
    'classifier__C': loguniform(1e-2, 1e2),
    'classifier__gamma': loguniform(1e-4, 1e-1),
    'classifier__kernel': ['rbf', 'linear']
}

# Search with parallel execution
random_search = RandomizedSearchCV(
    pipeline,
    param_distributions,
    n_iter=50,
    cv=5,
    n_jobs=-1,  # Parallel processing
    random_state=42,
    verbose=2,
    return_train_score=True  # Track overfitting
)

random_search.fit(X, y)

Set n_jobs=-1 to use all CPU cores. Monitor return_train_score=True to detect overfitting when training scores significantly exceed validation scores.

Evaluating and Visualizing Results

Extract detailed results to understand which hyperparameters matter most:

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Convert results to DataFrame
results_df = pd.DataFrame(random_search.cv_results_)

# Extract key columns
analysis_df = results_df[[
    'param_n_estimators', 
    'param_max_depth',
    'param_min_samples_split',
    'mean_test_score',
    'std_test_score',
    'rank_test_score'
]].copy()

# Sort by performance
analysis_df = analysis_df.sort_values('rank_test_score')
print(analysis_df.head(10))

# Visualize parameter importance
fig, axes = plt.subplots(1, 3, figsize=(15, 4))

for idx, param in enumerate(['param_n_estimators', 'param_max_depth', 'param_min_samples_split']):
    axes[idx].scatter(
        analysis_df[param], 
        analysis_df['mean_test_score'],
        alpha=0.6
    )
    axes[idx].set_xlabel(param.replace('param_', ''))
    axes[idx].set_ylabel('CV Score')
    axes[idx].set_title(f'Impact of {param.replace("param_", "")}')

plt.tight_layout()
plt.show()

# Score distribution
plt.figure(figsize=(10, 6))
plt.hist(results_df['mean_test_score'], bins=30, edgecolor='black')
plt.xlabel('Cross-Validation Score')
plt.ylabel('Frequency')
plt.title('Distribution of Model Performance')
plt.axvline(random_search.best_score_, color='red', linestyle='--', 
            label=f'Best: {random_search.best_score_:.4f}')
plt.legend()
plt.show()

These visualizations reveal which hyperparameters have the strongest impact on performance and whether you’ve adequately explored the search space. If all your best results cluster at distribution boundaries, expand those ranges and search again.

Random search provides an excellent balance between exploration and computational efficiency. Use it as your default hyperparameter tuning approach, reserving grid search for final refinement of a narrow parameter range or when you need exhaustive guarantees. The probabilistic nature means you might miss the absolute optimum, but you’ll find near-optimal solutions much faster—a worthwhile trade-off in practice.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.