Go HTTP Client: Making HTTP Requests

Key Insights

Go’s net/http package provides a production-ready HTTP client out of the box, but the default client lacks timeouts and should never be used in production code
Always close response bodies with defer resp.Body.Close() immediately after checking for errors to prevent resource leaks, even when the request fails
Custom HTTP clients with proper timeout configurations, connection pooling, and retry logic are essential for building reliable distributed systems

Introduction to Go’s HTTP Client

Go’s net/http package is one of the standard library’s strongest offerings, providing everything you need to make HTTP requests without external dependencies. Unlike many languages that require third-party libraries for HTTP communication, Go gives you a robust, production-ready client from day one.

The package offers two primary approaches: convenience functions like http.Get() for quick operations, and custom clients for production use. Here’s the critical distinction: the default client (http.DefaultClient) has no timeout configuration. A misbehaving server can hang your application indefinitely. For any production code, you should always create a custom client with appropriate timeouts.

Making Basic HTTP Requests

Let’s start with the fundamentals. Go provides convenience functions for common HTTP methods, but they all use the default client under the hood.

package main

import (
    "fmt"
    "io"
    "net/http"
)

func simpleGet() error {
    resp, err := http.Get("https://api.github.com/users/golang")
    if err != nil {
        return fmt.Errorf("request failed: %w", err)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return fmt.Errorf("reading response: %w", err)
    }

    fmt.Printf("Status: %d\n", resp.StatusCode)
    fmt.Printf("Body: %s\n", body)
    return nil
}

For POST requests with JSON payloads, use http.Post() or construct a request manually for more control:

import (
    "bytes"
    "encoding/json"
    "net/http"
)

type CreateUserRequest struct {
    Name  string `json:"name"`
    Email string `json:"email"`
}

func createUser() error {
    user := CreateUserRequest{
        Name:  "John Doe",
        Email: "john@example.com",
    }

    jsonData, err := json.Marshal(user)
    if err != nil {
        return err
    }

    resp, err := http.Post(
        "https://api.example.com/users",
        "application/json",
        bytes.NewBuffer(jsonData),
    )
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    // Handle response...
    return nil
}

For full control over the request method and configuration, use http.NewRequest():

func updateUser(userID string) error {
    payload := map[string]string{"status": "active"}
    jsonData, _ := json.Marshal(payload)

    req, err := http.NewRequest(
        http.MethodPut,
        fmt.Sprintf("https://api.example.com/users/%s", userID),
        bytes.NewBuffer(jsonData),
    )
    if err != nil {
        return err
    }

    req.Header.Set("Content-Type", "application/json")
    
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    return nil
}

Working with Request and Response Objects

Real-world HTTP clients need to handle headers, query parameters, and authentication. Here’s how to do it properly:

import (
    "net/url"
)

func authenticatedRequest(apiToken string) error {
    // Build URL with query parameters
    baseURL, _ := url.Parse("https://api.example.com/data")
    params := url.Values{}
    params.Add("limit", "100")
    params.Add("offset", "0")
    params.Add("filter", "active")
    baseURL.RawQuery = params.Encode()

    req, err := http.NewRequest(http.MethodGet, baseURL.String(), nil)
    if err != nil {
        return err
    }

    // Set headers
    req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", apiToken))
    req.Header.Set("Accept", "application/json")
    req.Header.Set("User-Agent", "MyApp/1.0")

    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    // Parse JSON response
    var result map[string]interface{}
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return fmt.Errorf("parsing response: %w", err)
    }

    fmt.Printf("Data: %+v\n", result)
    return nil
}

The critical pattern here is defer resp.Body.Close() immediately after the error check. This ensures proper resource cleanup even if subsequent operations fail. Using json.NewDecoder is more efficient than io.ReadAll followed by json.Unmarshal for most use cases.

Creating Custom HTTP Clients

Production applications require custom clients with timeouts, connection pooling, and controlled redirect behavior:

import (
    "net"
    "time"
)

func newProductionClient() *http.Client {
    return &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            MaxIdleConns:        100,
            MaxIdleConnsPerHost: 10,
            IdleConnTimeout:     90 * time.Second,
            DialContext: (&net.Dialer{
                Timeout:   10 * time.Second,
                KeepAlive: 30 * time.Second,
            }).DialContext,
            TLSHandshakeTimeout:   10 * time.Second,
            ResponseHeaderTimeout: 10 * time.Second,
            ExpectContinueTimeout: 1 * time.Second,
        },
    }
}

This configuration provides:

Overall request timeout of 30 seconds
Connection pooling with up to 100 idle connections
Separate timeouts for dialing, TLS handshake, and response headers
Connection reuse for better performance

To disable automatic redirect following:

func noRedirectClient() *http.Client {
    return &http.Client{
        Timeout: 30 * time.Second,
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
}

Error Handling and Best Practices

Robust error handling separates production code from examples. Always check status codes and handle errors appropriately:

func robustRequest(url string) error {
    client := newProductionClient()
    
    resp, err := client.Get(url)
    if err != nil {
        // Network errors, timeouts, etc.
        return fmt.Errorf("request failed: %w", err)
    }
    defer resp.Body.Close()

    // Check status code
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        body, _ := io.ReadAll(resp.Body)
        return fmt.Errorf("unexpected status %d: %s", resp.StatusCode, body)
    }

    // Process successful response
    return nil
}

Implement retry logic with exponential backoff for transient failures:

func requestWithRetry(url string, maxRetries int) error {
    client := newProductionClient()
    
    for attempt := 0; attempt < maxRetries; attempt++ {
        resp, err := client.Get(url)
        if err != nil {
            // Network error - retry
            backoff := time.Duration(attempt*attempt) * time.Second
            time.Sleep(backoff)
            continue
        }
        defer resp.Body.Close()

        if resp.StatusCode == http.StatusTooManyRequests || 
           resp.StatusCode >= 500 {
            // Retryable status codes
            backoff := time.Duration(attempt*attempt) * time.Second
            time.Sleep(backoff)
            continue
        }

        if resp.StatusCode >= 400 {
            // Client error - don't retry
            return fmt.Errorf("client error: %d", resp.StatusCode)
        }

        // Success
        return nil
    }

    return fmt.Errorf("max retries exceeded")
}

Advanced Techniques

For request cancellation and deadline management, use context:

import "context"

func cancelableRequest(ctx context.Context, url string) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return err
    }

    client := newProductionClient()
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    // Process response...
    return nil
}

// Usage with timeout
func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    
    cancelableRequest(ctx, "https://api.example.com/slow")
}

For file uploads with multipart form data:

import (
    "mime/multipart"
    "os"
)

func uploadFile(filepath string) error {
    file, err := os.Open(filepath)
    if err != nil {
        return err
    }
    defer file.Close()

    body := &bytes.Buffer{}
    writer := multipart.NewWriter(body)
    
    part, err := writer.CreateFormFile("file", filepath)
    if err != nil {
        return err
    }
    
    io.Copy(part, file)
    writer.Close()

    req, err := http.NewRequest(
        http.MethodPost,
        "https://api.example.com/upload",
        body,
    )
    if err != nil {
        return err
    }
    
    req.Header.Set("Content-Type", writer.FormDataContentType())

    client := newProductionClient()
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    return nil
}

For streaming large responses without loading everything into memory:

func streamResponse(url string) error {
    client := newProductionClient()
    resp, err := client.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    // Process response in chunks
    buffer := make([]byte, 4096)
    for {
        n, err := resp.Body.Read(buffer)
        if n > 0 {
            // Process chunk
            fmt.Printf("Received %d bytes\n", n)
        }
        if err == io.EOF {
            break
        }
        if err != nil {
            return err
        }
    }

    return nil
}

Go’s HTTP client is powerful and flexible. Start with custom clients that have proper timeouts, implement robust error handling, and leverage contexts for cancellation. These patterns will serve you well in production systems where reliability matters.