Encoding: UTF-8, Base64, and URL Encoding

Key Insights

• UTF-8 is character encoding (how text becomes bytes), while Base64 and URL encoding are data encoding schemes (how bytes become safe ASCII strings for specific contexts)
• Base64 adds 33% overhead to your data size—use it when you need binary data in text-only contexts, not as a general-purpose encoding
• Double-encoding is the most common encoding bug in web applications; always know whether your framework auto-encodes before manually encoding

Introduction: Why Encoding Matters

Every time you send an emoji in a message, embed an image in an email, or pass a search query through a URL, encoding is happening behind the scenes. Yet most developers treat encoding as an afterthought—until they’re debugging garbled text or broken API calls at 2 AM.

Encoding is simply data transformation. We encode data to make it compatible with systems that have constraints on what bytes or characters they can handle. The confusion starts when developers conflate two distinct concepts:

Character encoding (like UTF-8) defines how human-readable text maps to bytes. It answers: “How do I store the letter ‘é’ or the emoji ‘🔥’ as binary data?”

Data encoding schemes (like Base64 and URL encoding) transform arbitrary bytes into a restricted set of safe ASCII characters. They answer: “How do I transmit binary data through a channel that only accepts certain characters?”

Understanding this distinction prevents most encoding headaches.

UTF-8: The Universal Character Encoding

UTF-8 is the dominant character encoding on the web, used by over 98% of websites. It encodes Unicode code points using a variable-width scheme of 1 to 4 bytes.

The brilliance of UTF-8 lies in its backward compatibility with ASCII. Any valid ASCII text is also valid UTF-8—the first 128 characters use identical single-byte representations. This made adoption painless for systems built on ASCII.

Here’s how the variable-width encoding works:

Let’s see this in practice:

# ASCII character: 1 byte
ascii_char = 'A'
print(ascii_char.encode('utf-8'))  # b'A'
print(list(ascii_char.encode('utf-8')))  # [65]

# Accented character: 2 bytes
accented = 'é'
print(accented.encode('utf-8'))  # b'\xc3\xa9'
print(list(accented.encode('utf-8')))  # [195, 169]

# CJK character: 3 bytes
chinese = '中'
print(chinese.encode('utf-8'))  # b'\xe4\xb8\xad'
print(list(chinese.encode('utf-8')))  # [228, 184, 173]

# Emoji: 4 bytes
emoji = '🔥'
print(emoji.encode('utf-8'))  # b'\xf0\x9f\x94\xa5'
print(list(emoji.encode('utf-8')))  # [240, 159, 148, 165]

In JavaScript, you can inspect UTF-8 bytes using TextEncoder:

const encoder = new TextEncoder();

console.log(encoder.encode('A'));      // Uint8Array [65]
console.log(encoder.encode('é'));      // Uint8Array [195, 169]
console.log(encoder.encode('中'));     // Uint8Array [228, 184, 173]
console.log(encoder.encode('🔥'));     // Uint8Array [240, 159, 148, 165]

The key insight: string length doesn’t equal byte length. The string “Hello 🔥” has 7 characters but requires 10 bytes in UTF-8. This matters for storage calculations and protocol limits.

Base64: Binary-to-Text Encoding

Base64 converts arbitrary binary data into a string using 64 ASCII characters: A-Z, a-z, 0-9, +, and /. The = character pads the output to ensure the length is a multiple of 4.

Why do we need this? Many protocols and formats—email (SMTP), JSON, HTML attributes—are designed for text, not binary. Base64 bridges this gap.

import base64

# Encode binary data
binary_data = b'\x00\x01\x02\xff\xfe\xfd'
encoded = base64.b64encode(binary_data)
print(encoded)  # b'AAEC//79'

# Decode back to binary
decoded = base64.b64decode(encoded)
print(decoded)  # b'\x00\x01\x02\xff\xfe\xfd'

The most common use case is embedding images directly in HTML or CSS:

import base64

with open('icon.png', 'rb') as f:
    image_data = f.read()
    
encoded_image = base64.b64encode(image_data).decode('ascii')
data_uri = f'data:image/png;base64,{encoded_image}'

# Use in HTML: <img src="{data_uri}">

// In the browser, convert a file to Base64
async function fileToBase64(file) {
    return new Promise((resolve, reject) => {
        const reader = new FileReader();
        reader.onload = () => {
            const base64 = reader.result.split(',')[1];
            resolve(base64);
        };
        reader.onerror = reject;
        reader.readAsDataURL(file);
    });
}

// Decode Base64 in Node.js
const decoded = Buffer.from('SGVsbG8gV29ybGQ=', 'base64');
console.log(decoded.toString());  // "Hello World"

The 33% overhead is real. Base64 uses 4 characters to represent every 3 bytes of input. For a 1 MB image, the Base64 version is approximately 1.33 MB. Don’t embed large files as Data URIs—use proper asset hosting.

import base64

original = b'ABC'  # 3 bytes
encoded = base64.b64encode(original)
print(f'Original: {len(original)} bytes')   # 3 bytes
print(f'Encoded: {len(encoded)} bytes')     # 4 bytes
print(f'Overhead: {len(encoded) / len(original):.0%}')  # 133%

URL Encoding (Percent-Encoding)

URLs have a restricted character set. RFC 3986 defines unreserved characters that need no encoding: A-Z a-z 0-9 - _ . ~. Everything else—including spaces, query delimiters, and Unicode—must be percent-encoded.

Percent-encoding converts each byte to %XX where XX is the hexadecimal value:

from urllib.parse import quote, quote_plus, unquote

# Basic encoding
text = 'hello world'
print(quote(text))       # 'hello%20world'
print(quote_plus(text))  # 'hello+world' (form encoding)

# Unicode characters get UTF-8 encoded first, then percent-encoded
unicode_text = 'café'
print(quote(unicode_text))  # 'caf%C3%A9'

# Special characters
special = 'price=100&tax=20%'
print(quote(special, safe=''))  # 'price%3D100%26tax%3D20%25'

JavaScript provides two functions with critical differences:

const url = 'https://example.com/search?q=hello world&category=books';

// encodeURI: Encodes a complete URI, preserves reserved characters
console.log(encodeURI(url));
// 'https://example.com/search?q=hello%20world&category=books'

// encodeURIComponent: Encodes a URI component, encodes reserved characters
const query = 'hello world&category=books';
console.log(encodeURIComponent(query));
// 'hello%20world%26category%3Dbooks'

Use encodeURIComponent for query parameter values. Use encodeURI only when encoding a complete URL where you want to preserve the structure.

// Correct: encode each parameter value separately
const searchTerm = 'C++ programming';
const url = `https://api.example.com/search?q=${encodeURIComponent(searchTerm)}`;
// 'https://api.example.com/search?q=C%2B%2B%20programming'

// Wrong: using encodeURI on user input
const badUrl = `https://api.example.com/search?q=${encodeURI(searchTerm)}`;
// 'https://api.example.com/search?q=C++%20programming' (+ not encoded!)

Common Pitfalls and Debugging

Double-encoding is the most frequent encoding bug. It happens when you encode data that’s already encoded:

from urllib.parse import quote, unquote

original = 'hello world'
encoded_once = quote(original)
print(encoded_once)  # 'hello%20world'

# Bug: encoding again
encoded_twice = quote(encoded_once)
print(encoded_twice)  # 'hello%2520world' (%25 is the encoding of %)

# Now decoding once gives you the wrong result
print(unquote(encoded_twice))  # 'hello%20world' (not the original!)

Mojibake occurs when text is decoded with the wrong character encoding:

# Text encoded as UTF-8
utf8_bytes = 'café'.encode('utf-8')  # b'caf\xc3\xa9'

# Incorrectly decoded as Latin-1
wrong = utf8_bytes.decode('latin-1')
print(wrong)  # 'café' (mojibake!)

# Correct decoding
correct = utf8_bytes.decode('utf-8')
print(correct)  # 'café'

Base64 URL-safe variants matter for JWTs and URLs. Standard Base64 uses + and /, which have special meaning in URLs:

import base64

data = b'\xfb\xff\xfe'

# Standard Base64
standard = base64.b64encode(data)
print(standard)  # b'+//+'

# URL-safe Base64 (replaces + with -, / with _)
urlsafe = base64.urlsafe_b64encode(data)
print(urlsafe)  # b'--_-'

Choosing the Right Encoding

Here’s a decision framework:

Use UTF-8 when storing or transmitting text. It’s the default for HTML5, JSON, and most modern APIs. Never use UTF-16 or UTF-32 for web content.

Use Base64 when you need to embed binary data in text-only contexts: JSON payloads, HTML/CSS, email bodies, or anywhere binary isn’t allowed. Don’t use it for “security” or obfuscation—it’s trivially reversible.

Use URL encoding when constructing URLs with dynamic values. Always encode user input in query parameters. Let your HTTP client library handle this when possible.

Quick Reference Summary

Encoding isn’t glamorous, but understanding it prevents entire categories of bugs. The next time you see garbled text or a broken URL, you’ll know exactly where to look.