Eric Campbell 5 min read Python

Python - Read File (Complete Guide)

The `with` statement is the standard way to read files in Python. It automatically closes the file even if an exception occurs, preventing resource leaks.

Key Insights

  • Python offers multiple methods to read files: read(), readline(), readlines(), and iteration, each optimized for different use cases and memory constraints
  • The context manager (with statement) automatically handles file closing and exception handling, preventing resource leaks in production code
  • Binary mode, encoding specification, and buffering parameters are critical for handling non-text files, international character sets, and performance optimization

Basic File Reading with Context Managers

The with statement is the standard way to read files in Python. It automatically closes the file even if an exception occurs, preventing resource leaks.

# Read entire file as a single string
with open('data.txt', 'r') as file:
    content = file.read()
    print(content)

# File is automatically closed here, even if an error occurred

The open() function returns a file object. The 'r' mode opens the file for reading (this is the default mode). Without the context manager, you’d need explicit cleanup:

# Not recommended - manual file handling
file = open('data.txt', 'r')
try:
    content = file.read()
    print(content)
finally:
    file.close()  # Must explicitly close

Reading Files Line by Line

For large files, reading line by line prevents loading the entire file into memory. Python provides three approaches:

# Method 1: Iterate directly over file object (most memory efficient)
with open('large_log.txt', 'r') as file:
    for line in file:
        print(line.strip())  # strip() removes trailing newline

# Method 2: readline() for manual control
with open('data.txt', 'r') as file:
    line = file.readline()
    while line:
        print(line.strip())
        line = file.readline()

# Method 3: readlines() loads all lines into a list
with open('data.txt', 'r') as file:
    lines = file.readlines()
    for line in lines:
        print(line.strip())

The iteration method (Method 1) is preferred for large files because it uses an internal buffer and doesn’t load the entire file into memory. The readlines() method (Method 3) creates a list of all lines, consuming memory proportional to file size.

Reading Specific Amounts of Data

Control exactly how much data to read using size parameters:

# Read first 100 characters
with open('data.txt', 'r') as file:
    chunk = file.read(100)
    print(f"First 100 chars: {chunk}")

# Read file in chunks (useful for processing large files)
chunk_size = 1024  # 1KB chunks
with open('large_file.txt', 'r') as file:
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            break
        process_chunk(chunk)  # Your processing function

# Read specific number of lines
with open('data.txt', 'r') as file:
    first_five_lines = [file.readline() for _ in range(5)]
    print(first_five_lines)

Binary File Reading

Use binary mode ('rb') for non-text files like images, PDFs, or serialized data:

# Read binary file
with open('image.png', 'rb') as file:
    binary_data = file.read()
    print(f"File size: {len(binary_data)} bytes")
    print(f"First 10 bytes: {binary_data[:10]}")

# Read binary file in chunks
with open('video.mp4', 'rb') as file:
    chunk = file.read(4096)  # Read 4KB
    while chunk:
        # Process binary chunk
        chunk = file.read(4096)

# Combine with struct for binary data parsing
import struct

with open('data.bin', 'rb') as file:
    # Read 4 bytes as an integer
    int_data = struct.unpack('i', file.read(4))[0]
    # Read 8 bytes as a double
    float_data = struct.unpack('d', file.read(8))[0]
    print(f"Integer: {int_data}, Float: {float_data}")

Handling File Encodings

Specify encoding explicitly to handle international characters correctly:

# Read UTF-8 encoded file (recommended to be explicit)
with open('unicode_data.txt', 'r', encoding='utf-8') as file:
    content = file.read()
    print(content)

# Read file with different encoding
with open('legacy_data.txt', 'r', encoding='latin-1') as file:
    content = file.read()

# Handle encoding errors gracefully
with open('mixed_encoding.txt', 'r', encoding='utf-8', errors='ignore') as file:
    content = file.read()  # Skips invalid characters

# Replace invalid characters with a placeholder
with open('data.txt', 'r', encoding='utf-8', errors='replace') as file:
    content = file.read()  # Invalid chars become �

Common encoding values: 'utf-8', 'ascii', 'latin-1', 'cp1252' (Windows), 'utf-16'.

File Reading with Path Objects

Use pathlib for modern, object-oriented file path handling:

from pathlib import Path

# Read using Path object
file_path = Path('data/config.txt')
content = file_path.read_text(encoding='utf-8')
print(content)

# Read binary file
binary_content = file_path.read_bytes()

# Check if file exists before reading
if file_path.exists():
    content = file_path.read_text()
else:
    print(f"File not found: {file_path}")

# Iterate over lines using Path
for line in file_path.open('r'):
    print(line.strip())

Reading CSV and JSON Files

Python’s standard library provides specialized readers for structured data:

import csv
import json

# Read CSV file
with open('data.csv', 'r', newline='', encoding='utf-8') as file:
    csv_reader = csv.reader(file)
    headers = next(csv_reader)  # First row as headers
    for row in csv_reader:
        print(row)

# Read CSV as dictionaries
with open('data.csv', 'r', newline='', encoding='utf-8') as file:
    csv_reader = csv.DictReader(file)
    for row in csv_reader:
        print(row['column_name'])  # Access by column name

# Read JSON file
with open('config.json', 'r', encoding='utf-8') as file:
    data = json.load(file)
    print(data['key'])

# Read JSON lines format (one JSON object per line)
with open('data.jsonl', 'r', encoding='utf-8') as file:
    for line in file:
        obj = json.loads(line)
        print(obj)

Error Handling and Edge Cases

Robust file reading requires handling common error scenarios:

from pathlib import Path

def safe_read_file(filepath, encoding='utf-8'):
    """Read file with comprehensive error handling."""
    try:
        path = Path(filepath)
        
        if not path.exists():
            raise FileNotFoundError(f"File not found: {filepath}")
        
        if not path.is_file():
            raise ValueError(f"Path is not a file: {filepath}")
        
        # Check file size before reading
        file_size = path.stat().st_size
        if file_size > 100 * 1024 * 1024:  # 100MB
            raise ValueError(f"File too large: {file_size} bytes")
        
        return path.read_text(encoding=encoding)
    
    except PermissionError:
        print(f"Permission denied: {filepath}")
        return None
    except UnicodeDecodeError as e:
        print(f"Encoding error: {e}")
        return None
    except Exception as e:
        print(f"Unexpected error: {e}")
        return None

# Usage
content = safe_read_file('data.txt')
if content:
    print(content)

Performance Optimization

Optimize file reading for large files and high-performance scenarios:

# Use buffering parameter for I/O optimization
with open('large_file.txt', 'r', buffering=8192) as file:
    # 8KB buffer size
    content = file.read()

# Memory-mapped files for very large files
import mmap

with open('huge_file.txt', 'r+b') as file:
    with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mmapped:
        # Access file as if it were in memory
        content = mmapped.read(1000)
        print(content)

# Generator for memory-efficient processing
def read_large_file(filepath, chunk_size=8192):
    """Generator that yields chunks of a file."""
    with open(filepath, 'r') as file:
        while True:
            chunk = file.read(chunk_size)
            if not chunk:
                break
            yield chunk

# Process file without loading it all into memory
for chunk in read_large_file('large_log.txt'):
    # Process each chunk
    word_count = len(chunk.split())
    print(f"Chunk word count: {word_count}")

Reading from Multiple Files

Handle multiple file operations efficiently:

from pathlib import Path
import glob

# Read all text files in a directory
directory = Path('data')
for filepath in directory.glob('*.txt'):
    content = filepath.read_text()
    print(f"File: {filepath.name}, Size: {len(content)}")

# Read multiple files using glob
for filepath in glob.glob('logs/*.log'):
    with open(filepath, 'r') as file:
        lines = file.readlines()
        print(f"{filepath}: {len(lines)} lines")

# Combine multiple files
def concatenate_files(file_list, output_file):
    """Concatenate multiple files into one."""
    with open(output_file, 'w') as outfile:
        for filepath in file_list:
            with open(filepath, 'r') as infile:
                outfile.write(infile.read())
                outfile.write('\n')  # Separator

concatenate_files(['file1.txt', 'file2.txt'], 'combined.txt')

File reading is fundamental to Python programming. Choose the appropriate method based on file size, format, and memory constraints. Always use context managers, specify encoding explicitly, and implement proper error handling for production code.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.