Linux strace: System Call Tracing

Key Insights

System calls are the fundamental interface between user applications and the Linux kernel, and tracing them reveals exactly what your programs are doing under the hood—from file access to network operations.
strace adds significant overhead (often 100x slowdown) making it unsuitable for production profiling, but invaluable for debugging configuration issues, missing dependencies, and understanding opaque binaries.
Effective strace usage requires aggressive filtering with -e flags and understanding common syscall patterns—the raw output is verbose, but systematic filtering reveals actionable insights quickly.

Introduction to System Calls and strace

Every time your application reads a file, allocates memory, or sends data over the network, it makes a system call—a controlled transition from user space to kernel space where the actual work happens. Your program doesn’t directly access hardware or kernel resources; instead, it asks the kernel politely through this well-defined interface.

Understanding what system calls your application makes is critical for debugging. When something fails mysteriously—a configuration file isn’t found, a network connection times out, or permissions are denied—the system call trace tells you exactly what happened, not what you thought should happen.

strace is the standard Linux tool for tracing system calls. It uses the ptrace system call to intercept and record every syscall a process makes, along with arguments, return values, and errors. Let’s see it in action:

$ strace ls
execve("/usr/bin/ls", ["ls"], 0x7ffd9c8a3d40 /* 67 vars */) = 0
brk(NULL)                               = 0x55a9c8f3a000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc8e9a1a80) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8b5e9a1000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=67284, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 67284, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8b5e990000
close(3)                                = 0
...

Even a simple ls command makes dozens of syscalls. This is normal—and exactly what we need to see when debugging.

Basic strace Usage

The basic syntax is straightforward: strace [options] command [args]. The most useful flags for everyday work are:

Summary statistics with -c show you aggregate counts and timing:

$ strace -c python3 -c "print('hello')"
hello
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 22.45    0.000089          89         1           execve
 18.37    0.000073          14         5           mmap
 12.24    0.000049          12         4           openat
 10.20    0.000040          13         3           read
  8.16    0.000032          10         3           newfstatat
  7.14    0.000028          28         1           munmap
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000396          7        56         8 total

This gives you a high-level view of where time is spent without drowning in details.

Filtering specific syscalls with -e keeps output manageable:

$ strace -e open,openat,read cat /etc/hostname
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0"..., 832) = 832
openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
read(3, "myserver\n", 131072)           = 9
read(3, "", 131072)                     = 0
myserver
+++ exited with 0 +++

Output to file with -o prevents strace output from mixing with program output:

$ strace -o trace.log ./myapp
$ grep "openat.*ENOENT" trace.log
openat(AT_FDCWD, "/opt/myapp/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)

Attach to running processes with -p:

$ strace -p 1234
strace: Process 1234 attached
epoll_wait(5, [], 128, 1000)            = 0
epoll_wait(5, [{EPOLLIN, {u32=12, u64=12}}], 128, 1000) = 1
read(12, "GET / HTTP/1.1\r\n...", 8192) = 234

This is invaluable for debugging live applications without restarting them.

Interpreting strace Output

Each line of strace output follows a consistent format:

syscall_name(arg1, arg2, ...) = return_value [error]

Let’s break down a real example:

openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 3
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 2847
close(3)                                = 0

openat(AT_FDCWD, "/etc/passwd", O_RDONLY): Opens /etc/passwd for reading. AT_FDCWD means “relative to current directory.”
= 3: Returns file descriptor 3 (0, 1, 2 are stdin/stdout/stderr)
read(3, "root:x:0:0...", 4096): Reads up to 4096 bytes from fd 3
= 2847: Actually read 2847 bytes
close(3) = 0: Closes the file descriptor, returns 0 for success

When syscalls fail, you see error codes:

openat(AT_FDCWD, "/nonexistent", O_RDONLY) = -1 ENOENT (No such file or directory)

Common syscalls you’ll encounter:

File operations: openat, read, write, close, stat, access
Memory: mmap, munmap, brk
Process: fork, execve, wait4, clone
Network: socket, connect, send, recv
I/O multiplexing: select, poll, epoll_wait

Advanced Filtering and Analysis

Filtering by syscall category dramatically improves signal-to-noise ratio:

$ strace -e trace=file ls
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

Categories include: file, process, network, signal, ipc, desc (file descriptors), memory.

For network debugging:

$ strace -e trace=network curl -s https://example.com > /dev/null
socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(443), sin6_flowinfo=htonl(0),
        inet_pton(AF_INET6, "2606:2800:220:1:248:1893:25c8:1946", &sin6_addr),
        sin6_scope_id=0}, 28) = 0

Timing analysis reveals performance bottlenecks:

$ strace -T -e read,write cp large_file /tmp/
read(3, "..."..., 131072)               = 131072 <0.000015>
write(4, "..."..., 131072)              = 131072 <0.002847>
read(3, "..."..., 131072)               = 131072 <0.000012>
write(4, "..."..., 131072)              = 131072 <0.002791>

The -T flag shows time spent in each syscall. Here, writes take 200x longer than reads—possibly indicating slow disk I/O.

Use -r for relative timestamps:

$ strace -r -e openat cat file1 file2
     0.000000 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
     0.000234 openat(AT_FDCWD, "file1", O_RDONLY) = 3
     0.000156 openat(AT_FDCWD, "file2", O_RDONLY) = 3

Follow child processes with -f:

$ strace -f -e trace=process bash -c 'ls | wc -l'
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
      child_tidptr=0x7f8b5e9a2a10) = 12345
[pid 12345] execve("/usr/bin/ls", ["ls"], 0x55a9c8f3a000 /* 67 vars */) = 0

This is essential for debugging shell scripts or applications that spawn subprocesses.

Real-World Debugging Scenarios

Debugging missing configuration files:

Your application fails with a vague error. Where is it looking for config?

$ strace -e openat ./myapp 2>&1 | grep -i config
openat(AT_FDCWD, "/etc/myapp/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/user/.config/myapp/config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "./config.yml", O_RDONLY) = -1 ENOENT (No such file or directory)

Now you know exactly which paths it checks and in what order.

Investigating slow startup:

$ strace -c -e trace=file slow_starting_app
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 89.23    2.847123      284712        10           openat
  8.45    0.269847         123      2193      2180 stat
  2.32    0.074012          74      1000           access

The app is making 2193 stat calls, mostly failing (2180 errors). It’s probably searching for files in many locations. Time to optimize that search path.

Understanding third-party binaries:

What does this proprietary tool actually do?

$ strace -e trace=network mystery_binary
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(443), 
        sin_addr=inet_addr("192.0.2.1")}, 16) = 0
sendto(3, "POST /api/telemetry HTTP/1.1\r\n"..., 156, 0, NULL, 0) = 156

Ah, it’s sending telemetry to a remote server. Good to know.

Performance Considerations and Alternatives

strace has significant overhead—typically 100-200x slowdown. It stops the traced process at every syscall to record information. This makes it unsuitable for production performance profiling or high-frequency syscalls.

For production environments, consider:

perf for system-wide profiling with minimal overhead:

$ perf trace -p 1234 --duration 5000
     0.000 ( 0.003 ms): nginx/1234 epoll_wait(epfd: 6, maxevents: 512, timeout: 25000) ...
  1247.891 ( 0.012 ms): nginx/1234 accept4(fd: 7, upeer_sockaddr: 0x7ffd..., flags: SOCK_CLOEXEC) = 12

bpftrace for low-overhead custom tracing using eBPF:

$ bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[str(args->filename)] = count(); }'
Attaching 1 probe...
^C
@[/proc/sys/kernel/random/boot_id]: 42
@[/etc/resolv.conf]: 128
@[/var/log/app.log]: 1247

This counts openat calls by filename with negligible overhead, even in production.

ltrace traces library calls instead of syscalls:

$ ltrace -e malloc,free ./app
malloc(1024)                                     = 0x55a9c8f3a000
free(0x55a9c8f3a000)                             = <void>

Use strace for debugging and understanding behavior. Use perf or bpftrace for performance analysis in production.

Quick Reference

Common patterns:

# See what files a program accesses
strace -e trace=file command

# Debug network issues
strace -e trace=network command

# Find slow syscalls
strace -T command 2>&1 | grep '<[0-9]\.[0-9]'

# Attach to running process
strace -p PID

# Follow all child processes
strace -f command

# Save output for later analysis
strace -o output.txt command

# Count syscalls only
strace -c command

# Filter multiple syscalls
strace -e openat,read,write command

# See relative timestamps
strace -r command

Debugging checklist:

Start with -c for overview
Filter with -e trace=category to reduce noise
Add -T if you suspect performance issues
Use -f for multi-process applications
Save to file with -o for complex analysis

The key to effective strace usage is aggressive filtering. The raw output is overwhelming, but focused traces reveal exactly what your application is doing—no guessing required.