Eric Campbell 5 min read Python

PySpark - Sort in Descending Order

Sorting data in descending order is one of the most common operations in data analysis. Whether you're identifying top-performing sales representatives, analyzing the most recent transactions, or...

Key Insights

  • PySpark offers two primary methods for descending sorts: orderBy() with desc() and sort() with ascending=False, with orderBy() being the more flexible and widely-used approach
  • Null value handling in descending sorts requires explicit control using desc_nulls_first() or desc_nulls_last() to avoid unexpected data positioning in your results
  • For large datasets, use sortWithinPartitions() instead of global orderBy() when you only need sorted data within partitions, significantly reducing shuffle operations and improving performance

Introduction

Sorting data in descending order is one of the most common operations in data analysis. Whether you’re identifying top-performing sales representatives, analyzing the most recent transactions, or building leaderboards, you’ll constantly need to arrange data from highest to lowest. PySpark provides multiple methods to accomplish descending sorts, each with specific use cases and performance characteristics.

Unlike pandas where you might use sort_values(ascending=False), PySpark’s distributed nature requires understanding how sorting operations trigger expensive shuffle operations across your cluster. This article covers the practical approaches to descending sorts in PySpark, from basic single-column operations to advanced null handling and performance optimization techniques.

Basic Descending Sort with orderBy()

The orderBy() method combined with the desc() function is the standard approach for descending sorts in PySpark. This method returns a new DataFrame with rows sorted by the specified columns.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, desc

spark = SparkSession.builder.appName("DescendingSort").getOrCreate()

# Sample sales data
data = [
    ("Alice", "Electronics", 15000),
    ("Bob", "Clothing", 8000),
    ("Charlie", "Electronics", 22000),
    ("Diana", "Furniture", 12000),
    ("Eve", "Clothing", 18000)
]

df = spark.createDataFrame(data, ["name", "department", "revenue"])

# Sort by revenue in descending order
sorted_df = df.orderBy(col("revenue").desc())
sorted_df.show()

# Output:
# +-------+------------+-------+
# |   name|  department|revenue|
# +-------+------------+-------+
# |Charlie|Electronics |  22000|
# |    Eve|    Clothing|  18000|
# |  Alice|Electronics |  15000|
# |  Diana|   Furniture|  12000|
# |    Bob|    Clothing|   8000|
# +-------+------------+-------+

You can also use the string-based syntax with desc() directly:

from pyspark.sql.functions import desc

sorted_df = df.orderBy(desc("revenue"))

Both approaches produce identical results. The col() syntax is generally preferred when you’re performing transformations on the column, while the string syntax works well for simple sorting operations.

Multiple Column Sorting

Real-world scenarios often require sorting by multiple columns with different sort orders. PySpark allows you to chain multiple sort specifications in a single orderBy() call.

# Extended employee dataset
employee_data = [
    ("Alice", "Engineering", 95000),
    ("Bob", "Engineering", 87000),
    ("Charlie", "Sales", 75000),
    ("Diana", "Sales", 82000),
    ("Eve", "Engineering", 92000),
    ("Frank", "Marketing", 68000),
    ("Grace", "Marketing", 71000)
]

employees_df = spark.createDataFrame(
    employee_data, 
    ["name", "department", "salary"]
)

# Sort by department (ascending) then salary (descending)
result = employees_df.orderBy(
    col("department").asc(),
    col("salary").desc()
)

result.show()

# Output:
# +-------+------------+------+
# |   name|  department|salary|
# +-------+------------+------+
# |  Alice| Engineering| 95000|
# |    Eve| Engineering| 92000|
# |    Bob| Engineering| 87000|
# |  Grace|   Marketing| 71000|
# |  Frank|   Marketing| 68000|
# |  Diana|       Sales| 82000|
# |Charlie|       Sales| 75000|
# +-------+------------+------+

This pattern is extremely useful for creating grouped rankings or organizing hierarchical data. Each department appears together, with employees sorted by salary from highest to lowest within their department.

Alternative Syntax with sort()

PySpark provides the sort() method as an alias to orderBy(). While functionally equivalent, sort() offers a slightly different syntax that some developers prefer for simple cases.

# Using sort() with ascending parameter
df_sorted_1 = df.sort("revenue", ascending=False)

# Using orderBy() with desc()
df_sorted_2 = df.orderBy(desc("revenue"))

# Both produce identical results
df_sorted_1.show()
df_sorted_2.show()

For multiple columns with mixed sort orders, sort() accepts a list of booleans:

# Sort by department (asc) and salary (desc) using sort()
employees_df.sort(["department", "salary"], ascending=[True, False]).show()

While sort() works well for simple cases, orderBy() provides more explicit control and is generally preferred in production code for its clarity. The desc() function makes the intent immediately obvious, whereas ascending=False requires mental translation.

Handling Null Values

Null handling in descending sorts can produce unexpected results if not explicitly controlled. By default, PySpark places nulls at the end for ascending sorts and at the beginning for descending sorts (this behavior can vary by Spark version and configuration).

# Data with null values
data_with_nulls = [
    ("Alice", 95),
    ("Bob", None),
    ("Charlie", 88),
    ("Diana", None),
    ("Eve", 92)
]

scores_df = spark.createDataFrame(data_with_nulls, ["name", "score"])

# Default descending sort - nulls appear first
scores_df.orderBy(col("score").desc()).show()

# Output:
# +-------+-----+
# |   name|score|
# +-------+-----+
# |    Bob| null|
# |  Diana| null|
# |  Alice|   95|
# |    Eve|   92|
# |Charlie|   88|
# +-------+-----+

# Force nulls to appear last
scores_df.orderBy(col("score").desc_nulls_last()).show()

# Output:
# +-------+-----+
# |   name|score|
# +-------+-----+
# |  Alice|   95|
# |    Eve|   92|
# |Charlie|   88|
# |    Bob| null|
# |  Diana| null|
# +-------+-----+

The null handling functions available are:

  • asc_nulls_first() - Ascending with nulls first
  • asc_nulls_last() - Ascending with nulls last
  • desc_nulls_first() - Descending with nulls first
  • desc_nulls_last() - Descending with nulls last

Always use explicit null handling in production code to ensure consistent behavior across different Spark environments.

Performance Considerations

Sorting in PySpark triggers a full shuffle operation, moving data across the cluster to ensure global ordering. This is expensive. Understanding when and how to optimize sorting operations is critical for production workloads.

# Global sort - expensive, shuffles all data
df.orderBy(desc("revenue")).show()

# Sort within partitions - much faster, no global shuffle
df.sortWithinPartitions(desc("revenue")).show()

Use sortWithinPartitions() when you don’t need global ordering but want sorted data within each partition. This is common when:

  • You’re writing partitioned data to storage and want each partition file sorted
  • You’re performing window operations that only require partition-level ordering
  • You’re doing downstream processing that benefits from local ordering
# Example: Sort within partitions before writing
(df
 .repartition("department")
 .sortWithinPartitions(desc("salary"))
 .write
 .partitionBy("department")
 .parquet("output/employees"))

For very large datasets, consider:

  • Limiting results early: Use limit() after sorting only when you need top-N results
  • Partition pruning: Filter data before sorting to reduce shuffle volume
  • Appropriate partition counts: Too few partitions create memory pressure; too many increase overhead
# Efficient top-10 query
top_customers = (df
    .orderBy(desc("total_purchases"))
    .limit(10)
    .collect())

# Better: Filter before sorting when possible
high_value_sorted = (df
    .filter(col("total_purchases") > 10000)
    .orderBy(desc("total_purchases")))

Practical Use Cases

Top-N Queries: Finding the highest or most recent values is a classic use case.

# Top 10 customers by purchase amount
top_10_customers = (customers_df
    .orderBy(desc("total_spent"))
    .limit(10))

top_10_customers.show()

Time-Series Analysis: Working with temporal data almost always requires descending date sorts.

from pyspark.sql.functions import to_date

# Most recent transactions first
recent_transactions = (transactions_df
    .withColumn("date", to_date("timestamp"))
    .orderBy(desc("date"), desc("timestamp"))
    .limit(100))

Ranking and Leaderboards: Combine sorting with window functions for sophisticated ranking.

from pyspark.sql.window import Window
from pyspark.sql.functions import rank

# Rank employees by salary within each department
window_spec = Window.partitionBy("department").orderBy(desc("salary"))

ranked_employees = employees_df.withColumn(
    "rank",
    rank().over(window_spec)
)

ranked_employees.show()

# Output:
# +-------+------------+------+----+
# |   name|  department|salary|rank|
# +-------+------------+------+----+
# |  Alice| Engineering| 95000|   1|
# |    Eve| Engineering| 92000|   2|
# |    Bob| Engineering| 87000|   3|
# |  Grace|   Marketing| 71000|   1|
# |  Frank|   Marketing| 68000|   2|
# |  Diana|       Sales| 82000|   1|
# |Charlie|       Sales| 75000|   2|
# +-------+------------+------+----+

Data Quality Checks: Sort by timestamp descending to identify the most recent issues or anomalies.

# Find most recent data quality issues
quality_issues = (data_quality_log
    .filter(col("status") == "FAILED")
    .orderBy(desc("check_timestamp"))
    .select("table_name", "check_name", "check_timestamp", "error_message"))

Descending sorts are fundamental to PySpark data analysis. Master the orderBy() method with desc(), understand null handling, and know when to use sortWithinPartitions() for performance. These techniques will serve you in virtually every PySpark application you build.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.