Watermark

PySpark - Streaming Watermark and Late Data

Watermarks solve a fundamental problem in stream processing: when can you safely finalize an aggregation? In batch processing, you know when all data has arrived. In streaming, data arrives…