Polars Lazy Streaming: When to Use collect or sink_parquet

Consider a hypothetical developer working from one source table. An analyst wants a small category summary in Python; a dataset pipeline needs selected records saved on disk. Same input, different jobs. The destination should drive the execution choice.

For Polars lazy streaming, start with a file-backed scan and inspect the query. Use collect when the consumer needs a deliberately bounded DataFrame, or sink_parquet when the result belongs on disk. The sources and sinks guide explains how reading and processing connect to writing. Streaming does not guarantee a memory ceiling.

This guide follows the documentation. The local-file example is hypothetical and unexecuted, and there are no benchmark results.

Start with a scan, not an already loaded DataFrame

What scan_parquet and scan_csv let the optimizer see

A file-backed LazyFrame describes work before the result is materialized. With scan_parquet or scan_csv, the optimizer can move eligible filters and column selections toward the reader. The lazy API guide illustrates predicate and projection pushdown with a filtered aggregation.

Projection keeps the columns a query needs. Predicate pushdown moves suitable row conditions closer to the source. Neither allows you to change what the consumer receives. If a downstream process needs an identifier, removing it changes the data contract. So does dropping missing values that carry meaning.

Not every expression can move into the reader. CSV and Parquet need not deliver identical I/O savings. Inspect the plan, then measure I/O before claiming savings.

Why read().lazy() cannot undo an eager read

By the time read_parquet(...).lazy() reaches .lazy(), the eager load has happened. Later transformations can be lazy; the initial read cannot be undone. The sources and sinks documentation recommends scan_* over read_* for intended lazy file workflows.

That does not make an eager DataFrame a mistake. The lazy API guide describes its value during exploration, where intermediate results help you understand the data. Choose it for that reason rather than rewriting useful exploratory work just to call it streaming.

The example here assumes a hypothetical local events.parquet file. Its record_id and category columns are String; value is Float64. No schema has been inspected. The two possible consumers need either selected row-level records or a category summary derived from them.

Read the logical plan and the streaming physical plan separately

Use explain to inspect optimization

Compare query.explain(optimized=False) with query.explain(optimized=True). The query-plan guide shows a filter moving into a scan after optimization. These views describe planned work. They do not measure runtime or peak memory, and they do not establish correctness.

Check that the expected projection and selection appear. If either is missing, investigate before interpreting resource measurements. Then check the filter itself against the business rule: a filter in the right place can still select the wrong rows.

Use the physical graph to investigate memory-sensitive work

The streaming guide documents query.show_graph(plan_stage="physical", engine="streaming") and a legend for potentially memory-intensive operations. Rendering requires Graphviz available on PATH, as the query-plan guide explains. Without it, textual explain is still useful, but does not provide the same physical view.

Streaming works in batches. Some operations are inherently non-streaming; others do not yet have a streaming implementation and fall back to in-memory execution. Use the physical graph to find those concerns. It is not a peak-RSS guarantee.

Check version-specific defaults too. The current stable collect reference says engine="auto" follows Config.set_engine_affinity or POLARS_ENGINE_AFFINITY, otherwise defaulting to streaming. Older installations can differ. Explicit engine="streaming" states your intent, but does not prove every operation streams. Record the installed version and configuration before relying on the current documentation.

Choose collect or sink_parquet by the output you need

Ask what the next consumer requires. Source-file size alone cannot answer that.

A bounded in-memory result

collect materializes a LazyFrame into a DataFrame. That returned object needs memory even if upstream work streams. Aggregation alone does not make it small: the number of categories determines how many summary rows remain.

A persisted result without collecting the whole output

The sink_parquet reference describes streaming evaluation and Parquet writing without first requiring the caller to obtain the full final DataFrame. The default lazy=False executes the sink. With lazy=True, execution waits for collection; the sink's engine and optimizations arguments do not apply when building that deferred plan. Choose execution settings when collecting it. The deferred option is marked unstable.

Keep the two endpoints' fallback rules separate. The current sink reference follows engine affinity for automatic selection, otherwise streaming, and describes selected-engine fallback to streaming. The collect reference describes selected-engine fallback to in-memory. Operation-level limits still apply. Compression and row-group settings do not establish a fixed memory budget.

The historical community question about DuckDB and Polars raises PostgreSQL change-data capture. Read it as a terminology question, not verified architecture advice. Batch streaming here does not supply an always-on service with offsets, event-time handling or delivery guarantees.

The Result-First Polars Playbook

Define the result contract before choosing execution

The Result-First Polars Playbook is an original editorial checklist, not a library or validated deployment. Start by naming the consumer. Agree on the required columns and types, then settle filters, null handling and any ordering requirement. Those decisions give you something concrete to validate after execution.

The diagram shows the destination choice conceptually. It is not an observed Polars physical plan.

Scan, narrow and inspect

This illustration assumes Polars is already available and uses the current stable documented API. It has not been run. No setup or dataset contents have been checked, and no printed output is claimed. The filter drops null value rows only to illustrate a policy; your dataset may require a different one.

import polars as pl

rows = (
    pl.scan_parquet("events.parquet")
    .select("record_id", "category", "value")
    .filter(pl.col("value").is_not_null())
)

planned_schema = rows.collect_schema()
unoptimized_plan = rows.explain(optimized=False)
optimized_plan = rows.explain(optimized=True)

collect_schema resolves the planned schema. This may read disk metadata or make remote requests, so schema resolution is not necessarily free. It also does not validate every row against domain rules. Compare names and types with the contract, then inspect the logical plan and, where tooling permits, the physical graph. The schema guide explains why known schemas matter during lazy planning.

Choose one of the following endpoints for the required output. They are unexecuted alternatives, not a sequence. The summary branch assumes you have independently established a manageable category count. You must also decide what a null category means.

Alternative A produces the category summary needed in Python.

summary = rows.group_by("category").agg(
    pl.col("value").mean().alias("mean_value")
)
result = summary.collect(engine="streaming")

Alternative B writes the selected records needed on disk.

rows.sink_parquet("events-selected-candidate.parquet", engine="streaming")

The collect API and sink API support these endpoint descriptions. Comparing their timings would compare different tasks: one returns a category summary, while the other exports row-level records. That is not an engine benchmark.

Propose a comparable check, then validate the destination

A useful comparison holds the output constant. Use the same input snapshot and Polars version, with comparable settings, machine and storage conditions. Optijara's guide to keeping benchmark protocols attached to results discusses that discipline in another domain. Keep the proposed protocol separate from the results you eventually record.

None of these measurements has been performed here. For a persisted candidate, check schema and row counts without loading the entire output merely to validate it. Compare values against agreed tolerances; compare order only where the contract requires it.

If execution fails, treat partial files as unvalidated. Investigate the error and the operations involved. Use a fresh candidate destination for the next attempt, and validate it before any approved handoff. A file's presence is not an acceptance check.

What teams get wrong about lazy streaming

The tempting shortcut is to treat lazy planning and streaming execution as one decision. They are separate. Adding .lazy() after a read cannot change that completed load. Choosing streaming does not remove the final DataFrame returned by collect.

Inspection can also look more conclusive than it is. A plan explains intended work; a resolved schema gives names and types. Neither answers whether memory use is acceptable or whether the rows retain their intended meaning. Keep those checks separate from inspection.

A performance comparison becomes hard to interpret if you change the filters or projection between trials. Changing the snapshot or destination creates the same problem. Record the changed task rather than attributing every difference to the engine. And keep continuous-ingestion requirements out of a batch endpoint decision.

For rows extracted from PDFs, check that headers and units still belong to the correct values, and retain source references before tabular transformation. Optijara's guide to preserving extracted PDF table meaning and source references covers that upstream work. It does not establish a Polars integration. A different sink cannot recover meaning lost during extraction.

Limits to check before changing a data pipeline

Memory, version and storage trade-offs

Input size is only part of the memory question. Account for intermediate state and the final result, and check which operations can stream. The streaming and endpoint documentation explain those limits. Adoption also costs implementation effort and adds storage I/O or operational complexity. No runtime or cost improvement is promised here.

The PythonSpeed memory investigation is a concrete historical example of workload-specific analysis. Its measurements apply to its dataset, versions and platform. They are neither universal current results nor Optijara tests.

Before using defaults or optional parameters, check the documentation for your installed version. The sink reference marks several options unstable or experimental. If you want to understand a change, avoid adjusting compression and ordering at the same time as execution settings.

Data meaning and privacy still need review

During preparation, preserve identifiers and split boundaries. Make null handling explicit, and retain ordering where the consumer requires it. A valid Parquet file does not prove that the intended examples survived. The adjacent guide to preserving dataset selection and temporal identity explains those checks for time-dependent datasets; it does not claim a Polars integration.

Use authorized input snapshots. Decide who can access candidate outputs and how long to retain them. A local destination does not settle privacy obligations, even if the query executes successfully.

Choose the destination before you collect

Begin with the consumer's required result. Use a scan for intended file-backed lazy work, then inspect logical and physical plans for their different purposes. Validate the chosen endpoint and retain actual measurement evidence before changing an operational pipeline. For help planning an AI workflow, Optijara offers AI advisory.