Choosing a notebook kernel in Microsoft Fabric

Notebooks in Microsoft Fabric support three kernel types: Python, Spark, and T-SQL. The Spark kernel supports four languages—PySpark, SparkSQL, Scala, and SparkR—all backed by the same Spark compute. This guide focuses on the decision between the Python kernel and the Spark kernel, as these kernels are the most common choices for data engineering workloads. Both run in the same notebook experience but differ in compute model, scalability, engine capabilities, and Delta Lake compatibility. This guide provides a balanced evaluation to help you choose the right kernel—and avoid common misconceptions about cost and performance.

Understand your compute options

A common misconception is that the Python kernel is always cheaper than the Spark kernel for small data workloads. In reality, cost depends on how you configure compute for each kernel.

Python kernel compute

The Python kernel runs on a single-node machine that defaults to 2 vCores (1 CU) and can be configured to start at up to 64 vCores (32 CU). This environment has no distributed execution. The starter pool initializes in approximately 5 seconds, making it fast for interactive work.

Spark kernel compute

The Spark kernel uses Spark pools with several configuration options:

When using the starter pool, a single-node 8-vCore Spark session starts in approximately 5 seconds—comparable to the Python kernel. A single-node Spark cluster keeps costs similar to the Python kernel while providing access to all Spark-native capabilities, including the Native Execution Engine (NEE).

Performance by workload scale

Benchmarks comparing Fabric Spark with the Native Execution Engine against single-machine Python engines (like Pandas, DuckDB, or Polars) across end-to-end ELT workloads show clear patterns based on data scale:

Scaling beyond small data

Consider the rate of data growth when selecting an engine. Non-distributed Python engines work well for truly small data, but migrating your code when data exceeds their limits is costly. Starting with Spark on a single-node configuration lets you scale out seamlessly to a multi-node configuration without rewriting your data engineering pipelines.

Delta Lake compatibility

Delta Lake compatibility is a critical consideration when selecting an engine. Fabric Spark has native, full-featured Delta Lake support, while Python engines may have meaningful gaps:

Key implications:

• Newer Delta features: Support for newer Delta Lake features—including type widening, v2 checkpoints, liquid clustering, identity columns, Change Data Feed writes, and shallow clone reads—is inconsistent or absent across OSS Python engines. If your data pipeline depends on any of these features, use Fabric Spark. Treat Python engines as a complement to Spark for specific workloads (lightweight reads, local development, simple appends) rather than a general-purpose replacement.

• Deletion vectors: Deletion vectors are a best practice for Delta tables (enabled by default starting in Fabric Spark Runtime 2.0) as they greatly improve the performance of MERGE, UPDATE, and DELETE operations via a merge-on-read strategy. No Python engine (delta-rs, DuckDB, or Polars) supports writing deletion vectors. If you use any Python engine to write to tables that have deletion vectors enabled, you encounter compatibility errors.

• ACID guarantees: Not all Python engines provide native ACID guarantees. Delta-rs supports optimistic concurrency control (OCC) for merge, update, and delete operations. However, cross-engine pipelines where DuckDB or Polars performs the read and delta-rs performs the write require explicit version pinning on both sides to maintain read-write isolation. DuckDB INSERT is an append-only operation with no conflict detection.

• Checkpointing: Not all Python engines write checkpoints, and those engines that do default to every 100 commits rather than Spark's default of every 10 commits. DuckDB INSERT never writes checkpoints, causing the Delta transaction log to grow unbounded. Consider setting a lower checkpoint interval in delta-rs and Polars, and run periodic delta-rs maintenance for tables written by DuckDB.

• Row tracking: All Python engines can read from and write to tables with row tracking enabled, but the _metadata column containing row_id and row_commit_version isn't accessible outside Spark. Pipelines that rely on row_id for deduplication or CDC must use Spark to read.

• OPTIMIZE and VACUUM: Python engines rely on the deltalake library for compaction and vacuum. While delta-rs can be fast for these operations, this approach introduces extra dependency management and the operations aren't natively orchestrated the way they are in Spark. Tables written exclusively via Python engines accumulate small files and unbounded transaction logs without explicit maintenance.

• Shallow clones: Python engines don't support reading tables created via shallow clone due to absolute path resolution limitations. No Python engine supports creating shallow clones.

Engine maturity and Microsoft support

Fabric Spark support

Fabric Spark is Microsoft's own fork of open-source Apache Spark. Microsoft maintains and ships the runtime, meaning:

• Microsoft supports Spark and Delta Lake internals end-to-end, including the Native Execution Engine (NEE), which is built on Velox and Apache Gluten.
• You can open support tickets for Spark behavior, query plans, memory issues, and engine bugs.
• Performance improvements are continuously shipped as part of Fabric runtime updates—your existing code gets faster without code changes.

Python engine support

Microsoft doesn't maintain a fork of OSS Python engines like DuckDB or Polars. Support is limited to issues in the OneLake integrations shipped as part of the Fabric runtime, such as authentication or file system access. If you encounter a performance regression, an engine bug, or a broken API between library versions, you need to engage directly with the open-source communities for those libraries.

Operational maturity

Real-world experience building end-to-end ELT benchmarks with these engines highlights meaningful differences in operational maturity:

• Spark: Code written for one runtime version runs without modification on newer versions and performs faster due to continuous Microsoft engineering investment. The Spark UI and Fabric telemetry provide live monitoring with full visibility into active queries, execution plans, and historical job runs.
• DuckDB and Polars: API and behavior changes between versions can require code refactoring as engines mature and APIs evolve. Both engines lack live monitoring—when a job runs longer than expected, there's no equivalent to the Spark UI to understand what's happening. Authentication to OneLake can require version-specific workarounds.
• Composable data stack overhead: Using DuckDB or Polars for a full ELT workflow typically means stitching together multiple libraries (for example, DuckDB for data scan and transformation, delta-rs for writes and maintenance). Library compatibility needs to be maintained between the components and should be considered anytime the version of the library is upgraded beyond what ships in the runtime.

Decision guidance

Use the Python kernel when

• Your data is small—under approximately 1 GB compressed—and raw performance on single-machine engines matters most.
• You're building lightweight API orchestration, REST/gRPC integrations, or control-flow automation where distributed compute adds unnecessary overhead.
• You're doing rapid interactive exploration of small datasets where ad-hoc query latency is the priority.
• Your workload requires an older Python version than what ships in the current Fabric Spark runtime.
• You understand and accept the Delta Lake feature limitations of the Python engine you're using.

Use the Spark kernel when

• Your data is 1 GB or larger in compressed form, or you expect data to grow to that scale.
• You need full Delta Lake compatibility, including deletion vectors, column mapping, type widening, OPTIMIZE, VACUUM, and ACID guarantees.
• You require production-grade features such as environment variables, item-based library management, high-concurrency, and FAIR or first-in, first-out (FIFO) job scheduling.
• You need live monitoring and full operational visibility into running jobs.
• You rely on Spark-native APIs such as MLlib, Spark SQL, or Spark Streaming.
• You want Microsoft end-to-end support for your data processing engine.
• You want the ability to scale from single-node to multi-node compute without rewriting code.
• You need to author notebooks in PySpark, SparkSQL, Scala, or SparkR.

Key differences at a glance

Glossary

• ACID transactions: A set of properties (Atomicity, Consistency, Isolation, Durability) that guarantee database operations are processed reliably. Delta Lake implements ACID semantics using optimistic concurrency control and transaction logs.
• Optimistic Concurrency Control (OCC): A concurrency strategy where transactions proceed without locking, then verify at commit time that no conflicting changes occurred. Delta Lake uses OCC; cross-engine pipelines (for example, Polars reads followed by delta-rs writes) require explicit version pinning to maintain isolation.
• Auto compaction: A Delta Lake feature in the Fabric Spark runtime that automatically merges small files into larger ones after write operations, reducing file fragmentation without a separate OPTIMIZE step.
• Change Data Feed (CDF): A Delta Lake feature that records row-level changes (insert, update, delete) in a table, enabling incremental data processing and CDC pipelines. Only Fabric Spark supports writing Change Data Feed (CDF) metadata; OSS Python engines can read but not produce it.
• Column mapping: A Delta Lake feature that allows columns to be renamed or dropped without rewriting the underlying Parquet files. Supported by Fabric Spark; not supported by delta-rs or Polars.
• delta-rs: An open-source Rust implementation of the Delta Lake protocol with Python bindings (the deltalake PyPI package). Provides Delta read/write support in OSS Python environments but has narrower feature coverage than delta-spark which back the Fabric Spark runtime.
• Deletion vectors: A Delta Lake optimization that uses merge-on-read to reduce the amount of data rewritten during MERGE, UPDATE, and DELETE operations. Enabled by default in Fabric Spark Runtime 2.0; not supported for writes by any OSS Python engine.
• FAIR scheduling: A Spark scheduling policy that allocates cluster resources fairly across concurrent jobs, ensuring no single job monopolizes the cluster.
• FIFO scheduling: A Spark scheduling policy that executes jobs in First-In-First-Out order, giving priority to the first submitted job.
• Liquid Clustering: A Delta Lake feature that incrementally reorganizes data for optimal query performance without requiring explicit partitioning. Only supported by Fabric Spark.
• NEE (Native Execution Engine): A vectorized C++ query engine built on Velox and Apache Gluten that accelerates Fabric Spark workloads. NEE is available at no extra compute cost and requires no code changes.
• Row tracking: A Delta Lake feature that assigns a stable row_id and row_commit_version to each row via a _metadata column. All engines can read from and write to tables with row tracking enabled, but only Fabric Spark can access the _metadata column contents.
• Spark pool: A shared compute resource for running distributed Spark workloads. The starter pool provides prewarmed nodes for near-instant session start times (~5 seconds) with autoscaling enabled by default.
• V-Order: A Fabric write optimization that sorts and compresses Parquet data in a way that improves read performance for Power BI Direct Lake semantic models and other Fabric read paths.
• Intelligent cache: An intelligent disk cache in Fabric Spark that speeds up repeated reads of the same Delta table files by caching file data locally on the executor nodes.

Related content

• How to use Microsoft Fabric notebooks
• Use Python experience on Notebook
• Develop, execute, and manage Microsoft Fabric notebooks
• Introduction of Fabric NotebookUtils
• Native execution engine for Fabric Data Engineering
• Configure and manage starter pools in Fabric Spark
• Apache Spark compute for Data Engineering and Data Science
• Delta table maintenance in Microsoft Fabric
• Deletion vectors for Delta tables
• Concurrency control for Delta tables