Spark SQL: from a query to an RDD of rows

SparkSession.sql(...) parses text into an unresolved LogicalPlan and wraps it in a Dataset via Dataset.ofRows. A Dataset is just a QueryExecution plus an Encoder; every DataFrame transformation produces a new logical plan. Nothing runs until an action like collect calls withAction, which forces the executed plan and collects rows.

SparkSession.scala L528-L559 — sql()

Dataset.scala L110-L138 — ofRows

Each phase is a lazy val: analyzed, optimizedPlan, sparkPlan, executedPlan, and finally toRdd. Accessing one forces the previous, so the whole pipeline runs on demand and the planning time of each phase is tracked separately.

analyzed       = analyzer.executeAndCheck(logical)
optimizedPlan  = optimizer.executeAndTrack(withCachedData)
sparkPlan      = createSparkPlan(planner, optimizedPlan)
executedPlan   = prepareForExecution(preparations, sparkPlan)
toRdd          = new SQLExecutionRDD(executedPlan.execute(), conf)

QueryExecution.scala L313-L394 — optimized, spark, executed, toRdd

Every plan, expression, and data type in Catalyst is a TreeNode: an immutable node with children. All transformations are expressed as transformDown/transformUp traversals that return a rewritten tree. A Rule is a single rewrite step (apply(tree): tree), and the RuleExecutor runs batches of rules until they reach a fixed point. This tiny abstraction is why adding an optimization is just writing a new rule.

TreeNode.scala L70-L75 — base node

Rule.scala L24-L35 — Rule

The parser produces a plan full of UnresolvedRelation and UnresolvedAttribute placeholders. The Analyzer (a RuleExecutor) consults the catalog to bind table and column references, resolve functions, add casts for type coercion, and validate the result. Its output is a fully typed, analyzed logical plan.

Analyzer.scala L300-L309 — class

Analyzer.scala L506-L609 — resolution batches

The Optimizer applies rule batches that make the plan cheaper without changing its meaning: predicate pushdown, column pruning, constant folding, filter combination, boolean simplification, join reordering, and subquery rewriting. These are heuristic (rule-based) rewrites; statistics-driven choices like join strategy happen during physical planning and Adaptive Query Execution.

Optimizer.scala L100-L291 — default rule batches

Physical planning applies a list of strategies that pattern-match logical operators and emit physical SparkPlan operators (the *Exec classes). JoinSelection decides between broadcast, sort-merge, and shuffle-hash joins; FileSourceStrategy turns a scan into a file read with pushed-down filters; BasicOperators maps the simple cases. The planner takes the first complete candidate.

SparkPlanner.scala L31-L57 — the strategy list

SparkStrategies.scala L906-L949 — BasicOperators

A SparkPlan is a physical operator. Its doExecute() returns an RDD[InternalRow] — the bridge back to the RDD engine. InternalRow (and its binary form UnsafeRow) is Tungsten's compact, off-heap-friendly row layout that flows through these RDDs, avoiding Java object overhead.

SparkPlan.scala L197-L202 — execute

SparkPlan.scala L338-L343 — doExecute returns RDD[InternalRow]

A naive operator tree calls next() through many virtual method calls per row (the "volcano" model). CollapseCodegenStages, a preparation rule, fuses a chain of codegen-capable operators (scan → filter → project) into a single generated Java class with one tight loop. WholeStageCodegenExec.doExecute() compiles that source at runtime and runs it, falling back to normal execution if compilation fails. This is the largest single performance feature in modern Spark SQL.

WholeStageCodegenExec.scala L673-L749 — doCodeGen & doExecute

WholeStageCodegenExec.scala L914-L994 — CollapseCodegenStages