Apache Spark — Tests & Learning Path

10. Tests

Frameworks

Test types

• Unit: Catalyst rule tests, serializer tests, individual RDD ops with local[2]
• Integration: Full SQL queries, Hive metastore (often Derby embedded), streaming with memory sinks
• Cluster-ish: local-cluster mode simulates multiple executors in one JVM
• Tagged slow/extended: @Tag(SlowSQLTest), ExtendedYarnTest — skipped in default CI
• Docker/K8s IT: connector/docker-integration-tests, kubernetes/integration-tests

What tests reveal about intended behavior

• DAGSchedulerSuite — stage boundaries, fetch failure recovery, cache tracking
• QueryExecutionSuite / SQLQueryTestSuite — golden SQL plans and results
• StreamingQuerySuite — micro-batch offset progression
• SparkConnect* suites — client/server protocol parity with classic API

What appears less covered

• Full multi-node failure chaos (partial — mostly simulated)
• Every connector combination with real cloud services (Docker ITs cover some)
• Performance regressions (separate benchmarking, not unit tests)

How to run tests

# Full suite (long)
./dev/run-tests

# Specific modules
./dev/run-tests --parallelism 1 --modules core,sql

# Maven single module
./build/mvn -pl :spark-core_2.13 test

# Python
./python/run-tests --modules pyspark-sql --python-executables=python3.12

# With tags
./dev/run-tests --included-tags org.apache.spark.tags.ExtendedSQLTest

Test fixtures

SparkContext often created per suite with local[2] and spark.testing=true. Temporary dirs via Utils.createTempDir. Shared test JARs in core/target/.... Module dependency graph in dev/sparktestsupport/modules.py determines which suites run when files change.

11. Build, run, and deploy

Install dependencies

• JDK 17+ (Temurin recommended per README)
• Maven (bootstrapped by ./build/mvn)
• Python 3.11+ for PySpark development
• Node.js for UI tests (optional)

Build

# Standard build
./build/mvn -DskipTests clean package

# With cluster profiles
./build/mvn -Pyarn -Phive -Pkubernetes -DskipTests clean package

# SBT (dev)
./build/sbt package

# Distribution tarball
./dev/make-distribution.sh --tgz -Pyarn -Pkubernetes

Run locally

./bin/spark-shell
./bin/pyspark
./bin/spark-submit --class org.apache.spark.examples.SparkPi \
  examples/jars/spark-examples_*.jar

# Standalone mini-cluster
./sbin/start-master.sh
./sbin/start-worker.sh spark://localhost:7077

Configuration

• Copy conf/spark-defaults.conf.template → conf/spark-defaults.conf
• --conf key=value on submit; SparkConf in code
• SPARK_HOME, JAVA_HOME, HADOOP_CONF_DIR (for YARN/HDFS)

Deployment

CI

63 GitHub Actions workflows. Primary: build_main.yml → reusable build_and_test.yml (SBT precompile + dev/run-tests matrix). Separate Maven path in maven_test.yml. Python/Java version matrices on branch-specific workflows.

12. Most important files to understand first

13. Things that are confusing or risky

• Two streaming systems — legacy DStreams vs Structured Streaming; docs and imports differ.
• Classic vs Connect API — Connect client has no local SparkContext; debugging requires server logs.
• Hive vs V2 catalog — table resolution paths differ; spark.sql.catalogImplementation matters.
• Global SparkEnv — implicit state complicates testing and embedding.
• Closure serialization — capturing non-serializable objects fails at runtime, not compile time.
• Shuffle + dynamic allocation — requires external shuffle service or risk losing shuffle files.
• AQE — plan at runtime may differ from explain() without AdaptiveSparkPlanExec awareness.
• Version skew — PySpark pip package must match cluster Spark version.

14. Suggested learning path for a new engineer

1. Read README.md, run ./bin/spark-shell and spark.range(1000).count().
2. Trace submit: bin/spark-submit → launcher/Main.java → SparkSubmit.scala.
3. Core model: RDD.scala (doc comment) → Dependency.scala → DAGScheduler.scala (header comment).
4. Run one test: DAGSchedulerSuite in IDE or Maven to see stage creation.
5. SQL path: SparkSession.sql → QueryExecution lazy vals → SparkPlan.execute.
6. Catalyst: Pick one optimizer rule (e.g. PushDownPredicates) and follow RuleExecutor.
7. Executor: CoarseGrainedExecutorBackend + Executor.TaskRunner.
8. Storage/shuffle: BlockManager, SortShuffleManager.
9. Deploy: Skim YARN Client.scala or K8s submit path for your target environment.
10. Connect (if relevant): SparkConnectService → SparkConnectPlanExecution.
11. Streaming (if relevant): MicroBatchExecution.scala + checkpoint layout.

15. Open questions

These cannot be fully answered from code alone without running clusters at scale:

• Exact production tuning for a specific cloud (instance types, shuffle service sizing) — docs give guidelines, not proofs.
• Real-world performance vs Databricks Runtime optimizations (some APIs are OSS stubs or differ in commercial builds).
• Complete connector compatibility matrix for every cloud vendor fork of Hadoop FS.
• Which deprecated code paths will be removed in which release — check JIRA/SPARK tickets and release notes.
• Operational SLOs for Spark Connect at N concurrent sessions — requires load testing.

← Back to overview