Apache Spark Expert Engineer

What it does

This skill transforms the agent into a senior Apache Spark engineer capable of architecting and optimizing large-scale distributed data processing pipelines. It provides deep expertise in PySpark and Scala, focusing on production-grade reliability and performance.

When to use it

Use this skill when writing new Spark jobs, debugging performance bottlenecks (like shuffle spill or data skew), configuring cluster memory, or implementing structured streaming analytics.

What's included

• References: Includes detailed guidance on Spark SQL, RDD operations, partitioning/caching strategies, and performance tuning.
• Instructions: Provides a rigorous 5-step workflow (Analyze $\rightarrow$ Design $\rightarrow$ Implement $\rightarrow$ Optimize $\rightarrow$ Validate) and strict constraints to avoid common pitfalls like collect() OOMs or inefficient UDFs.
• Code Patterns: Includes implementation templates for Broadcast Joins and Salting techniques to handle data skew.

Compatible agents

Compatible with any LLM-based agent capable of writing and executing Python/Scala code (Claude Code, Cursor, etc.).