Senior Data Engineer

What it does

Provides a comprehensive, production-focused data engineering toolkit: Airflow DAG generation, streaming job scaffolds (Flink/Spark), Kafka config generators, data quality validation, and performance analysis. It helps build end-to-end ETL/ELT pipelines and real-time streaming architectures with monitoring and CI/CD integration.

When to use it

Use this skill when you need to design or implement production data pipelines (batch or streaming), add automated data quality checks, scaffold streaming/Flink jobs, generate Kafka configurations, or profile and optimize pipeline performance. Good for enterprise ETL, data-lake/warehouse builds, and production ML data infra.

What's included

• Scripts: pipeline_orchestrator.py, data_quality_validator.py, stream_processor.py, kafka_config_generator.py, etl_performance_optimizer.py, streaming_quality_validator.py (has_scripts=true)
• References: frameworks, templates, tools documentation (has_references=true)
• Instructions: YAML-driven pipeline generation, Airflow DAG templates, dbt model guidance, data-quality rule patterns, streaming validation and monitoring workflows.

Compatible agents

Best suited to agents that can run Python tooling and integrate with Git/CI systems (Copilot/Code assistants, Claude Code/Cursor), and that can operate with containerized or cloud data stacks.