Senior Data Engineer

Trust Score 75/100

Production-ready data engineering skill for designing ETL/ELT and real-time streaming pipelines, data quality, and pipeline performance optimization.

triggers:build etl pipelinestreaming pipelinedata qualitygenerate airflow dagkafka configoptimize spark

GitHub SKILL.md

What it does

Provides a comprehensive, production-focused data engineering toolkit: Airflow DAG generation, streaming job scaffolds (Flink/Spark), Kafka config generators, data quality validation, and performance analysis. It helps build end-to-end ETL/ELT pipelines and real-time streaming architectures with monitoring and CI/CD integration.

When to use it

Use this skill when you need to design or implement production data pipelines (batch or streaming), add automated data quality checks, scaffold streaming/Flink jobs, generate Kafka configurations, or profile and optimize pipeline performance. Good for enterprise ETL, data-lake/warehouse builds, and production ML data infra.

What's included

Scripts: pipeline_orchestrator.py, data_quality_validator.py, stream_processor.py, kafka_config_generator.py, etl_performance_optimizer.py, streaming_quality_validator.py (has_scripts=true)
References: frameworks, templates, tools documentation (has_references=true)
Instructions: YAML-driven pipeline generation, Airflow DAG templates, dbt model guidance, data-quality rule patterns, streaming validation and monitoring workflows.

Compatible agents

Best suited to agents that can run Python tooling and integrate with Git/CI systems (Copilot/Code assistants, Claude Code/Cursor), and that can operate with containerized or cloud data stacks.

Audit Summary

Comprehensive data engineering skill covering ETL, streaming, Kafka, Flink, and data quality. SKILL.md is well-structured and detailed (23K+ chars). However, 4 of 6 scripts are broken stubs — they define classes with __init__(config: Dict) but main() passes args.target as positional arg and calls tool.run() which doesn't exist (should be tool.process()). Only kafka_config_generator.py and stream_processor.py are substantive and functional. The Kafka config generator is genuinely useful with multi-mode CLI support.

Watch Out

4 of 6 scripts (data_quality_validator, etl_performance_optimizer, pipeline_orchestrator, streaming_quality_validator) are broken stubs that crash with AttributeError
SKILL.md references references/frameworks.md, references/templates.md, and references/tools.md but these are not bundled
stream_processor.py Docker Compose template contains hardcoded POSTGRES_PASSWORD: streaming (dev-only but still a bad practice)
kafka_config_generator.py security configs correctly use env vars for credentials, not hardcoded values

Notes

Skill has impressive documentation breadth but implementation depth is uneven. The 4 stub scripts are clearly template-generated boilerplate that were never filled in — they share identical structure and all have the same bug (calling .run() instead of .process()). kafka_config_generator.py (18K) and stream_processor.py (60K) are the real value. The skill would benefit from either completing the stubs or removing them.

Information

Repository: skillsbench
Stars: 1,301

Trust Score

Overall75

Security95

Code Quality45

Architecture68

Usefulness52

Related Skills

OpenTestAI

Automated, high-confidence AI testing: bug detection, persona feedback, and prioritized test-case generation using many specialized tester profiles.

SSE Stream Resilience

Provide robust server-sent-events (SSE) stream management with Redis-backed registry, heartbeat monitoring, completion persistence, and background guardian clea

Go Data Structures

Authoritative guidance on choosing and using Go built-in and standard-library data structures, with practical best practices for slices, maps, arrays, container

React Development Expert

Provides authoritative React guidance on hooks, state patterns, Server Components, performance optimization, and common architectural patterns.

Code Reviewer

Perform structured, prioritized code reviews that find correctness, security, performance, reliability, and testing issues and provide concrete fix suggestions.

dotLottie Web

Guidelines and patterns for implementing performant dotLottie/Lottie animations on the web (vanilla JS and React), including workers, state machines, and themin

Party Engine Skill

Guidance and examples for using the @cazala/party particle engine (engine lifecycle, modules, WebGPU vs CPU patterns) in custom apps.

Party Skill

Programmatic guide for the @cazala/party library: engine setup, modules, particle APIs, and performance tips for WebGPU and CPU runtimes.

Back to Skills