MaxText Performance Analysis

What it does

This skill provides a step-by-step performance analysis pipeline for MaxText training runs on Slurm-managed clusters. The dispatcher examines logs and artifacts, runs appropriate tools (tgs_tagger, TraceLens, IRLens), and produces a structured analysis.json and CSV artifacts. The assistant uses these outputs to summarize throughput (TGS/MFU), GPU compute/comm/idle breakdowns, kernel-level time composition, and suggested next steps.

When to use it

Use when investigating slow training runs, unexplained throughput regressions, per-kernel hotspots, or when comparing two jobs to find system-level or GPU-side causes. It supports both single-job deep dives and multi-job comparison workflows (the SKILL.md links to complementary skills for multi-job TSDB comparisons and profile-drill per-kernel verification).

What's included

• Scripts: dispatcher utilities (utils/analyze_job.py) and helper scripts to extract key metrics from analysis.json and TraceLens CSVs
• References: patch notes and operational guidance for TraceLens; instructions for installing/patching TraceLens when needed
• Instructions: runbook for using dispatcher, handling TraceLens install/patch, reading analysis.json, and interpreting CSV outputs (gpu_events_averages.csv, kernel breakdowns)

Compatible agents

Best for agents that can run shell/python tooling and inspect generated JSON/CSV outputs. Targets ML infra engineers and performance investigators working with TraceLens/IRLens artifacts.