VAKRA
by ibm
Enterprise-grade benchmark for evaluating multi-hop, multi-source tool-calling in AI agents.
What it does
VAKRA (eValuating API and Knowledge Retrieval Agents) is a tool-grounded executable benchmark designed to evaluate AI agents' compositional reasoning in complex enterprise environments. It measures how well agents can chain decisions across multiple APIs and documents, reconciling mismatched schemas and following natural-language tool-use constraints.
Tools
VAKRA provides an executable environment with over 8,000 locally hosted APIs across 62 domains. These are categorized by capabilities:
- API Chaining: Nested and compositional API calls.
- Tool Selection: Large-scale selection over query-aligned endpoints.
- Multi-hop Reasoning: Dependent reasoning chains where earlier outputs parameterize later calls.
- Multi-turn Reasoning: Complex dialogues combining APIs and document retrieval with policy adherence.
Installation
Requires Docker and Python 3.11+. Run the following:
make download
make build
docker compose up -d
Supported hosts
- claude
Quick install
make download && make build && docker compose up -dInformation
- Pricing
- free
- Published
- 8/4/2026
- stars
- 0
Categories
Choose your AI client and follow the steps below.
Claude Desktop
Follow the repository guide: make download, make build, and docker compose up -d to start the benchmark environment.Related Apps
nanobot
MCP ServerUltra-lightweight personal AI agent framework with built-in MCP, long-term memory, and multi-channel chat support.
FinanceToolkit
MCP ServerProfessional-grade financial analysis toolkit for equities, options, and risk management.
DiffSitter MCP
MCP ServerAI-powered structural code navigation using tree-sitter ASTs for semantic understanding across 14+ languages.
OpenAI Apps SDK Examples
MCP AppOfficial example gallery of interactive MCP widgets for ChatGPT — 3D viewers, maps, carousels, shopping carts, and more.
Human MCP
MCP ServerGive AI agents human-like senses: visual analysis, image/video generation, speech synthesis, browser automation, and advanced reasoning — 29 MCP tools in one se
Containarium
MCP ServerSelf-hostable agent runtime with SSH-native isolation, eBPF egress policy, and MCP-native CLI.
Shopify MCP Server
MCP ServerDirect interaction with Shopify store data via GraphQL API for managing products, customers, and orders.
Git MCP Server
MCP ServerFull-featured Git MCP server exposing 28 tools for AI agents to clone, commit, branch, diff, merge, rebase, and more via STDIO or Streamable HTTP.