Benchmark
SWE-Bench Pro
A long-horizon software-engineering benchmark built from public and proprietary repositories.
Scale · 100% completeConfirmed
Benchmark·Private commercial
SWE-Bench Pro - Private Dataset
The proprietary-repository subset of SWE-Bench Pro, documented publicly but not distributed as an open dataset.
What it is
This private subset evaluates coding agents against commercial-grade proprietary repositories. Its existence and leaderboard are public; the underlying repository contents are not presented as publicly downloadable by EnvIndex.
Editorial note: Private describes access, not verification. EnvIndex has not inspected the proprietary repository contents.
Capabilities
Only evidenced capabilities are shown as Yes. Missing evidence remains Unknown.
StatefulUnknown
Long horizonYes
Multi-turnUnknown
Multi-agentUnknown
AdversarialUnknown
MultimodalUnknown
Tool useUnknown
Browser useUnknown
Computer useUnknown
Code executionYes
File manipulationUnknown
API interactionUnknown
MCP supportUnknown
SandboxedYes
Deterministic resetUnknown
Stochastic behaviorUnknown
Human in the loopYes
Expert authoredUnknown
SyntheticUnknown
Production derivedYes
Real application replicaUnknown
Partial observabilityUnknown
Persistent stateUnknown
Cross-applicationUnknown
Sources & provenance
- Scale Labs · paper · accessed 2026-08-16 · confirmed
- Scale Labs · official · accessed 2026-08-16 · company-reported
Related artifacts
Public sample
RL Environment
RL Environments and Agents
Custom RL environments and verifier design for training and evaluating agentic models.
Surge · 100% completeCompany-reported
Environment Bundle
Off-The-Shelf Data Catalog
Pre-built commercial datasets and RL environments spanning coding, enterprise agents, STEM, tool use, and reasoning.
Surge · 100% completeCompany-reported