Benchmark
SWE-Bench Pro - Private Dataset
The proprietary-repository subset of SWE-Bench Pro, documented publicly but not distributed as an open dataset.
Scale · 100% completeCompany-reported
Benchmark·Public sample
SWE-Bench Pro
A long-horizon software-engineering benchmark built from public and proprietary repositories.
What it is
SWE-Bench Pro evaluates agents on realistic software-engineering problems. The published paper reports 1,865 problems from 41 actively maintained repositories; the public dataset and leaderboard are separately accessible.
Capabilities
Only evidenced capabilities are shown as Yes. Missing evidence remains Unknown.
StatefulUnknown
Long horizonYes
Multi-turnUnknown
Multi-agentUnknown
AdversarialUnknown
MultimodalUnknown
Tool useUnknown
Browser useUnknown
Computer useUnknown
Code executionYes
File manipulationUnknown
API interactionUnknown
MCP supportUnknown
SandboxedYes
Deterministic resetUnknown
Stochastic behaviorUnknown
Human in the loopYes
Expert authoredYes
SyntheticUnknown
Production derivedUnknown
Real application replicaUnknown
Partial observabilityUnknown
Persistent stateUnknown
Cross-applicationUnknown
Sources & provenance
- Scale Labs · paper · accessed 2026-08-16 · confirmed
- Scale Labs · official · accessed 2026-08-16 · confirmed
Related artifacts
Private commercial
RL Environment
RL Environments and Agents
Custom RL environments and verifier design for training and evaluating agentic models.
Surge · 100% completeCompany-reported
Environment Bundle
Off-The-Shelf Data Catalog
Pre-built commercial datasets and RL environments spanning coding, enterprise agents, STEM, tool use, and reasoning.
Surge · 100% completeCompany-reported