RLTasksby ulam.ai
Sign inAll tasks
Software engineeringSWE-0147

SWE-0147

A TypeScript software-engineering benchmark for credential epochs under hostile JavaScript re-entry. The visible suite starts green while private cases expose interacting authority, liveness, retry, boundary, validation, and resource failures. Agents must repair stale-token rotation, refresh ownership, cross-context races, logout, and exact public declarations under isolated, mutation-audited verification.

Coding agentPrivate verifierRL environment
Version6.0.0
Environments1
RewardScalar · 0–1
DeliveryPrivate ZIP
01 Task contract

A real, versioned RL environment.

A TypeScript software-engineering benchmark for credential epochs under hostile JavaScript re-entry. The visible suite starts green while private cases expose interacting authority, liveness, retry, boundary, validation, and resource failures. Agents must repair stale-token rotation, refresh ownership, cross-context races, logout, and exact public declarations under isolated, mutation-audited verification.

Agent objective

  • Interact with the supplied stateful environment.
  • Produce verifier-checkable actions or artifacts.
  • Maximise scalar reward under the package contract.

Evaluation

  • Private verifier.
  • Reported reward range 0–1.
  • Package-specific public and private checks.

Delivery boundary

  • Private object stored in Cloudflare R2.
  • Authenticated entitlement required.
  • Short-lived signed URL per download.
02 What you will work on

Distinct environments, one demanding research contract.

Enough detail to understand the intellectual terrain; generated instances, hidden mechanisms, and solution paths remain inside the private package.

EnvironmentMathematical or technical frontierAdaptive research problem
Credential authorityEpochs, rotation, stale-token rejectionEnsure only the newest credential epoch can authorize work across re-entrant JavaScript and cross-context state.
Refresh ownershipSingle-flight and retry semanticsCoordinate refresh callers, retries, cancellation, and terminal failures without duplicate authority or lost wakeups.
Lifecycle boundariesLogout, disposal, validationPrevent late completions from resurrecting sessions and preserve exact errors and public declarations.
Hostile schedulingMicrotasks, timers, callbacksRemain live and deterministic under generated schedules that interleave callbacks at adversarial boundaries.
Resource safetyListeners, promises, timersClean up waiters and resources across success, failure, cancellation, logout, and repeated lifecycle transitions.
03 Why it is interesting

What the supplied evaluation reveals.

The seeded repository already passes all 14 visible tests but only 122/211 private cases and receives zero reward. The oracle passes every case, the production reproduction, strict build, and declaration checks; the verifier kills all 14 audited near-miss mutations. A perfect-score patch is supplied, although no separate normalized model-run artifact identifies its actor. The benchmark is valuable precisely because a green public suite is not evidence of correct concurrent authority semantics.

We publish aggregate behavior and task structure, while withholding generated instances, hidden labels, exact successful probes, private checks, and solution trajectories.

04 Supplied evaluation

Observed evaluation result.

Shown with its provenance and limitations; it is not a performance guarantee.

i
Methodology matters

A perfect-score patch is supplied, but no separate normalized score artifact is present.

Evaluated system / policyGPT-5.6 Pro

As identified by the supplied artifact.

OutcomeReference patch

1 reported runs.

Result artifactIncluded

swe0147-perfect-score.patch

Public result record

Machine-readable provenance and the exact displayed metric are available in results.json.

Open result JSON
05 Private delivery

The package stays off the public website.

The paid ZIP will live in a private R2 bucket. Vercel authorizes the buyer and issues a 2–5 minute object URL; R2 serves the bytes directly.

Included with purchase

  • Exact package version 6.0.0
  • Environment and task contracts
  • Verifier or scoring interface
  • Supplied reference/evaluation artifacts
  • Purchase record and licence v1.1
delivery flow
Authenticated buyer + entitlement check
+ private R2 object + 2–5 minute signed URL
= direct, auditable download

Package SHA-256
7a18ec976a45742409da97b74212c688321a2fbbd17c5dd2f9455be59c9e8a1a
06 Licence v1.1

Commercial use, without exclusivity.

One purchase licenses this identified item to one legal organisation for worldwide, perpetual commercial model training, evaluation, research and development. Redistribution and resale of the package are not permitted.

Read the full licenceYotta Content LTD · business customers only
SWE-0147

Ready to add this environment?

Back to marketplace
Stripe checkout

Business purchase confirmation

Sign in or create an account, then complete secure Stripe Checkout. Access is granted only by the verified payment webhook.