RLTasksby ulam.ai
Sign inAll tasks
Software engineeringSWE-0194

SWE-0194

A production-style Go benchmark for reconnect ownership, protocol recovery, event integrity, concurrency, and resource safety in a resilient WebSocket client. Agents must repair a state machine that behaves plausibly in simple tests but fails race-enabled fixed cases and held-out deterministic traces under a confidential operator verifier.

Coding agentPrivate verifierRL environment
Version2.0.0
Environments1
RewardScalar · 0–1
DeliveryPrivate ZIP
01 Task contract

A real, versioned RL environment.

A production-style Go benchmark for reconnect ownership, protocol recovery, event integrity, concurrency, and resource safety in a resilient WebSocket client. Agents must repair a state machine that behaves plausibly in simple tests but fails race-enabled fixed cases and held-out deterministic traces under a confidential operator verifier.

Agent objective

  • Interact with the supplied stateful environment.
  • Produce verifier-checkable actions or artifacts.
  • Maximise scalar reward under the package contract.

Evaluation

  • Private verifier.
  • Reported reward range 0–1.
  • Package-specific public and private checks.

Delivery boundary

  • Private object stored in Cloudflare R2.
  • Authenticated entitlement required.
  • Short-lived signed URL per download.
02 What you will work on

Distinct environments, one demanding research contract.

Enough detail to understand the intellectual terrain; generated instances, hidden mechanisms, and solution paths remain inside the private package.

EnvironmentMathematical or technical frontierAdaptive research problem
Reconnect ownershipConcurrent state machinesGuarantee one authoritative connection and reconnect loop across overlapping failures and callers.
Protocol recoveryResume, resubscribe, and replayRestore negotiated state and subscriptions in the right order without duplicate or missing protocol effects.
Event integrityOrdering, deduplication, stale generationsReject obsolete connection events and preserve exactly-once visible behavior across reconnect boundaries.
Cancellation and shutdownGo contexts and goroutinesTerminate blocked work, prevent late resurrection, and keep close behavior deterministic and race-free.
Resource safetyTimers, channels, backoffAvoid leaks, deadlocks, busy loops, and runaway retry state across held-out deterministic traces.
03 Why it is interesting

What the supplied evaluation reveals.

The canonical seed scores 0.105 in development and 0.3822 finally, passing 17/29 fixed cases and none of 32 secret traces. The human reference scores 1.0 across all fixed and secret cases under the race detector. The supplied independent candidate also records 1.0 across development and three final holdout runs. This makes the task a strong test of concurrency and state-machine generalization, not simply making visible reconnect tests green.

We publish aggregate behavior and task structure, while withholding generated instances, hidden labels, exact successful probes, private checks, and solution trajectories.

04 Supplied evaluation

Observed evaluation result.

Shown with its provenance and limitations; it is not a performance guarantee.

i
Methodology matters

Final development reward 1.0; all three holdout rewards were 1.0, with 29/29 fixed checks and 32/32 secret-trace checks.

Evaluated system / policyGPT-5.6 Pro

As identified by the supplied artifact.

Final reward1.000

3 reported runs.

Result artifactIncluded

swe0194_v2_score_summary.json

Public result record

Machine-readable provenance and the exact displayed metric are available in results.json.

Open result JSON
05 Private delivery

The package stays off the public website.

The paid ZIP will live in a private R2 bucket. Vercel authorizes the buyer and issues a 2–5 minute object URL; R2 serves the bytes directly.

Included with purchase

  • Exact package version 2.0.0
  • Environment and task contracts
  • Verifier or scoring interface
  • Supplied reference/evaluation artifacts
  • Purchase record and licence v1.1
delivery flow
Authenticated buyer + entitlement check
+ private R2 object + 2–5 minute signed URL
= direct, auditable download

Package SHA-256
7387f94a8537593166111109f17ac3a5b2962e6a9a136687114c628d5a484fd0
06 Licence v1.1

Commercial use, without exclusivity.

One purchase licenses this identified item to one legal organisation for worldwide, perpetual commercial model training, evaluation, research and development. Redistribution and resale of the package are not permitted.

Read the full licenceYotta Content LTD · business customers only
SWE-0194

Ready to add this environment?

Back to marketplace
Stripe checkout

Business purchase confirmation

Sign in or create an account, then complete secure Stripe Checkout. Access is granted only by the verified payment webhook.