OSWorld-Compatible Computer-Use Training & Evaluation Environments

Computer-use agents are moving beyond text and code into environments where models must see, reason, interact with, and finish work on real desktop systems.

For these agents, model capability alone is not enough. High-quality post-training needs realistic environments, diverse tasks, reliable execution, deterministic evaluation, and trajectories that can be generated and measured again and again.

Cognyzer proposes to build a private, OSWorld-compatible task and training environment for Z.ai, designed for computer-use post-training and evaluation.

This is not a one-time set of exam questions. It is a renewable supply of executable tasks across desktop applications and operating systems. Each task is a complete unit: instruction, metadata, a restoreable start state, configuration, automated verification, difficulty measured by repeated model runs, a human reference trajectory, and the logs that prove the score.

We build the executable environments and training tasks around the model, not simply the annotations inside them.

That matches the bottleneck Z.ai has already named. GLM-5.3’s published post-training story is more environments, more diverse tasks, more compute spent on them. As the model improves, difficulty moves from the weights to the environment: the task has to be executable, verifiable, and close to real work, and there have to be many of them. Environment generation and verification still need human-in-the-loop work. That is the work this proposal is about.

Traditional benchmarks are built to measure capability. Training environments have a different job.

A training task must be:

Executable to Reproducible to Verifiable to Calibrated to Renewable

A task the model always solves teaches little. A task it never solves teaches little either. The highest-value tasks sit on the current capability boundary: hard enough to need real reasoning and interaction, still demonstrably achievable.

OSWorld is the right execution model because the agent works on a real desktop, not a mock UI, and success is read from the machine it leaves behind.

The agent can be asked to navigate applications, edit documents and spreadsheets, manage files, use a browser, configure software, edit media, run multi-step workflows, and produce a specific final state. A fluent description of the plan does not score.

That is why the framework fits reinforcement learning and automated eval. A public exam is still the wrong training resource:

What is needed is a private, renewable task supply built around the OSWorld execution model, calibrated against Z.ai’s own models as they move.

A private OSWorld-compatible computer-use task and evaluation dataset for Z.ai. Four uses, one pipeline.

The fourth use is continuous difficulty expansion: new tasks authored and calibrated as GLM improves. The system is a pipeline, not a one-time dump.

Each task is a complete, reusable unit. Not a prompt sitting in a spreadsheet.

That unit can be executed, evaluated, debugged and reused independently. It drops into an OSWorld-style harness. You do not need our authoring console to train on it.

Volume without these five is noise.

1. Realistic. Meaningful computer work, not an artificial click sequence.
2. Verifiable. Outcome measurable from environment state.
3. Reproducible. Same start every time.
4. Challenging. Reasoning, planning and interaction, not transcription.
5. Calibrated. Difficulty measured by model execution, not assigned by length.

Difficulty is determined by measured agent performance against the agreed Z.ai configuration.

Each task is run repeatedly. Successful runs place it in a band. The bands, model, run count, step budget and harness version are aligned with you before production.

In the sample in section 16 we used a practical version of this: GLM on every Ubuntu RL task, and Claude Sonnet 4.5 on three of them as a solvability check. Sonnet is a mid-tier model. If it can finish the work some of the time, a GLM zero is headroom, not a broken checker.

The mix is aligned to Z.ai’s computer-use roadmap, not frozen to the public OSWorld list. The applications we already author against include:

The public OSWorld exam is Ubuntu. Agents shipped to real users meet all three desktops. Interaction patterns do not transfer cleanly.

Windows and macOS are not a config flag. They are separate images, licensing and automation. That pool already exists on our side.

The dataset can span four levels. Long-horizon work is where post-training gains currently concentrate, including in GLM-5.3’s own published results.

The evaluator inspects the resulting machine state. It does not grade the model’s claim that it finished.

Depending on the task, checks look at files and directories, spreadsheet values and formulas, document properties, application configuration, image properties, rendered output, file metadata, or application-specific state. Where possible the check is deterministic. Complex tasks can combine several checks into a weighted score.

Each verifier is tested both ways: a successful human trajectory must pass, a deliberately incorrect trajectory must fail. That stops weak evaluators that reward incomplete work.

Every RL task can ship with a human-created reference trajectory: actions, screenshots, step sequence, final state, metadata.

That trajectory proves the task is feasible, gives a reference for debugging, supplies SFT data where required, and is the baseline that validates the automated evaluator. The same task therefore supports several stages of the post-training pipeline. Reasoning traces from models are not included unless you ask for them.

Five independent layers. A task that fails any of them is reworked before it enters a delivery.

• 1. Task authoring Instruction clarity, realism, completeness, application fit, diversity, safety.
• 2. Environment and verifier Restoreable start, stability, correct apps, verifier correctness, deterministic score.
• 3. Trajectory Correct completion, action integrity, screenshot alignment, no wasted detours, successful final state.
• 4. Difficulty Model results, target band, step budget, failure patterns, repeatability.
• 5. Final audit Senior review of a sample. Any task can be sent back.

Illustrative, not from the scored sample. This is the shape of end-to-end computer-use work the dataset is meant to train.

Objective. Open the quarterly sales workbook, identify products whose revenue declined for two consecutive quarters, create a summary table of those products and their percentage decline, apply the required formatting, and save the workbook to the specified location.

Environment. Windows. Microsoft Excel. Starting state: a pre-populated workbook with multiple sheets and realistic sales data.

The agent must open the workbook, inspect the relevant sheets, identify qualifying products, calculate the decline, create the summary table, apply formatting, and save.

Verification checks that the required products are present, values are correct, table structure and formatting hold, and the file exists at the expected path.

Training assets for that unit: instruction, snapshot, config, verifier, human trajectory, execution logs, difficulty results.

This is the part a written methodology cannot replace. A ten-task pack exists today. The Ubuntu slice has been executed inside the OSWorld harness. The numbers below are those runs, including where GLM scored zero.

How every Ubuntu rollout was run

Identical conditions, so the model is the only variable.

GLM V-series models report click coordinates on a 0 to 1000 grid. Those are scaled to real pixels before execution. Without that step every click lands in the top-left of the screen. The scores below include the rescale.

The six Ubuntu RL tasks

These are real tickets, not toy click sequences. Each has a human trajectory that the evaluator accepts as a pass.

Measured pass rates

Pass rate is successes out of three runs. 1 1 0 is 0.67. GLM did not pass these six. We are showing that rather than hiding it. A pack that comes back 3/3 everywhere is a certificate, not a training resource.

Near misses, not incomprehension

Failed GLM runs are stored step by step. Three examples from the trajectories:

• LibreOffice Impress (related Ubuntu task in the wider pack). Both template tokens replaced, figures preserved, then one neighbouring word deleted with the token.
• Pinta. Correct 1280×720 redacted image; the black bar missed by one to three pixel rows.
• LibreOffice Calc. Every sub-task done. The save never landed.

Those are execution misses. That is the signal an RL environment is supposed to carry.

Windows and macOS in the same pack

Four further tasks ship with a passing human trajectory. They have not yet been put through the GLM three-run grid. That is stated plainly. They are in the pack so the three-OS claim is a folder you can open, not a slide.

sample/
├── ubuntu/     6 RL tasks, SFT, 27 scored runs
├── windows/    2 tasks + human trajectory
└── macos/      2 tasks + human trajectory

The pack will be shared from a private Google Cloud Storage bucket as a time-limited signed URL, not as an email attachment. If a number in this section cannot be recomputed from the files in runs/, the number is wrong.

A bounded technical pilot before volume. It locks applications, OS mix, taxonomy, difficulty bands, evaluation model, verifier standard, trajectory format, QA, delivery format and integration.

Exact volume and mix are set in technical alignment. 100 is an illustration, not a quote.

A repeatable production loop, not a one-off generation exercise.

Authoring, VM allocation and SFT recording run on tooling we already operate (environment core, trainer CLI, SFT console). You receive the task folders. You do not have to run that stack.

The result is a reusable asset across training, evaluation and later model iterations.

Cognyzer builds data and evaluation infrastructure for advanced AI systems: RL environments, computer-use tasks, agentic trajectories, SFT and eval datasets, automated verification, and human-in-the-loop quality. Across the shop we have delivered 4,500+ tasks, with 24/7 QA coverage and a 500+ contributor network that is application-only and credential-verified before anyone touches a production sample.

Delivery runs across global time zones, with contributors routed by verified domain credential rather than drawn from a general pool. Environment work is staffed by engineers, because verifier construction and difficulty tuning are software tasks. OSWorld-style production is not annotation. It is environment engineering, task design, application expertise, verifier development, model evaluation and QA under one workflow.

That is the operation we already run on other suites the frontier is measured on. Named clients stay unnamed here. Where a programme was delivered under contract, it is described as work for a frontier lab.

Terminal Bench 2

For a frontier lab we delivered Terminal Bench 2 environments: real command-line, DevOps and scientific-computing tasks with executable setups and deterministic graders. Volume on this family is 2,000+ verified terminal tasks shipped. The measurable outcome is a private, execution-grounded eval and training set that sits on the same suite labs already use to rank terminal agents, rather than a proxy checklist.

SWE-Bench

For a frontier lab we delivered SWE-Bench-style software-engineering instances: GitHub-form issues, gold patches, fail-to-pass and pass-to-pass tests, and a harness command that actually runs. Volume on this family is 500+ tasks shipped. The measurable outcome is instances a coding agent can be trained and scored on end to end, with hidden tests going from red to green without wrecking the existing suite.

Terminal Bench 3

We are currently delivering Terminal Bench 3.0 for a frontier lab: longer-horizon terminal and science environments, Harbor-packaged, with oracle solutions and repeated model scoring on the same tasks. This is live production, not a completed archive. It is the same shape of work as TB2, moved to the harder public exam the field is now using.

Computer-use (this proposal) is the next family in that line. The other programmes are context, not a bundle being sold in this document.

1. Technical alignment. Target models, agent framework, OSWorld harness version, applications, operating systems, difficulty method, eval criteria.
2. Pilot. Bounded first dataset. Validate task quality, environment stability, verifier reliability, calibration, delivery format, integration.
3. Scale. More applications, more OS coverage, longer workflows, higher volume, new difficulty bands, additional training or eval needs.

Commercial terms belong after phase 1, once the technical fit is agreed. They are not in this document.

The next generation of computer-use agents will need better environments, better tasks and better feedback loops, not only stronger models.

Cognyzer’s proposal is a private, renewable OSWorld-compatible task ecosystem that can move with GLM. Not a static collection of benchmark questions. A continuous source of executable, verifiable, difficulty-calibrated computer-use work for post-training and evaluation across model generations.

We would welcome a review of the sample with the Z.ai team, and alignment on the technical requirements for an initial pilot.

Contact

Cognyzer / Hyderabad, India / Global delivery
Pragnasya S / Co-founder & CTO