Beyond the Ticket

IT leaders can now extend technician expertise far beyond the tickets any one technician handles, changing how an IT team organizes its work.

Studying the performance of today’s leading models on challenging tickets shows where they still fall short, and how technicians help define success and test whether models meet that standard.

We released Shield’s first set of IT evals as part of Thrive Holdings’ HoldingsBench*. They are one piece of our efforts to rethink how IT firms operate from first principles as agents take on more work.

We measured leading AI model performance on a set of 52 tickets that AI Engineer, our helpdesk agent, had previously handled incorrectly. Those earlier attempts involved a range of models. AI Engineer functions within Forge, our AI operating system for IT services.

AI systems can already handle a wide range of routine helpdesk work end-to-end, including password resets, employee onboarding and offboarding, license renewals, and firewall repairs. This is part of why Forge is already handling over 50% of all actionable tickets for our providers (ShieldCos), with new models pushing performance further. To study the harder work that eludes complete automation, we tested new models on those 52 tickets. We plan to build on these findings with additional benchmarks.

GPT-6 Astra with extra high reasoning passed the evaluation for 61.5% of tickets (32 out of 52). Claude Opus 5.5 max and Astra on high effort followed, passing the evaluation for 59.6% of the same set of tickets (31 out of 52). These configurations were separated by just one ticket.

Technicians and engineers across Shield’s ecosystem defined an ideal outcome for every ticket: what success required in that context. Each model was tested in Forge’s proprietary harness. All testing involved a human reviewer verifying the completeness and accuracy of an LLM as a judge, which outputted a pass or fail for each run, based on the defined outcomes.

Most importantly, we learned that high model intelligence does not guarantee consistent, professional service. Agents could complete difficult technical work and still fail to follow a customer’s instructions or established procedures.

Ticket Tokenomics

The results make a point IT leaders know well: performance is a tradeoff between quality, time, and cost.

Claude Opus 5.5 high passed 53.8% of cases, taking 13 minutes per ticket and costing $3.30, while Gemini 3.8 Flash high passed 36.5% in 24 minutes for $1.60. Opus was faster and did a better job, but at about double the agent cost.

Switching from low to xhigh with Astra resulted in two additional passes (30 to 32), but the time taken jumped from under 10 minutes to nearly 20 minutes, and cost from $5.10 to $7.60. At the other end, DeepSeek V4.1 Flash low passed 25 cases at a much lower $0.13. These tests provide valuable data that feeds into our model selection, which balances time, cost, and accuracy depending on the ticket.

The times and costs don’t factor in human verification, correction, and follow-through. A quick but sloppy first pass often adds more work to the technician’s plate and could greatly increase the total cost and time of using the lower-performing models.

The Jagged Frontier in IT

In 2023, Wharton professor Ethan Mollick and his colleagues popularized the “jagged frontier” of AI to describe how tasks that seem equally challenging for humans can fall on either side of a model’s abilities. Within IT, that sense of unevenness is visible even within familiar workflows.

AI capabilities don’t fit neatly into the L1-L3 tiers used to organize support teams. For example, AI can verify a hundred logins with subagents dividing the checks, working in parallel, and consolidating them for review. This would occupy a project engineer for hours and require painstaking attention to do well. Yet an agent can overlook a customer’s instructions and go down the wrong path.

In one benchmark case, a customer asked the team to contact her phone extension regarding a slow computer, failed updates, and some printing issues. When investigating the case, Claude Opus 5.5 high pointed to an earlier ticket that had been resolved remotely and explicitly concluded: “I treated this as a contact request, not an explicit request for a phone call.”

It emailed the customer attempting to schedule remote maintenance. Although that approach may have helped them, the agent cited a different ticket to justify this approach, when the ideal outcome would have been to simply escalate and arrange a phone call.

A second ticket exposes a different issue. A customer requested an email account for a new employee. The documented procedure required an Active Directory account first, followed by a separate Microsoft 365 account; these systems did not automatically synchronize.

GPT-6 Sol high acknowledged that procedure, but only proposed M365 setup, assuming that the AD account was optional, though the opposite was written.

Making AI Work for IT

These cases were not simply failures to find information. The relevant customer procedures were available, but the agents did not carry them into their decisions. Evaluating success therefore requires more than checking whether an agent performed technical work; it requires checking whether that work meets the customer’s requirements.

Strong evals help turn technician standards into tests that guide improvements. Andrej Karpathy framed verifiability as an engine of AI progress. Models can be evaluated against outcomes we can check in a reliable way. For us, the technician’s definition of “done” is that engine, giving engineers a concrete target to orient the system around.

We can test how better models and improvements to our harness help agents follow the customer’s established procedures and escalate in the right situations. These evals are already showing frontier labs where models struggle in IT workflows, and researchers have told us they plan to use them to guide improvements to their models. This is the last mile of making models effective for IT support.

From Contributor to Conductor

This continuously improving IT service desk expands a technician’s role. Their reach broadens beyond individual tickets to help define how agents should handle the work. As Jon Katz, co-founder and COO of ShieldCo LISS Technologies, put it, “You go from owning individual tickets to owning the process of a ticket.”

In practice, technicians help define these ideal outcomes and document what can (and should) be delegated. To make this happen, agents need authority to act within agreed limits, and engineers need to act on technician feedback. Technician expertise is why we begin every partnership by spending weeks onsite with providers so we can build around how their teams actually work.

Our work deploying Forge with ShieldCos also raises a broader question: with a team of conductors, how should the service desk and the wider IT firm be organized?

We’re experimenting with our ShieldCos on how teams should be organized as agents take on more work. At one provider, some helpdesk capacity has been shifted into account management, projects, and managed AI services. At another, the restructured workflow pencils in a larger role for escalation engineers, and a new position for team leaders who oversee AI Engineer.

The goal is to offer customers fast resolution for a growing range of IT problems at any time of the day, with technicians jumping in when judgment or direct involvement is needed. With agent delegation creating new capacity, providers can prioritize root-cause fixes, advise businesses on technology planning, and help them implement AI for their end customers.

Our ambition is to keep building with our IT firms and expand what they can do for customers. Better models give technician knowledge more reach, with careful evaluation making their expertise legible to our systems.

Building on that foundation to rethink the entire IT firm is the work ahead.

For the full results and methodology, explore HoldingsBench.