Two Mobile AI runs can both be marked “successful” and still answer different questions. They may have used different app states, devices, network conditions, agent versions, or completion checks. A comparison is useful only when the team can say what it intended to hold constant, what it changed, and what evidence supports the conclusion.
The goal is not to make every run identical. It is to make the relationship between runs explicit enough to interpret.
Start with the decision the comparison must support
Before comparing results, state the decision. Is the team checking whether the same workflow behaves consistently under a defined baseline? Testing how it responds to a particular device or network condition? Or deciding whether a new agent version improves a task set?
Those questions need different comparison designs. A repeatability check holds relevant conditions steady and observes variation across attempts. A robustness check keeps the task and completion rule stable while deliberately changing a condition. A release comparison changes the agent version while preserving the conditions needed to make that comparison fair.
Repeatability vs. Reality in Mobile AI Data Collection explains why controlled repetition and physical execution context support different claims. Why Standardized Device Fleets Can Improve Data Collection Consistency discusses how to define a stable cohort. The next step is to state what a particular comparison allows the team to conclude.
Write down the comparison contract
A useful comparison contract records four things:
- The question and task. Keep the task definition, input, starting fixture, and completion criterion identifiable. If any of them changes, record a new revision.
- The intended baseline. Name the app and relevant software versions, device cohort or configuration, account and data state, and any configured operating conditions that matter to the question.
- The planned variation. State which factor or set of factors is allowed to differ and why. If several factors change together, the result may describe that combined scenario, but it cannot isolate the effect of each factor.
- The evidence, outcome, and uncertainty rules. Record what observations will support completion, how missing or conflicting evidence will be handled, and how each attempt is linked to its result.
Benchmark design can make this discipline concrete. AndroidWorld gives tasks dedicated initialization, success-checking, and teardown logic, and its robustness analysis reports that task variations can materially affect agent performance. That is a useful reminder to define both the starting point and the check used to judge an outcome, rather than comparing labels detached from their conditions. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Use one task to see how the question changes the comparison
Imagine an agent asked to save a selected item to a reading list and verify that it appears there.
If the question is whether a new agent version behaves differently, keep the task revision, app build, account fixture, device cohort, configured network condition, and completion check the same. The intended difference is the agent version. If other conditions also changed, record them; do not attribute the outcome to the agent alone.
If the question is whether the workflow tolerates a network interruption, keep the task, agent, app, device cohort, and completion rule stable, then introduce a defined interruption. Record the configured condition and what the device actually observed during the attempt. A run where the interruption never occurred is not evidence about recovery from that interruption.
If the question is how the workflow behaves across device configurations, select the device variants because they represent conditions the team wants to test. Keep the task and outcome rule consistent, report results by configuration, and avoid collapsing distinct device groups into one unexplained score.
The task is recognizable in all three comparisons, but the claim changes. In the first, the team is comparing agent versions. In the second, it is testing a particular network condition. In the third, it is examining selected device configurations. Sharing the same task does not, by itself, make two runs comparable. The task, controlled conditions, and evidence need to fit the question being asked.
Separate configured conditions from observed conditions
A test plan may specify an app build, device cohort, locale, or network profile. A particular run may still encounter a different state: an app update may not have applied, a device may be recovering, or connectivity may have changed during execution. The planned setup describes what the team intended. The run record describes what was present and observed.
Keep those descriptions separate. When they differ, preserve the deviation with the attempt and decide whether it belongs in the planned comparison, needs a separate analysis, or warrants a rerun. A successful rerun should not erase the earlier attempt if that attempt helps explain the result.
This is where physical execution context matters. A named device cohort can reduce avoidable differences, but it does not prove that every device or run had identical conditions. Record the device and software context that could affect the question, together with the observations needed to interpret the attempt.
Treat incomplete evidence as a result in its own right
A pass/fail label is not enough when the evidence path is incomplete. If the expected screen was not captured, the state check failed to run, or the device stopped responding before the task could be judged, the team may not know whether the agent failed or the outcome was simply unobserved.
Keep the task and attempt identifiers, agent and app versions, device context, configured and observed conditions, action records, resulting observations, timestamps, and any error or recovery event connected to the result. Mark an unresolved check as inconclusive or incomplete according to the evaluation rule. Do not silently count it as either a pass or a task failure.
Match the conclusion to the comparison
Two runs can be compared as individual observations, but two observations alone do not establish stable behavior across repeated attempts, devices, or operating conditions. A result from one defined cohort supports a claim about that cohort and setup. A deliberate network variation supports a claim about that tested condition. A release comparison supports a claim about the tested task set and versions.
When the decision requires a broader claim, plan the needed repetitions or coverage explicitly. Report the number of attempts, the conditions represented, unresolved outcomes, and gaps. If the task, criterion, or relevant condition changed, explain how that change limits the comparison.
Make the comparison useful to the next decision
Before running a comparison, a team should be able to complete this sentence: “We are comparing ___, holding ___ stable, varying ___, and will use ___ as evidence.” If those blanks cannot be filled in, the comparison design is not ready.
Real Android infrastructure helps when it provides the execution context the question requires and keeps that context connected to observations and outcomes. The comparison logic can remain in the team’s existing benchmark or evaluation workflow. A comparison is only as useful as the conditions and evidence behind its conclusion.