A mobile task can look simple: open an application, find an item, save it, and confirm the result.
Yet the same goal does not always produce the same interaction sequence. A response may arrive late. An application may return to a different state after an interruption. A permission request may introduce a decision that was absent in a previous run.
These differences matter when they change what an AI agent should do next—or how its result should be judged.
The previous article examined when emulators are sufficient and when physical devices add value. The next question is what to observe once a workflow runs: which changes reveal useful behavior, and which merely add noise?
The Goal Can Stay the Same While the Path Changes
Consider an illustrative task: find an item in an application and save it to a list. Completion means confirming that the intended item appears in that list.
In a straightforward run, the agent searches, selects the item, taps Save, and checks the list. In another run, a delayed response or an interruption changes that sequence.
The following scenarios are examples for designing a collection workflow, not results from an ARMARRAY device test. Their value is in showing how conditions, decisions, and outcomes relate.
Delayed Feedback Changes the Next Decision
Suppose the agent taps Save, but confirmation does not appear immediately.
Should it wait, inspect the destination list, or retry? A second tap might be unnecessary if the first request is still being processed. Ending the task immediately might leave the outcome unresolved.
A useful record preserves the action, the feedback that followed, the agent's response, and the final check. That sequence helps distinguish an agent that handles uncertainty from one that simply repeats an action.
It also places a limit on interpretation. A loading indicator alone does not establish a network failure. Without supporting evidence, the record should describe delayed feedback rather than assign an unverified cause.
The additional value is not the delay itself. It is the opportunity to examine the decision made while the result was uncertain.
Returning to an App Can Require Recovery
Now imagine that the application leaves the foreground before the task is complete. When the agent returns, the item may still be open, or the workflow may need to resume from an earlier state.
Android distinguishes activity and process lifecycles; a background process can be killed when the system needs resources. Returning to an app therefore should not be assumed to preserve every part of its previous state. The exact behavior depends on the application and circumstances. Android lifecycle documentation
For data collection, the useful question is whether the agent recognizes the state it actually encounters. Does it continue appropriately, recover lost progress, or act as though nothing changed?
Capturing the state before the interruption, the state after return, and the subsequent decisions makes recovery behavior visible. A final success label alone would hide the route taken to reach it.
Permission State Can Introduce a Different Branch
In a separate example, a task invokes a feature that needs a runtime permission. A run with permission already granted may proceed directly; another may present a request or an unavailable feature after denial. Android's permission guidance explicitly accounts for checking access and handling the user's response. Runtime permission documentation
The collection question is how the agent responds within the task's allowed scope. Does it recognize the request, follow the permitted choice, and interpret the resulting state correctly?
This is not a reason to grant every request. An appropriate outcome may be a clearly recorded limitation rather than task completion.
For comparison, the starting permission state and allowed behavior need to be known. Otherwise, two different paths may be mistaken for inconsistent agent performance when they began under different conditions.
Which Variations Are Worth Collecting?
Not every environmental difference deserves another sample. Prioritize variations that change at least one of three things:
- The next decision: the agent must wait, inspect, recover, or choose a permitted branch.
- The execution path: reaching the goal requires a different sequence of interactions.
- The interpretation of the result: an apparent completion needs further confirmation, or a limitation must be distinguished from failure.
This gives each scenario a purpose. A collection of unusual runs is not automatically a useful dataset. The variation should answer a defined question about behavior.
Preserve the Connection, Not Just the Event
A compact example shows the relationship worth retaining:
Task: Save the selected item to a list
Action: Tap Save
Immediate feedback: Confirmation has not appeared
Agent response: Wait, then inspect the destination list
Verified outcome: The intended item is present
Remaining uncertainty: The cause of delayed feedback was not established
This is an example of evidence relationships, not a complete recording schema. It preserves enough context to understand the decision without claiming more than was observed.
Physical Execution Adds Evidence Within a Defined Scope
These scenarios are not exclusive to physical devices. The Android Emulator supports controls for conditions such as network characteristics, location, and battery state, and controlled environments can be useful for studying specific variations. Android Emulator controls
A physical run becomes relevant when the question requires evidence from the target hardware and its configured execution environment. It shows what happened under those particular conditions; it does not prove behavior across every Android device or every real-world setting.
For the same reason, a device in a controlled facility does not automatically reproduce a person's everyday mobile experience. The collection scope still needs to identify which conditions were present and which were outside the experiment.
ARMARRAY's role in this discussion is the physical Android execution layer. The usefulness of the resulting data still depends on the collection workflow, the observations it can actually capture, and the way outcomes are checked.
From Execution Behavior to Usable Data
The choice of data source determines what an experiment can observe. Within that environment, behavior reveals how a task progresses when conditions change.
Useful collection connects those conditions to decisions, paths, and outcomes. It preserves uncertainty where the cause is unknown and verifies results where appearance alone is insufficient.
That leads to the next question: if an automation system already produces logs, what is still missing before those records can support AI training or evaluation?
The next article examines the gap between UI automation logs and usable AI data.