A screenshot captures a moment.
A mobile task, however, unfolds over time.
An agent observes an interface, chooses an action, encounters a new state, observes again, and continues until the task reaches an outcome.
If we record only individual screenshots, we may preserve what the interface looked like at several points during execution.
But we may lose something equally important:
how those moments were connected.
For Mobile AI, the order of observations and actions is itself useful information.
This is where individual interaction records begin to form something more meaningful:
a mobile interaction trajectory.
A Screenshot Is a Moment, Not a Process
Screenshots are useful.
They can preserve what was visible on a screen at a particular moment and provide evidence of the interface state an agent encountered.
But a screenshot is still a snapshot.
Consider four screenshots captured during a mobile task:
Screen A → Screen B → Screen C → Screen D
Looking at them individually, we may be able to infer that the interface changed.
What we cannot necessarily determine is why.
Did the agent tap something between Screen A and Screen B?
Did it type a query?
Did the app transition automatically?
Did the agent navigate backward and try again?
Without the interactions connecting those observations, several moments from the same task can remain little more than a collection of isolated states.
The missing information is not another screenshot.
It is the relationship between the screenshots.
Mobile Interaction Data Has an Order
A Mobile AI task naturally produces a sequence.
At a simplified level:
Observation → Action → State Change → Observation → Action → State Change → …
Each new observation is influenced by what happened before it.
The agent sees a state.
It performs an action.
The environment responds.
A new state becomes available for observation.
That ordering matters.
Suppose we preserve all the observations and actions from a task but remove their sequence.
We might still know which screens appeared and which actions occurred, but it becomes much harder to reconstruct the execution itself.
Was the search query entered before or after the product page appeared?
Did the agent navigate back before retrying an action?
Which action produced a particular state change?
The individual records still exist.
Their temporal relationships do not.
From Individual Records to a Trajectory
This gives us a useful conceptual definition:
A mobile interaction trajectory is a time-ordered record of how a task unfolds through observations, actions, and resulting state changes.
The important idea here is not a particular file format or data schema.
It is structure.
A set of observations tells us what appeared during execution.
A set of actions tells us what the agent did.
A trajectory connects those records so that we can understand how one interaction led to the next.
The difference is subtle but important.
At the top, we have individual moments:
t0, t1, t2, … tn
At the bottom, those same moments are connected through actions and ordered over time.
The underlying observations may be the same.
What has been added is:
order + relationships.
That additional structure turns isolated interaction records into an interaction history.
One Task, Multiple Steps
Consider the same simple task from the previous discussion:
Find a product and add it to the cart.
The task may unfold like this:
Observation 1
The agent sees the home screen.
Action 1
Tap Search.
Observation 2
The search interface appears.
Action 2
Type the product name.
Observation 3
Search results appear.
Action 3
Select the product.
Observation 4
The product page appears.
Action 4
Tap “Add to Cart.”
Outcome
The item is added—or the execution reaches another resulting state.
Each observation and action is useful on its own.
But together, in order, they describe how the task actually unfolded.
That ordered history is much closer to the interaction itself than any individual screenshot or final result.
Why Sequence Matters
Preserving sequence gives Mobile AI interaction data several useful properties.
First, it supports reconstruction.
We can follow the execution from an earlier state to a later one and understand which interactions occurred between them.
Second, it supports comparison.
Two executions of the same task may contain similar observations and reach the same final outcome while following different paths.
Third, it supports diagnosis.
If an execution enters an unexpected state, an ordered history can help identify where its path began to differ from what was expected.
None of these requires us to decide whether one trajectory is inherently better than another.
The point is simpler:
sequence preserves information that isolated records cannot.
The Final Outcome Does Not Tell the Whole Story
Imagine two agents performing the same task.
Both eventually reach:
SUCCESS
If we record only the final outcome, the two executions appear identical.
But their interaction histories might look very different.
One may move directly through the expected sequence of states.
Another may encounter an unexpected state, navigate back, retry an action, and then reach the same result.
The outcome is the same.
The trajectories are not.
This does not automatically tell us that one execution is better.
It tells us that the final result alone does not preserve the path that produced it.
The same principle applies to failure.
Two failed tasks may stop at entirely different points for entirely different reasons.
If we want to understand the execution rather than only classify its ending, the path matters.
From Interaction History to Structured Mobile AI Data
This brings us back to Mobile AI data collection.
Screenshots can preserve observations.
Action records can preserve what the agent did.
State information can preserve what changed.
Outcome records can preserve how the task ended.
A trajectory provides the temporal structure connecting those pieces.
That distinction also helps clarify the role of execution environments.
Synthetic environments can produce interaction sequences.
Emulators can produce interaction sequences.
Physical devices can produce interaction sequences while adding real-device execution context when that context matters to the task.
A physical Android device does not automatically turn raw interactions into a trajectory.
The infrastructure still needs to preserve the relevant observations, actions, state changes, and their relationships over time.
For real-device workflows, this becomes increasingly important as execution scales.
Running one interaction manually is one problem.
Running repeated tasks across many physical Android environments while preserving consistent interaction histories is a different infrastructure problem.
ARMARRAY is building Real Android infrastructure for Mobile AI Data Collection & Evaluation, providing the physical execution layer on which these real-device workflows can run.
The trajectory itself, however, is a data concept:
an ordered history of interaction.
Next: Observation → Action → Outcome
We have now moved from individual screenshots and actions to the idea of a time-ordered interaction trajectory.
But there is a smaller relationship inside every trajectory that deserves closer attention:
what the agent observed, what it did next, and what happened as a result.
An observation provides the context for an action.
An action changes—or attempts to change—the environment.
The resulting state provides new evidence about what happened.
In the next article, we’ll look more closely at this Observation → Action → Outcome relationship and why it forms a useful unit for understanding Mobile AI interaction data.