A tap by itself tells us very little.
We may know that an agent tapped a particular coordinate, pressed a button, entered text, or performed a swipe.
But an action record alone does not necessarily tell us what the interaction meant.
What was visible when the action was chosen?
What information was available to the agent?
And what happened after the action was performed?
For Mobile AI, these relationships matter.
An action becomes more meaningful when we connect what the agent observed before it with what happened after it.
This gives us a simple interaction unit:
Observation → Action → Outcome
Observation provides context for the action.
Action represents what the agent did.
Outcome provides evidence of what happened next.
Together, they describe more than an isolated event. They describe a relationship inside the interaction.
An Action Alone Has Limited Meaning
Consider a simple action record:
Tap
By itself, this tells us almost nothing about the interaction.
Add a target:
Tap “Submit”
Now we know more.
But important context is still missing.
Was the form complete when the agent tapped Submit?
Was a required field empty?
Was the button active?
Was the interface already showing an error?
And after the tap, did the application move forward, display a validation message, remain unchanged, or enter another state?
The action itself may be identical in all of these situations.
The interaction is not.
This is why Mobile AI data becomes more useful when an action is connected to the state that preceded it and the state that followed it.
Observation Provides Context
Before an agent acts, it has some representation of the current environment.
At the conceptual level, we can call this an observation.
The observation tells us what information was available when the action was chosen.
For example, an agent may observe:
a search interface,
a product page,
a form waiting for input,
a confirmation dialog,
or an unexpected application state.
The exact representation of an observation can vary between systems.
For this discussion, the important point is simpler:
the observation provides context.
Without that context, the same action record can be difficult to interpret.
A tap is not meaningful only because a coordinate was touched.
Its meaning depends partly on what existed at that coordinate, what state the application was in, and what the agent was responding to.
Action Connects Decision to Execution
An action is where an agent moves from observing the environment to interacting with it.
The action may be a tap, text input, swipe, back navigation, or another supported interaction.
But the important relationship is not the action type by itself.
It is the connection:
Observation → Action
The agent encounters a state and performs an action in response to that state.
Once the action is executed, the environment may change.
That gives us the next relationship:
Action → Outcome
Together:
Observation → Action → Outcome
The action sits between what the agent encountered and what happened afterward.
Outcome Tells Us What Happened Next
In this context, “outcome” does not necessarily mean the final result of the task.
It means the observable consequence that follows an action.
Suppose an agent sees a login screen and taps “Sign In.”
The immediate outcome might be:
a loading indicator appears,
an error message is displayed,
the dashboard opens,
or no visible change occurs.
Each is an outcome of that interaction.
The overall task may continue for many more steps before reaching a final success, failure, or another terminal state.
This distinction matters because Mobile AI interaction data exists at more than one level.
A task can have a final outcome.
But individual actions also produce intermediate consequences throughout the execution.
Those intermediate outcomes help connect one moment of interaction to the next.
One Interaction Unit
We can now describe a useful interaction unit with three questions:
What did the agent observe?
What did the agent do?
What happened next?
Or, more compactly:
Observation → Action → Outcome
This unit does not require us to judge whether the action was correct.
It does not require a score.
And it does not require us to decide whether the overall task succeeded.
It simply preserves a meaningful relationship between the state before an action, the action itself, and the state that followed.
That relationship can then become part of a longer interaction history.
From Interaction Units to a Trajectory
In the previous article, we described a mobile interaction trajectory as a time-ordered record of how observations, actions, and state changes unfold during a task.
The Observation → Action → Outcome relationship gives us a closer view of what happens inside that trajectory.
Consider:
O₁ → A₁ → O₂ → A₂ → O₃ → A₃ → O₄
Each action connects two moments in the interaction.
O₁ provides context for A₁.
A₁ contributes to what becomes observable at O₂.
O₂ then provides context for A₂.
The process continues.
From one perspective, O₂ is the outcome following A₁.
From the next perspective, O₂ is the observation that provides context for A₂.
This is how individual interaction units connect into a sequence.
A trajectory is therefore not only ordered in time.
It is built from relationships between what came before an action and what came after it.
Same Action, Different Context
Now consider two executions that contain exactly the same action:
Tap “Continue”
In the first execution:
Observation:
The form is complete.
Action:
Tap “Continue.”
Outcome:
A confirmation screen appears.
In the second execution:
Observation:
A required field is missing.
Action:
Tap “Continue.”
Outcome:
A validation error appears.
The action record is the same.
The surrounding interaction is different.
This is why an action cannot always be understood in isolation.
What came before the action changes its context.
What came after the action provides evidence of its consequence.
The same action can therefore participate in very different interaction histories.
From Interaction Records to Useful Mobile AI Data
Mobile AI systems generate many individual records during execution.
Observations capture states.
Actions capture interactions.
Outcomes capture what happened afterward.
The value comes not only from collecting these records, but also from preserving the relationships between them.
This applies across different execution environments.
Synthetic environments can produce observation-action relationships.
Emulators can produce them.
Physical Android devices can produce them while adding real-device execution context when that context matters to the task.
A real device does not automatically make interaction data structured or useful.
The relevant observations, actions, outcomes, and their relationships still need to be preserved.
For real-device Mobile AI workflows, the physical device is the execution environment in which these interaction cycles occur.
ARMARRAY is building Real Android infrastructure for Mobile AI Data Collection & Evaluation, providing the physical execution layer where these observation-action-outcome cycles can occur on real Android devices.
The underlying data principle remains simple:
what the agent observed,
what it did,
and what happened next
belong together.
Next: Why a Screenshot Alone Is Not Enough
A screenshot can preserve an observation.
But an interaction contains more than a single observed state.
Once we care about the relationship between observation, action, and outcome, an important question follows:
What do we lose when we preserve only the screenshot?
In the next article, we’ll look at why a screenshot alone is not enough for many Mobile AI training and evaluation workflows.