A mobile agent taps Submit in an Android app. The screen keeps spinning, then reports a timeout. A few seconds later, the device is connected again. Did the task fail? Did the server accept the submission while its response was lost? Should the agent try again?
Those questions cannot be answered from the network profile alone. A device may have been prepared for Wi-Fi or cellular access, but its effective connection can change during the attempt. The app's request, the UI feedback, and the resulting record may each tell a different part of the story.
The useful distinction is between configured conditions—the network path and controls a team planned—and observed conditions—the connectivity and application events that actually occurred. Both belong in a run record, but they answer different questions.
A configured network is a test input
Before an attempt, a team can specify the connection it intends to use: a particular Wi-Fi access point, a cellular subscription, a network policy, or a controlled delay or interruption. It can verify that the device starts on the intended path. These choices make the scenario reproducible enough to investigate.
They do not guarantee that every request will experience the planned condition. Wi-Fi may lose its place as the default network; a cellular path may become available; a connection may remain nominally present while the target service is unreachable; and request time can vary after the test begins. A configured delay is also not a measurement of every request's observed latency.
Android's network-state documentation makes an important distinction: NET_CAPABILITY_INTERNET says a network is configured for internet access, while NET_CAPABILITY_VALIDATED reflects an Android connectivity probe. Even a validated network does not prove that a particular app endpoint was reachable at the moment a request was made. Network capabilities and the default transport can change during an app's lifetime.
For a run that depends on connectivity, record the intended profile and its revision separately from the starting check. Then preserve the observations available during the attempt. Do not turn a setup label such as “cellular run” into a claim that every consequential request used cellular service.
Follow one submission across the change
Consider a mobile agent booking a service appointment in an Android app. The completion criterion is one confirmed appointment for the intended customer and time slot. The team intends to run on a specified Wi-Fi network with cellular available as a fallback.
An illustrative attempt might unfold this way:
- The pre-run check records the app build, booking fixture, intended Wi-Fi profile, and the observed default network for the app.
- The agent taps Submit. The app shows a submitting state. If request instrumentation is available, it records that a submission request started; no completion has been confirmed.
- Android changes the default network for the app from Wi-Fi to cellular. The app later reports a request timeout.
- The agent sees the timeout and considers submitting again. Before that action, the workflow checks whether an appointment was created, if such a check is available.
- A final result is assigned using the record and UI evidence, with any remaining uncertainty retained.
The third step does not mean an in-flight request moved to the new network. Existing connections on the previous default network may then be terminated, but the request could already have reached the service. The timeout alone cannot tell whether the server processed it. Another Submit action could create a duplicate unless the application or service makes repetition safe. HTTP Semantics, RFC 9110 explains why a client should not automatically retry a non-idempotent request without a way to know it is safe or that the original request was not applied.
This is a network-sensitive interpretation problem, not simply a question of whether the agent tapped the right button. The appointment may exist even when the app shows a timeout; conversely, a restored connection does not establish that the booking was saved. Where the backend result cannot be checked, the attempt should retain that uncertainty rather than be flattened into an unqualified success or failure.
Observe the device, the app, and the result separately
Different observations support different conclusions. A useful record can link four layers to one task attempt:
- Intended condition: the selected network path or test profile, including when a planned change should occur.
- Device observation: default-network or capability changes and their times, when those signals can be collected. For Android's default-network callback,
onLost()means a network lost default status; it does not necessarily mean the radio disconnected. - Application observation: request start, timeout or response where instrumented, plus the UI state and the agent's next action. A spinner alone does not identify the cause of delay.
- Outcome evidence: the application or service state that can establish whether the intended appointment exists, and whether it exists once or more than once.
No single layer substitutes for the others. A transport label is not a latency measurement. An Android validation signal is not an application response. A UI error does not prove the service did nothing. An eventual success screen may not explain which submission produced the result.
The capture method depends on the test environment. An instrumented app may expose request timings and errors; a black-box app may provide only visible state and whatever device or service observations the workflow is authorized to collect. The article's method does not require pretending that every run has packet-level visibility. It requires naming which observations are available and limiting the conclusion to them.
Design a network comparison that answers one question
“Test under a bad network” is too broad to interpret. Start with a specific question: can the agent complete the booking if the connection drops after it taps Submit? Or does a controlled delay change when the agent decides to retry? These are different tests.
For each comparison, state the task and fixture, the network condition intended at the start, the change to introduce, and the point in the task when it should happen. Keep the app build, account, booking data, and completion rule stable where the question requires it. This applies the baseline-and-variation distinction from When Are Two Mobile AI Runs Comparable? to a changing network. Record whether the intended network change actually occurred and how the app behaved around it.
A delayed response, a lost default network, an app-level timeout, and a service failure are not interchangeable labels. If two runs diverge, compare their event sequences before attributing the difference to “network quality.” When an attempt leaves the final service state unknown, keep it visible as an unresolved attempt; a later retry should have its own action history and should not erase the original evidence.
This also gives a practical rule for data collection. Preserve the configured condition, observed connection events, app feedback, agent decisions, and result check with aligned timestamps and the same attempt ID. The record can then show whether a network change exposed a useful recovery behavior, invalidated a controlled comparison, or left an outcome unverified. The deeper questions of outcome classification and safe retry policy can be handled separately; here the goal is to avoid mistaking a network label for an execution history.
What physical execution contributes
Controlled network conditions are useful for isolating a mechanism. A real Android device adds evidence from the particular hardware, radio and connection arrangement under which the task ran. Neither setting automatically represents every field condition. The claim should identify the device and network context actually exercised, not imply a universal “real-world” result.
ARMARRAY provides real Android infrastructure for Mobile AI data collection and evaluation. It can be the physical execution layer for a customer's network-sensitive task and evidence workflow. The customer can define the network question, choose the observations it can collect, and interpret the result against its own completion rule.
A configured network tells us what we tried to test. The observed connection, app behavior, and outcome evidence tell us what happened. Keeping those records distinct is what makes a network-affected run useful rather than merely unpredictable.