A real-device proof of concept is easy to underscope. A team may connect a phone, run one task, see a plausible result, and call the integration validated. That demonstrates that a command can reach a device. It does not yet show that a useful data workflow can be repeated, observed, evaluated, and recovered when something goes wrong.
A better small POC answers a bounded decision: Can this team run one representative mobile task on selected physical Android devices and produce evidence that another person can inspect and use to make the same outcome judgment?
That question is deliberately narrower than “Does the agent work?” A POC can validate a workflow slice and expose integration gaps. It cannot, by itself, establish broad task success, statistical reliability, or coverage across devices and applications.
Begin with the decision the POC must support
Before choosing devices or writing automation, state what decision will follow the POC. For example: proceed to a larger evaluation, revise the capture design, add a recovery step, or stop because a required part of the workflow is unavailable.
Then write one testable hypothesis. A useful example is: “For this selected task and app build, the runner can start from a known state, execute the requested interaction on a physical device, retain the resulting observations, and determine whether a fixed completion condition was met.”
This hypothesis has two separate concerns:
- Workflow readiness: Can the system coordinate the task, device, execution, evidence, and evaluation without losing their relationship?
- Agent quality: Does the agent complete the task accurately across realistic variation?
A small POC can provide evidence about the first concern and an initial example for the second. It should not collapse them into one pass/fail claim. An agent may fail a task while the workflow correctly records the attempt and its evidence. Conversely, a task may appear successful while the workflow has failed to preserve enough evidence to support that conclusion.
Bound one representative slice
Keep the first slice narrow enough that a failure can be diagnosed. Record the task instruction, app and build, account or test fixture, device configuration, starting condition, expected end condition, and what observations will be captured. Define what is explicitly outside the POC—for example, other apps, locales, device families, long-duration soak testing, or a broad agent benchmark.
Choose the slice because it exercises a meaningful path through the intended workflow, not because it is the easiest demo. A form submission, for example, might require the runner to reach a screen, enter fields, submit, observe the resulting state, and check a completion condition. The POC need not test every form or every device. It does need to make clear which part of that path it is validating.
When the question includes device or environment coverage, define the matrix explicitly. A matrix might vary device model, operating-system version, app build, orientation, or locale. Firebase Test Lab’s Android documentation uses dimensions such as device model, Android version, orientation, and locale to define test matrices; the useful lesson is to name the dimensions under test rather than describe a run as “representative” without saying what that means. Firebase Test Lab: Get started with Android testing
For a workflow POC, begin with a controlled baseline and add only the variation needed to answer the decision. One selected device and one fixed configuration can be appropriate if the open question is end-to-end orchestration. It is insufficient if the decision depends on comparing device or environment behavior.
Validate the full chain: collect, execute, record, evaluate
Treat the POC as four connected stages. Each stage should leave evidence that lets the next stage proceed and lets a reviewer understand what happened.
Collect. Confirm that the run has an identifiable task, the intended device and software context, and a known starting condition. Record the configuration that materially affects interpretation. If a device is unavailable or the starting state is wrong, surface that before treating the run as a valid attempt.
Execute. Issue the planned interaction and observe the device-side result. Separate “request accepted,” “action dispatched,” and “expected state observed” as distinct events. A successful HTTP response can confirm that a service accepted a request; it does not alone prove that the device completed the interaction.
Record. Preserve the evidence needed to interpret the attempt: relevant screenshots or state snapshots, action records and resulting observations, timestamps, task and attempt identifiers, device context, and any error or recovery event. Keep identifiers at the right level. A task can contain multiple attempts; an attempt can contain multiple actions; a device identifier describes the execution target. Linking these records makes partial completion visible instead of flattening it into a single ambiguous status.
Evaluate. Apply a completion condition written before the run. State what evidence is sufficient, what constitutes failure, and when the result is inconclusive. The evaluator should be able to point from its judgment back to the observations on which it depends.
The chain is only as useful as its joins. A POC reviewer should be able to follow one run from the initial task through dispatch, device observation, artifact storage, and final assessment. Distributed tracing offers a familiar pattern for correlating work across service boundaries: propagate a shared trace context and retain spans for the operations that form the request path. OpenTelemetry describes traces as a way to follow a request through services and spans as the individual operations within that trace. OpenTelemetry: Traces
The same principle applies even if the POC uses a simpler event log rather than a tracing backend: identifiers and timestamps must connect the stages. Avoid inventing one universal status field that hides whether a task was queued, an action was attempted, evidence was captured, or an outcome was assessed.
Include one controlled interruption
A happy-path demo rarely tests the hardest integration question. Include one realistic interruption that matters to the proposed workflow: a delayed response, a temporary connection loss, an incomplete observation, or a device that begins outside the expected state.
The goal is not to manufacture failure. It is to see whether the system preserves what is known and makes uncertainty visible. A network timeout does not prove that a device action never ran. Before retrying, the workflow should inspect available state or evidence where possible, then decide whether to continue, recover, create a new attempt, or mark the result inconclusive. It should not silently overwrite the first attempt or treat an ambiguous response as a confirmed failure.
For example, if a submit action times out after dispatch, retain the original attempt and its request context. Inspect the app state and available observations. If the completion condition is visible, evaluate that evidence. If it is not, record the uncertainty and apply a defined recovery or retry policy. This small test reveals whether retry behavior is deliberate and auditable rather than an accidental second execution.
Set acceptance gates before running
Agree on pass conditions before the POC begins. The thresholds depend on the customer’s task and operating needs; a generic article should not invent a universal success percentage or latency target. The following gates provide a practical starting point:
| Stage | Evidence to inspect | Example acceptance condition |
|---|---|---|
| Collect | Task definition, device/software context, starting state | The intended task and execution context are identifiable before dispatch. |
| Execute | Request, dispatch record, observed app state | The reviewer can distinguish acceptance of a request from an observed device result. |
| Record | Attempt-linked observations, artifacts, timestamps | Required evidence is present and linked to the correct task and attempt. |
| Evaluate | Predefined completion condition and supporting evidence | A second reviewer can reproduce the outcome judgment from the retained evidence. |
| Recover | Interruption record, retry or recovery decision | The original attempt remains inspectable; uncertainty is not silently converted to success or failure. |
For every gate, define three outcomes: pass, fail, and inconclusive. “Inconclusive” is important. If the app state cannot be observed or the required artifact is missing, the POC may not show that the agent failed; it may show that the evidence path is insufficient. Naming that distinction points to the right next change.
Keep the conclusion proportional to the evidence
A small POC can establish that a selected workflow slice is implementable and inspectable under stated conditions. It can identify missing joins, unclear ownership, capture gaps, recovery behavior, and evaluation ambiguity. It can also show whether the team can repeat the run with the same configuration.
It cannot establish broad reliability from a handful of executions. Nor does one device prove fleet coverage, one app flow prove cross-app performance, or a successful integration prove that the agent generalizes. If statistical confidence or coverage is the next decision, plan a larger evaluation with defined cohorts, repeated trials, and explicit variation dimensions. Keep that separate from the initial workflow-readiness gate.
At the end, choose a next action tied to the observed gap:
- Proceed when the required stages pass and reviewers can inspect the same evidence chain.
- Adjust when the workflow is viable but a bounded gap remains, such as an unlinked artifact or unclear retry rule.
- Re-scope when the selected task did not exercise the decision the team actually needs to make.
- Stop when a required stage cannot be performed or the evidence cannot support the intended judgment.
Record the tested configuration, acceptance results, unresolved questions, and the next owner. That makes the POC a decision instrument rather than a polished demo.
A practical POC is small in scope and complete in evidence
The most useful first POC is not the one with the most devices or the most elaborate dashboard. It is the smallest run that tests the relevant operational path end to end and retains enough evidence to explain its result. Start with a clear decision, bound the task and conditions, connect collection to evaluation, include one meaningful interruption, and agree on acceptance gates in advance.
That gives a team more than a successful demo: a grounded answer about what is ready, what remains uncertain, and what the next investment should test. Real-device infrastructure is most useful when it supports more than device access: it should help teams connect execution, observations, and evaluation into a workflow they can inspect, repeat, and improve.