A workflow that succeeds on a small study area may encounter different constraints when the number of observations, locations, or users increases. Scaling requires attention to processing capacity, data organisation, reference coverage, and the effort needed to review results. This perspective looks beyond whether a calculation can run on a larger dataset. It asks whether the complete service can preserve its meaning, quality, and delivery commitments as its scope expands. The objective is a design that identifies bottlenecks early and increases throughput without silently weakening the evidence available to users.
This working guide considers designing reproducible cloud processing workflows within cloud-based analysis teams. The immediate decision is to make analytical results reproducible. Useful reference evidence includes input versions, processing parameters and independent product checks. The case is an illustrative planning context, with assumptions to check against the actual sensor, landscape and project requirements.
The setting: Cloud-based analysis teams
A cloud-based analysis team needs to connect a scientific method with repeatable processing and an interpretable result. The same nominal dataset can produce different outputs when input versions, masks, resampling choices or analytical parameters differ. Begin by defining the product and the decisions it is intended to support. A visually successful demonstration is useful, but it does not establish that the workflow will behave consistently across regions, seasons or repeated processing runs.
The processing record should identify input observations, acquisition times, quality screening, software or method versions and the parameters relevant to interpretation. Preserve missing-data conditions and distinguish an analytical result from a task that merely completed without a technical error. For field-level products, boundary versions and the spatial support used for summaries also matter. These details make it possible to explain why two apparently similar runs produced different maps or numerical values.
Validation should include checks of both the analytical claim and the delivery process. Compare representative outputs with suitable reference observations, and test whether another operator can reproduce the result from the recorded inputs. Retain limitations in the product metadata rather than relying on informal knowledge held by one team member. A dependable cloud service makes a chain of evidence accessible, allowing users to understand what was observed, how it was processed and which conclusions the available validation supports.
Move the computation with a clear plan
Cloud processing can place analysis near a large satellite archive and reduce repeated transfers to a local computer. Its value depends on the workload. A small study may be easier to complete with a few downloaded scenes, while a continental time series benefits from selective reads and distributed processing. Begin with the geographic extent, dates, variables and output resolution that the analysis actually needs.
Inspect the storage format before choosing an execution pattern. Tiled imagery can support reading only intersecting blocks, and chunked multidimensional arrays can support selected time and spatial slices. These benefits depend on how data are organized and how the reader accesses them. Repeatedly opening complete remote files for tiny requests can waste time and bandwidth. Match processing chunks to source layout where practical, and avoid creating more small tasks than the scheduler can manage efficiently.
Make each run explainable and recoverable
Keep source identifiers, processing versions, input checksums where available, parameters and software dependencies with the workflow. Explicitly record projections, resampling methods, scale factors, masks and time aggregation rules. An output filename alone cannot capture these decisions. A small manifest linking inputs to outputs is useful when an archive is reprocessed or a result must be reproduced months later.
Develop on a representative subset that includes awkward cases such as missing bands, empty intersections and scenes with extensive cloud. Check numerical results and output metadata before expanding the workload. Design individual tasks so they can safely restart without duplicating completed results. Write intermediate output to a temporary location, verify it, and only then mark the task complete. Retain useful logs without recording access credentials.
Estimate storage, compute and data transfer costs together. Compression, unnecessary copies and repeated intermediate materialization can change the budget substantially. Monitor resource use during a pilot run and establish sensible limits before launching a large batch. Permissions should give each process the access it needs, and credentials should come from an appropriate secure mechanism. Finally, export an accessible result with provenance and quality information. A fast calculation becomes useful science only when another person can understand and inspect what was produced.
Represent gaps without inventing observations
A monitoring sequence often contains periods without a usable observation. The reason may be cloud, shadow, missing coverage, a failed transfer, or a quality condition that excludes the data. Begin by representing these states explicitly. A missing observation is different from a low measurement, an unchanged surface, or a confirmed absence of the feature being studied. The distinction should survive both the analysis and the presentation of the result.
Define when a temporal gap is acceptable for the intended decision. A workflow may continue using the latest qualified observation, request another source, or suspend an alert until sufficient evidence is available. Whichever response is selected, show the age of the supporting observation. If interpolation is used for a suitable analytical purpose, preserve the distinction between an estimated value and a directly observed measurement, together with the assumptions needed to interpret it.
Review how gaps affect the coverage of the final evaluation. Locations with frequent missing data may differ systematically from those with regular observations. A performance estimate based only on clear, complete records can therefore describe a narrower population than the service is expected to support. Report usable coverage alongside analytical results and use recurring gaps to guide acquisition planning, reference collection, and the selection of complementary observation methods.
Design an evaluation that reflects deployment
A model evaluation should resemble the conditions in which the model will be used. Start by identifying which relationships the development data share: nearby locations, repeated observations of the same unit, common collection teams, or measurements from a single period. These relationships can make records less independent than they appear. An evaluation that ignores them may test the ability to recognise familiar conditions rather than the ability to operate in a new setting.
Choose a split strategy that matches the intended transfer. Holding out locations, seasons, operational units, or acquisition conditions can answer different questions. Document which question each split is designed to address, and keep the held-out evidence separate while selecting features and tuning the method. Repeatedly inspecting the same evaluation set to improve a model gradually makes that set part of development, even if it retains its original label.
Report results at a scale that is meaningful to the user. A strong average can hide poor performance for important subgroups or unusual conditions. Examine error patterns and retain examples of both successful and unsuccessful predictions. Where the evidence covers only a limited range of conditions, describe that range explicitly. The evaluation then becomes a practical guide to suitable use, further testing, and the collection of additional reference observations.
Bring the evidence back to the decision
Expansion should be guided by measured workload, visible quality conditions, and an understanding of which tasks require expert attention. Test the workflow under representative data volumes before treating a successful demonstration as evidence of operational capacity. Preserve a route for investigating individual observations even when most processing is automated. A scalable service keeps its results traceable and useful as it grows, with clear choices about where to automate, where to review, and where to invest in additional evidence.
Sources & further reading
Explore the bhumisat spacecraft concept and our vision towards 2030, or return to the journal.



