Large Dataset Performance Benchmarking in Agisoft Metashape

A benchmark should answer a precise question rather than generate an isolated time.

• State whether the test compares hardware, settings, software versions or architectures.
• Identify the single primary variable intended to change.
• Define primary metrics such as stage duration, peak RAM, VRAM, storage growth or energy where available.
• Define output-quality and accuracy criteria that must remain satisfied.
• State the hypothesis before the run.
• Define repetition count and treatment of failed or interrupted runs.
• Identify the audience and the conclusions the test is not designed to support.
• Register the test identifier, author, reviewer and date.

Benchmark record: research question, controlled variable, metrics, acceptance criteria and exclusions.

The dataset determines how widely the result can be interpreted.

• Record image count, dimensions, megapixels, bit depth, format and total volume.
• Describe aerial, corridor, oblique, terrestrial, close-range or mixed capture geometry.
• Record cameras, lenses, sensors, flights, sessions and metadata.
• Describe scene complexity, texture, vegetation, reflections, movement and elevation range.
• List GCPs, checkpoints, masks, laser scans or other inputs.
• State why the dataset represents the intended production workload.
• Separate “large by image count” from “large by resolution, geometry or output complexity”.
• Confirm permission to use and, if applicable, publish information about the dataset.

Benchmark record: dataset profile, permitted use and explanation of representativeness.

Every compared run must begin from the same controlled state.

• Create a read-only or otherwise controlled copy of source files.
• Record file counts, folder structure and integrity checks where required.
• Preserve metadata and reference files without conversion between runs.
• Freeze masks, calibration groups, camera status and marker projections.
• Define the project state from which each stage begins.
• State whether intermediate data are rebuilt or reused.
• Clear or control cached results when they could affect timing.
• Use consistent region, chunks and enabled items.

Benchmark record: immutable input manifest and restorable project baseline.

A component name alone is not a complete test environment.

• Record Metashape edition, version and build.
• Record operating system, updates and power profile.
• Record CPU model, core configuration, memory capacity and memory configuration.
• Record GPU model, VRAM, enabled devices and driver version.
• Record active project storage, filesystem, free space and connection type.
• For network or cloud tests, record server, workers, shared storage, network and instance details.
• Record relevant firmware, BIOS and thermal-management settings.
• Identify monitoring tools and their sampling interval.

Benchmark record: environment manifest sufficient to understand and repeat the setup.

The settings actually executed—not the intended settings—must match the record.

• Record the exact order of processing stages.
• Record matching, alignment, optimization, depth-map and filtering parameters.
• Record point-cloud, mesh, DEM, orthomosaic, tiled-model and texture settings as applicable.
• Record camera calibration, reference accuracy and control/checkpoint state.
• Use the same batch configuration or script where practical.
• Record whether CPU and GPU devices are enabled for supported stages.
• Prevent unrecorded manual edits between comparable runs.
• Export or capture the parameter record with the result package.

Benchmark record: executable workflow specification and verified run parameters.

Background load, thermal state and storage conditions can distort the comparison.

• Close or record unrelated applications and scheduled tasks.
• Use a consistent power and performance profile.
• Record ambient and component temperatures where they may affect throttling.
• Define whether runs begin from a cold, warm or stabilized system state.
• Maintain comparable free storage and project location.
• Control network traffic and concurrent storage use for distributed tests.
• Record reboots, driver resets, failed tasks, retries and operator intervention.
• Avoid changing hardware configuration between repetitions unless it is the tested variable.

Benchmark record: pre-run checklist and execution-condition log.

Total project time can hide where a difference actually occurs.

• Record start, finish and elapsed time for every tested stage.
• Capture peak and representative system RAM use.
• Capture CPU utilization, frequency and temperature behaviour.
• Capture GPU utilization, VRAM allocation, clock and temperature behaviour.
• Record storage read/write activity, project growth and minimum free space.
• Record network throughput and worker utilization where applicable.
• Separate automated processing time from manual review or transfer time.
• Retain Metashape and system logs required to explain anomalies.

Benchmark record: stage-level telemetry and event log synchronized with each run.

One successful run may not represent typical behaviour.

• Perform the predefined number of comparable repetitions.
• Use the same reset or preparation procedure before each run.
• Report individual observations rather than only the best result.
• Calculate an appropriate summary and variability measure for the test design.
• Investigate outliers before deciding whether they are valid, failed or excluded.
• Record thermal drift, cache effects, background events and storage changes.
• Repeat both comparison conditions under equivalent rules.
• State when resource cost or duration prevented sufficient repetition.

Benchmark record: all repeated observations, variability and documented treatment of anomalies.

A performance gain is not comparable if the result is materially different.

• Compare aligned camera count, tie points and calibration state.
• Compare checkpoint or scale-bar evidence where relevant.
• Compare depth-map, point-cloud, mesh, DEM, orthomosaic or texture statistics.
• Inspect completeness, holes, noise, artefacts and spatial coverage.
• Confirm resolution, region, CRS and export settings are equivalent.
• Use identical acceptance criteria for all compared runs.
• Identify nondeterministic differences and assess whether they affect the conclusion.
• Reject a ranking when output equivalence cannot be demonstrated.

Benchmark record: output-equivalence assessment linked to every accepted run.

A reader should be able to understand exactly what the benchmark proves.

• Publish the research question and comparison variable.
• Describe the dataset without exposing restricted information.
• List software, hardware, drivers, settings and execution conditions.
• Report every valid repetition and summary method.
• Report failures, exclusions, anomalies and output-equivalence checks.
• Separate observed results from interpretation and recommendation.
• State that results apply to the tested conditions and may not scale linearly.
• Provide a review date and preserve the benchmark package.

Benchmark record: transparent report plus archived dataset manifest, workflow, logs and validation evidence.