What Must You Optimize to Run a Detector on Edge Hardware?

• On-device camera
• Local inference chip
• Cloud outside this processing step

Start with the application's limits

Before changing a model, decide what its answer will be used for. If an application only needs to detect cars in individual images, a late frame may still be useful. If it must track a moving object, a result that arrives too late may describe an outdated position. Mark when a frame is handed to the program and when its result is ready for the application; that interval is the inference latency of the chosen pipeline, not necessarily the full time from camera exposure.

Choose quality criteria first: for example, which object groups must be detected, which missed detections need to be reduced, and which errors cannot be accepted. There is no universal threshold for every task. Record the model version, input image size, device configuration, measurement method and test images so later comparisons use the same conditions.

Establish a baseline, then change one factor

Run the initial configuration on the same representative image set or video. Measure inference time, frame rate for continuous processing, peak RAM and energy use over the same observation interval. Record quality against known labels, especially in difficult scenes. FPS is the number of frames processed per second; by itself it does not say how long one frame took, so report per-frame time and how overall throughput was measured.

Then try reversible changes, one at a time:

• Reduce input image size. Fewer pixels usually mean less computation, but small objects lose detail. Compare recognition rates by object-size group, not only runtime.
• Choose a smaller model. A compact model may need less memory and computation, but may make more mistakes. Accept it only if the test set still meets the application's requirements.
• Quantise. A more compact number representation may reduce compute or storage cost on supported hardware. Numerical changes can alter outputs, so measure quality again on the actual runtime and device.
• Use an accelerator. A GPU or AI processor helps only when the model and runtime can use it; copying data between the CPU and accelerator also costs time. Compare the full pipeline rather than only kernel time.

The LiteRT quantization guide provides a concrete example: full integer quantization needs representative inputs to calibrate changing tensor ranges. A model that converts successfully still needs quality checks; a calibration set that does not resemble deployment inputs may give a misleading evaluation. Keep a non-quantized baseline for comparison. The performance guide also recommends profiling operators to identify where computation time is spent. These are methods to test, not guaranteed improvements for every board.

An illustrative optimisation pass

Suppose a camera feeds frames to an edge computer and the application displays detection boxes. The team records a baseline on a fixed video, then reduces input resolution. Processing time falls, but small objects start to disappear; the team keeps the original size or tries a compact model instead of accepting those losses. Next, it tries quantisation and reruns the same video with the same configuration. If speed improves but errors rise in dim scenes, the change has not met the chosen criteria.

This is a proposed workflow, not a result measured by the sources. ML Kit documentation says images are scaled and may be stretched to fit model input, which shows that preprocessing is part of the quality question. The NVIDIA Jetson guide introduces developer kits for AI and robotics applications but does not provide one score that applies to every model.

When should optimisation stop?

Choose a configuration that stays within all three limits: required quality, response time and hardware budget. FPS without quality, RAM, energy use or a measurement boundary is not enough to call a system optimised. Temperature, concurrent load and lighting changes should also be checked in the intended scenario. Results on one board do not automatically apply to another computer.

Summary

• Set quality targets and a latency boundary before optimising.
• Measure a baseline, change one factor, then compare quality, time, memory and energy.
• Smaller images, compact models and quantisation can all change speed and recognition quality.