How Does Visual Servoing Guide a Drone?

• Camera feedback image
• Image error to reduce
• Drone receiving a correction

What you will learn

• Trace the cycle from image to feature error to drone velocity.
• Understand how an interaction matrix maps image motion to camera motion.
• Compare image-space control with control based on estimated position.

Before you start

It helps to know that an image is a two-dimensional projection of the scene; without depth information, one pixel can correspond to multiple positions in space.

How does visual servoing close the loop with images?

Each camera frame is processed to find one or more features, such as the centre of a target landing region. The controller has a desired feature, calculates the difference between current and desired positions, and computes a camera motion that should reduce the error. After the drone moves, a new image yields new features and the loop is calculated again. The image is both a measurement of task state and feedback on the previous action.

An object detector only answers “where does an object appear in this image?” Visual servoing needs another step to use that information to choose velocity. If a detector publishes a bounding box but the controller expects its centre and size, the components cannot be joined correctly even if both refer to the same target. Feature definitions, image coordinates and timestamps are part of the control interface.

How does an interaction matrix connect image error to velocity?

An interaction matrix, also called an image Jacobian, approximates how image features change as the camera moves. Let the current feature vector be s, the target be s*, and the error be e = s - s*. In a region where the model is reasonable, the matrix relates feature change rate s_dot to camera velocity v. A feedback law chooses v to shrink e; a matrix pseudoinverse can map a desired feature-change rate to camera velocity.

This is not a pixel-to-metre conversion that works everywhere. The matrix depends on feature choice, camera geometry, depth or depth assumptions, and relative target-camera state. If a camera motion direction barely changes the features in a distinguishable way, the problem becomes ill-conditioned: the computed velocity may become very large or unstable. A real controller needs velocity limits and checks on rank, noise and confidence before issuing commands.

In a study of landing on a moving vehicle, the authors use image features and an interaction matrix to calculate quadrotor velocity and send it to a lower-level controller. The study chooses a constant matrix computed at the target to reduce dependence on UAV position-estimation error, while still requiring camera configuration and target depth information. This is a design choice in that paper, not a general rule for all visual servoing.

How does image-space control differ from world-coordinate control?

This comparison concerns the main feedback variable; it does not mean one method is always better. If the task is to keep a target in an image region, direct image feedback may be enough. If the drone must reach a point at a specified distance or avoid objects in space, image centre alone is usually insufficient. The methods can be combined: image features refine local alignment, while position estimates and a planner constrain motion in space.

How does one image update become a flight command?

Proposed illustrative design; not a test result. In simulation, a camera sees a target marker. A tracker publishes feature coordinates with capture time; a servoing node calculates e, uses the interaction matrix to produce desired velocity in the camera frame, then a frame transform converts it into drone-body velocity. An adapter sends a velocity setpoint to the flight stack. After the virtual vehicle moves, the image processor takes the next frame and recalculates.

Success means image error shrinks steadily while velocity remains inside model limits, timestamps are valid and the target remains identified. Failure cases include lost tracking, a feature switching to another object, an old frame arriving after a newer one, camera vibration or missing depth required by the model. On invalid timing or low confidence, the node should stop updating the approach command and report an error to the coordinator; a stop or cancellation request still has to be handled according to the flight-stack interface.

What can destabilize the loop or lose the target?

If tracking loses the target, image error no longer represents the original target. Delay means a command is computed from an old frame; the drone can continue moving while the target has already shifted. Vibration makes features jump, while lighting changes or occlusion can make detection switch objects. The system can also saturate: requested velocity exceeds limits or the matrix does not provide enough information to solve the task. The last image value should not be treated as though it were a current observation.

Common misunderstandings

Image-space control means no model is needed. It still needs a relationship between camera motion and image features, plus calibration and depth assumptions. Putting a target in the image centre always takes the drone to the right distance. Image centre describes direction; distance and depth need additional information.

Frequently asked questions

Is an interaction matrix a neural network?

No. It is a matrix model of how image features respond to camera velocity; it can be derived from geometry or estimated.

Can control use only the target centre in an image?

It can support simple directional alignment, but the centre alone usually does not give distance, depth or obstacles along the route.

Why use a new frame after every command?

Real motion, noise and delay make earlier predictions less accurate; new feedback lets the controller recalculate error.

Should the previous command continue if tracking is lost?

Old features should not be treated as a current position. The system should mark the data invalid and pass failure handling to the coordinator.

Summary

Visual servoing repeats feature detection, image-error calculation, mapping that error to camera or drone velocity, and measurement again. The interaction matrix is the mapping step; tracking, depth, calibration, delay and coordinate frames determine its quality. Image-based and world-position-based control use different feedback variables and suit different goals.