aruco | Aruco marker detector and pose estimation | Robotics library

Intel® Distribution of OpenVINO™ Toolkit can also be installed through PyPI repository instead of installing the binary package that had compiled OpenCV community version.

• Runtime package with the Inference Engine inside.
• Developer package that includes the runtime package as a dependency, Model Optimizer, Accuracy Checker and Post-Training Optimization Tool.

Next, you can get the ArUco library that included in the opencv-contrib-python package which you can install via:

I managed to solve the problem myself. The problem was I was running a too old version of OpenCV. The version needs to be 3.4.x or above since it was a bug that got fixed in 3.4.

The problem was in OpenCV library. Maybe it was badly compiled, or the problem is in the library itself. I have used OpenCV version 4.5.3 before. Then I compiled 4.5.4 version, and the program started to work well.

The picture is a mirror image of the markers.

ArUco markers will not be decoded when they're mirrored.

Prevent/undo the mirroring. Find the device/driver setting that does this.

flipped = cv.flip(img, 1) flips the image around the Y-axis.

Rt is just the transformation (rotation, translation) from world to camera frame.

You expected a projection?

If you want to project the point, you still need to apply the camera matrix (and then maybe the distortion coefficients to be precise, but let's not get into that).

Without checking all the code (looks roughly okay), a few basics about OpenCV and aruco:

Both use right-handed coordinate systems. Thumb X, index Y, middle Z.

OpenCV uses X right, Y down, Z far, for screen/camera frames. Origin for screens and pictures is the top left corner. For cameras, the origin is the center of the pinhole model, which would be the center of the aperture. I can't comment on lenses or lens systems. Assume the lens center is the origin. That's probably close enough.

Aruco uses X right, Y far, Z up, if the marker is lying flat on a table. Origin is in the center of the marker. The top left corner of the marker is considered the "first" corner.

The marker can be considered to have its own coordinate system/frame.

The pose given by rvec and tvec is the pose of the marker in the camera frame. That means np.linalg.norm(tvec) gives you the direct distance from the camera to the marker's center. tvec's Z is just the component parallel to optical axis.

If the marker is in the right half of the picture ("half" defined by camera matrix's cx,cy), you'd expect tvec's X to grow. Lower half, Y positive/growing.

Conversely, that transformation transforms marker-local coordinates to camera-local. Try transforming some marker-local points, such as origin or points on the axes. I believe that cv::transform can help with that. Using OpenCV's projectPoints to map 3D space points to 2D image points, you can then draw the marker's axes, or a cube on top of it, or anything you like.

Say the marker sits upright and faces the camera dead-on. When you consider the frame triads of the marker and the camera in space ("world" space), both would be X "right", but one's Y and Z are opposite the other's Y and Z, so you'd expect to see a rotation around the X axis by half a turn (rotating Z and Y).

You could imagine the transformation to happen like this:

• initially the camera looks through the marker, from the marker's back out into the world. The camera would be "upside down". The camera sees marker-space.
• the pose's rotation component rotates the whole marker-local world around the camera's origin. Seen from the world frame (point of reference), the camera rotates, into an attitude you'd find natural.
• the pose's translation moves the marker's world out in front of the camera (Z being positive), or equivalently, the camera backs away from the marker.

If you get implausible values, check aruco_marker_side_length and camera matrix. f would be around 500-3000 for typical resolutions (VGA-4k) and fields of view (60-80 degrees).

Yes, always use the same resolution.

One could recalculate the camera matrix and distortion coefficients to fit a different resolution but that's a hassle, and requires some knowledge of how the camera made these pictures (binning, cropping). Unless you understand the math behind it, just stick with same resolution.

Now everything working, code after improvements:

Without looking at your code I may say that the image quality and perspective you selected is a bit poor. You may try to work with more clear view of your markers. For instance, hang the markers on the wall take one or two step back and try to take photo of it with better light and if not necessary do not add extra rotation, and keep the contrast high :). This will probably give better results.