ViewFinder | browser made into a web app

if you took photo on Pixel probably using default cam app (GCam) - this app is fulfilled with quaility improvements backed up by some AI. tough task to comptetite with the biggest in quality... try to take a photo with some 3rd party like OpenCamera and compare this picture with one got by your app

Without checking all the code (looks roughly okay), a few basics about OpenCV and aruco:

Both use right-handed coordinate systems. Thumb X, index Y, middle Z.

OpenCV uses X right, Y down, Z far, for screen/camera frames. Origin for screens and pictures is the top left corner. For cameras, the origin is the center of the pinhole model, which would be the center of the aperture. I can't comment on lenses or lens systems. Assume the lens center is the origin. That's probably close enough.

Aruco uses X right, Y far, Z up, if the marker is lying flat on a table. Origin is in the center of the marker. The top left corner of the marker is considered the "first" corner.

The marker can be considered to have its own coordinate system/frame.

The pose given by rvec and tvec is the pose of the marker in the camera frame. That means np.linalg.norm(tvec) gives you the direct distance from the camera to the marker's center. tvec's Z is just the component parallel to optical axis.

If the marker is in the right half of the picture ("half" defined by camera matrix's cx,cy), you'd expect tvec's X to grow. Lower half, Y positive/growing.

Conversely, that transformation transforms marker-local coordinates to camera-local. Try transforming some marker-local points, such as origin or points on the axes. I believe that cv::transform can help with that. Using OpenCV's projectPoints to map 3D space points to 2D image points, you can then draw the marker's axes, or a cube on top of it, or anything you like.

Say the marker sits upright and faces the camera dead-on. When you consider the frame triads of the marker and the camera in space ("world" space), both would be X "right", but one's Y and Z are opposite the other's Y and Z, so you'd expect to see a rotation around the X axis by half a turn (rotating Z and Y).

You could imagine the transformation to happen like this:

• initially the camera looks through the marker, from the marker's back out into the world. The camera would be "upside down". The camera sees marker-space.
• the pose's rotation component rotates the whole marker-local world around the camera's origin. Seen from the world frame (point of reference), the camera rotates, into an attitude you'd find natural.
• the pose's translation moves the marker's world out in front of the camera (Z being positive), or equivalently, the camera backs away from the marker.

If you get implausible values, check aruco_marker_side_length and camera matrix. f would be around 500-3000 for typical resolutions (VGA-4k) and fields of view (60-80 degrees).

to make the screenshot of a cell "work", I had to replace EnvironmentObject with ObservedObject in PrintableCells. I also added a few bits of code to show it is working. Here is the code that works for me:

Thanks to Adrian's comment.

It worked after the following changes.

In BarcodeAnalyser

1. Removed imageProxy.close() from addOnSuccessListener and addOnFailureListener. Added it to addOnCompleteListener.
2. Added imageProxy.close() in else condition as well.

You are suffering from buffering. OpenCV VideoCapture() reads a few frames into a buffer - I think it is 5 but have not checked for a while and it may differ between platforms or versions.

There are a few possible work-arounds depending on your situation:

• call read() 4-5 times when you want a frame - it will only take a couple of hundred milliseconds
• call grab() either repeatedly in another thread or just before you want a frame
• reduce the size of the buffer so it can only hold a single frame.

Sorry for the somewhat woolly answer as I am not set up to test more at the moment.

The difficulty that you are having is getting a good mapping from the image in the ImageProxy to what is displayed by the PreviewView. Although this sounds easy, I don't believe there is straightforward way to do this mapping. See the answer to a similar question. I took a look at implementing each of the suggestions in this answer and, although they worked in some situations, they failed in others. Of course, I could have taken the wrong approach.

I have come to the conclusion that extracting and analyzing a bitmap extracted from the preview area and identifying those words that are completely enclosed by the red rectangle is the simplest. I circumscribe those words with their own red rectangle to show that they have been correctly identified.

The following is the reworked activity, a graphic overlay the produces the word boxes and the XML for the display. Comments are in the code. Good luck!

TestPhotoscan.kt

The issue is: In the first example, you are updating the global variable angle, but in the second you are updating this.angle, In the first case, you are recreating the object viewFinder each frame, and since the value of angle is changed, the values of both dx & dy are re-calculated for the new object.

However, in the second case you are also recreating the object viewFinder, which uses the global variable angle to set the new angle and dx and dy but you are not updating the angle anymore, which makes the element look static, To fix this:

1. don't recreate the object for each frame, create it once and update it for each frame.

2. update the value of dx and dy after updating the value of this.angle

your code will look something like:

You can access the QCamera using the mediaObject property of the Camera item and then use the QCameraImageCapture where the QCameraImageCapture::CaptureToBuffer mode is set and use the imageCaptured signal to get the QImage.

With regards to the image analysis resolution, the documentation of ImageAnalysis.Builder.setTargetResolution() states that:

The maximum available resolution that could be selected for an ImageAnalysis is limited to be under 1080p.

So setting a size of 2560x800 won't work as you expect. In return CameraX seems to be selecting the maximum ImageAnalysis resolution that has the same aspect ratio you requested (2560/800 = 1280/400).