StereoCalib | 实现摄像头的标定，双目摄像头的立体矫正、获取深度图和点云坐标，估计三维重建后的体积

If you already know which points correspond to each other on two images you are right, stereo rectification is not needed. You can just triangulate the points to get the 3D coordinate.

But if you are trying to estimate a depth map from a whole image you will need to solve the stereo correspondence problem, i.e. you will need to find for each point in one image the corresponding point in the second image.

By stereo rectifying you ensure that epipolar lines are horizontal and corresponding points only can be on the same horizontal pixel row in both images. This means your search space for correspondences is radically smaller and thus the problem becomes tractable.

Let me know if this answers your question or if you need a few more pointers into the right direction!