cv2.stereoCalibrate 是否提供了合并初始估计的方法

这是我的代码：

def stereo_calibrate(img, correspondences_left, correspondences_right, camera_matrix_L, camera_matrix_R, dist_coeffs=None):
    if dist_coeffs is None:
        dist_coeffs = np.zeros((4, 1))

    # Extract image points and object points from correspondences
    image_points_left = correspondences_left[:, :2].astype(np.float32)
    image_points_right = correspondences_right[:, :2].astype(np.float32)
    object_points = correspondences_left[:, 2:].astype(np.float32)  # Assuming object points are the same for both sets

    # Stereo calibration to find the extrinsic parameters
    ret, camera_matrix_L, dist_coeffs_L, camera_matrix_R, dist_coeffs_R, R, T, E, F = cv2.stereoCalibrate(
        [object_points], [image_points_left], [image_points_right],
        camera_matrix_L, dist_coeffs, camera_matrix_R, dist_coeffs,
        imageSize=(img.shape[1], image_points_left.shape[0]),
        criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-6),
        flags=cv2.CALIB_FIX_INTRINSIC
    )

    if ret:
        print("Stereo Calibration Successful")
        print("Rotation Matrix between the cameras:")
        print(R)
        print("Translation Vector between the cameras:")
        print(T)
    else:
        raise ValueError("stereoCalibrate failed to find a solution")

    # Reproject object points to both cameras
    rvec_L, tvec_L = np.zeros((3, 1)), np.zeros((3, 1))  # Left camera is at origin
    reprojected_points_left, _ = cv2.projectPoints(object_points, rvec_L, tvec_L, camera_matrix_L, dist_coeffs_L)
    reprojected_points_right, _ = cv2.projectPoints(object_points, R, T, camera_matrix_R, dist_coeffs_R)

    return R, T, reprojected_points_left, reprojected_points_right


def visualize_reprojection(original_points, reprojected_points, title, img):
    total_residual = [0.0, 0.0]
    for i in range(len(original_points)):
        dx = original_points[i][0] - reprojected_points[i][0][0]
        dy = original_points[i][1] - reprojected_points[i][0][1]
        residual_x = dx * dx
        residual_y = dy * dy
        total_residual[0] += residual_x
        total_residual[1] += residual_y

        cv2.circle(img, (int(original_points[i][0]), int(original_points[i][1])), 3, (0, 0, 255), -1)  # Red for original
        cv2.circle(img, (int(reprojected_points[i][0][0]), int(reprojected_points[i][0][1])), 3, (255, 0, 0), -1)  # Blue for reprojected

    print("Total residual x: {:.4f}, y: {:.4f}".format(total_residual[0], total_residual[1]))

    cv2.imwrite(title + "output.png", img)

correspondences_left = np.array([
    [671.0889955686854, 193.80354505169868, 0, -4.0, 11],
    [776.4436176302232, 915.4922724670864, 1.4031329107584791, 3.7458267491631854, 11],

    # points along 3 point projected circle
    [929.34693878, 231.39735894, 3.0701740089441083, -2.5639874326533003, 11],
    [539.25672372, 254.15158924, -1.6947050288157524, -3.6232547337037455, 11],
    [449.60294118, 345.16666667, -3.070174008944108, -2.5639874326533008, 11],
    [393.13690476, 466.13095238, -3.867306226784721, -1.021735067555247, 11],
    [388.53170732, 601.62926829, -3.9359437874957477, 0.7129842225979681, 11],
    [434.3463035, 728.29961089, -3.263157253033188, 2.3133967973473335, 11],
    [517.17673049, 832.67746686, -1.975681194440357, 3.4780287258639375, 11],
    [904.77595628, 871.08306011, 2.8579948474501466, 2.7985470251450866, 11],
], dtype=np.float32)

correspondences_right = np.array([
    [436.2245592329106, 193.79399938137954, 0, -4.0, 11.0],
    [536.1735742118314, 916.4208289054197, 1.4031329107584791, 3.7458267491631854, 11.0],

    # points along 3 pt projected circle
    [698.08370044, 233.78414097, 3.0701740089441083, -2.5639874326533003, 11.0],
    [309.2157969, 251.30324401, -1.694705028815752, -3.623254733703746, 11.0],
    [213.93006993, 346.26923077, -3.070174008944108, -2.5639874326533008, 11.0],
    [160, 468, -3.867306226784721, -1.021735067555247, 11.0],
    [152.14662447, 602.2943038, -3.9359437874957477, 0.7129842225979681, 11.0],
    [198.33038348, 728.98328417, -3.263157253033188, 2.3133967973473335, 11.0],
    [285.64635473, 833.48966268, -1.975681194440357, 3.4780287258639375, 11.0],
    [667.77506775, 870.36856369, 2.8579948474501466, 2.7985470251450866, 11.0],
], dtype=np.float32)

# Camera intrinsic parameters, replace with your actual camera matrix
camera_matrix_L = np.array([
    [1018.9, 0, 601.447],
    [0, 1018.9, 517.462],
    [0, 0, 1]
], dtype=np.float32)

camera_matrix_R = np.array([
    [1018.9, 0, 690.392],
    [0, 1018.9, 517.462],
    [0, 0, 1]
], dtype=np.float32)

# cv2.stereoCalibrate example usage
img_left, img_right = cv2.imread("a1_capture_6_26_L.png"), cv2.imread("a1_capture_6_26_R.png")
R, T, reprojected_points_left, reprojected_points_right = stereo_calibrate(img_left, correspondences_left, correspondences_right, camera_matrix_L, camera_matrix_R)

# Print out the results
print("Stereo Calibration - Rotation Matrix (R):")
print(R)
print("Stereo Calibration - Translation Vector (T):")
print(T)

# Visualize the reprojection
visualize_reprojection(correspondences_left[:, :2], reprojected_points_left, "Left_Side_Reprojection", img_left)
visualize_reprojection(correspondences_right[:, :2], reprojected_points_right, "Right_Side_Reprojection", img_right)

立体右图（红色是特征检测算法检测到的原始点，类似于findChessboardCorners）：

立体左图：

如您所见，蓝色点相差很远。总残留量也非常高。数千个（尽管这些是高分辨率图像）。

编辑：我已经确认我的 3d 点是正确的。我通过基于 Ceres-solver 的程序和 cv2.solvePnP 运行了我的立体问题，它产生了更好的结果（为简洁起见，仅显示右图像。左图像输出相同）：

是正确的，的代码中存在一些可能导致立体校准不佳的问题：

1. cv2.stereoCalibrate 的使用不正确： 在 stereoCalibrate 函数中传递内参和畸变系数，但是使用的是 cv2.CALIB_FIX_INTRINSIC 标志，这告诉函数在校准过程中保持内参不变。这意味着传递的内参和畸变系数将被忽略，并且不会在校准过程中更新。

2. 初始估计不足： cv2.stereoCalibrate 函数受益于外参（旋转和平移）的良好初始估计。当前正在传递零矩阵作为旋转和平移的初始估计，这可能会导致陷入局部最小值并产生较差的校准结果。

可以尝试以下操作来改进代码：

1. 从 stereoCalibrate 调用中删除 cv2.CALIB_FIX_INTRINSIC 标志：

ret, camera_matrix_L, dist_coeffs_L, camera_matrix_R, dist_coeffs_R, R, T, E, F = cv2.stereoCalibrate(
    [object_points], [image_points_left], [image_points_right],
    camera_matrix_L, dist_coeffs, camera_matrix_R, dist_coeffs,
    imageSize=(img.shape[1], image_points_left.shape[0]),
    criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-6)
)

这将允许 stereoCalibrate 函数优化内参和畸变系数以及外参。

2. 提供外参的初始估计：

可以使用 cv2.solvePnP 或 cv2.stereoCalibrate （不使用 cv2.CALIB_FIX_INTRINSIC 标志）的先前结果来获取旋转和平移的初始估计。

例如，可以首先使用 cv2.solvePnP 获取旋转和平移的初始估计，如下所示：

# Assume object_points, image_points_left, camera_matrix_L, dist_coeffs are defined

# Find the rotation and translation vectors.
_, rvec, tvec = cv2.solvePnP(object_points, image_points_left, camera_matrix_L, dist_coeffs)

# Convert rotation vector to rotation matrix
R, _ = cv2.Rodrigues(rvec)

然后，可以将 R 和 tvec 作为初始估计值传递给 stereoCalibrate 函数：

ret, camera_matrix_L, dist_coeffs_L, camera_matrix_R, dist_coeffs_R, R, T, E, F = cv2.stereoCalibrate(
    [object_points], [image_points_left], [image_points_right],
    camera_matrix_L, dist_coeffs, camera_matrix_R, dist_coeffs,
    imageSize=(img.shape[1], image_points_left.shape[0]),
    criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-6),
    flags=cv2.CALIB_USE_INTRINSIC_GUESS,
    R=R, T=tvec 
)

通过提供初始估计值，将帮助 stereoCalibrate 函数更快、更准确地收敛到解决方案。

3. 使用更多对应点：

通常，用于校准的对应点越多，结果越好。尝试增加用于校准的对应点的数量。

通过进行这些更改，应该能够改进立体校准的结果并减少重投影误差。