勉強しないとな~blog

ちゃんと勉強せねば…な電気設計エンジニアです。

OpenCVやってみる - 35. 複数データで点数判定実施

春のパン祭りが始まりましたが、まだ点数自動計算ができず…
むずい…

方式変更

前まで検討していた点数判定方法を実際のデータで試したところ、うまくいかず。
特に"5"の文字の判定が困難でした。

経過は省きますが、以下の点を変更しました。

  • "0"以外での比較方法の修正
    "0"以外の文字での比較ですが、前回まではテンプレート→対象輪郭の変換行列を求めて、変換後テンプレートと対象輪郭の塗りつぶし画像を比較していました。
    ただ、変換行列を求めた後に逆変換行列を求め、テンプレートの塗りつぶし画像(1回生成すればいい)と逆変換をかけた対象輪郭の塗りつぶしを比較する、というほうが処理が軽くなるかと考えました。
  • 輪郭検出時、cv2.findContours()cv2.CHAIN_APPROX_NONEによる検出を実施
    今まではcv2.CHAIN_APPROX_SIMPLEを使っていましたが、これだと輪郭上の一部の点しか取得できません。
    下記の理由で、輪郭のカーブ上の点も得たかったので、変更しました。
  • テンプレート輪郭上の点は、カーブ上のものも選ぶ
    判定に失敗したデータを見ると、変換行列推定のICPアルゴリズムの結果、テンプレートの点と比較対象画像の点の対応が期待通りになっていませんでした。
    前回までで選んだテンプレート上の点は、文字の角の点にしていましたが、配置に偏りがあったのが問題かと。
    なので、テンプレートの輪郭上の点を全体的に取るように変更しました。
  • 初期変換行列の評価方法
    最近傍点との距離の総和を使っていましたが、輪郭点が増えるので処理が重くなると考え、テンプレートマッチングによる評価に変更しました。
  • 初期変換行列の時点でのふるい分け
    ICPで使うテンプレート輪郭点が増えたので、処理が重くなっています。
    初期変換行列の時点である程度判断してよさそうかと思われるので、これにより処理を軽減します。 閾値はデータを見て決める必要があるかと思われます。
    • 初期変換行列で十分な一致度が得られればそれで終了
    • 一致度が十分な値に達しなければ、それでも終了

下準備

今まで通りのライブラリインポート、画像読み込みを行います。

import cv2
import numpy as np
%matplotlib inline
from matplotlib import pyplot as plt
import math

img1 = cv2.imread('harupan_190428_1.jpg')
img2 = cv2.imread('harupan_190428_2.jpg')
img3 = cv2.imread('harupan_200317_1.jpg')
img4 = cv2.imread('harupan_210227_2.jpg')
img5 = cv2.imread('harupan_210402_1.jpg')
img6 = cv2.imread('harupan_210402_2.jpg')
img7 = cv2.imread('harupan_210414_1.jpg')
plt.figure(figsize=(12.8,9.6), dpi=100)
plt.subplot(2,4,1), plt.imshow(cv2.cvtColor(img1,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.subplot(2,4,2), plt.imshow(cv2.cvtColor(img2,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.subplot(2,4,3), plt.imshow(cv2.cvtColor(img3,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.subplot(2,4,4), plt.imshow(cv2.cvtColor(img4,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.subplot(2,4,5), plt.imshow(cv2.cvtColor(img5,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.subplot(2,4,6), plt.imshow(cv2.cvtColor(img6,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.subplot(2,4,7), plt.imshow(cv2.cvtColor(img7,cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.show()

f:id:nokixa:20220223004508p:plain

輪郭検出処理

変更した輪郭検出処理です。
若干の変更もあります。

  • 関数名
  • リサイズ画像を毎回返す(デバッグモード指定の引数は廃止)
  • 解像度閾値(リサイズするかどうか)をオプション引数にした

処理内容を改めてまとめておきます。

  • 解像度があまり大きくならないように調整する (画像の縦、横いずれかが閾値を超えていたらその閾値になるよう、縦横比を維持して縮小)
  • HSVフォーマットに変換
  • Hue(色相)、Saturation(彩度)で2値化 (Hueは、赤周辺の範囲で2値化するため、値の範囲を回転させる)
  • 輪郭検出を実施、階層情報も全て取得するようにする
  • 第2レベルの輪郭を取り出す
  • 所定の面積以下の輪郭をフィルタする、この結果の輪郭を返す
  • リサイズした画像データも返す
def detect_candidate_contours(image, res_th=800):
    h, w, chs = image.shape
    if h > res_th or w > res_th:
        k = float(res_th)/h if w > h else float(res_th)/w
    else:
        k = 1.0
    img = cv2.resize(image, None, fx=k, fy=k, interpolation=cv2.INTER_AREA)
    if __debug__:
        print('Resized to ', img.shape)
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    # Convert hue value (rotation, mask by saturation)
    hsv[:,:,0] = np.where(hsv[:,:,0] < 50, hsv[:,:,0]+180, hsv[:,:,0])
    hsv[:,:,0] = np.where(hsv[:,:,1] < 100, 0, hsv[:,:,0])
    # Thresholding with cv2.inRange()
    th_hue = cv2.inRange(hsv[:,:,0], 135, 190)
    # Retrieve all points on the contours (cv2.CHAIN_APPROX_NONE)
    contours, hierarchy = cv2.findContours(th_hue, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    indices0 = [i for i,hier in enumerate(hierarchy[0,:,:]) if hier[3] == -1]
    indices1 = [i for i,hier in enumerate(hierarchy[0,:,:]) if hier[3] in indices0]
    if __debug__:
        print('Number of contours: ', len(contours))
        print('Number of indices0: ', len(indices0), 'indices1: ', len(indices1))
    contours1 = [contours[i] for i in indices1]
    contours1_filtered = [ctr for ctr in contours1 if cv2.contourArea(ctr) > float(res_th)*float(res_th)/4000]
    return contours1_filtered, img

文字判定処理

いくつかの処理の組み合わせになります。

補助処理

輪郭周辺の小画像を作成します。
原点をこの小画像に合わせた輪郭データも返します。

def create_contour_area_image(img, ctrs, idx):
    x,y,w,h = cv2.boundingRect(ctrs[idx])
    rtn_img = img[y:y+h,x:x+w,:].copy()
    rtn_ctr = ctrs[idx].copy()
    origin = np.array([x,y])
    for c in rtn_ctr:
        c[0,:] -= origin
    return rtn_img, rtn_ctr

輪郭の塗りつぶし画像を作成します。

# ctr: Should be output of create_contour_area_image() (Origin of points is the origin of bounding box)
# img_shape: Optional, tuple of (image_height, image_width), if omitted, calculated from ctr
def create_solid_contour(ctr, img_shape=(int(0),int(0))):
    if img_shape == (int(0),int(0)):
        _,_,w,h = cv2.boundingRect(ctr)
    else:
        h,w = img_shape
    img = np.zeros((h,w), 'uint8')
    img = cv2.drawContours(img, [ctr], -1, 255, -1)
    return img

変換行列計算

2つの輪郭データ間の最適な変換を求める処理です。
以下を含みます。

  • 最近傍点探索
  • 外接矩形(回転あり)に基づいた変換行列の計算
  • 2次元アフィン変換行列計算(3組以上の対応点を使用、最小二乗誤差)
  • ICPアルゴリズム
# pts: list of 2D points, or ndarray of shape (n,2)
# query: 2D point to find nearest neighbor
def find_nearest_neighbor(pts, query):
    min_distance = float('inf')
    min_idx = 0
    for i, p in enumerate(pts):
        d = np.linalg.norm(query - p)
        if(d < min_distance):
            min_distance = d
            min_idx = i
    return min_idx, min_distance

def get_initial_transform(src_ctr, src_img, dst_ctr, dst_img):
    src_box = cv2.boxPoints(cv2.minAreaRect(src_ctr))
    dst_box = cv2.boxPoints(cv2.minAreaRect(dst_ctr))
    # Rotated patterns are created when starting index is slided
    dst_box = np.vstack([dst_box, dst_box])
    
    src_pts = [p for p in src_ctr[:,0,:]]
    dst_pts = [p for p in dst_ctr[:,0,:]]
    max_similarity = 0.0
    for i in range(4):
        M = cv2.getAffineTransform(src_box[0:3], dst_box[i:i+3])
        converted_img = cv2.warpAffine(src_img, M, dsize=(dst_img.shape[1], dst_img.shape[0]), flags=cv2.INTER_NEAREST)
        similarity = cv2.matchTemplate(converted_img, dst_img, cv2.TM_CCORR_NORMED)
        if similarity[0,0] > max_similarity:
            M_rtn = M
            max_similarity = similarity[0,0]
    return M_rtn, max_similarity

# src, dst: ndarray, shape is (n,2) (n: number of points)
def estimate_affine_2d(src, dst):
    n = min(src.shape[0], dst.shape[0])
    x = dst[0:n].flatten()
    A = np.zeros((2*n,6))
    for i in range(n):
        A[i*2,0] = src[i,0]
        A[i*2,1] = src[i,1]
        A[i*2,2] = 1
        A[i*2+1,3] = src[i,0]
        A[i*2+1,4] = src[i,1]
        A[i*2+1,5] = 1
    M = np.linalg.inv(A.T @ A) @ A.T @ x
    return M.reshape([2,3])

# Find optimum affine matrix using ICP algorithm
# src_pts: ndarray, shape is (n_s,2) (n_s: number of points)
# dst_pts: ndarray, shape is (n_d,2) (n_d: number of points, n_d should be larger or equal to n_s)
# initial_matrix: ndarray, shape is (2,3)
def icp(src_pts, dst_pts, max_iter=100, initial_matrix=np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]])):
    default_affine_matrix = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]])
    if dst_pts.shape[0] < src_pts.shape[0]:
        print("icp: Insufficient destination points")
        return default_affine_matrix, False
    if initial_matrix.shape != (2,3):
        print("icp: Illegal shape of initial_matrix")
        return default_affine_matrix, False
    M = initial_matrix
    # Store indices of the nearest neighbor point of dst_pts to the converted point of src_pts
    nn_idx = []
    for i in range(max_iter):
        nn_idx_tmp = []
        dst_pts_list = [p for p in dst_pts]
        idx_list = list(range(0,dst_pts.shape[0]))
        for p in src_pts:
            p2 = M @ np.array([p[0], p[1], 1])
            idx, d = find_nearest_neighbor(dst_pts_list, p2)
            nn_idx_tmp += [idx_list[idx]]
            del dst_pts_list[idx]
            del idx_list[idx]
        if __debug__:
            print("icp: nn_idx: ", nn_idx_tmp)
        if nn_idx != [] and nn_idx == nn_idx_tmp:
            if __debug__:
                print("icp: converged in ", i, " iteration(s)")
            break
        dst_pts2 = np.zeros_like(src_pts)
        for j,idx in enumerate(nn_idx_tmp):
            dst_pts2[j,:] = dst_pts[idx,:]
        M = estimate_affine_2d(src_pts, dst_pts2)
        nn_idx = nn_idx_tmp
        if i == max_iter -1:
            print("icp: Not converged")
            return M, False
    return M, True

# src_selected_pt_idx: Indices of points in src_ctr to find matching points
# sim_th: Threshold of similarity
def get_optimum_transform(src_ctr, src_selected_pt_idx, dst_ctr, sim_th):
    src_img = create_solid_contour(src_ctr)
    dst_img = create_solid_contour(dst_ctr)
    src_pts = np.array([src_ctr[idx,0,:] for idx in src_selected_pt_idx])
    dst_pts = np.array([p for p in dst_ctr[:,0,:]])
    M_init, sim_init = get_initial_transform(src_ctr, src_img, dst_ctr, dst_img)
    if sim_init > sim_th:
        print('get_optimum_transform: ICP skipped')
        return M_init, True
    else:
        return icp(src_pts, dst_pts, initial_matrix=M_init)

一致度計算 ("0"以外の判定用)

ここで、上で検討した比較方法の変更を入れます。
cv2.invertAffineTransform()逆行列を計算してくれます。

OpenCV: Geometric Image Transformations

輪郭点の座標にアフィン変換を実施、その後に変換した輪郭点で塗りつぶし画像を作る、という形で実装しました。
先に輪郭の塗りつぶし画像を作ってからアフィン変換する、という方式も考えられますが、こちらのほうが少し処理が軽くなるかなと。

def get_contours_similarity(ctr, ctr_tmp, solid_tmp, selected_pt_idx_tmp, sim_th):
    M, result = get_optimum_transform(ctr_tmp, selected_pt_idx_tmp, ctr, sim_th)
    Minv = cv2.invertAffineTransform(M)
    converted_ctr = np.zeros_like(ctr)
    for i in range(ctr.shape[0]):
        converted_ctr[i,0,:] = (Minv[:,0:2] @ ctr[i,0,:]) + Minv[:,2]
    ctr_img = create_solid_contour(converted_ctr, img_shape=solid_tmp.shape)
    val = cv2.matchTemplate(solid_tmp, ctr_img, cv2.TM_CCORR_NORMED)
    return val[0,0], ctr_img

テンプレート画像生成 ("0"の判定用)

前回やったものと比べて、実装を変えています。
あとで一度確認してみます。

ここでは、create_contour_area_image()関数を実施して、Bounding Boxの原点を原点とした輪郭点データを扱うこととします。

  • テンプレートの輪郭を楕円近似
  • 楕円の角度をまっすぐ(0deg)にする変換行列を求める、回転中心は楕円の中心
  • 変換行列には並進の成分も含まれるが、近似楕円の外接矩形の原点が原点に来るように変更する
  • テンプレートの塗りつぶし画像を用意しておく
  • 変換後の画像サイズを近似楕円の外接矩形の縦横サイズとして、変換を実施

この変換は、比較対象にも実施します。

# ctr: Should be output of create_contour_area_image() (Origin of points is the origin of bounding box)
# img_shape: Optional, tuple of (image_height, image_width), determined from fitted ellipse if omitted
def create_upright_solid_contour(ctr,img_shape=(int(0),int(0))):
    (cx,cy),(w,h),angle = cv2.fitEllipse(ctr)
    if img_shape == (int(0),int(0)):
        # Default: same as fitted ellipse
        img_shape = (math.ceil(w), math.ceil(h))
    ctr_img = create_solid_contour(ctr)
    Mrot = cv2.getRotationMatrix2D((cx,cy), angle, 1)
    Mrot[0,2] -= cx - w/2
    Mrot[1,2] -= cy - h/2
    rotated_ctr_img = cv2.warpAffine(ctr_img, Mrot, dsize=img_shape, flags=cv2.INTER_NEAREST)
    return rotated_ctr_img

一致度計算 ("0"の判定用)

比較対象の輪郭をテンプレート同様にまっすぐに回転させた後、テンプレートの縦横比と同じになるようにリサイズして、テンプレートマッチングを実施します。

def get_contours_similarity_zero(solid_tmp, ctr):
    img = create_upright_solid_contour(ctr)
    img = cv2.resize(img, dsize=(solid_tmp.shape[1], solid_tmp.shape[0]), interpolation=cv2.INTER_NEAREST)
    val = cv2.matchTemplate(img, solid_tmp, cv2.TM_CCORR_NORMED)
    return val[0,0], img

最後に判定

比較対象の輪郭を各数字のテンプレートと比較します。
一致度が閾値以上のものが複数あれば、最も一致度の高いものを選びます。
どれも閾値を超えなければ、どの数字でもない、ということになります。

閾値は、全数字共通で、0.92にしておきたいと思います。が、要検討だなー…

# ctr: Single contour to compare
# solid_zero: template image of "Zero"
# solid_other: list of template images of other numbers (1,2,3,5), if template does not exist, fill corresponding element with ndarray with shape (1)
# ctr_other: list of contours of other numbers (1,2,3,5), if template does not exist, fill corresponding element with None
# pts_idx_other: list of list of edge points of other numbers (1,2,3,5), if template does not exist, fill corresponding element with None
# debug_number: Optional, if specified, comparing image for the number is returned
# return: determined number (0,1,2,3,5), -1 if none corresponds
def determine_number(ctr, solid_zero, solid_other, ctr_other, pts_idx_other, debug_number=-1):
    # Threshold value of similarity, should be adjusted
    val_th = 0.92
    sim, img = get_contours_similarity_zero(solid_zero, ctr)
    max_val = sim
    max_number = 0
    # For evaluation
    similarities = [sim]
    if debug_number==0:
        dbg_img = img.copy()
    
    numbers = [1,2,3,5]
    for i in range(4):
        if solid_other[i].shape == (1,):
            similarities += [0.0]
            if debug_number == numbers[i]:
                dbg_img = np.zeros((1,1), 'uint8')
        else:
            sim, img = get_contours_similarity(ctr, ctr_other[i], solid_other[i], pts_idx_other[i], val_th)
            similarities += [sim]
            if sim > max_val:
                max_val = sim
                max_number = numbers[i]
            if debug_number == numbers[i]:
                dbg_img = img.copy()
    rtn_number = -1 if max_val < val_th else max_number
    if debug_number != -1:
        return rtn_number, similarities, dbg_img
    else:
        return rtn_number

その他処理

以下の関数は、文字テンプレートの選択等で使用します。

def draw_contour(img, ctrs, idx):
    img_with_ctr = cv2.drawContours(img.copy(), [ctrs[idx]], -1, (0,255,0), 2)
    plt.figure(figsize=(6.4,4.8), dpi=100)
    plt.imshow(cv2.cvtColor(img_with_ctr, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
    plt.show()
def draw_contour_point(img, ctr, idx):
    img_with_pt = cv2.drawMarker(img.copy(), ctr[idx,0,:], (0,255,0), markerType=cv2.MARKER_CROSS, markerSize=3)
    plt.imshow(cv2.cvtColor(img_with_pt, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
    plt.show()

まず輪郭検出

ここから実際の画像で作業をやっていきます。
まずは、上で出した7枚の画像全てで輪郭検出を行います。
テンプレートを選ぶにしろ、正解ラベル付けしていくにしろ必要になるので。

imgs = [img1, img2, img3, img4, img5, img6, img7]
resized_imgs = []
ctrs_all = []
for img in imgs:
    ctrs, im = detect_candidate_contours(img)
    resized_imgs += [im]
    ctrs_all += [ctrs]
Resized to  (1067, 800, 3)
Number of contours:  2514
Number of indices0:  1448 indices1:  875
Resized to  (1067, 800, 3)
Number of contours:  2269
Number of indices0:  1265 indices1:  818
Resized to  (1067, 800, 3)
Number of contours:  2062
Number of indices0:  1154 indices1:  718
Resized to  (1067, 800, 3)
Number of contours:  1204
Number of indices0:  450 indices1:  664
Resized to  (1067, 800, 3)
Number of contours:  1613
Number of indices0:  698 indices1:  795
Resized to  (1067, 800, 3)
Number of contours:  1242
Number of indices0:  373 indices1:  777
Resized to  (1067, 800, 3)
Number of contours:  1258
Number of indices0:  555 indices1:  595

テンプレート選択

既に実施していますが、上の関数を使った形で1つだけやり直してみたいと思います。
interactでは複数の引数も扱うことができるようです。

上のdraw_contour()そのままでやろうとしましたが、だめでした。
引数がinteractで指定できるもの(bool値、整数、浮動小数点値、など)でないといけないようです。

こちらは公式ドキュメント

Using Interact

from ipywidgets import interact, fixed

def draw_contour_interact(i_img, idx):
    draw_contour(resized_imgs[i_img], ctrs_all[i_img], idx)

interact(draw_contour_interact, i_img=fixed(4), idx=(0, len(ctrs_all[4])-1));

f:id:nokixa:20220223004718p:plain

前にテンプレートとして選んだ輪郭を再度示します。
また、比較用の2値画像も作成します。

1つ目の画像の"5"の文字のテンプレートだけ、前に選んだものがあまり良くなかったので、別のものを選んでいます。
また、3つ目の画像、5つ目の画像では"3"の文字がなかったので、1つ目の画像のテンプレートを使うことにします。(一度0埋めもやりましたが、都合が悪かったので。)

ctrs1_idx_zero = 26
ctrs1_idx_one = 27
ctrs1_idx_two = 24
ctrs1_idx_three = 33
# ctrs1_idx_five = 35
ctrs1_idx_five = 8
ctrs1_idx_numbers = [ctrs1_idx_zero, ctrs1_idx_one, ctrs1_idx_two, ctrs1_idx_three, ctrs1_idx_five]

subimgs1 = []
subctrs1 = []
binimgs1 = []
for i,idx in enumerate(ctrs1_idx_numbers):
    img, ctrs = create_contour_area_image(resized_imgs[0], ctrs_all[0], idx)
    if i == 0:
        binimg = create_upright_solid_contour(ctrs)
    else:
        binimg = create_solid_contour(ctrs)
    subimgs1 += [img.copy()]
    subctrs1 += [ctrs.copy()]
    binimgs1 += [binimg.copy()]
    ctr_img = cv2.drawContours(img, [ctrs], -1, (0,255,0), 2)
    plt.subplot(2,5,1+i), plt.imshow(cv2.cvtColor(ctr_img, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
    plt.subplot(2,5,6+i), plt.imshow(binimg,cmap='gray'), plt.xticks([]), plt.yticks([])
plt.show()

f:id:nokixa:20220223004513p:plain

ctrs3_idx_zero = 7
ctrs3_idx_one = 4
ctrs3_idx_two = 17
ctrs3_idx_five = 6
ctrs3_idx_numbers = [ctrs3_idx_zero, ctrs3_idx_one, ctrs3_idx_two, ctrs3_idx_five]

subimgs3 = []
subctrs3 = []
binimgs3 = []
for i,idx in enumerate(ctrs3_idx_numbers):
    img, ctrs = create_contour_area_image(resized_imgs[2], ctrs_all[2], idx)
    if i == 0:
        binimg = create_upright_solid_contour(ctrs)
    else:
        binimg = create_solid_contour(ctrs)
    subimgs3 += [img.copy()]
    subctrs3 += [ctrs.copy()]
    binimgs3 += [binimg.copy()]
    ctr_img = cv2.drawContours(img, [ctrs], -1, (0,255,0), 2)
    plt.subplot(2,4,1+i), plt.imshow(cv2.cvtColor(ctr_img, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
    plt.subplot(2,4,5+i), plt.imshow(binimg,cmap='gray'), plt.xticks([]), plt.yticks([])
plt.show()

subimgs3.insert(3, subimgs1[3])
subctrs3.insert(3, subctrs1[3])
binimgs3.insert(3, binimgs1[3])

f:id:nokixa:20220223004515p:plain

ctrs5_idx_zero = 3
ctrs5_idx_one = 4
ctrs5_idx_two = 2
ctrs5_idx_five = 5
ctrs5_idx_numbers = [ctrs5_idx_zero, ctrs5_idx_one, ctrs5_idx_two, ctrs5_idx_five]

subimgs5 = []
subctrs5 = []
binimgs5 = []
for i,idx in enumerate(ctrs5_idx_numbers):
    img, ctrs = create_contour_area_image(resized_imgs[4], ctrs_all[4], idx)
    if i == 0:
        binimg = create_upright_solid_contour(ctrs)
    else:
        binimg = create_solid_contour(ctrs)
    subimgs5 += [img.copy()]
    subctrs5 += [ctrs.copy()]
    binimgs5 += [binimg.copy()]
    ctr_img = cv2.drawContours(img, [ctrs], -1, (0,255,0), 2)
    plt.subplot(2,4,1+i), plt.imshow(cv2.cvtColor(ctr_img, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
    plt.subplot(2,4,5+i), plt.imshow(binimg,cmap='gray'), plt.xticks([]), plt.yticks([])
plt.show()

subimgs5.insert(3, subimgs1[3])
subctrs5.insert(3, subctrs1[3])
binimgs5.insert(3, binimgs1[3])

f:id:nokixa:20220223004518p:plain

テンプレート輪郭点の選択

文字テンプレートの輪郭点から、ICPで使うものを選択します。
輪郭点の数を確認、適当に間引きます。

[print(subctrs1[i].shape[0], ', ') for i in range(len(ctrs1_idx_numbers))];
133 , 
141 , 
212 , 
214 , 
131 , 
[print(subctrs3[i].shape[0], ', ') for i in range(len(ctrs3_idx_numbers))];
139 , 
149 , 
214 , 
214 , 
[print(subctrs5[i].shape[0], ', ') for i in range(len(ctrs5_idx_numbers))];
100 , 
88 , 
159 , 
214 , 

間引きは1/5ぐらいでどうかな。
リスト内包表記が便利です。1行で1/5の間引きを記述できます。

Pythonのリスト(配列)の特定の要素を抽出、置換、変換

subctrs1_selected_pts_one = [i for i in range(subctrs1[1].shape[0]) if i % 5 == 0]
subctrs1_selected_pts_two = [i for i in range(subctrs1[2].shape[0]) if i % 5 == 0]
subctrs1_selected_pts_three = [i for i in range(subctrs1[3].shape[0]) if i % 5 == 0]
subctrs1_selected_pts_five = [i for i in range(subctrs1[4].shape[0]) if i % 5 == 0]

subctrs1_selected_pts = [subctrs1_selected_pts_one, subctrs1_selected_pts_two, subctrs1_selected_pts_three, subctrs1_selected_pts_five]
for i in range(4):
    img = subimgs1[i+1].copy()
    for p in subctrs1_selected_pts[i]:
        img = cv2.drawMarker(img, subctrs1[i+1][p,0,:], (0,255,0), markerType=cv2.MARKER_CROSS, markerSize=3)
    plt.subplot(1,4,1+i), plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.show()

f:id:nokixa:20220223004520p:plain

subctrs3_selected_pts_one = [i for i in range(subctrs3[1].shape[0]) if i % 5 == 0]
subctrs3_selected_pts_two = [i for i in range(subctrs3[2].shape[0]) if i % 5 == 0]
subctrs3_selected_pts_three = [i for i in range(subctrs3[3].shape[0]) if i % 5 == 0]
subctrs3_selected_pts_five = [i for i in range(subctrs3[4].shape[0]) if i % 5 == 0]

subctrs3_selected_pts = [subctrs3_selected_pts_one, subctrs3_selected_pts_two, subctrs3_selected_pts_three, subctrs3_selected_pts_five]
for i in range(4):
    if subimgs3[i+1].shape == (1,):
        continue
    img = subimgs3[i+1].copy()
    for p in subctrs3_selected_pts[i]:
        img = cv2.drawMarker(img, subctrs3[i+1][p,0,:], (0,255,0), markerType=cv2.MARKER_CROSS, markerSize=3)
    plt.subplot(1,4,1+i), plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.show()

f:id:nokixa:20220223004523p:plain

subctrs5_selected_pts_one = [i for i in range(subctrs5[1].shape[0]) if i % 5 == 0]
subctrs5_selected_pts_two = [i for i in range(subctrs5[2].shape[0]) if i % 5 == 0]
subctrs5_selected_pts_three = [i for i in range(subctrs5[3].shape[0]) if i % 5 == 0]
subctrs5_selected_pts_five = [i for i in range(subctrs5[4].shape[0]) if i % 5 == 0]

subctrs5_selected_pts = [subctrs5_selected_pts_one, subctrs5_selected_pts_two, subctrs5_selected_pts_three, subctrs5_selected_pts_five]
for i in range(4):
    if subimgs5[i+1].shape == (1,):
        continue
    img = subimgs5[i+1].copy()
    for p in subctrs5_selected_pts[i]:
        img = cv2.drawMarker(img, subctrs5[i+1][p,0,:], (0,255,0), markerType=cv2.MARKER_CROSS, markerSize=3)
    plt.subplot(1,4,1+i), plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)), plt.xticks([]), plt.yticks([])
plt.show()

f:id:nokixa:20220223004525p:plain

正解ラベル付け

全画像について、検出した各輪郭にそれぞれラベルを付けていきます。
ラベルは、0,1,2,3,5,どれでもなければ-1とします。

interact()も使用しますが、それでも面倒な作業…

interact(draw_contour_interact, i_img=fixed(0), idx=(0, len(ctrs_all[0])-1));

f:id:nokixa:20220223004722p:plain

labels1 = [-1,-1,-1,-1,-1
           ,-1,5,0,5,1
           ,5,0,2,1,2
           ,-1,-1,1,1,5
           ,0,2,5,0,2
           ,5,0,1,2,-1
           ,5,1,2,3,1
           ,5,0,-1]
interact(draw_contour_interact, i_img=fixed(1), idx=(0, len(ctrs_all[1])-1));

f:id:nokixa:20220223004725p:plain

labels2 = [-1,-1,-1,-1,-1
           ,-1,5,0,5,1
           ,5,0,2,1,2
           ,-1,-1,1,1,5
           ,0,-1,2,5,0
           ,2,5,0,1,2
           ,5,0,1,-1,2
           ,-1,-1,-1,3,-1
           ,5,0,-1,1,-1
           ,-1]
interact(draw_contour_interact, i_img=fixed(2), idx=(0, len(ctrs_all[2])-1));

f:id:nokixa:20220223004728p:plain

labels3 = [-1,-1,-1,-1,1
           ,1,5,0,1,1
           ,5,0,5,0,-1
           ,-1,-1,2,-1,-1
           ,-1,1,1,1,-1
           ,1,-1,-1,1,1
           ,-1,2,-1,1,-1
           ,1,2,-1,1,-1
           ,-1,2,5,-1,0
           ,-1,1,1]
interact(draw_contour_interact, i_img=fixed(3), idx=(0, len(ctrs_all[3])-1));

f:id:nokixa:20220223004731p:plain

labels4 = [-1,-1,-1,-1,-1
           ,-1,-1,-1,-1,-1
           ,-1,-1,-1,-1,-1
           ,-1,-1,-1,1,1
           ,1,1,1,1,1
           ,-1,5,0,2,5
           ,0,2,1,2,2
           ,-1,-1,-1,1,1
           ,1]
interact(draw_contour_interact, i_img=fixed(4), idx=(0, len(ctrs_all[4])-1));

f:id:nokixa:20220223004734p:plain

labels5 = [-1,-1,2,0,1
           ,5,-1,1,1,1
           ,1,1,1,1,1
           ,1,-1,5,1,0
           ,5,1,2,0,5
           ,0,2,1,2,2
           ,-1,-1,1,1,1
           ]
interact(draw_contour_interact, i_img=fixed(5), idx=(0, len(ctrs_all[5])-1));

f:id:nokixa:20220223004738p:plain

labels6 = [-1,0,1,5,2
                ,-1,1,1,1,1
                ,5,1,0,5,0
                ,2,1,5,0,2
                ,2,2,1,-1,-1
                ,1,1,1,1,1
                ,1,1,1]
interact(draw_contour_interact, i_img=fixed(6), idx=(0, len(ctrs_all[6])-1));

f:id:nokixa:20220223004741p:plain

labels7 = [-1,-1,-1,-1,-1
           ,-1,1,2,2,2
           ,2,1,2,2,2
           ,1,-1,-1,-1,2
           ,1,2,1,1]

各輪郭で点数判定の実施

点数の判定をやっていきます。
ここでは確認のため、いくつか判定結果以外の情報も見ておきます。

まずは1つ目の画像から。

subimgs = []
subctrs = []
det_numbers1 = []
similarities1 = []
dbg_imgs = []
for i in range(len(ctrs_all[0])):
    subimg, subctr = create_contour_area_image(resized_imgs[0], ctrs_all[0], i)
    debug_number= 0 if labels1[i] == -1 else labels1[i]
    det_number, sim, img = determine_number(subctr, binimgs1[0], binimgs1[1:5], subctrs1[1:5], subctrs1_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers1 += [det_number]
    similarities1 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [81, 91, 102, 112, 123, 132, 141, 149, 158, 166, 175, 183, 192, 201, 209, 222, 233, 243, 248, 249, 22, 32, 39, 48, 56, 61, 62, 71, 79]
icp: nn_idx:  [80, 86, 99, 110, 123, 131, 140, 148, 157, 166, 175, 184, 193, 202, 210, 224, 235, 247, 252, 11, 20, 32, 38, 47, 56, 61, 62, 70, 79]
icp: nn_idx:  [81, 85, 98, 110, 123, 131, 140, 148, 157, 166, 175, 184, 193, 201, 217, 227, 240, 252, 255, 12, 21, 32, 38, 47, 56, 61, 62, 70, 79]
icp: nn_idx:  [81, 86, 98, 111, 124, 131, 140, 149, 158, 166, 175, 184, 193, 202, 219, 229, 242, 254, 258, 12, 21, 32, 39, 48, 56, 61, 62, 71, 79]
icp: nn_idx:  [81, 86, 99, 112, 125, 131, 140, 149, 158, 167, 175, 184, 193, 202, 220, 230, 243, 256, 4, 12, 21, 32, 39, 48, 56, 61, 62, 71, 80]
icp: nn_idx:  [82, 87, 100, 113, 126, 132, 141, 149, 158, 167, 175, 184, 193, 202, 221, 232, 244, 257, 4, 13, 22, 32, 39, 48, 57, 62, 63, 71, 80]
icp: nn_idx:  [82, 87, 100, 113, 126, 132, 141, 150, 158, 167, 176, 184, 193, 202, 222, 232, 245, 258, 5, 14, 22, 32, 40, 48, 57, 62, 63, 72, 80]
icp: nn_idx:  [82, 87, 100, 113, 126, 133, 141, 150, 159, 167, 176, 184, 193, 202, 222, 233, 246, 259, 5, 14, 23, 32, 40, 48, 57, 62, 63, 72, 81]
icp: nn_idx:  [82, 87, 100, 113, 126, 133, 142, 150, 159, 167, 176, 185, 193, 202, 225, 233, 246, 259, 6, 14, 23, 32, 40, 49, 57, 62, 64, 72, 81]
icp: nn_idx:  [82, 88, 100, 113, 126, 133, 142, 150, 159, 167, 176, 185, 193, 202, 225, 233, 246, 259, 6, 14, 23, 32, 40, 49, 57, 62, 64, 72, 81]
icp: nn_idx:  [82, 88, 101, 113, 126, 133, 142, 150, 159, 167, 176, 185, 193, 202, 225, 233, 246, 259, 6, 14, 23, 32, 40, 49, 57, 62, 64, 72, 81]
icp: nn_idx:  [82, 88, 101, 114, 126, 133, 142, 150, 159, 167, 176, 185, 193, 202, 225, 233, 246, 259, 6, 14, 23, 32, 40, 49, 57, 62, 64, 72, 81]
icp: nn_idx:  [82, 88, 101, 114, 127, 133, 142, 150, 159, 167, 176, 185, 193, 202, 225, 233, 246, 259, 6, 14, 23, 32, 40, 49, 57, 62, 64, 72, 81]
icp: nn_idx:  [82, 88, 101, 114, 127, 133, 142, 150, 159, 167, 176, 185, 193, 202, 225, 233, 246, 259, 6, 14, 23, 32, 40, 49, 57, 62, 64, 72, 81]
icp: converged in  13  iteration(s)
icp: nn_idx:  [21, 32, 41, 52, 63, 74, 91, 95, 100, 105, 111, 149, 158, 148, 118, 119, 124, 128, 138, 150, 159, 170, 180, 191, 201, 212, 207, 197, 188, 179, 169, 48, 50, 47, 38, 33, 26, 248, 243, 246, 251, 8, 17]
icp: nn_idx:  [20, 30, 41, 52, 64, 76, 87, 93, 98, 104, 111, 148, 157, 146, 120, 121, 126, 129, 136, 147, 158, 169, 181, 191, 203, 213, 209, 199, 189, 179, 170, 48, 51, 47, 38, 32, 25, 248, 245, 247, 254, 6, 16]

...

icp: nn_idx:  [44, 48, 50, 56, 59, 70, 78, 71, 72, 0, 2, 3, 4, 5, 7, 11, 14, 18, 19, 30, 33, 34, 35, 31, 28, 26, 20, 21, 22, 23, 24, 25, 53, 54, 52, 51, 46, 45, 42, 40, 39, 41, 43]
icp: converged in  5  iteration(s)
icp: nn_idx:  [5, 9, 13, 16, 17, 18, 20, 22, 19, 30, 33, 34, 40, 43, 48, 57, 64, 66, 60, 55, 25, 24, 74, 71, 0, 2, 4]
icp: nn_idx:  [4, 7, 13, 16, 17, 18, 20, 22, 19, 29, 33, 35, 40, 44, 48, 57, 63, 67, 60, 55, 25, 24, 74, 71, 0, 2, 3]
icp: nn_idx:  [4, 7, 12, 16, 17, 18, 20, 22, 19, 29, 32, 35, 40, 44, 48, 57, 63, 67, 60, 55, 25, 24, 74, 71, 0, 1, 3]
icp: nn_idx:  [4, 7, 12, 16, 17, 18, 21, 74, 20, 29, 32, 35, 40, 44, 48, 57, 63, 67, 60, 55, 25, 24, 73, 71, 0, 1, 3]
icp: nn_idx:  [4, 7, 12, 16, 17, 19, 21, 74, 20, 29, 32, 35, 40, 44, 48, 57, 63, 67, 60, 55, 53, 24, 73, 71, 0, 1, 3]
icp: nn_idx:  [4, 7, 12, 16, 17, 19, 21, 74, 20, 28, 32, 35, 40, 44, 48, 57, 63, 67, 60, 55, 53, 24, 73, 71, 0, 1, 3]
icp: nn_idx:  [4, 7, 12, 16, 17, 19, 21, 74, 20, 28, 32, 35, 40, 44, 48, 57, 63, 67, 60, 55, 53, 24, 73, 71, 0, 1, 3]
icp: converged in  6  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.0)
pltx = 5
plty = np.int(len(ctrs_all[0]) / pltx)
if len(ctrs_all[0]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[0])):
    if det_numbers1[i] == labels1[i]:
        results[labels1[i]][0] += 1
    results[labels1[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers1[i])
    for s in similarities1[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004528p:plain

-1 :  9  /  10
0 :  6  /  6
1 :  7  /  7
2 :  6  /  6
3 :  1  /  1
5 :  6  /  8

まあまあな結果になりました。
"0"、"1"、"2"、"3"の文字については全て正しく判定できています。
"3"は1つしかないので当たり前ですが…

問題なのは、

  • 数字でない輪郭で、数字として認識されてしまったものがある
    見てみると、"白"の文字が"0"と認識されています。ただ、実は"0"の判定は最終的な点数計算には不要なので、ひとまず不問にしておきます。
  • "5"の文字の認識不良
    2つ誤認識があります。1つは輪郭取得が悪くて、"点"の文字までくっついてきてしまったのが原因のようです。もう1つは白い傷かノイズのようなものが入っているような?ただ、一致度は閾値すれすれなので、閾値の調整でなんとかなりそう。

というところです。

ちなみに、まだ初期変換行列での一致度不十分判定は入れていません。
ICPの処理は結構時間がかかっています。
後で閾値決定をします。

他の画像でも点数判定をやっていきます。
画像ごとに、対応する年のテンプレートデータを選びます。

subimgs = []
subctrs = []
det_numbers2 = []
similarities2 = []
dbg_imgs = []
for i in range(len(ctrs_all[1])):
    subimg, subctr = create_contour_area_image(resized_imgs[1], ctrs_all[1], i)
    debug_number= 0 if labels2[i] == -1 else labels2[i]
    det_number, sim, img = determine_number(subctr, binimgs1[0], binimgs1[1:5], subctrs1[1:5], subctrs1_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers2 += [det_number]
    similarities2 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [208, 220, 230, 240, 251, 3, 12, 20, 29, 37, 45, 54, 62, 71, 79, 88, 99, 109, 112, 141, 150, 158, 165, 175, 184, 189, 190, 198, 206]
icp: nn_idx:  [207, 215, 227, 239, 251, 3, 11, 20, 29, 37, 46, 54, 63, 72, 80, 92, 104, 116, 132, 140, 149, 157, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [207, 214, 227, 240, 252, 3, 12, 20, 29, 37, 46, 54, 65, 72, 82, 94, 107, 120, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 215, 227, 240, 253, 4, 12, 21, 29, 38, 46, 55, 65, 72, 86, 96, 109, 121, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 215, 228, 240, 253, 4, 13, 21, 30, 38, 47, 55, 65, 73, 87, 97, 110, 122, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 215, 228, 240, 254, 4, 13, 22, 30, 39, 47, 56, 65, 73, 87, 97, 110, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 216, 228, 240, 254, 5, 13, 22, 30, 39, 47, 56, 65, 73, 87, 98, 110, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 216, 228, 240, 254, 5, 13, 22, 31, 39, 48, 56, 65, 73, 88, 98, 110, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 216, 228, 240, 254, 5, 14, 22, 31, 39, 48, 56, 65, 73, 88, 98, 111, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 216, 228, 241, 254, 5, 14, 22, 31, 39, 48, 56, 65, 73, 88, 98, 111, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 216, 229, 241, 254, 5, 14, 22, 31, 39, 48, 56, 65, 73, 88, 98, 111, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: nn_idx:  [208, 216, 229, 241, 254, 5, 14, 22, 31, 39, 48, 56, 65, 73, 88, 98, 111, 123, 132, 140, 149, 158, 165, 175, 183, 188, 189, 199, 206]
icp: converged in  11  iteration(s)
icp: nn_idx:  [223, 229, 234, 240, 246, 8, 15, 23, 32, 40, 49, 55, 65, 66, 64, 63, 73, 81, 86, 92, 98, 103, 109, 115, 121, 126, 134, 141, 147, 155, 164, 37, 28, 241, 233, 185, 177, 176, 178, 181, 190, 198, 221]
icp: nn_idx:  [220, 225, 234, 240, 248, 5, 13, 22, 30, 39, 49, 55, 65, 66, 64, 63, 73, 81, 84, 90, 97, 104, 111, 118, 124, 126, 134, 142, 148, 155, 165, 35, 27, 241, 232, 186, 179, 178, 180, 183, 192, 201, 218]
icp: nn_idx:  [219, 225, 233, 240, 248, 4, 12, 21, 30, 39, 48, 54, 65, 66, 64, 63, 73, 81, 84, 90, 97, 104, 111, 118, 125, 126, 134, 142, 148, 156, 165, 35, 27, 241, 231, 187, 180, 179, 181, 184, 193, 203, 217]

...

icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 9, 4, 3, 1]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 9, 4, 3, 1]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 9, 4, 3, 1]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 9, 4, 3, 1]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: Not converged
icp: nn_idx:  [42, 0, 1, 2, 4, 5, 7, 8, 10, 11, 13, 16, 18, 17, 15, 14, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 34, 9, 6, 3, 40, 38, 36, 35, 37, 33, 39, 41, 12]
icp: nn_idx:  [42, 0, 2, 3, 5, 6, 7, 8, 10, 11, 13, 15, 17, 16, 18, 14, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 9, 12, 4, 40, 39, 37, 36, 35, 38, 41, 1, 25]

...

icp: nn_idx:  [21, 22, 23, 24, 25, 27, 28, 30, 32, 33, 35, 36, 38, 40, 42, 0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 4, 41, 39, 37, 34, 11, 13, 15, 31, 26, 19, 18, 16, 14, 17, 20, 12, 29]
icp: nn_idx:  [21, 22, 23, 24, 25, 27, 28, 30, 32, 33, 34, 36, 38, 40, 42, 0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 4, 41, 39, 37, 35, 11, 13, 15, 31, 26, 19, 18, 16, 14, 17, 20, 12, 29]
icp: nn_idx:  [21, 22, 23, 24, 25, 27, 28, 30, 32, 33, 34, 36, 38, 40, 42, 0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 4, 41, 39, 37, 35, 11, 13, 15, 31, 26, 19, 18, 16, 14, 17, 20, 12, 29]
icp: converged in  3  iteration(s)
icp: nn_idx:  [5, 7, 10, 14, 17, 16, 11, 9, 13, 19, 22, 24, 26, 30, 33, 36, 40, 42, 41, 37, 32, 34, 38, 0, 1, 2, 4]
icp: nn_idx:  [5, 7, 10, 14, 17, 16, 11, 9, 13, 19, 22, 24, 26, 30, 33, 36, 40, 42, 41, 37, 32, 34, 38, 0, 1, 3, 4]
icp: nn_idx:  [5, 7, 10, 14, 17, 16, 11, 9, 13, 19, 22, 24, 26, 30, 33, 36, 40, 42, 41, 37, 32, 34, 38, 0, 1, 3, 4]
icp: converged in  2  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.2)
pltx = 5
plty = np.int(len(ctrs_all[1]) / pltx)
if len(ctrs_all[1]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[1])):
    if det_numbers2[i] == labels2[i]:
        results[labels2[i]][0] += 1
    results[labels2[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers2[i])
    for s in similarities2[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004533p:plain

-1 :  11  /  17
0 :  7  /  7
1 :  7  /  7
2 :  6  /  6
3 :  1  /  1
5 :  7  /  8
subimgs = []
subctrs = []
det_numbers3 = []
similarities3 = []
dbg_imgs = []
for i in range(len(ctrs_all[2])):
    subimg, subctr = create_contour_area_image(resized_imgs[2], ctrs_all[2], i)
    debug_number= 0 if labels3[i] == -1 else labels3[i]
    det_number, sim, img = determine_number(subctr, binimgs3[0], binimgs3[1:5], subctrs3[1:5], subctrs3_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers3 += [det_number]
    similarities3 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [101, 110, 119, 131, 138, 146, 154, 161, 169, 176, 184, 191, 199, 207, 218, 227, 236, 235, 237, 25, 33, 40, 48, 56, 57, 52, 58, 65, 72, 94]
icp: nn_idx:  [95, 106, 118, 128, 136, 145, 153, 161, 169, 177, 185, 193, 202, 209, 221, 233, 243, 242, 15, 23, 31, 39, 47, 55, 56, 51, 57, 65, 72, 78]
icp: nn_idx:  [94, 105, 118, 128, 137, 145, 153, 161, 169, 177, 185, 193, 202, 213, 224, 236, 247, 246, 15, 23, 31, 39, 47, 55, 56, 51, 57, 64, 72, 78]
icp: nn_idx:  [94, 106, 118, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 215, 226, 238, 249, 7, 15, 23, 31, 39, 47, 55, 56, 51, 57, 64, 72, 78]
icp: nn_idx:  [95, 107, 119, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 216, 228, 240, 251, 8, 16, 23, 31, 39, 47, 55, 56, 52, 57, 65, 72, 78]
icp: nn_idx:  [95, 107, 120, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 217, 229, 241, 252, 8, 16, 24, 32, 40, 48, 56, 57, 52, 58, 65, 72, 78]
icp: nn_idx:  [95, 107, 120, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 218, 229, 242, 253, 8, 16, 24, 32, 40, 48, 56, 57, 52, 58, 66, 73, 78]
icp: nn_idx:  [96, 108, 120, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 218, 230, 242, 253, 8, 16, 24, 32, 40, 48, 56, 57, 53, 58, 66, 73, 78]
icp: nn_idx:  [96, 108, 120, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 218, 230, 243, 253, 8, 16, 24, 32, 40, 48, 56, 57, 53, 58, 66, 73, 78]
icp: nn_idx:  [96, 108, 120, 129, 137, 145, 153, 161, 169, 177, 185, 193, 202, 218, 230, 243, 253, 8, 16, 24, 32, 40, 48, 56, 57, 53, 58, 66, 73, 78]
icp: converged in  9  iteration(s)
icp: nn_idx:  [231, 236, 242, 246, 14, 22, 30, 37, 44, 51, 57, 64, 65, 63, 66, 73, 80, 86, 92, 97, 103, 108, 114, 120, 125, 132, 138, 145, 151, 158, 164, 39, 33, 25, 234, 191, 187, 180, 186, 195, 202, 220, 226]
icp: nn_idx:  [230, 237, 244, 250, 12, 20, 28, 36, 43, 50, 57, 64, 63, 65, 66, 73, 77, 82, 88, 95, 102, 109, 116, 123, 128, 131, 138, 145, 151, 158, 165, 38, 31, 24, 234, 193, 188, 181, 187, 196, 204, 217, 224]

...

icp: nn_idx:  [150, 0, 1, 3, 6, 20, 23, 30, 34, 41, 46, 51, 57, 62, 71, 73, 75, 78, 80, 84, 89, 94, 99, 100, 101, 97, 93, 95, 98, 104, 105, 110, 115, 118, 122, 123, 121, 120, 126, 128, 137, 144, 148]
icp: nn_idx:  [150, 151, 1, 3, 6, 20, 23, 30, 34, 41, 46, 51, 57, 62, 71, 73, 75, 78, 80, 84, 89, 94, 99, 100, 101, 97, 93, 95, 98, 104, 105, 110, 115, 118, 122, 123, 121, 120, 126, 128, 137, 144, 148]
icp: nn_idx:  [150, 151, 1, 3, 6, 20, 23, 30, 34, 41, 46, 51, 57, 62, 71, 73, 75, 78, 80, 84, 89, 94, 99, 100, 101, 97, 93, 95, 98, 104, 105, 110, 115, 118, 122, 123, 121, 120, 126, 128, 137, 144, 148]
icp: converged in  11  iteration(s)
icp: nn_idx:  [82, 83, 88, 94, 95, 96, 98, 99, 101, 106, 130, 132, 137, 144, 150, 1, 6, 10, 18, 25, 23, 122, 123, 118, 113, 115, 34, 38, 45, 51, 58, 64, 72, 77, 81]
icp: nn_idx:  [83, 82, 88, 95, 94, 96, 98, 100, 101, 106, 129, 131, 137, 146, 150, 1, 5, 10, 18, 25, 23, 122, 123, 118, 113, 115, 34, 38, 45, 52, 58, 65, 71, 75, 79]
icp: nn_idx:  [83, 82, 88, 95, 94, 96, 98, 100, 101, 106, 129, 131, 137, 146, 151, 1, 5, 10, 18, 25, 23, 122, 123, 118, 113, 115, 34, 38, 45, 52, 58, 65, 70, 74, 78]
icp: nn_idx:  [82, 83, 88, 95, 94, 96, 98, 100, 101, 106, 129, 131, 137, 146, 151, 1, 5, 10, 18, 25, 23, 122, 123, 119, 113, 115, 34, 38, 45, 52, 58, 65, 70, 74, 78]
icp: nn_idx:  [83, 82, 89, 95, 94, 96, 98, 100, 101, 106, 129, 131, 137, 146, 151, 1, 5, 10, 18, 25, 23, 122, 123, 119, 113, 115, 34, 38, 45, 52, 58, 65, 70, 74, 78]
icp: nn_idx:  [83, 82, 89, 95, 94, 96, 98, 100, 101, 106, 129, 131, 137, 146, 151, 1, 5, 10, 18, 25, 23, 122, 123, 119, 113, 115, 34, 38, 45, 52, 58, 65, 70, 74, 78]
icp: converged in  5  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.2)
pltx = 5
plty = np.int(len(ctrs_all[2]) / pltx)
if len(ctrs_all[2]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[2])):
    if det_numbers3[i] == labels3[i]:
        results[labels3[i]][0] += 1
    results[labels3[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers3[i])
    for s in similarities3[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004538p:plain

-1 :  18  /  21
0 :  4  /  4
1 :  13  /  15
2 :  4  /  4
3 :  0  /  0
5 :  4  /  4
subimgs = []
subctrs = []
det_numbers4 = []
similarities4 = []
dbg_imgs = []
for i in range(len(ctrs_all[3])):
    subimg, subctr = create_contour_area_image(resized_imgs[3], ctrs_all[3], i)
    debug_number= 0 if labels4[i] == -1 else labels4[i]
    det_number, sim, img = determine_number(subctr, binimgs3[0], binimgs3[1:5], subctrs3[1:5], subctrs3_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers4 += [det_number]
    similarities4 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [32, 34, 36, 38, 39, 41, 44, 45, 47, 51, 53, 55, 1, 3, 5, 6, 9, 10, 12, 15, 50, 48, 46, 26, 25, 23, 24, 27, 28, 30]
icp: nn_idx:  [32, 34, 36, 37, 39, 42, 44, 45, 47, 51, 53, 55, 1, 3, 5, 7, 9, 10, 12, 15, 50, 48, 46, 26, 25, 23, 24, 27, 28, 30]
icp: nn_idx:  [32, 34, 35, 37, 39, 42, 44, 45, 47, 51, 53, 54, 1, 3, 5, 7, 9, 10, 12, 15, 50, 48, 46, 26, 25, 23, 24, 27, 28, 30]
icp: nn_idx:  [32, 34, 35, 37, 39, 42, 43, 45, 47, 51, 53, 54, 1, 3, 5, 7, 9, 10, 12, 15, 50, 48, 46, 26, 25, 23, 24, 27, 28, 30]
icp: nn_idx:  [32, 34, 35, 37, 38, 42, 43, 45, 47, 51, 53, 54, 1, 3, 5, 7, 9, 10, 12, 15, 50, 48, 46, 26, 25, 23, 24, 27, 28, 30]
icp: nn_idx:  [32, 34, 35, 37, 38, 42, 43, 45, 47, 51, 53, 54, 1, 3, 5, 7, 9, 10, 12, 15, 50, 48, 23, 26, 25, 22, 24, 27, 28, 30]
icp: nn_idx:  [32, 33, 35, 37, 38, 42, 43, 45, 47, 51, 53, 54, 1, 3, 5, 7, 9, 10, 12, 15, 50, 20, 23, 26, 25, 22, 24, 27, 28, 30]
icp: nn_idx:  [31, 33, 35, 37, 38, 42, 43, 45, 47, 51, 53, 54, 0, 3, 6, 7, 9, 11, 12, 15, 19, 20, 23, 26, 25, 22, 24, 27, 28, 30]
icp: nn_idx:  [31, 33, 35, 37, 38, 42, 43, 45, 47, 51, 52, 54, 0, 3, 6, 8, 10, 11, 13, 15, 19, 20, 23, 25, 26, 22, 24, 27, 28, 29]
icp: nn_idx:  [31, 33, 35, 37, 38, 42, 43, 45, 47, 49, 52, 54, 0, 3, 7, 8, 10, 12, 13, 15, 19, 20, 23, 25, 26, 22, 24, 27, 28, 29]
icp: nn_idx:  [31, 33, 35, 37, 38, 42, 43, 45, 47, 49, 52, 54, 0, 3, 7, 9, 11, 12, 13, 15, 19, 20, 23, 25, 26, 22, 24, 27, 28, 29]
icp: nn_idx:  [31, 33, 35, 37, 38, 42, 43, 45, 47, 49, 52, 54, 0, 3, 7, 9, 11, 12, 13, 15, 19, 20, 23, 25, 26, 22, 24, 27, 28, 29]
icp: converged in  11  iteration(s)
icp: nn_idx:  [7, 8, 10, 11, 13, 15, 17, 19, 21, 23, 26, 27, 28, 25, 24, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 43, 46, 48, 50, 51, 14, 12, 54, 52, 53, 55, 0, 2, 4, 6]
icp: nn_idx:  [7, 8, 9, 11, 13, 15, 17, 19, 20, 23, 26, 27, 25, 28, 24, 29, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 46, 48, 50, 51, 12, 10, 54, 53, 52, 55, 0, 2, 4, 6]
icp: nn_idx:  [7, 8, 9, 11, 13, 15, 17, 19, 20, 23, 26, 27, 25, 28, 24, 29, 30, 31, 32, 33, 34, 36, 37, 38, 39, 40, 41, 42, 44, 46, 48, 50, 51, 12, 10, 54, 53, 52, 55, 0, 2, 4, 6]

...

icp: nn_idx:  [2, 4, 7, 10, 13, 25, 30, 35, 39, 46, 51, 56, 61, 66, 75, 77, 79, 82, 84, 87, 93, 97, 103, 102, 104, 100, 96, 98, 99, 106, 108, 112, 117, 119, 125, 126, 127, 122, 123, 130, 131, 139, 0]
icp: nn_idx:  [2, 4, 7, 10, 12, 25, 30, 35, 39, 46, 51, 56, 61, 66, 75, 77, 79, 82, 84, 87, 93, 98, 103, 102, 104, 100, 96, 97, 99, 106, 108, 112, 117, 119, 125, 126, 127, 123, 122, 130, 131, 139, 0]
icp: nn_idx:  [2, 4, 7, 10, 12, 25, 30, 35, 39, 46, 51, 56, 61, 66, 75, 77, 79, 82, 84, 87, 93, 98, 103, 102, 104, 100, 96, 97, 99, 106, 108, 112, 117, 119, 125, 126, 127, 123, 122, 130, 131, 139, 0]
icp: converged in  9  iteration(s)
icp: nn_idx:  [84, 85, 89, 95, 96, 97, 100, 99, 102, 105, 112, 120, 132, 137, 3, 7, 12, 17, 22, 28, 30, 125, 124, 119, 114, 39, 38, 41, 47, 54, 61, 68, 73, 77, 81]
icp: nn_idx:  [83, 85, 89, 96, 95, 97, 56, 100, 102, 106, 112, 120, 131, 138, 4, 8, 12, 17, 21, 29, 30, 125, 124, 119, 114, 39, 38, 42, 47, 55, 61, 67, 73, 77, 81]
icp: nn_idx:  [83, 85, 89, 96, 95, 97, 56, 100, 102, 106, 113, 120, 130, 138, 5, 8, 12, 17, 22, 29, 30, 125, 124, 119, 114, 39, 38, 42, 47, 55, 61, 68, 73, 76, 80]
icp: nn_idx:  [82, 85, 89, 96, 95, 97, 56, 100, 102, 106, 113, 120, 130, 138, 5, 8, 12, 17, 23, 29, 30, 125, 124, 119, 114, 39, 38, 42, 48, 55, 61, 68, 72, 76, 80]
icp: nn_idx:  [82, 84, 90, 96, 95, 97, 56, 100, 102, 106, 113, 120, 130, 138, 5, 8, 12, 17, 23, 29, 31, 125, 124, 119, 114, 39, 38, 42, 49, 55, 62, 68, 72, 76, 80]
icp: nn_idx:  [82, 84, 90, 96, 95, 97, 56, 100, 102, 106, 113, 120, 130, 138, 5, 8, 12, 17, 23, 29, 31, 125, 124, 119, 114, 39, 38, 42, 49, 55, 62, 68, 72, 76, 80]
icp: converged in  5  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.2)
pltx = 5
plty = np.int(len(ctrs_all[3]) / pltx)
if len(ctrs_all[3]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[3])):
    if det_numbers4[i] == labels4[i]:
        results[labels4[i]][0] += 1
    results[labels4[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers4[i])
    for s in similarities4[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004543p:plain

-1 :  14  /  22
0 :  2  /  2
1 :  10  /  11
2 :  4  /  4
3 :  0  /  0
5 :  1  /  2
subimgs = []
subctrs = []
det_numbers5 = []
similarities5 = []
dbg_imgs = []
for i in range(len(ctrs_all[4])):
    subimg, subctr = create_contour_area_image(resized_imgs[4], ctrs_all[4], i)
    debug_number= 0 if labels5[i] == -1 else labels5[i]
    det_number, sim, img = determine_number(subctr, binimgs5[0], binimgs5[1:5], subctrs5[1:5], subctrs5_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers5 += [det_number]
    similarities5 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [44, 47, 50, 53, 56, 63, 67, 72, 78, 85, 89, 91, 3, 18, 28, 38, 37, 42]
icp: nn_idx:  [44, 47, 50, 53, 57, 63, 67, 72, 78, 84, 89, 91, 4, 18, 28, 38, 37, 42]
icp: nn_idx:  [44, 48, 50, 53, 57, 63, 67, 72, 78, 84, 89, 91, 4, 18, 31, 38, 37, 42]
icp: nn_idx:  [44, 48, 50, 53, 57, 63, 67, 72, 78, 84, 89, 92, 4, 18, 31, 38, 37, 42]
icp: nn_idx:  [44, 48, 50, 53, 57, 63, 67, 72, 78, 84, 89, 92, 4, 18, 31, 38, 37, 42]
icp: converged in  4  iteration(s)
icp: nn_idx:  [83, 88, 89, 90, 92, 3, 6, 24, 32, 35, 38, 37, 40, 43, 46, 47, 48, 49, 51, 53, 54, 56, 59, 62, 29, 28, 19, 10, 11, 13, 74, 79]

...

icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 22, 21, 20]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 20, 21, 22]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 22, 21, 20]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 20, 21, 22]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 22, 21, 20]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 20, 21, 22]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 22, 21, 20]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 20, 21, 22]
icp: nn_idx:  [24, 25, 27, 29, 31, 32, 35, 37, 40, 41, 44, 45, 43, 46, 0, 1, 2, 4, 6, 8, 10, 11, 12, 14, 18, 39, 33, 30, 23, 22, 21, 20]
icp: Not converged
icp: nn_idx:  [1, 2, 4, 5, 7, 9, 11, 13, 14, 16, 18, 19, 21, 22, 24, 25, 26, 28, 29, 31, 32, 34, 36, 35, 33, 30, 27, 23, 20, 17, 39, 40, 41, 12, 6, 3, 46, 44, 42, 43, 45, 0, 8]
icp: nn_idx:  [1, 3, 5, 6, 7, 8, 10, 12, 14, 16, 17, 20, 21, 22, 24, 25, 27, 28, 29, 31, 32, 34, 35, 36, 33, 30, 26, 23, 19, 18, 39, 40, 41, 13, 9, 4, 46, 44, 43, 42, 45, 0, 2]
icp: nn_idx:  [1, 3, 5, 6, 7, 8, 10, 12, 14, 16, 17, 20, 21, 22, 24, 25, 27, 28, 29, 31, 32, 34, 35, 36, 33, 30, 26, 23, 19, 18, 39, 40, 41, 13, 9, 4, 46, 44, 43, 42, 45, 0, 2]
icp: converged in  2  iteration(s)
icp: nn_idx:  [18, 20, 19, 14, 13, 17, 23, 27, 29, 31, 34, 39, 43, 46, 0, 44, 40, 37, 41, 2, 3, 5, 7, 10, 15]
icp: nn_idx:  [17, 20, 19, 14, 13, 18, 23, 27, 29, 31, 34, 38, 43, 45, 0, 44, 39, 40, 41, 2, 3, 5, 7, 10, 15]

...

icp: nn_idx:  [0, 2, 4, 6, 8, 17, 21, 24, 27, 31, 34, 37, 40, 44, 48, 50, 52, 54, 56, 58, 61, 65, 68, 69, 67, 66, 63, 64, 36, 70, 71, 74, 77, 78, 81, 82, 83, 80, 84, 86, 88, 91, 94]
icp: nn_idx:  [0, 2, 4, 6, 8, 17, 21, 24, 27, 31, 34, 37, 40, 44, 48, 50, 52, 54, 56, 58, 61, 65, 68, 69, 67, 66, 63, 64, 36, 70, 71, 74, 77, 78, 81, 82, 83, 80, 84, 86, 88, 91, 94]
icp: converged in  8  iteration(s)
icp: nn_idx:  [56, 59, 64, 65, 66, 67, 70, 75, 86, 90, 0, 6, 10, 14, 20, 21, 81, 77, 74, 29, 32, 38, 44, 49, 54]
icp: nn_idx:  [56, 59, 64, 65, 66, 67, 70, 75, 86, 90, 0, 6, 10, 14, 20, 21, 81, 77, 74, 29, 32, 38, 44, 50, 54]
icp: nn_idx:  [56, 59, 64, 65, 66, 67, 70, 75, 86, 90, 1, 6, 10, 14, 20, 21, 81, 77, 74, 29, 32, 38, 44, 50, 54]
icp: nn_idx:  [56, 59, 64, 65, 66, 67, 70, 75, 86, 90, 1, 6, 10, 14, 20, 21, 81, 77, 74, 29, 32, 38, 44, 50, 54]
icp: converged in  3  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.2)
pltx = 5
plty = np.int(len(ctrs_all[4]) / pltx)
if len(ctrs_all[4]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[4])):
    if det_numbers5[i] == labels5[i]:
        results[labels5[i]][0] += 1
    results[labels5[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers5[i])
    for s in similarities5[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004548p:plain

-1 :  5  /  6
0 :  4  /  4
1 :  14  /  16
2 :  4  /  5
3 :  0  /  0
5 :  2  /  4
subimgs = []
subctrs = []
det_numbers6 = []
similarities6 = []
dbg_imgs = []
for i in range(len(ctrs_all[5])):
    subimg, subctr = create_contour_area_image(resized_imgs[5], ctrs_all[5], i)
    debug_number= 0 if labels6[i] == -1 else labels6[i]
    det_number, sim, img = determine_number(subctr, binimgs5[0], binimgs5[1:5], subctrs5[1:5], subctrs5_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers6 += [det_number]
    similarities6 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [41, 45, 48, 51, 56, 65, 73, 82, 97, 103, 105, 109, 119, 85, 21, 32, 34, 38]
icp: nn_idx:  [41, 45, 48, 51, 56, 65, 73, 82, 97, 103, 105, 109, 119, 10, 21, 32, 34, 38]
icp: nn_idx:  [41, 45, 48, 51, 56, 65, 73, 82, 97, 103, 105, 109, 0, 10, 21, 32, 34, 38]
icp: nn_idx:  [41, 45, 48, 51, 56, 65, 73, 82, 97, 103, 105, 109, 0, 10, 21, 32, 34, 38]
icp: converged in  3  iteration(s)
icp: nn_idx:  [40, 46, 47, 48, 52, 56, 63, 70, 76, 88, 92, 94, 91, 99, 103, 104, 105, 108, 109, 110, 111, 0, 2, 85, 16, 72, 68, 59, 34, 30, 33, 38]
icp: nn_idx:  [41, 46, 47, 48, 53, 57, 63, 71, 76, 88, 92, 91, 95, 99, 102, 103, 105, 108, 109, 110, 111, 0, 3, 85, 16, 72, 68, 59, 34, 32, 33, 38]
icp: nn_idx:  [41, 46, 47, 48, 53, 57, 64, 71, 76, 88, 92, 91, 95, 99, 102, 103, 105, 108, 109, 110, 111, 0, 3, 85, 16, 72, 68, 59, 34, 32, 33, 38]
icp: nn_idx:  [41, 46, 47, 48, 53, 57, 64, 71, 76, 88, 92, 91, 95, 99, 102, 103, 105, 108, 109, 110, 111, 0, 3, 85, 16, 72, 68, 59, 34, 32, 33, 38]
icp: converged in  3  iteration(s)
icp: nn_idx:  [12, 21, 30, 34, 39, 42, 45, 46, 37, 48, 49, 50, 52, 55, 61, 66, 73, 81, 90, 98, 101, 103, 105, 115, 92, 83, 76, 72, 68, 27, 78, 16, 15, 26, 25, 24, 14, 85, 3, 112, 111, 0, 8]

...

icp: nn_idx:  [74, 75, 77, 79, 0, 9, 12, 15, 17, 20, 23, 25, 28, 31, 33, 35, 36, 38, 40, 43, 45, 48, 51, 50, 52, 49, 47, 27, 26, 53, 54, 56, 59, 60, 62, 63, 64, 61, 65, 66, 67, 70, 72]
icp: nn_idx:  [73, 75, 77, 79, 0, 10, 12, 15, 17, 20, 23, 25, 28, 31, 33, 35, 36, 38, 40, 43, 45, 48, 51, 50, 52, 49, 47, 27, 26, 53, 54, 56, 59, 60, 62, 63, 64, 61, 65, 66, 67, 69, 72]
icp: nn_idx:  [73, 75, 77, 78, 0, 10, 12, 15, 17, 20, 23, 25, 28, 31, 33, 34, 36, 38, 40, 43, 45, 48, 51, 50, 52, 49, 47, 27, 26, 53, 54, 56, 59, 60, 62, 63, 64, 61, 65, 66, 67, 69, 72]
icp: nn_idx:  [73, 75, 77, 78, 0, 10, 12, 15, 17, 20, 23, 25, 28, 31, 33, 34, 36, 38, 40, 43, 45, 48, 51, 50, 52, 49, 47, 27, 26, 53, 54, 56, 59, 60, 62, 63, 64, 61, 65, 66, 67, 69, 72]
icp: converged in  7  iteration(s)
icp: nn_idx:  [41, 44, 48, 49, 51, 50, 54, 57, 67, 71, 75, 79, 1, 6, 11, 12, 62, 59, 17, 19, 21, 27, 31, 34, 38]
icp: nn_idx:  [41, 44, 48, 49, 51, 50, 53, 57, 67, 71, 75, 79, 2, 6, 11, 12, 62, 59, 58, 17, 21, 26, 31, 35, 39]
icp: nn_idx:  [42, 44, 48, 49, 51, 50, 53, 57, 67, 70, 75, 79, 1, 6, 11, 12, 62, 59, 58, 17, 21, 26, 31, 36, 40]
icp: nn_idx:  [42, 44, 48, 49, 51, 50, 53, 57, 67, 70, 75, 79, 1, 6, 11, 12, 62, 59, 58, 17, 21, 26, 31, 36, 40]
icp: converged in  3  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.2)
pltx = 5
plty = np.int(len(ctrs_all[5]) / pltx)
if len(ctrs_all[5]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[5])):
    if det_numbers6[i] == labels6[i]:
        results[labels6[i]][0] += 1
    results[labels6[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers6[i])
    for s in similarities6[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004553p:plain

-1 :  4  /  4
0 :  4  /  4
1 :  7  /  16
2 :  4  /  5
3 :  0  /  0
5 :  0  /  4
subimgs = []
subctrs = []
det_numbers7 = []
similarities7 = []
dbg_imgs = []
for i in range(len(ctrs_all[6])):
    subimg, subctr = create_contour_area_image(resized_imgs[6], ctrs_all[6], i)
    debug_number= 0 if labels7[i] == -1 else labels7[i]
    det_number, sim, img = determine_number(subctr, binimgs5[0], binimgs5[1:5], subctrs5[1:5], subctrs5_selected_pts, debug_number=debug_number)
    subimgs += [subimg]
    subctrs += [subctr]
    det_numbers7 += [det_number]
    similarities7 += [sim]
    dbg_imgs += [img]
icp: nn_idx:  [141, 154, 176, 1, 9, 15, 20, 26, 33, 40, 63, 82, 84, 79, 157, 158, 130, 133]
icp: nn_idx:  [140, 153, 175, 1, 8, 14, 20, 27, 33, 41, 62, 82, 84, 79, 157, 158, 145, 135]
icp: nn_idx:  [141, 154, 176, 1, 8, 14, 21, 27, 34, 41, 62, 81, 83, 79, 157, 158, 146, 137]
icp: nn_idx:  [142, 154, 176, 1, 8, 15, 21, 28, 34, 41, 62, 81, 83, 79, 158, 159, 146, 137]
icp: nn_idx:  [142, 155, 176, 1, 8, 15, 21, 28, 35, 41, 62, 81, 83, 79, 158, 159, 146, 137]
icp: nn_idx:  [143, 155, 176, 1, 8, 15, 21, 28, 35, 41, 62, 81, 83, 79, 158, 159, 146, 138]
icp: nn_idx:  [143, 155, 177, 1, 8, 15, 21, 28, 35, 41, 62, 81, 79, 78, 158, 159, 146, 138]
icp: nn_idx:  [143, 155, 177, 1, 8, 15, 21, 28, 35, 41, 61, 80, 79, 78, 159, 160, 146, 138]
icp: nn_idx:  [144, 155, 177, 1, 8, 15, 21, 28, 35, 41, 61, 79, 78, 77, 159, 160, 146, 139]
icp: nn_idx:  [144, 156, 177, 1, 9, 15, 21, 28, 35, 41, 61, 79, 78, 77, 160, 159, 146, 139]
icp: nn_idx:  [144, 156, 177, 1, 9, 15, 21, 28, 35, 41, 61, 79, 78, 77, 160, 159, 146, 140]
icp: nn_idx:  [144, 156, 177, 1, 9, 15, 21, 28, 35, 41, 61, 78, 79, 77, 160, 161, 146, 140]
icp: nn_idx:  [145, 156, 177, 1, 9, 15, 21, 28, 35, 41, 61, 78, 79, 77, 160, 161, 146, 140]
icp: nn_idx:  [145, 156, 177, 1, 9, 15, 21, 28, 35, 41, 61, 78, 79, 77, 160, 161, 146, 140]
icp: converged in  13  iteration(s)
icp: nn_idx:  [146, 157, 167, 177, 189, 0, 13, 18, 25, 53, 60, 51, 30, 34, 41, 52, 62, 72, 83, 93, 104, 99, 91, 79, 74, 176, 181, 175, 163, 156, 128, 132]


...

icp: nn_idx:  [1, 3, 5, 7, 9, 18, 22, 25, 29, 32, 36, 39, 43, 46, 50, 52, 54, 56, 58, 62, 66, 69, 73, 72, 71, 70, 67, 68, 38, 74, 75, 78, 82, 83, 86, 87, 88, 85, 89, 91, 93, 96, 0]
icp: nn_idx:  [1, 3, 5, 7, 9, 18, 22, 25, 29, 32, 36, 39, 43, 46, 50, 52, 54, 56, 58, 62, 66, 69, 73, 72, 71, 70, 67, 68, 38, 74, 75, 78, 82, 83, 86, 87, 88, 85, 89, 91, 93, 96, 0]
icp: converged in  8  iteration(s)
icp: nn_idx:  [59, 63, 68, 69, 70, 71, 74, 79, 91, 95, 2, 6, 10, 14, 21, 22, 86, 82, 78, 30, 33, 40, 46, 52, 56]
icp: nn_idx:  [59, 62, 68, 69, 70, 71, 75, 79, 91, 95, 2, 6, 10, 15, 21, 22, 86, 82, 78, 30, 34, 40, 46, 53, 57]
icp: nn_idx:  [59, 63, 68, 67, 70, 71, 75, 79, 91, 95, 2, 7, 10, 15, 21, 22, 86, 82, 78, 31, 34, 40, 46, 53, 57]
icp: nn_idx:  [59, 63, 68, 67, 70, 71, 75, 79, 91, 95, 2, 7, 10, 15, 22, 21, 86, 82, 78, 31, 34, 41, 47, 53, 57]
icp: nn_idx:  [59, 63, 68, 67, 70, 71, 75, 80, 91, 96, 2, 7, 10, 15, 22, 21, 86, 82, 78, 31, 34, 41, 47, 53, 57]
icp: nn_idx:  [59, 63, 68, 67, 70, 71, 75, 80, 91, 97, 2, 7, 10, 15, 22, 21, 86, 82, 78, 31, 34, 41, 47, 53, 57]
icp: nn_idx:  [59, 63, 68, 67, 70, 71, 75, 80, 91, 97, 2, 7, 10, 15, 22, 21, 86, 82, 78, 31, 34, 41, 47, 53, 57]
icp: converged in  6  iteration(s)
plt.figure(figsize=(12.8,20),dpi=100)
plt.subplots_adjust(wspace=2, hspace=1.2)
pltx = 5
plty = np.int(len(ctrs_all[6]) / pltx)
if len(ctrs_all[6]) % pltx:
    plty +=1
# Dictionary containing number of correct answers and number of same labels
results = {-1:[0,0], 0:[0,0], 1:[0,0], 2:[0,0], 3:[0,0], 5:[0,0]}
for i in range(len(ctrs_all[6])):
    if det_numbers7[i] == labels7[i]:
        results[labels7[i]][0] += 1
    results[labels7[i]][1] += 1
    title = 'Number:%d\n (' %(det_numbers7[i])
    for s in similarities7[i]:
        title += '%.2f ' %(s)
    title += ')'
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+1), plt.imshow(cv2.cvtColor(subimgs[i],cv2.COLOR_BGR2RGB)), plt.title(title),plt.xticks([]),plt.yticks([])
    plt.subplot(plty*2, pltx, np.int(i/5)*10+i%5+6), plt.imshow(dbg_imgs[i], cmap='gray'),plt.xticks([]),plt.yticks([])
plt.show()
for k,v in results.items():
    print(k, ': ', v[0], ' / ', v[1])

f:id:nokixa:20220223004558p:plain

-1 :  8  /  9
0 :  0  /  0
1 :  6  /  6
2 :  4  /  9
3 :  0  /  0
5 :  0  /  0

ざっと全画像で試してみた感じ、うまく検出できていないところもありますが、なんとか閾値等調整すればいけそうな気が。

あと、時々ICPが収束しないことがありました。
様子を見てみると、最近傍点セットが2つのパターンを繰り返しています。
下に一部取り出して掲載します。
収束条件も後で見直します。

icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 9, 4, 3, 1]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 9, 4, 3, 1]
icp: nn_idx:  [5, 8, 11, 17, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 37, 42, 41, 0, 40, 39, 2, 6, 7, 4, 3, 1, 9]
icp: Not converged

一致度の分布を確認

上で見た各数字への一致度を、ヒストグラムで出してみたいと思います。
データは一致データ、不一致データの2要素に分けて表示します。
また、年ごとの点数文字フォントの違いで状況も違うかもしれないので、別々に表示します。

labels = labels1 + labels2
sims = similarities1 + similarities2
numbers = [0,1,2,3,5]
plt.figure(figsize=(20, 4.8), dpi=100)
plt.suptitle('Year 2019')
for i,n in enumerate(numbers):
    t = [s[i] for j,s in enumerate(sims) if labels[j]==n]
    f = [s[i] for j,s in enumerate(sims) if labels[j]!=n]
    plt.subplot(1,5,1+i), plt.hist([t,f], 20, [0.5,1.0], stacked=False, color=['orange', 'green'])
    plt.title('Number: %d' %(n))
plt.show()

f:id:nokixa:20220223004602p:plain

labels = labels3 + labels4
sims = similarities3 + similarities4
plt.figure(figsize=(20, 4.8), dpi=100)
plt.suptitle('Year 2020')
for i,n in enumerate(numbers):
    t = [s[i] for j,s in enumerate(sims) if labels[j]==n]
    f = [s[i] for j,s in enumerate(sims) if labels[j]!=n]
    plt.subplot(1,5,1+i), plt.hist([t,f], 20, [0.5,1.0], stacked=False, color=['orange', 'green'])
    plt.title('Number: %d' %(n))
plt.show()

f:id:nokixa:20220223004605p:plain

labels = labels5 + labels6 + labels7
sims = similarities5 + similarities6 + similarities7
plt.figure(figsize=(20, 4.8), dpi=100)
plt.suptitle('Year 2021')
for i,n in enumerate(numbers):
    t = [s[i] for j,s in enumerate(sims) if labels[j]==n]
    f = [s[i] for j,s in enumerate(sims) if labels[j]!=n]
    plt.subplot(1,5,1+i), plt.hist([t,f], 20, [0.5,1.0], stacked=False, color=['orange', 'green'])
    plt.title('Number: %d' %(n))
plt.show()

f:id:nokixa:20220223004608p:plain

一致データと不一致データとで、一致度がきっちり分かれているものもありますが、そうでないものもあります。
そうなると確実な閾値が設定できない…特に"1"と"5"が難しい。

一旦ここまで

長くなったのでここで区切ります。
Jupyter notebook的にはまだ続いているので、次回は今回の結果を引き継いだ状態でスタートします。