OpenCV和Tesseract关于门标签检测

Question

我对OpenCv和tesseract相当陌生。我最近正在建立一个使用计算机视觉检测门标签的项目。希望这对视障群体有益。

该方案的思想是对输入图像进行二值化预处理，然后用canny边缘检测门牌轮廓，最后对canny边缘结果进行扩展。之后，将图像输入到tesseract，同时尝试显示用方框检测到的文本。

预期结果是文本上的绿色矩形。同时打印出文本本身。

问题是缺少矩形和文本检测失败。

我尝试过这些：

1. 在 Opencv 中使用精明边缘检测识别图像中的文本
2. OpenCv pytesseract for OCR
3. 在执行字符识别之前使用 OpenCV 对图像进行预处理（镶嵌）

问题和解决方案要么过于简单，要么不那么相关。有些人也不在蟒蛇中。

下面是我对代码的尝试：

import pytesseract as pytess
import cv2 as cv
import numpy as np
from PIL import Image
from pytesseract import Output

img = cv.imread(r"C:\Users\User\Desktop\dataset\p\Image_31.jpg", 0)

# edges store the canny version of img
edges = cv.Canny(img, 100, 200)

# ker as in kernel
# (5, 5) is the matrix while uint8 is datatype
ker = np.ones((3, 3), np.uint8)

# dil as in dilation
# edges as the src, ker is the kernel we set above, number of dilation
dil = cv.dilate(edges, ker, iterations=1)

# setup pytesseract parameters
configs = r'--oem 3 --psm 6'

# feed image to tesseract
result = pytess.image_to_data(dil, output_type=Output.DICT, config=configs, lang='eng')
print(result.keys())

boxes = len(result['text'])

# make a new copy of edges
new_item = dil.copy()

for sequence_number in range(boxes):
    if int(result['conf'][sequence_number]) > 30: # removed constraints
        (x, y, w, h) = (result['left'][sequence_number], result['top'][sequence_number],
                        result['width'][sequence_number], result['height'][sequence_number])
        new_item = cv.rectangle(new_item, (x, y), (x + w, y + h), (0, 255, 0), 2)

# detect sentence with tesseract
# pending as rectangle not achieved

cv.imshow("original", img)
cv.imshow("canny", edges)
cv.imshow("dilation", dil)
cv.imshow("capturedText", new_item)

#ignore below this line, it is only for testing

#testobj = Image.fromarray(dil)
#testtext = pytess.image_to_string(testobj, lang='eng')
#print(testtext)


cv.waitKey(0)
cv.destroyAllWindows()

代码的测试部分返回结果，如下所示：

a)
Meets

这显然不符合目标。

编辑

发布问题后，我意识到我一开始可能做错了。在发送矩形中的任何内容进行OCR识别之前，我应该尝试使用OpencV检测门标签的轮廓并隔离包含文本的部分。

第二版

由于我们的堆栈溢出成员，现在我发现了这个问题，现在我正尝试添加图像校正/图像包装技术来检索直前视图，以获得更好的系统准确性。很快更新。

编辑3

在修复了某些错误之后，在允许函数在原始图像上绘制的同时减少约束，我实现了以下结果。还附加了更新的代码。

import cv2 as cv
import numpy as np
import pytesseract as pytess
from pytesseract import Output

# input of img source
img = cv.imread(r"C:\Users\User\Desktop\dataset\p\Image_31.jpg")

# necessary image color conversion
img2 = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

# edges store the canny version of img
edges = cv.Canny(img2, 100, 200)

# ker as in kernel
# (5, 5) is the matrix while uint8 is datatype
ker = np.ones((3, 3), np.uint8)

# dil as in dilation
# edges as the src, ker is the kernel we set above, number of dilation
dil = cv.dilate(edges, ker, iterations=1)

# setup pytesseract parameters
configs = r'--oem 3 --psm 6'

# feed image to tesseract
result = pytess.image_to_data(dil, output_type=Output.DICT, config=configs, lang='eng')

# number of boxes that encapsulate the boxes
boxes = len(result['text'])

# make a new copy of edges
new_item = dil.copy()

for sequence_number in range(boxes):
    if int(result['conf'][sequence_number]) > 0: #removed constraints
        (x, y, w, h) = (result['left'][sequence_number], result['top'][sequence_number],
                        result['width'][sequence_number], result['height'][sequence_number])

        # draw rectangle boxes on the original img
        cv.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 3)
        # Crop the image
        crp = new_item[y:y + h, x:x + w]
        # OCR
        txt = pytess.image_to_string(crp, config=configs)
        # returns recognised text
        print(txt)
        cv.imshow("capturedText", crp)
        cv.waitKey(0)

# cv.imshow("original", img)
# cv.imshow("canny", edges)
# cv.imshow("dilation", dil)
cv.imshow("results", img)
cv.waitKey(0)
cv.destroyAllWindows()

Answer 1

我认为你在这里寻找的是图像矫正（扭曲图像，使其看起来像是从另一个角度拍摄的），而python中似乎有这样的工具。然而，问题变得更加复杂，因为在您的情况下，您需要检测您想要如何纠正它。我不知道你该怎么做。

Answer 2

您已在图像中找到所有检测到的文本：

for sequence_number in range(boxes):
    if int(result['conf'][sequence_number]) > 30:
        (x, y, w, h) = (result['left'][sequence_number], result['top'][sequence_number],
                        result['width'][sequence_number], result['height'][sequence_number])
        new_item = cv.rectangle(new_item, (x, y), (x + w, y + h), (0, 255, 0), 2)

但你也说目前的信心应该在70%以上。

< li >如果我们取消限制 < li >如果我们对每个新项目进行OCR each >

结果将是:

现在，如果您阅读：

txt = pytesseract.image_to_string(new_item, config="--psm 6")
print(txt)

OCR将是：

Meeting Room §

当前pytesseract版本0.3.7的输出

代码：

# Load the libraries
import cv2
import pytesseract

# Load the image
img = cv2.imread("fsUSw.png")

# Convert it to the gray-scale
gry = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# OCR detection
d = pytesseract.image_to_data(gry, config="--psm 6", output_type=pytesseract.Output.DICT)

# Get ROI part from the detection
n_boxes = len(d['level'])

# For each detected part
for i in range(1, 2):
    # Get the localized region
    (x, y, w, h) = (d['left'][i], d['top'][i], d['width'][i], d['height'][i])

    # Draw rectangle to the detected region
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 5)

    # Crop the image
    crp = gry[y:y + h, x:x + w]
    
    # OCR
    txt = pytesseract.image_to_string(crp, config="--psm 6")
    print(txt)

    # Display the cropped image
    cv2.imshow("crp", crp)
    cv2.waitKey(0)

# Display
cv2.imshow("img", img)
cv2.waitKey(0)