在python中针对json模式验证tsv

Question

我试图根据Python中的JSON模式验证TSV文件的行。下面是该模式的一个示例：

{
  "title": "employee",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "surname": {
      "type": "string"
    },
    "position": {
      "type": "string"
    },
    "telephone": {
      "type": "integer"
    },
  "required": [
    "name",
    "surname",
    "position",
    "telephone"
  ]
}

TSV文件如下所示：

NAME\tSURNAME\tPOSITION\tTELEPHONE
JOHN\tJOHNSON\tSECRETARY\t11111

null

Answer 1

您可以使用一些pandas和JSON进行自己的验证。请注意，您的模式示例无效，因为您有一个打开的花括号。我假设您打算在required标记之前关闭properties。下面的示例使用了以下内容：

{
  "title": "employee",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "surname": {
      "type": "string"
    },
    "position": {
      "type": "integer"
    },
    "telephone": {
      "type": "integer"
    }
  },
  "required": [
    "name",
    "surname",
    "position",
    "telephone"
  ]
}

注意，我还将position的数据类型更改为integer，作为一个不验证的示例。json模式存储在名为input.json的文件中，表格数据存储在名为input.tsv的文件中：

name  surname   position  telephone
John  Johnson  SECRETORY     111111

首先，我们导入模块并读取数据，如下所示：

import pandas as pd
import json

# Read tabular data
df = pd.read_csv("input.tsv", sep="\t")

# Read JSON data
with open("input.json") as json_file:
    data = json.load(json_file)

# Check that all required columns are present
required = data["required"]
cols_ok = all([col in df.columns for col in required])  # True
print("All required columns present: %s" % (cols_ok))

# Restructure the data types of our tabular data into a separate dataframe
dtypes = df.dtypes.reset_index()
dtypes.columns = ["column", "data_type"]
# There are discrepancies in naming conventions between your JSON file
# and how pandas reports data types, so we need to change them to a common format
dtypes.data_type = dtypes.data_type.replace("object", "string")  # Pandas calls it "object", you call it "string"
dtypes.data_type = dtypes.data_type.replace("int64", "integer")  # Pandas calls it "int64", you call it "integer"

# Extract the column names and the expected data type for that column
for column_name, subdict in data["properties"].items():
    expected_dtype = subdict["type"]
    actual_dtype = dtypes[dtypes.column == column_name].data_type.iloc[0]
    # Check if it matches our actual data types
    if expected_dtype == actual_dtype:
        print("Correct data type for column '%s' (%s)" % (column_name, expected_dtype))
    else:
        print("WRONG data type for column '%s' (expected '%s', found '%s')" % (column_name, expected_dtype, actual_dtype))

输出：

Correct data type for column 'name' (string)
Correct data type for column 'surname' (string)
WRONG data type for column 'position' (expected 'integer', found 'string')
Correct data type for column 'telephone' (integer)