PyTorch dataloader显示字符串数据集的奇怪行为

Question

我正在处理NLP问题，正在使用PyTorch。由于某些原因，我的数据加载器返回了格式错误的批。我有由句子和整数标签组成的输入数据。这些句子可以是句子列表，也可以是标记列表。稍后我将在下游组件中将标记转换为整数。

list_labels = [ 0, 1, 0]

# List of sentences.
list_sentences = [ 'the movie is terrible',
                   'The Film was great.',
                   'It was just awful.']

# Or list of list of tokens.
list_sentences = [['the', 'movie', 'is', 'terrible'],
                  ['The', 'Film', 'was', 'great.'],
                  ['It', 'was', 'just', 'awful.']]

我创建了以下自定义数据集：

import torch
from torch.utils.data import DataLoader, Dataset

class MyDataset(torch.utils.data.Dataset):

    def __init__(self, sentences, labels):

        self.sentences = sentences
        self.labels = labels

    def __getitem__(self, i):
        result = {}
        result['sentences'] = self.sentences[i]
        result['label'] = self.labels[i]
        return result

    def __len__(self):
        return len(self.labels)

当我以句子列表的形式提供输入时，数据加载器正确地返回成批完整的句子。请注意，batch\u size=2：

list_sentences = [ 'the movie is terrible', 'The Film was great.', 'It was just awful.']
list_labels = [ 0, 1, 0]


dataset = MyDataset(list_sentences, list_labels)
dataloader = DataLoader(dataset, batch_size=2)

batch = next(iter(dataloader))
print(batch)
# {'sentences': ['the movie is terrible', 'The Film was great.'], <-- Great! 2 sentences in batch!
#  'label': tensor([0, 1])}

批次正确地包含两句话和两个标签，因为batch\u size=2。

然而，当我将句子作为标记列表的预标记列表输入时，我得到了奇怪的结果：

list_sentences = [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.'], ['It', 'was', 'just', 'awful.']]
list_labels = [ 0, 1, 0]


dataset = MyDataset(list_sentences, list_labels)
dataloader = DataLoader(dataset, batch_size=2)

batch = next(iter(dataloader))
print(batch)
# {'sentences': [('the', 'The'), ('movie', 'Film'), ('is', 'was'), ('terrible', 'great.')], <-- WHAT?
#  'label': tensor([0, 1])}

请注意，此批的句子是一个包含单词对元组的单一列表。我希望句子是两个列表的列表，如下所示：

{'sentences': [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.']

到底是怎么回事？




Answer 1

另一种解决方案是在数据集中将字符串编码为字节，然后在转发过程中对其进行解码。如果您希望包含元数据的字符串（如数据来自的文件路径），但实际上不需要将数据传递到模型中，这将非常有用。

例如：

class MyDataset(torch.utils.data.Dataset):
    def __next__(self):
        return np.array("this is a sentence").bytes()

然后在向前传球时，你会做：

sentences: List[str] = []
for sentence in batch:
    sentences.append(sentence.decode("ascii"))

Answer 2

此行为是因为默认的collate\u fn在必须整理列表时会执行以下操作（这是[“句子”]的情况）：

# [...]
elif isinstance(elem, container_abcs.Sequence):
    # check to make sure that the elements in batch have consistent size
    it = iter(batch)
    elem_size = len(next(it))
    if not all(len(elem) == elem_size for elem in it):
        raise RuntimeError('each element in list of batch should be of equal size')
    transposed = zip(*batch)
    return [default_collate(samples) for samples in transposed]

出现“问题”的原因是，在最后两行中，当批处理是容器时，它将递归调用zip（*batch）。序列（和列表是），而zip的行为如下。

如你所见：

batch = [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.']]
list(zip(*batch))

# [('the', 'The'), ('movie', 'Film'), ('is', 'was'), ('terrible', 'great.')]

在您的案例中，除了实现一个新的collator并将其传递给数据加载程序（…，collate\u fn=mycollator）之外，我看不到任何解决方法。例如，一个简单丑陋的例子可能是：

def mycollator(batch):
    assert all('sentences' in x for x in batch)
    assert all('label' in x for x in batch)
    return {
        'sentences': [x['sentences'] for x in batch],
        'label': torch.tensor([x['label'] for x in batch])
    }