MapReale-保留输入顺序

Question

包含由管道分隔的数字列表的文件可以有重复项。需要编写map reduce程序，在原始输入顺序中列出不重复的数字。我可以删除重复项，但不保留输入顺序。

Answer 1

这很简单，假设你的文本是：

Line 1 -> On the top of the Crumpetty Tree
Line 2 -> The Quangle Wangle sat,
Line 3 -> But his face you could not see,
Line 4 -> On account of his Beaver Hat.
Line 5 -> But his face you could not see,
Line 6 -> The Quangle Wangle sat,

其中，第2行和第3行在第5行和第6行重复。

映射器应该类似于wordcount程序，其中映射器的输入类似于

键值对：

(0, On the top of the Crumpetty Tree)
(33, The Quangle Wangle sat,)
(57, But his face you could not see,)
(89, On account of his Beaver Hat.)
(113, But his face you could not see,)
(146, The Quangle Wangle sat,)

映射器的输出

(NullWritable, 0_On the top of the Crumpetty Tree)
(NullWritable, 33_The Quangle Wangle sat,)
(NullWritable, 57_But his face you could not see,)
(NullWritable, 89_On account of his Beaver Hat.)
(NullWritable, 113_But his face you could not see,)
(NullWritable, 146_The Quangle Wangle sat,)

现在，确保您只有一个减速器，这样单个减速器的输入将是

减速机输入

Key: NullWritable
Iterable<value>: [(0_On the top of the Crumpetty Tree), 
(33_The Quangle Wangle sat,), 
(57_But his face you could not see,), 
(89_On account of his Beaver Hat.), 
(113_But his face you could not see,), 
(146_The Quangle Wangle sat,)]

请注意，减速机的输入是按升序排序的，在这种情况下，减速机保持原始顺序，因为文本输入格式中的偏移量到行总是按升序排序。

在reducer中，只需遍历列表，剔除重复项，并在删除开头的偏移量和分隔符后写入行。减速器的输出类似于：

减速机键值

NullWritable, value.split("_")[1]

减速机输出

Line 1 -> On the top of the Crumpetty Tree
Line 2 -> The Quangle Wangle sat,
Line 3 -> But his face you could not see,
Line 4 -> On account of his Beaver Hat.