在pig中尝试使用Python UDF时，无法存储别名C

Question

#commaFormat- format a number with commas, 12345-> 12,345
 @outputSchema("numformat:chararray")
 def commaFormat(num):
   return '{:,}'.format(num)

DEFINE CSVExcelStorage org.apache.pig.piggybank.storage.CSVExcelStorage;
A = LOAD '/result.csv' using CSVExcelStorage() As (id:int,lastvisitedtime:chararray,title:chararray,typedcount:int,URL:chararray,visitcount:int,bytes:int);
B = limit A 15;
REGISTER '/data/pyudf/test.py' USING streaming_python AS myudfs;
C = FOREACH B generate myudfs.commaFormat($1);

在org.apache.pig.impl.builtin.streamingUDF$processerrorthread.run(streamingUDF.java:503)

Answer 1

首先，在define语句中缺少（）。

REGISTER /path/piggybank.jar;
DEFINE CSVExcelStorage org.apache.pig.piggybank.storage.CSVExcelStorage();

您可能正在使用Mortar的cPython发行版，该发行版至少需要pig0.12。尝试使用jython脚本引擎。

REGISTER '/data/pyudf/test.py' USING jython AS myudfs;
C = FOREACH B generate myudfs.commaFormat($1);

或者，您可以使用REPLACE函数轻松地删除逗号，而不是编写UDF。

REGISTER /path/piggybank.jar;
DEFINE CSVExcelStorage org.apache.pig.piggybank.storage.CSVExcelStorage();
A = LOAD '/result.csv' using CSVExcelStorage() AS (id:int,lastvisitedtime:chararray,title:chararray,typedcount:int,URL:chararray,visitcount:int,bytes:int);
B = FOREACH A GENERATE id,REPLACE(lastvisitedtime,',',''),title,typedcount,URL,visitcount,bytes;
C = LIMIT B 15;
DUMP C;