Cloudera spark,RDD为空

Question

我试图在cloudera vm上使用pyspark和hive创建一个数据帧，但每次都出现这个错误。

编辑2-sc=SparkContext(appname=“pythonsentimentanalysis”）sqlCtx=HiveContext（sc）

filenameAFINN = "/home/cloudera/Desktop/TwitterSentimentAnalysis/AFINN/AFINN-111.txt"

 afinn = dict(map(lambda (w, s): (w, int(s)), [ ws.strip().split('\t') for ws in open(filenameAFINN) ]))

filenameCandidate = "file:///home/cloudera/Desktop/TwitterSentimentAnalysis/Candidates/Candidate Mapping.txt"

candidates = sc.textFile(filenameCandidate).map(lambda x: (x.strip().split(",")[0],x.strip().split(","))) \
                   .flatMapValues(lambda x:x).map(lambda y: (y[1],y[0])).distinct()


pattern_split = re.compile(r"\W+")

tweets = sqlCtx.sql("select id, text, entities.user_mentions.name from incremental_tweets")

def sentiment(text):
  words = pattern_split.split(text.lower())
  sentiments = map(lambda word: afinn.get(word, 0), words)
  if sentiments:
   sentiment = float(sum(sentiments))/math.sqrt(len(sentiments))
  else:
   sentiment = 0
   return sentiment

   sentimentTuple = tweets.rdd.map(lambda r: [r.id, r.text, r.name]) \
           .map(lambda r: [sentiment(r[1]),r[2]]) \
           .flatMapValues(lambda x: x) \
           .map(lambda y: (y[1],y[0])) \
           .reduceByKey(lambda x, y: x+y) \
           .sortByKey(ascending=True)

  scoreDF = sentimentTuple.join(candidates) \
        .map(lambda (x,y): (y[1],y[0])) \
        .reduceByKey(lambda a,b: a+b) \
        .toDF()

   scoreRenameDF =  scoreDF.withColumnRenamed("_1","Candidate").withColumnRenamed("_2","Score")

   sqlCtx.registerDataFrameAsTable(scoreRenameDF, "SCORE_TEMP")

   sqlCtx.sql("INSERT OVERWRITE TABLE candidate_score SELECT Candidate, Score FROM SCORE_TEMP")

Answer 1

如果正确创建了中间RDD，请使用下面的代码检查它们：

for i in rdd.take(10):   print(i)

这将显示RDD的前10个条目