Spark - объект «LDAModel» имеет атрибута «describeTopics»

Question

я в настоящее время с помощью CDH Спарк 1.5.0, Python 2.6.6 Hadoop 2.6

Я пытаюсь построить модель LDA со ссылкой на эту ссылку Saprk1.5.0-Latent Dirichlet allocation (LDA)

Цитирование от того, что написано в документации:

Все модели поддерживают LDA MLlib в:

• descri beTopics: Возвращает темы, как массивы наиболее важных терминов и долгосрочных весов
• topicsMatrix: Возвращает vocabSize на к матрице, где каждый столбец является темой

Я хочу осуществить LDA в режиме describeTopics

Код (воспроизводимые):

from pyspark.mllib.clustering import LDA, LDAModel 
from pyspark.mllib.linalg import Vectors 
from pyspark import SparkConf, SparkContext 
from pyspark.sql import SQLContext, Row 

conf = SparkConf().setAppName("test").set("spark.executor.memory", "512m") 
sc = SparkContext(conf = conf) 
sc.setLogLevel('ERROR') 
sqlContext = SQLContext(sc) 


# Load and parse the data 
data = sc.parallelize([[0,0,0,0,0,1,0,1,0,0,0,0,0,0,1,0,0,0,0,0,1,0,1,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,1,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,1,0,1,1,0,1],[0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,1,0,0,0,1,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,1,0,0,0,0,0,2,0,0,0,0,0,1,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0],[0,1,1,1,0,0,0,0,0,1,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,1,1,0,1,0,0,0,1,0,0,0,0,0,0,0,0,1,1,0,0,0,1,0,0,1,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,1,0,0,1,0,0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,1,0,0,0,1,0,0,0,1,1,0,0,1,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0],[1,0,0,0,1,0,0,0,0,0,1,0,0,1,0,1,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0,0,1,0,0,1,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,1,0,0,0,1,0,1,0,0,0,1,0,1,0,0,0,0,1,0]]) 
#data = sc.textFile("file://data.txt") 

parsedData = data.map(lambda line: Vectors.dense([float(x) for x in line.strip().split(' ')])) 
# Index documents with unique IDs 
corpus = parsedData.zipWithIndex().map(lambda x: [x[1], x[0]]).cache() 

# Cluster the documents into three topics using LDA 
ldaModel = LDA.train(corpus, k=3) 

# Output topics. Each is a distribution over words (matching word count vectors) 
print("Learned topics (as distributions over vocab of " + str(ldaModel.vocabSize()) + " words):") 
#topics = ldaModel.topicsMatrix() 
topics = ldaModel.describeTopics(maxTermsPerTopic = 10) 

for topic in range(3): 
    print("Topic " + str(topic) + ":") 
    for word in range(0, ldaModel.vocabSize()): 
    print(" " + str(topics[word][topic])) 

Однако я получаю следующее сообщение об ошибке

AttributeError: 'LDAModel' object has no attribute 'describeTopics' 

ОписатьТопики не поддерживаются Spark? Здесь что-то не хватает?

Answer 1

Ожидаемое поведение. describeTopics в PySpark MLLib была введена в Спарк 1.6:

• SPARK-8467Добавить LDAModel.describeTopics() в Python.
• describeTopics documentatation.
• describeTopics source.