Как распечатать фрагменты RDD в искровой оболочке/pyspark?

Question

При работе в искровой оболочке я часто хочу проверить RDD (аналогично использованию head в unix).

Например:

scala> val readmeFile = sc.textFile("input/tmp/README.md") 
scala> // how to inspect the readmeFile? 

и ...

scala> val linesContainingSpark = readmeFile.filter(line => line.contains("Spark")) 
scala> // how to inspect linesContainingSpark? 

Answer 1

я узнал, как это сделать (here) и думал, что это будет полезно для других пользователей, поэтому делиться здесь. take(x) выбирает первый х предметов и Foreach печатает их:

scala> val readmeFile = sc.textFile("input/tmp/README.md") 
scala> readmeFile.take(5).foreach(println) 
# Apache Spark 

Spark is a fast and general cluster computing system for Big Data. It provides 
high-level APIs in Scala, Java, and Python, and an optimized engine that 
supports general computation graphs for data analysis. It also supports a 

и ...

scala> val linesContainingSpark = readmeFile.filter(line => line.contains("Spark")) 
scala> linesContainingSpark.take(5).foreach(println) 
# Apache Spark 
Spark is a fast and general cluster computing system for Big Data. It provides 
rich set of higher-level tools including Spark SQL for SQL and structured 
and Spark Streaming. 
You can find the latest Spark documentation, including a programming 

---

Приведенные ниже примеры являются эквивалентом, но с использованием pyspark:

>>> readmeFile = sc.textFile("input/tmp/README.md") 
>>> for line in readmeFile.take(5): print line 
... 
# Apache Spark 

Spark is a fast and general cluster computing system for Big Data. It provides 
high-level APIs in Scala, Java, and Python, and an optimized engine that 
supports general computation graphs for data analysis. It also supports a 

и

>>> linesContainingSpark = readmeFile.filter(lambda line: "Spark" in line) 
>>> for line in linesContainingSpark.take(5): print line 
... 
# Apache Spark 
Spark is a fast and general cluster computing system for Big Data. It provides 
rich set of higher-level tools including Spark SQL for SQL and structured 
and Spark Streaming. 
You can find the latest Spark documentation, including a programming