To initialize Apache Spark (2.0.+) R on R Studio (1.0.+) on Ubuntu OS (14.04+)

SparkR

Objective

• To initialize Apache Spark (2.0.+) R on R Studio (1.0.+) on Ubuntu OS (14.04+)

Environment

• Java 1.8.0_111
• Ubuntu 14.04 LTS

How to install

• Download Apache Spark
• http://spark.apache.org/downloads.html
• spark-2.0.1-bin-hadoop2.7.tgz
• then, extract the tar ball

• Download RStudio Desktop
• https://www.rstudio.com/products/rstudio/download3/


• RStudio 1.0.44 - Ubuntu 12.04+/Debian 8+ (64-bit)
• NOTE: R and major R packages are already included in this file.
• then, install RStudio

• Load SparkR Package with RStudio

library(SparkR, lib.loc = "/home/[your spark directory]/spark-2.0.1-bin-hadoop2.7/R/lib")

• Open sparkR session

sparkR.session(sparkHome = "/home/[your spark directory]/spark-2.0.1-bin-hadoop2.7/",                master = "local[*]", sparkConfig = list(spark.driver.memory = "4g"))

• Test the following code.

df <- as.DataFrame(faithful)

head(df)

• References
• https://spark.apache.org/docs/latest/sparkr.html#sparkdataframe 

References

• SparkR API에 대한 고찰 (2017.07.20)
• Spark 프로그래밍 패러다임에 SparkR이 다소 잘 들어맞지 않는 특성들을 설명.
• SparkDataFrame(Spark 클러스터내의 복수 머신)과 R의 data.frame(메모리기반; 로컬 머신)은 차이가 있음.