R application, SVR

• Text mining
• Text mining in R: a tutorial
• https://www.springboard.com/blog/text-mining-in-r
• RSQLite, tm, SnowballC, Wordcloud, Syuzhet(text sentiment), quanteda

• Support Vector Regression에 대한 친철한 설명
• http://www.svm-tutorial.com/2014/10/support-vector-regression-r/
• 데이터 정제
• Metric 설정: RMSE
• SVR의 최적 parameter 설정 위한 grid search

R packages

• R installation: v3.3.2 (2017.02.13)
• https://cran.r-project.org/
• $ sudo apt-get install r-base
• 만약, r 최신 버전이 설치되지 않는다면, 직접 sources.list 파일에 cran 미러 주소 입력 및 apt-get update 해준 뒤 다시 실행 필요.
• $ sudo gedit /etc/apt/sources.list 
• 만약, 이 과정에서 해당 미러의 공개키 없다고 나오면, 다음 블로그 참조하여, gpg 만들기
• http://tip.daum.net/question/58336339

• dplyr
• 공식 튜토리얼: https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html
• 예시, 설명: https://github.com/justmarkham/dplyr-tutorial
• notebook, youtube
• 참고할 만한 블로그: http://blog.daum.net/buillee/269

• tidyverse
• Hadley Wickham의 주요 package를 모았음 (2016.9)
• ggplot2, purrr, readr, tibble, tidyr, dplyr
• install.packages("tidyverse")
• $ sudo apt install libxml2-dev libssl-dev libcurl4-openssl-dev

• R for data science [book]
• Hadley Wickham이 공동저자로 참여한 R 관련 도서(2016)
• http://r4ds.had.co.nz/index.html

• bookdown: Authoring Books and Technical Documents with R Markdown
• R Markdown으로 출간 도서 및 기술 문서 작성하는 방법
• https://bookdown.org/yihui/bookdown/

• TSdist: distance 계산
• error-in-installing-rgl-package because of X11
• $ sudo apt install r-cran-rgl
• $ sudo apt install xorg openbox
• $ sudo apt install libX11-dev freeglut3 freeglut3-dev

• rattle
• $ sudo apt install libxml2-dev r-cran-xml

Ubuntu test client setting, tips

Test client ip address example

• Address 10.100.0.112
• Netmask 255.255.255.0
• Gateway 10.100.0.1
• DNS server 10.100.0.1

Ip setting for performance testing

• sudo ifconfig eth0 10.100.0.112 netmask 255.255.255.0 broadcast 10.100.0.255

Jmeter multi server testing configuration

• jmeter-server.sh -Djava.rmi.server.hostname=[local ip address]
• jmeter.sh -Djava.rmi.server.hostname=10.100.0.112

Setting the limit (open files)

• ulimit -n
• gedit /etc/security/limits.conf

MongoDB

• sudo service mongod start
• sudo service mongod stop

redis-server

• sudo /etc/init.d/redis-server stop

R Studio (rstudio-server-0.99.893, 20160326) : localhost:8787

HTTP Server

• python -m SimpleHTTPServer 8080
• python -mSimpleHTTPServer

ipython notebook : 127.0.0.1:8888

• ipython notebook

JMT

• java -jar JMT.jar

file rename: rename s/^[0-9]{3}// *.mp3

system problem : sudo rm /var/crash/*

scp user@xxx.xxx.xxx.xxx:/home/user/datafile ./

How to set up R Notebook

Requirement:

• Install r (2016.11.14) v3.2.3
• $ sudo apt-get install r-base
• Download and install RStudio v1.0.136 (2017.02.13)
• Installer version

• As a new feature of RStudio since v1.0.
• An R Notebook is an R Markdown document.
• http://rmarkdown.rstudio.com/r_notebooks.html

• Short key
• ctrl + shift + enter : Run the code chunk
• ctrl + shift + k : preview
• ctrl + alt + i : new code chunk

Issue:

• For publishing R Markdown document, if you meet following message when installing "RCurl";

Cannot find curl-config  
ERROR: configuration failed for package ‘RCurl’ 

• Then, install libcurlxxxx-dev as following.
  
  sudo apt-get install libcurl4-gnutls-dev

• When installing R Markdown, if you could not install stringr package, then remove 00LOCK-XXX folder in the R package directory.

To initialize Apache Spark (2.0.+) R on R Studio (1.0.+) on Ubuntu OS (14.04+)

SparkR

Objective

• To initialize Apache Spark (2.0.+) R on R Studio (1.0.+) on Ubuntu OS (14.04+)

Environment

• Java 1.8.0_111
• Ubuntu 14.04 LTS

How to install

• Download Apache Spark
• http://spark.apache.org/downloads.html
• spark-2.0.1-bin-hadoop2.7.tgz
• then, extract the tar ball

• Download RStudio Desktop
• https://www.rstudio.com/products/rstudio/download3/


• RStudio 1.0.44 - Ubuntu 12.04+/Debian 8+ (64-bit)
• NOTE: R and major R packages are already included in this file.
• then, install RStudio

• Load SparkR Package with RStudio

library(SparkR, lib.loc = "/home/[your spark directory]/spark-2.0.1-bin-hadoop2.7/R/lib")

• Open sparkR session

sparkR.session(sparkHome = "/home/[your spark directory]/spark-2.0.1-bin-hadoop2.7/",                master = "local[*]", sparkConfig = list(spark.driver.memory = "4g"))

• Test the following code.

df <- as.DataFrame(faithful)

head(df)

• References
• https://spark.apache.org/docs/latest/sparkr.html#sparkdataframe 

References

• SparkR API에 대한 고찰 (2017.07.20)
• Spark 프로그래밍 패러다임에 SparkR이 다소 잘 들어맞지 않는 특성들을 설명.
• SparkDataFrame(Spark 클러스터내의 복수 머신)과 R의 data.frame(메모리기반; 로컬 머신)은 차이가 있음.