0

making my first steps connecting sparkR to AWS cluster, I come across a problem:

I cannot create sparkcontext ('sc') in Rstudio - 

> .libPaths( c( .libPaths(), '/root/spark/R/lib') ) 

> Sys.setenv(SPARK_HOME = '/root/spark')

> Sys.setenv(PATH = paste(Sys.getenv(c("PATH")), '/root/spark/bin', sep=':'))

> library(SparkR)

> sc <- sparkR.init('ec2-54-159-90-154.compute-1.amazonaws.com:7077')

and i get:

    15/07/16 13:20:55 INFO SparkContext: Running Spark version 1.4.0
    15/07/16 13:20:55 INFO SecurityManager: Changing view acls to: rootroot1
    15/07/16 13:20:55 INFO SecurityManager: Changing modify acls to: rootroot1
    15/07/16 13:20:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rootroot1); users with modify permissions: Set(rootroot1)
    15/07/16 13:20:56 INFO Slf4jLogger: Slf4jLogger started
    15/07/16 13:20:56 INFO Remoting: Starting remoting
    15/07/16 13:20:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.101.183.7:36597]
    15/07/16 13:20:56 INFO Utils: Successfully started service 'sparkDriver' on port 36597.
    15/07/16 13:20:56 INFO SparkEnv: Registering MapOutputTracker
    15/07/16 13:20:56 INFO SparkEnv: Registering BlockManagerMaster
    15/07/16 13:20:56 ERROR Utils: Failed to create local root dir in /mnt/spark. Ignoring this directory.
    15/07/16 13:20:56 ERROR Utils: Failed to create local root dir in /mnt2/spark. Ignoring this directory.
    15/07/16 13:20:56 ERROR DiskBlockManager: Failed to create any local dir.
    15/07/16 13:20:56 INFO Utils: Shutdown hook called
    Error in readTypedObject(con, type) : 
      Unsupported type for deserialization

I've tried many users

any help would be appreciated Tnx Z

4

1 回答 1

0

You'll need to change the permissions for /mnt/spark and /mnt2/spark to be writable by all users (its only writable by root by default). You can do this by running chmod a+w /mnt/spark. There are a few more steps you need to do to get RStudio to work on EC2. http://blog.godatadriven.com/spark-rstudio.html is a good tutorial for this

于 2015-07-17T23:25:11.830 回答