Not able to connect to a Remote spark cluster using Sparklyr Livy Method.
config <- livy_config(username="<username>", password="<password>")
sc <- spark_connect(master = "<address>", method = "livy", config = config)
I am getting an error:
Error in value[3L] : Failed to initialize livy connection: Unable to retrieve a spark_connection from object of class function
Livy server is started on the cluster. Other Livy commands are working fine. Remote cluster is a cloudera managed cluster.
The problem resolved when the sparklyr was reinstalled from CRAN, earlier the sparlyr was installed from using devtools::install_github("rstudio/sparklyr")