0

I installed apache spark and apache Livy in my system. When I am running a python code, it's giving error 

'u'java.lang.IllegalStateException: Session is in state starting''

By default Apchy Livy run on port number 8998. My python code is 

import json, pprint, requests, textwrap

host = 'http://localhost:8998'
data = {'kind': 'pyspark'}
headers = {'Content-Type': 'application/json'}
r = requests.post(host + '/sessions', data=json.dumps(data),   
headers=headers)

session_url = host + r.headers['location']
statements_url = session_url + '/statements'


data = {
  'code': textwrap.dedent("""
          import random
          NUM_SAMPLES = 100000
          def sample(p):
          x, y = random.random(), random.random()
          return 1 if x*x + y*y < 1 else 0

          count = sc.parallelize(xrange(0, 
                    NUM_SAMPLES)).map(sample).reduce(lambda a, b: a +b)
          print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
          """)
      }

r = requests.post(statements_url, data=json.dumps(data),   
    headers=headers)
pprint.pprint(r.json())

{u'id': 12,
 u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
            u'execution_count': 12,
            u'status': u'ok'},
 u'state': u'running'}

How I can resolve this error?

4

2 回答 2

1

创建会话后，您需要检查返回的状态。提交语句，只有在你的会话状态变成idle. 下面的代码对我有用。

您可以打印返回的值来调试您的设置。如果该状态持续存在starting或dead您的设置存在问题。也许是一些许可问题或任何其他关于通过 livy 启动 spark 上下文的问题。在 livy UI stderr 中查看。

    uri = "/sessions"
    data = {'kind': 'pyspark'
            }
    headers = {'Content-Type': 'application/json'}
    r = requests.post(url=self.host + uri, data=json.dumps(data),
                      headers=headers)

    response = r.json()
    sessionId = response['id']
    
    while response['state'] != "idle":
        r = requests.get(self.host + "/sessions/" + str(sessionId), headers=headers)
        response = r.json()
        # print(r.json())

    return response['id']

此返回 sessionId 现在已准备好用于提交语句。您只需致电POST /sessions/{sessionId}/statements提交您的报表即可。

于 2020-09-06T14:14:12.583 回答
1

每当您在 Livy 中创建新的 spark 会话时，它都需要一段时间才能进入空闲状态。在创建会话之后，您直接将代码发布到该特定的 spark 会话，该会话由于引发了异常而仍处于启动状态。

尝试做这样的事情-：

import json, pprint, requests, textwrap
host = 'http://192.168.0.56:8998'
data = {'kind': 'spark'}
headers = {'Content-Type': 'application/json'}
r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
print(headers['location'])

从标题中获取 url 位置后，请遵循此

session_url = "http://localhost/"+paste headers value
r = requests.get(session_url, headers=headers)
statements_url = session_url + '/statements'
data = {'code': '1 + 1'}
r = requests.post(statements_url, data=json.dumps(data), headers=headers)
print(r.json())
statement_url = session_url
r = requests.get(statement_url, headers=headers)
pprint.pprint(r.json())
于 2018-03-20T12:45:29.273 回答