Ошибка запуска Pyspark — ошибка hive.HiveSessionState

Я установил Spark 2.1 с Cloudera. Когда я запускаю spark-shell из /usr/bin/spark2-shell, он запускается (со scala). Когда я запускаю Pyspark, я получаю эту проблему

sudo -u hdfs ./pyspark2

Я получил:

java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool. Original Exception: ------
java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
......
Caused by: ERROR XBM0H: Directory /usr/bin/metastore_db cannot be created.
Caused by: java.sql.SQLException: Failed to create database
'metastore_db', see the next exception for details
.....
Caused by: ERROR XJ041: Failed to create database 'metastore_db', see the next exception for details.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 105 more
Caused by: ERROR XBM0H: Directory /usr/bin/metastore_db cannot be created.

Traceback (most recent call last):
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/pyspark/shell.py", line 43, in <module>
    spark = SparkSession.builder\
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/pyspark/sql/session.py", line 179, in getOrCreate
    session._jsparkSession.sessionState().conf().setConfString(key, value)
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/pyspark/sql/utils.py", line 79, in deco
    raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"

Я думаю, что это проблема при создании HiveContext из pyspark. Также как запустить pyspark без создания HiveContext. Любая помощь будет оценена по достоинству.