Orca Known Issues¶
Estimator Issues¶
UnkownError: Could not start gRPC server¶
This error occurs while running Orca TF2 Estimator with spark backend, which may because the previous pyspark tensorflow job was not cleaned completely. You can retry later or you can set spark config spark.python.worker.reuse=false
in your application.
If you are using init_orca_context(cluster_mode="yarn-client")
:
conf = {"spark.python.worker.reuse": "false"}
init_orca_context(cluster_mode="yarn-client", conf=conf)
If you are using init_orca_context(cluster_mode="spark-submit")
:
spark-submit --conf spark.python.worker.reuse=false
RuntimeError: Inter op parallelism cannot be modified after initialization¶
This error occurs if you build your TensorFlow model on the driver rather than on workers. You should build the complete model in model_creator
which runs on each worker node. You can refer to the following examples:
Wrong Example
model = ...
def model_creator(config):
model.compile(...)
return model
estimator = Estimator.from_keras(model_creator=model_creator,...)
...
Correct Example
def model_creator(config):
model = ...
model.compile(...)
return model
estimator = Estimator.from_keras(model_creator=model_creator,...)
...
OrcaContext Issues¶
Exception: Failed to read dashbord log: [Errno 2] No such file or directory: ‘/tmp/ray/…/dashboard.log’¶
This error occurs when initialize an orca context with init_ray_on_spark=True
. We have not locate the root cause of this problem, but it might be caused by an atypical python environment.
You could follow below steps to workaround:
If you only need to use functions in ray (e.g.
bigdl.orca.learn
withbackend="ray"
,bigdl.orca.automl
for pytorch/tensorflow model,bigdl.chronos.autots
for time series model’s auto-tunning), we may use ray as the first-class.Start a ray cluster by
ray start --head
. if you already have a ray cluster started, please direcetly jump to step 2.Initialize an orca context with
runtime="ray"
andinit_ray_on_spark=False
, please refer to detailed information here.If you are using
bigdl.orca.automl
orbigdl.chronos.autots
on a single node, please set:ray_ctx = OrcaContext.get_ray_context() ray_ctx.is_local=True
If you really need to use ray on spark, please install bigdl-orca under a conda environment. Detailed information please refer to here.