Cluster Serving Example#

There are some examples provided for new user or existing Tensorflow user.

Quick Start Example#

Following is the recommended quick start example to transfer a local Keras application to Cluster Serving.

keras-to-cluster-serving-example

End-to-end Example#

Model/Data Convert Guide#

This guide is for users who:

have written local code of Tensorflow, Pytorch(to be added)
have used specified data type of a specific framework, e.g. TFDataSet
want to deploy the local code on Cluster Serving but do not know how to write client code (Cluster Serving takes Numpy Ndarray as input, other types need to transform in advance).

**If you have the above needs but fail to find the solution below, please create issue here

Tensorflow#

Model - includes savedModel, Frozen Graph (savedModel is recommended).

Data - includes TFDataSet, Tokenizer, ImageDataGenerator

Notes - includes tips to notice, includes savedModel tips

Model - ckpt to savedModel#

tensorflow all version#

This method works in all version of TF

You need to create the graph, get the output layer, create place holder for input, load the ckpt then save the model

# --- code you need to write
input_layer = tf.placeholder(...)
model = YourModel(...)
output_layer = model.your_output_layer()
# --- code you need to write
with tf.Session() as sess:
    saver = tf.train.Saver()
    saver.restore(sess, tf.train.latest_checkpoint(FLAGS.ckpt_path))
    tf.saved_model.simple_save(sess,
                               FLAGS.export_path,
                               inputs={
                                   'input_layer': input_layer
                               },
                               outputs={"output_layer": output_layer})

tensorflow >= 1.15#

This method works if you are familiar with savedModel signature, and tensorflow >= 1.15

model graph could be load via .meta, and load ckpt then save the model, signature_def_map is required to provide

# provide signature first
inputs = tf.placeholder(...)
outputs = tf.add(inputs, inputs)
tensor_info_input = tf.saved_model.utils.build_tensor_info(inputs)
tensor_info_output = tf.saved_model.utils.build_tensor_info(outputs)

prediction_signature = (
  tf.saved_model.signature_def_utils.build_signature_def(
      inputs={'x_input': tensor_info_input},
      outputs={'y_output': tensor_info_output},
      method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))

      
# Your ckpt file is prefix.meta, prefix.index, etc
ckpt_prefix = 'model/model.ckpt-xxxx'
export_dir = 'saved_model'

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # load
    loader = tf.train.import_meta_graph(ckpt_prefix + '.meta')
    loader.restore(sess, ckpt_prefix)

    # export
    builder = tf.saved_model.builder.SavedModelBuilder(export_dir)
    builder.add_meta_graph_and_variables(sess,
                                         [tf.saved_model.tag_constants.TRAINING, tf.saved_model.tag_constants.SERVING],signature_def_map={
      tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
          prediction_signature 
      }
    )
    builder.save()

Model - Keras to savedModel#

tensorflow > 2.0#

model = tf.keras.models.load_model("./model.h5")
tf.saved_model.save(model, "saved_model")

Model - ckpt to Frozen Graph#

freeze checkpoint example

Notes - Use SavedModel#

If model has single tensor input, then nothing to notice.

If model has multiple input, please notice following.

When export, savedModel would store the inputs in alphabetical order. Use saved_model_cli show --dir . --all to see the order. e.g.

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['id1'] tensor_info:
        dtype: DT_INT32
        shape: (-1, 512)
        name: id1:0
    inputs['id2'] tensor_info:
        dtype: DT_INT32
        shape: (-1, 512)
        name: id2:0

when enqueue to Cluster Serving, follow this order

Data#

To transform following data type to Numpy Ndarray, following examples are provided