Trusted Cluster Serving with Graphene on Kubernetes

Prerequisites

Prior to deploying PPML Cluster Serving, please make sure the following is setup

  • Hardware that supports SGX

  • A fully configured Kubernetes cluster

  • Intel SGX Device Plugin to use SGX in K8S cluster (install following instructions here)

  • Java

Deploy Trusted Realtime ML for Kubernetes

  1. Pull docker image from dockerhub

    $ docker pull intelanalytics/bigdl-ppml-trusted-realtime-ml-scala-graphene:2.1.0-SNAPSHOT
    
  2. Pull the source code of BigDL and enter PPML graphene k8s directory

    $ git clone https://github.com/intel-analytics/BigDL.git
    $ cd BigDL/ppml/trusted-realtime-ml/scala/docker-graphene/kubernetes
    
  3. Generate secure keys and passwords, and deploy as secrets (Refer here for details)

    1. Generate keys and passwords

      Note: Make sure to add ${JAVA_HOME}/bin to $PATH to avoid keytool: command not found error.

      $ sudo ../../../../scripts/generate-keys.sh
      $ openssl genrsa -3 -out enclave-key.pem 3072
      $ ../../../../scripts/generate-password.sh <used_password_when_generate_keys>
      
    2. Deploy as secrets for Kubernetes

      $ kubectl apply -f keys/keys.yaml
      $ kubectl apply -f password/password.yaml
      
  4. In values.yaml, configure pulled image name, path of enclave-key.pem generated in step 3 and path of script start-all-but-flink.sh.

  5. If kernel version is 5.11+ with built-in SGX support, create soft links for SGX device

    $ sudo ln -s /dev/sgx_enclave /dev/sgx/enclave
    $ sudo ln -s /dev/sgx_provision /dev/sgx/provision
    

Configure SGX mode

In templates/flink-configuration-configmap.yaml, configure sgx.mode to sgx or nonsgx to determine whether to run the workload with SGX.

Configure Resource for Components

  1. Configure jobmanager resource allocation in templates/jobmanager-deployment.yaml

    ...
    env:
      - name: SGX_MEM_SIZE
        value: "16G"
    ...
    resources:
      requests:
        cpu: 2
        memory: 16Gi
        sgx.intel.com/enclave: "1"
        sgx.intel.com/epc: 16Gi
      limits:
        cpu: 2
        memory: 16Gi
        sgx.intel.com/enclave: "1"
        sgx.intel.com/epc: 16Gi
    ...
    
  2. Configure Taskmanager resource allocation

    • Memory allocation in templates/flink-configuration-configmap.yaml

      taskmanager.memory.managed.size: 4gb
      taskmanager.memory.task.heap.size: 5gb
      xmx.size: 5g
      
    • Pod resource allocation

      Use taskmanager-deployment.yaml instead of taskmanager-statefulset.yaml for functionality test

      $ mv templates/taskmanager-statefulset.yaml ./
      $ mv taskmanager-deployment.yaml.back templates/taskmanager-deployment.yaml
      

      Configure resource in templates/taskmanager-deployment.yaml (allocate 16 cores in this example, please configure according to scenario)

      ...
      env:
        - name: CORE_NUM
          value: "16"
        - name: SGX_MEM_SIZE
          value: "32G"
      ...
      resources:
        requests:
          cpu: 16
          memory: 32Gi
          sgx.intel.com/enclave: "1"
          sgx.intel.com/epc: 32Gi
        limits:
          cpu: 16
          memory: 32Gi
          sgx.intel.com/enclave: "1"
          sgx.intel.com/epc: 32Gi
      ...
      
  3. Configure Redis and client resource allocation

    • SGX memory allocation in start-all-but-flink.sh

       ...
       cd /ppml/trusted-realtime-ml/java
       export SGX_MEM_SIZE=16G
       test "$SGX_MODE" = sgx && ./init.sh
       echo "java initiated"
       ...
      
    • Pod resource allocation in templates/master-deployment.yaml

      ...
      env:
        - name: CORE_NUM  #batchsize per instance
          value: "16"
      ...
      resources:
        requests:
          cpu: 12
          memory: 32Gi
          sgx.intel.com/enclave: "1"
          sgx.intel.com/epc: 32Gi
        limits:
          cpu: 12
          memory: 32Gi
          sgx.intel.com/enclave: "1"
          sgx.intel.com/epc: 32Gi
      ...
      

Deploy Cluster Serving

  1. Deploy all components and start job

    1. Download helm from release page and install

    2. Deploy cluster serving

      $ helm install ppml ./
      
  2. Port forwarding

    Set up port forwarding of jobmanager Rest port for access to Flink WebUI on host

    1. Run kubectl port-forward <flink-jobmanager-pod> --address 0.0.0.0 8081:8081 to forward jobmanager’s web UI port to 8081 on host.

    2. Navigate to http://<host-IP>:8081 in web browser to check status of Flink cluster and job.

  3. Performance benchmark

    $ kubectl exec <master-deployment-pod> -it -- bash
    $ cd /ppml/trusted-realtime-ml/java/work/benchmark/
    $ bash init-benchmark.sh
    $ python3 e2e_throughput.py -n <image_num> -i ../data/ILSVRC2012_val_00000001.JPEG
    

    The e2e_throughput.py script pushes test image for -n times (default 1000 if not manually set), and time the process from push images (enqueue) to retrieve all inference results (dequeue), to calculate cluster serving end-to-end throughput. The output should look like Served xxx images in xxx sec, e2e throughput is xxx images/sec