# Develop your own Big Data & AI applications with BigDL PPML
### 0. Understand E2E Security with PPML
Basic design guidelines for PPML applications are as follows:
* Data in use/computation should be protected by SGX.
* Data in transmit/network should be protected by encryption or TLS.
* Data at rest/storage should be protected by encryption.
This design ensures plain text data only be used in SGX, while in all others stages data is fully encrypted.
![](../images/ppml_dev_basic.png)
To our knowledge, most existing big data frameworks or systems have already provided network or storage protection. You can find more details in [Secure Your Services](https://bigdl.readthedocs.io/en/latest/doc/PPML/QuickStart/secure_your_services.html).
Please check with your admin or security department for security features and services available. We recommend building PPML applications based on the following conditions:
1. If you have network and storage protection enabled, and you want to secure computation with SGX. Then you can directly migrate your application into SGX with BigDL PPML. Please jump to [Migrate existing applications with BigDL PPML](#1-migrate-existing-applications-with-bigdl-ppml).
2. If you don't have any security features enabled, especially storage protection. Then you can use PPMLContext and recommended KMS. Please jump to [Enhance your applications with PPMLContext](#2-enhance-your-applications-with-ppmlcontext).
### 1. Migrate existing applications with BigDL PPML
This working model doesn't require any code change. You can reuse existing code and applications. The only difference is that your cluster manager/admin needs to set up a new execution environment for PPML applications.
You can find more details in these articles:
* [Installation for PPML](https://bigdl.readthedocs.io/en/latest/doc/PPML/Overview/install.html).
* [Hello World Example](https://bigdl.readthedocs.io/en/latest/doc/PPML/Overview/quicktour.html).
* [Deployment for production](https://bigdl.readthedocs.io/en/latest/doc/PPML/QuickStart/deploy_ppml_in_production.html).
### 2. Enhance your applications with PPMLContext
In this section, we will introduce how to secure your applications with `PPMLContext`. It requires a few code changes and configurations for your applications.
First, you need to create a `PPMLContext`, which wraps `SparkSession` and provides methods to read encrypted data files into plain-text RDD/DataFrame and write DataFrame to encrypted data files. Then you can read & write data through `PPMLContext`.
If you are familiar with Spark, you may find that the usage of `PPMLContext` is very similar to Spark.
#### 2.1 Create PPMLContext
- create a PPMLContext with `appName`
This is the simplest way to create a `PPMLContext`. When you don't need to read/write encrypted files, you can use this way to create a `PPMLContext`.
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
val sc = PPMLContext.initPPMLContext("MyApp")
```
python
```python
from bigdl.ppml.ppml_context import *
sc = PPMLContext("MyApp")
```
If you want to read/write encrypted files, then you need to provide more information.
- create a PPMLContext with `appName` & `ppmlArgs`
`ppmlArgs` is ppml arguments in a Map, `ppmlArgs` varies according to the kind of Key Management Service (KMS) you are using. Key Management Service (KMS) is used to generate `primaryKey` and `dataKey` to encrypt/decrypt data. We provide 3 types of KMS ——SimpleKeyManagementService, EHSMKeyManagementService, AzureKeyManagementService.
Refer to [KMS Utils](https://github.com/intel-analytics/BigDL/blob/main/ppml/services/kms-utils/docker/README.md) to use KMS to generate `primaryKey` and `dataKey`, then you are ready to create **PPMLContext** with `ppmlArgs`.
- For `SimpleKeyManagementService`:
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
val ppmlArgs: Map[String, String] = Map(
"spark.bigdl.kms.type" -> "SimpleKeyManagementService",
"spark.bigdl.kms.simple.id" -> "your_app_id",
"spark.bigdl.kms.simple.key" -> "your_app_key",
"spark.bigdl.kms.key.primary" -> "/your/primary/key/path/primaryKey",
"spark.bigdl.kms.key.data" -> "/your/data/key/path/dataKey"
)
val sc = PPMLContext.initPPMLContext("MyApp", ppmlArgs)
```
python
```python
from bigdl.ppml.ppml_context import *
ppml_args = {"kms_type": "SimpleKeyManagementService",
"simple_app_id": "your_app_id",
"simple_app_key": "your_app_key",
"primary_key_path": "/your/primary/key/path/primaryKey",
"data_key_path": "/your/data/key/path/dataKey"
}
sc = PPMLContext("MyApp", ppml_args)
```
- For `EHSMKeyManagementService`:
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
val ppmlArgs: Map[String, String] = Map(
"spark.bigdl.kms.type" -> "EHSMKeyManagementService",
"spark.bigdl.kms.ehs.ip" -> "your_server_ip",
"spark.bigdl.kms.ehs.port" -> "your_server_port",
"spark.bigdl.kms.ehs.id" -> "your_app_id",
"spark.bigdl.kms.ehs.key" -> "your_app_key",
"spark.bigdl.kms.key.primary" -> "/your/primary/key/path/primaryKey",
"spark.bigdl.kms.key.data" -> "/your/data/key/path/dataKey"
)
val sc = PPMLContext.initPPMLContext("MyApp", ppmlArgs)
```
python
```python
from bigdl.ppml.ppml_context import *
ppml_args = {"kms_type": "EHSMKeyManagementService",
"kms_server_ip": "your_server_ip",
"kms_server_port": "your_server_port"
"ehsm_app_id": "your_app_id",
"ehsm_app_key": "your_app_key",
"primary_key_path": "/your/primary/key/path/primaryKey",
"data_key_path": "/your/data/key/path/dataKey"
}
sc = PPMLContext("MyApp", ppml_args)
```
- For `AzureKeyManagementService`
the parameter `clientId` is not necessary, you don't have to provide this parameter.
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
val ppmlArgs: Map[String, String] = Map(
"spark.bigdl.kms.type" -> "AzureKeyManagementService",
"spark.bigdl.kms.azure.vault" -> "key_vault_name",
"spark.bigdl.kms.azure.clientId" -> "client_id",
"spark.bigdl.kms.key.primary" -> "/your/primary/key/path/primaryKey",
"spark.bigdl.kms.key.data" -> "/your/data/key/path/dataKey"
)
val sc = PPMLContext.initPPMLContext("MyApp", ppmlArgs)
```
python
```python
from bigdl.ppml.ppml_context import *
ppml_args = {"kms_type": "AzureKeyManagementService",
"azure_vault": "your_azure_vault",
"azure_client_id": "your_azure_client_id",
"primary_key_path": "/your/primary/key/path/primaryKey",
"data_key_path": "/your/data/key/path/dataKey"
}
sc = PPMLContext("MyApp", ppml_args)
```
- create a PPMLContext with `sparkConf` & `appName` & `ppmlArgs`
If you need to set Spark configurations, you can provide a `SparkConf` with Spark configurations to create a `PPMLContext`.
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
import org.apache.spark.SparkConf
val ppmlArgs: Map[String, String] = Map(
"spark.bigdl.kms.type" -> "SimpleKeyManagementService",
"spark.bigdl.kms.simple.id" -> "your_app_id",
"spark.bigdl.kms.simple.key" -> "your_app_key",
"spark.bigdl.kms.key.primary" -> "/your/primary/key/path/primaryKey",
"spark.bigdl.kms.key.data" -> "/your/data/key/path/dataKey"
)
val conf: SparkConf = new SparkConf().setMaster("local[4]")
val sc = PPMLContext.initPPMLContext(conf, "MyApp", ppmlArgs)
```
python
```python
from bigdl.ppml.ppml_context import *
from pyspark import SparkConf
ppml_args = {"kms_type": "SimpleKeyManagementService",
"simple_app_id": "your_app_id",
"simple_app_key": "your_app_key",
"primary_key_path": "/your/primary/key/path/primaryKey",
"data_key_path": "/your/data/key/path/dataKey"
}
conf = SparkConf()
conf.setMaster("local[4]")
sc = PPMLContext("MyApp", ppml_args, conf)
```
#### 2.2 Read and Write Files
To read/write data, you should set the `CryptoMode`:
- `plain_text`: no encryption
- `AES/CBC/PKCS5Padding`: for CSV, JSON and text file
- `AES_GCM_V1`: for PARQUET only
- `AES_GCM_CTR_V1`: for PARQUET only
To write data, you should set the `write` mode:
- `overwrite`: Overwrite existing data with the content of dataframe.
- `append`: Append new content of the dataframe to existing data or table.
- `ignore: Ignore the current write operation if data/table already exists without any error.
- `error`: Throw an exception if data or table already exists.
- `errorifexists`: Throw an exception if data or table already exists.
scala
```scala
import com.intel.analytics.bigdl.ppml.crypto.{AES_CBC_PKCS5PADDING, PLAIN_TEXT}
// read data
val df = sc.read(cryptoMode = PLAIN_TEXT)
...
// write data
sc.write(dataFrame = df, cryptoMode = AES_CBC_PKCS5PADDING)
.mode("overwrite")
...
```
python
```python
from bigdl.ppml.ppml_context import *
# read data
df = sc.read(crypto_mode = CryptoMode.PLAIN_TEXT)
...
# write data
sc.write(dataframe = df, crypto_mode = CryptoMode.AES_CBC_PKCS5PADDING)
.mode("overwrite")
...
```
expand to see the examples of reading/writing CSV, PARQUET, JSON and text file
The following examples use `sc` to represent an initialized `PPMLContext`
**read/write CSV file**
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
import com.intel.analytics.bigdl.ppml.crypto.{AES_CBC_PKCS5PADDING, PLAIN_TEXT}
// read a plain csv file and return a DataFrame
val plainCsvPath = "/plain/csv/path"
val df1 = sc.read(cryptoMode = PLAIN_TEXT).option("header", "true").csv(plainCsvPath)
// write a DataFrame as a plain csv file
val plainOutputPath = "/plain/output/path"
sc.write(df1, PLAIN_TEXT)
.mode("overwrite")
.option("header", "true")
.csv(plainOutputPath)
// read a encrypted csv file and return a DataFrame
val encryptedCsvPath = "/encrypted/csv/path"
val df2 = sc.read(cryptoMode = AES_CBC_PKCS5PADDING).option("header", "true").csv(encryptedCsvPath)
// write a DataFrame as a encrypted csv file
val encryptedOutputPath = "/encrypted/output/path"
sc.write(df2, AES_CBC_PKCS5PADDING)
.mode("overwrite")
.option("header", "true")
.csv(encryptedOutputPath)
```
python
```python
# import
from bigdl.ppml.ppml_context import *
# read a plain csv file and return a DataFrame
plain_csv_path = "/plain/csv/path"
df1 = sc.read(CryptoMode.PLAIN_TEXT).option("header", "true").csv(plain_csv_path)
# write a DataFrame as a plain csv file
plain_output_path = "/plain/output/path"
sc.write(df1, CryptoMode.PLAIN_TEXT)
.mode('overwrite')
.option("header", True)
.csv(plain_output_path)
# read a encrypted csv file and return a DataFrame
encrypted_csv_path = "/encrypted/csv/path"
df2 = sc.read(CryptoMode.AES_CBC_PKCS5PADDING).option("header", "true").csv(encrypted_csv_path)
# write a DataFrame as a encrypted csv file
encrypted_output_path = "/encrypted/output/path"
sc.write(df2, CryptoMode.AES_CBC_PKCS5PADDING)
.mode('overwrite')
.option("header", True)
.csv(encrypted_output_path)
```
**read/write PARQUET file**
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
import com.intel.analytics.bigdl.ppml.crypto.{AES_GCM_CTR_V1, PLAIN_TEXT}
// read a plain parquet file and return a DataFrame
val plainParquetPath = "/plain/parquet/path"
val df1 = sc.read(PLAIN_TEXT).parquet(plainParquetPath)
// write a DataFrame as a plain parquet file
plainOutputPath = "/plain/output/path"
sc.write(df1, PLAIN_TEXT)
.mode("overwrite")
.parquet(plainOutputPath)
// read a encrypted parquet file and return a DataFrame
val encryptedParquetPath = "/encrypted/parquet/path"
val df2 = sc.read(AES_GCM_CTR_V1).parquet(encryptedParquetPath)
// write a DataFrame as a encrypted parquet file
val encryptedOutputPath = "/encrypted/output/path"
sc.write(df2, AES_GCM_CTR_V1)
.mode("overwrite")
.parquet(encryptedOutputPath)
```
python
```python
# import
from bigdl.ppml.ppml_context import *
# read a plain parquet file and return a DataFrame
plain_parquet_path = "/plain/parquet/path"
df1 = sc.read(CryptoMode.PLAIN_TEXT).parquet(plain_parquet_path)
# write a DataFrame as a plain parquet file
plain_output_path = "/plain/output/path"
sc.write(df1, CryptoMode.PLAIN_TEXT)
.mode('overwrite')
.parquet(plain_output_path)
# read a encrypted parquet file and return a DataFrame
encrypted_parquet_path = "/encrypted/parquet/path"
df2 = sc.read(CryptoMode.AES_GCM_CTR_V1).parquet(encrypted_parquet_path)
# write a DataFrame as a encrypted parquet file
encrypted_output_path = "/encrypted/output/path"
sc.write(df2, CryptoMode.AES_GCM_CTR_V1)
.mode('overwrite')
.parquet(encrypted_output_path)
```
**read/write JSON file**
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
import com.intel.analytics.bigdl.ppml.crypto.{AES_CBC_PKCS5PADDING, PLAIN_TEXT}
// read a plain json file and return a DataFrame
val plainJsonPath = "/plain/json/path"
val df1 = sc.read(PLAIN_TEXT).json(plainJsonPath)
// write a DataFrame as a plain json file
val plainOutputPath = "/plain/output/path"
sc.write(df1, PLAIN_TEXT)
.mode("overwrite")
.json(plainOutputPath)
// read a encrypted json file and return a DataFrame
val encryptedJsonPath = "/encrypted/parquet/path"
val df2 = sc.read(AES_CBC_PKCS5PADDING).json(encryptedJsonPath)
// write a DataFrame as a encrypted parquet file
val encryptedOutputPath = "/encrypted/output/path"
sc.write(df2, AES_CBC_PKCS5PADDING)
.mode("overwrite")
.json(encryptedOutputPath)
```
python
```python
# import
from bigdl.ppml.ppml_context import *
# read a plain json file and return a DataFrame
plain_json_path = "/plain/json/path"
df1 = sc.read(CryptoMode.PLAIN_TEXT).json(plain_json_path)
# write a DataFrame as a plain json file
plain_output_path = "/plain/output/path"
sc.write(df1, CryptoMode.PLAIN_TEXT)
.mode('overwrite')
.json(plain_output_path)
# read a encrypted json file and return a DataFrame
encrypted_json_path = "/encrypted/parquet/path"
df2 = sc.read(CryptoMode.AES_CBC_PKCS5PADDING).json(encrypted_json_path)
# write a DataFrame as a encrypted parquet file
encrypted_output_path = "/encrypted/output/path"
sc.write(df2, CryptoMode.AES_CBC_PKCS5PADDING)
.mode('overwrite')
.json(encrypted_output_path)
```
**read textfile**
scala
```scala
import com.intel.analytics.bigdl.ppml.PPMLContext
import com.intel.analytics.bigdl.ppml.crypto.{AES_CBC_PKCS5PADDING, PLAIN_TEXT}
// read from a plain csv file and return a RDD
val plainCsvPath = "/plain/csv/path"
val rdd1 = sc.textfile(plainCsvPath) // the default cryptoMode is PLAIN_TEXT
// read from a encrypted csv file and return a RDD
val encryptedCsvPath = "/encrypted/csv/path"
val rdd2 = sc.textfile(path=encryptedCsvPath, cryptoMode=AES_CBC_PKCS5PADDING)
```
python
```python
# import
from bigdl.ppml.ppml_context import *
# read from a plain csv file and return a RDD
plain_csv_path = "/plain/csv/path"
rdd1 = sc.textfile(plain_csv_path) # the default crypto_mode is "plain_text"
# read from a encrypted csv file and return a RDD
encrypted_csv_path = "/encrypted/csv/path"
rdd2 = sc.textfile(path=encrypted_csv_path, crypto_mode=CryptoMode.AES_CBC_PKCS5PADDING)
```
For more usage with `PPMLContext` Python API, please refer to [PPMLContext Python API](https://github.com/intel-analytics/BigDL/blob/main/python/ppml/src/bigdl/ppml/README.md).