# Evaluator#

## Evaluator#

bigdl.chronos.metric.forecast_metrics.mae(y_label, y_predict)[source]#

Calculate the mean absolute error (MAE).

$\text{MAE} = \frac{1}{n}\sum_{t=1}^n |y_t-\hat{y_t}|$
Parameters
• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.mse(y_label, y_predict)[source]#

Calculate the mean squared error (MSE).

$\text{MSE} = \frac{1}{n}\sum_{t=1}^n (y_t-\hat{y_t})^2$
Parameters
• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.rmse(y_label, y_predict)[source]#

Calculate square root of the mean squared error (RMSE).

$\text{RMSE} = \sqrt{(\frac{1}{n}\sum_{t=1}^n (y_t-\hat{y_t})^2)}$
Parameters
• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.mape(y_label, y_predict)[source]#

Calculate mean absolute percentage error (MAPE).

$\text{MAPE} = \frac{100\%}{n}\sum_{t=1}^n |\frac{y_t-\hat{y_t}}{y_t}|$
Parameters
• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.smape(y_label, y_predict)[source]#

Calculate Symmetric mean absolute percentage error (sMAPE).

$\text{sMAPE} = \frac{100\%}{n} \sum_{t=1}^n \frac{|y_t-\hat{y_t}|}{|y_t|+|\hat{y_t}|}$
Parameters
• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.r2(y_label, y_predict)[source]#

Calculate the r2 score.

$R^2 = 1-\frac{\sum_{t=1}^n (y_t-\hat{y_t})^2}{\sum_{t=1}^n (y_t-\bar{y})^2}$
Parameters
• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 1.0).

class bigdl.chronos.metric.forecast_metrics.Evaluator[source]#

Bases: object

Evaluate metrics for y_true and y_pred.

static evaluate(metrics, y_true, y_pred, aggregate='mean')[source]#

Evaluate a specific metrics for y_true and y_pred. :param metrics: String or list in [‘mae’, ‘mse’, ‘rmse’, ‘r2’, ‘mape’, ‘smape’] for built-in

metrics. If callable function, it signature should be func(y_true, y_pred), where y_true and y_pred are numpy ndarray.

Parameters
• y_true – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_pred – Array-like of shape = (n_samples, *). Estimated target values.

• aggregate – aggregation method. Currently, “mean” and None are supported, ‘mean’ represents aggregating by mean, while None will return the element-wise result. The value defaults to ‘mean’.

Returns

Float or ndarray of floats. A floating point value, or an array of floating point values, one for each individual target.

static get_latency(func, *args, num_running=100, **kwargs)[source]#

Return the time cost in milliseconds of a specific function by running multiple times.

Parameters
• func – The function to be tested for latency.

• args – arguments for the tested function.

• num_running – Int and the value is positive. Specify the running number of the function and the value defaults to 100.

• kwargs – other arguments for the tested function.

Returns

Dictionary of str:float. Show the information of the time cost in milliseconds.

Example

>>> # to get the inferencing performance of a trained TCNForecaster
>>> # run forecaster.predict(x.numpy()) for len(tsdata_test.df) times
>>> # to evaluate the time cost
>>> latency = Evaluator.get_latency(forecaster.predict, x.numpy(),                num_running = len(tsdata_test.df))
>>> # an example output:
>>> # {"p50": 3.853, "p90": 3.881, "p95": 3.933, "p99": 4.107}

static plot(y, std=None, ground_truth=None, x=None, feature_index=0, instance_index=None, layout=(1, 1), prediction_interval=0.95, figsize=(16, 16), output_file=None, **kwargs)[source]#

Evaluator.plot function helps users to visualize their forecasting result.

Parameters
• y – predict result, a 3-dim numpy ndarray with shape represented as (batch_size, predict_length, output_feature_dim).

• std – standard deviation, a 3-dim numpy ndarray with shape represented as (batch_size, predict_length, output_feature_dim). Same shape as y.

• ground_truth – ground truth, a 3-dim numpy ndarray with shape represented as (batch_size, predict_length, output_feature_dim). Same shape as y.

• x – input numpy array, a 3-dim numpy ndarray with shape represented as (batch_size, lookback_length, input_feature_dim).

• feature_index – int, the feature index (along last dim) to plot. Default to the first feature.

• instance_index – int/tuple/list, the instance index to show. Default to None which represents random number.

• layout – a 2-dim tuple, indicate the row_num and col_num to plot.

• prediction_internval – a float, indicates the confidence percentile. Default to 0.95 refer to 95% confidence. This only effective when std is not None.

• figsize – figure size to be inputed to pyplot. Default to (16,16).

• output_file – a path, indicates the save path of the output plot. Default to None, indicates no output file is needed.

• **kwargs

other paramters will be passed to matplotlib.pyplot.