Evaluator#

bigdl.chronos.metric.forecast_metrics.mae(y_label, y_predict)[source]#

Calculate the mean absolute error (MAE).

\[\text{MAE} = \frac{1}{n}\sum_{t=1}^n |y_t-\hat{y_t}|\]

Parameters

y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.mse(y_label, y_predict)[source]#

Calculate the mean squared error (MSE).

\[\text{MSE} = \frac{1}{n}\sum_{t=1}^n (y_t-\hat{y_t})^2\]

Parameters

y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.rmse(y_label, y_predict)[source]#

Calculate square root of the mean squared error (RMSE).

\[\text{RMSE} = \sqrt{(\frac{1}{n}\sum_{t=1}^n (y_t-\hat{y_t})^2)}\]

Parameters

y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.mape(y_label, y_predict)[source]#

Calculate mean absolute percentage error (MAPE).

\[\text{MAPE} = \frac{100\%}{n}\sum_{t=1}^n |\frac{y_t-\hat{y_t}}{y_t}|\]

Parameters

y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.smape(y_label, y_predict)[source]#

Calculate Symmetric mean absolute percentage error (sMAPE).

\[\text{sMAPE} = \frac{100\%}{n} \sum_{t=1}^n \frac{|y_t-\hat{y_t}|}{|y_t|+|\hat{y_t}|}\]

Parameters

y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.r2(y_label, y_predict)[source]#

Calculate the r2 score.

\[R^2 = 1-\frac{\sum_{t=1}^n (y_t-\hat{y_t})^2}{\sum_{t=1}^n (y_t-\bar{y})^2}\]

Parameters

y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 1.0).

class bigdl.chronos.metric.forecast_metrics.Evaluator[source]#

Bases: object

Evaluate metrics for y_true and y_pred.

static evaluate(metrics, y_true, y_pred, aggregate='mean')[source]#

Evaluate a specific metrics for y_true and y_pred. :param metrics: String or list in [‘mae’, ‘mse’, ‘rmse’, ‘r2’, ‘mape’, ‘smape’] for built-in

metrics. If callable function, it signature should be func(y_true, y_pred), where y_true and y_pred are numpy ndarray.

Parameters

y_true – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_pred – Array-like of shape = (n_samples, *). Estimated target values.
aggregate – aggregation method. Currently, “mean” and None are supported, ‘mean’ represents aggregating by mean, while None will return the element-wise result. The value defaults to ‘mean’.

Returns

Float or ndarray of floats. A floating point value, or an array of floating point values, one for each individual target.

static get_latency(func, *args, num_running=100, **kwargs)[source]#

Return the time cost in milliseconds of a specific function by running multiple times.

Parameters

func – The function to be tested for latency.
args – arguments for the tested function.
num_running – Int and the value is positive. Specify the running number of the function and the value defaults to 100.
kwargs – other arguments for the tested function.

Returns

Dictionary of str:float. Show the information of the time cost in milliseconds.

Example

>>> # to get the inferencing performance of a trained TCNForecaster
>>> x = next(iter(test_loader))[0]
>>> # run forecaster.predict(x.numpy()) for len(tsdata_test.df) times
>>> # to evaluate the time cost
>>> latency = Evaluator.get_latency(forecaster.predict, x.numpy(),                num_running = len(tsdata_test.df))
>>> # an example output:
>>> # {"p50": 3.853, "p90": 3.881, "p95": 3.933, "p99": 4.107}

static plot(y, std=None, ground_truth=None, x=None, feature_index=0, instance_index=None, layout=(1, 1), prediction_interval=0.95, figsize=(16, 16), output_file=None, **kwargs)[source]#

Evaluator.plot function helps users to visualize their forecasting result.

Parameters

y – predict result, a 3-dim numpy ndarray with shape represented as (batch_size, predict_length, output_feature_dim).
std – standard deviation, a 3-dim numpy ndarray with shape represented as (batch_size, predict_length, output_feature_dim). Same shape as y.
ground_truth – ground truth, a 3-dim numpy ndarray with shape represented as (batch_size, predict_length, output_feature_dim). Same shape as y.
x – input numpy array, a 3-dim numpy ndarray with shape represented as (batch_size, lookback_length, input_feature_dim).
feature_index – int, the feature index (along last dim) to plot. Default to the first feature.
instance_index – int/tuple/list, the instance index to show. Default to None which represents random number.
layout – a 2-dim tuple, indicate the row_num and col_num to plot.
prediction_internval – a float, indicates the confidence percentile. Default to 0.95 refer to 95% confidence. This only effective when std is not None.
figsize – figure size to be inputed to pyplot. Default to (16,16).
output_file – a path, indicates the save path of the output plot. Default to None, indicates no output file is needed.
**kwargs –
other paramters will be passed to matplotlib.pyplot.