beta_rec.core package

beta_rec.core.eval_engine module

This is the core implementation of the evaluation.

class beta_rec.core.eval_engine.EvalEngine(config)[source]

Bases: object

The base evaluation engine.

flush()[source]

Flush eval_engine.

predict(data_df, model, batch_eval=False)[source]

Make prediction for a trained model. :param data_df: A dataset to be evaluated. :type data_df: DataFrame :param model: A trained model. :param batch_eval: A signal to indicate if the model is evaluated in batches. :type batch_eval: Boolean

Returns:predicted scores.
Return type:array
record_performance(valid_result, test_result, epoch_id)[source]

Record perforance result on tensorboard.

Parameters:
  • valid_result (dict) – Performance result of validation set.
  • test_result (dict) – Performance result of testing set.
  • epoch_id (int) – epoch_id.
seq_predict(train_seq, data_df, model, maxlen)[source]

Make prediction for a trained model.

Parameters:
  • data_df (DataFrame) – A dataset to be evaluated.
  • model – A trained model.
  • batch_eval (Boolean) – A signal to indicate if the model is evaluated in batches.
Returns:

predicted scores.
Return type:

array
seq_predict_time(train_seq, data_df, model, maxlen, time_span)[source]

Make prediction for a trained model.

Parameters:
  • data_df (DataFrame) – A dataset to be evaluated.
  • model – A trained model.
  • batch_eval (Boolean) – A signal to indicate if the model is evaluated in batches.
Returns:

predicted scores.
Return type:

array
seq_train_eval(train_seq, valid_data_df, test_data_df, model, maxlen, epoch_id=0)[source]

Evaluate the performance for a (validation) dataset with multiThread.

Parameters:
  • valid_data_df (DataFrame) – A validation dataset.
  • test_data_df (DataFrame) – A testing dataset.
  • model – trained model.
  • epoch_id – epoch_id.
  • k (int or list) – top k result to be evaluate.
seq_train_eval_time(train_seq, valid_data_df, test_data_df, model, maxlen, time_span, epoch_id=0)[source]

Evaluate the performance for a (validation) dataset with multiThread.

Parameters:
  • valid_data_df (DataFrame) – A validation dataset.
  • test_data_df (DataFrame) – A testing dataset.
  • model – trained model.
  • epoch_id – epoch_id.
  • k (int or list) – top k result to be evaluate.
test_eval(test_df_list, model)[source]

Evaluate the performance for a (testing) dataset list with multiThread.

Parameters:
  • test_df_list (list) – (testing) dataset list.
  • model – trained model.
test_seq_predict(train_seq, valid_data_df, test_data_df, model, maxlen)[source]

Make prediction for a trained model.

Parameters:
  • data_df (DataFrame) – A dataset to be evaluated.
  • model – A trained model.
  • batch_eval (Boolean) – A signal to indicate if the model is evaluated in batches.
Returns:

predicted scores.
Return type:

array
test_seq_predict_time(train_seq, valid_data_df, test_data_df, model, maxlen, time_span)[source]

Make prediction for a trained model.

Parameters:
  • data_df (DataFrame) – A dataset to be evaluated.
  • model – A trained model.
  • batch_eval (Boolean) – A signal to indicate if the model is evaluated in batches.
Returns:

predicted scores.
Return type:

array
train_eval(valid_data_df, test_data_df, model, epoch_id=0)[source]

Evaluate the performance for a (validation) dataset with multiThread.

Parameters:
  • valid_data_df (DataFrame) – A validation dataset.
  • test_data_df (DataFrame) – A testing dataset.
  • model – trained model.
  • epoch_id – epoch_id.
  • k (int or list) – top k result to be evaluate.
class beta_rec.core.eval_engine.SeqEvalEngine(config)[source]

Bases: object

The base evaluation engine for sequential recommendation.

evaluate_sequence(recommender, seq, evaluation_functions, user, given_k, look_ahead, top_n)[source]

Compute metrics for each sequence.

Parameters:
  • recommender (object) – which recommender to use
  • seq (List) – the user_profile/ context
  • given_k (int) – last element used as ground truth. NB if <0 it is interpreted as first elements to keep
  • evaluation_functions (dict) – which function to use to evaluate the rec performance
  • look_ahead (int) – number of elements in ground truth to consider. If look_ahead = ‘all’ then all the ground_truth sequence is considered
Returns:

performance of recommender.
Return type:

np.array(tmp_results) (1d array)
get_test_sequences(test_data, given_k)[source]

Run evaluation only over sequences longer than abs(LAST_K).

Parameters:
  • test_data (pandas.DataFrame) – Test set.
  • given_k (int) – last element used as ground truth.
Returns:

list of sequences for testing.
Return type:

test_sequences (List)
sequence_sequential_evaluation(recommender, seq, evaluation_functions, user, given_k, look_ahead, top_n, step)[source]

Compute metrics for each sequence incrementally.

Parameters:
  • recommender (object) – which recommender to use
  • seq (List) – the user_profile/ context
  • given_k (int) – last element used as ground truth. NB if <0 it is interpreted as first elements to keep
  • evaluation_functions (dict) – which function to use to evaluate the rec performance
  • look_ahead (int) – number of elements in ground truth to consider. If look_ahead = ‘all’ then all the ground_truth sequence is considered
Returns:

performance of recommender.
Return type:

eval_res/eval_cnt (1d array)
sequential_evaluation(recommender, test_sequences, evaluation_functions, users=None, given_k=1, look_ahead=1, top_n=10, scroll=True, step=1)[source]

Run sequential evaluation of a recommender over a set of test sequences.

Parameters:
  • recommender (object) – the instance of the recommender to test.
  • test_sequences (List) – the set of test sequences
  • evaluation_functions (dict) – list of evaluation metric functions.
  • users (List) – (optional) the list of user ids associated to each test sequence.
  • given_k (int) – (optional) the initial size of each user profile, starting from the first interaction in the sequence. If <0, start counting from the end of the sequence. It must be != 0.
  • look_ahead (int) – (optional) number of subsequent interactions in the sequence to be considered as ground truth. It can be any positive number or ‘all’ to extend the ground truth until the end of the sequence.
  • top_n (int) – (optional) size of the recommendation list
  • scroll (boolean) – (optional) whether to scroll the ground truth until the end of the sequence. If True, expand the user profile and move the ground truth forward of step interactions. Recompute and evaluate recommendations every time. If False, evaluate recommendations once per sequence without expanding the user profile.
  • step (int) – (optional) number of interactions that will be added to the user profile at each step of the sequential evaluation.
Returns:

the list of the average values for each evaluation metric.
Return type:

metrics/len(test_sequences) (1d array)
test_eval_seq(test_data, recommender)[source]

Compute performance of the sequential models with test dataset.

Parameters:
  • test_data (pandas.DataFrame) – test dataset.
  • recommender (Object) – Sequential recommender.
  • k (int) – size of the recommendation list
Returns:

None
train_eval_seq(valid_data, test_data, recommender, epoch_id=0)[source]

Compute performance of the sequential models with validation and test datasets for each epoch during training.

Parameters:
  • valid_data (pandas.DataFrame) – validation dataset.
  • test_data (pandas.DataFrame) – test dataset.
  • recommender (Object) – Sequential recommender.
  • epoch_id (int) – id of the epoch.
  • k (int) – size of the recommendation list
Returns:

None
beta_rec.core.eval_engine.computeRePos(time_seq, time_span)[source]

Compute position matrix for a user.

Parameters:
  • time_seq ([type]) – [description]
  • time_span ([type]) – [description]
Returns:

[description]
Return type:

[type]
beta_rec.core.eval_engine.evaluate(data_df, predictions, metrics, k_li)[source]

Evaluate the performance of a prediction by different metrics.

Parameters:
  • data_df (DataFrame) – the dataset to be evaluated.
  • predictions (narray) – 1-D array. The predicted scores for each user-item pair in the dataset.
  • metrics (list) – metrics to be evaluated.
  • k_li (int or list) – top k (s) to be evaluated.
Returns:

Performance result.
Return type:

result_dic (dict)
beta_rec.core.eval_engine.test_eval_worker(testEngine, eval_data_df, prediction)[source]

Start a worker for the evaluation during training.

Prediction and evaluation on the testing set.

beta_rec.core.eval_engine.train_eval_worker(testEngine, valid_df, test_df, valid_pred, test_pred, epoch)[source]

Start a worker for the evaluation during training.

Parameters:
  • testEngine –
  • valid_df –
  • test_df –
  • valid_pred –
  • test_pred –
  • epoch (int) –
Returns:

dictionary with performances on validation and testing sets.
Return type:

(dict,dict)

beta_rec.core.train_engine module

class beta_rec.core.train_engine.TrainEngine(args)[source]

Bases: object

Training engine for all the models.

check_early_stop(engine, model_dir, epoch)[source]

Check if early stop criterion is triggered.

Save model if previous epoch have already obtained better result.

Parameters:epoch (int) – epoch num
Returns:True: if early stop criterion is triggered, False: else
Return type:bool
get_device()[source]

Get one gpu id that have the most available memory.

Returns:The gpu id (None if no available gpu) and the the device string (pytorch style).
Return type:(int, str)
initialize_folders(config)[source]

Initialize the whole directory structure of the project.

load_dataset()[source]

Load dataset.

prepare_env()[source]

Prepare running environment.

  • Load parameters from json files.
  • Initialize system folders, model name and the paths to be saved.
  • Initialize resource monitor.
  • Initialize random seed.
  • Initialize logging.
test()[source]

Evaluate the performance for the testing sets based on the best performing model.

tune(runable)[source]

Tune parameters using ray.tune.

Module contents

Core Module.