Trainers System

Overview

Objects within the [Trainers] block are derived from SurrogateTrainer and are designed for creating training data for use with a model (see Surrogates System).

Creating a SurrogateTrainer

To create a trainer the new object should inherit from SurrogateTrainer, which is derived from GeneralUserObject. SurrogateTrainer overrides the execute() function to loop through the rows of a given sampler, specified by the "sampler" parameter:

void
SurrogateTrainer::execute()
{
  if (_doing_cv)
    for (const auto & trial : make_range(_cv_n_trials))
    {
      std::vector<Real> trial_score = crossValidate();

      // Expand _cv_trial_scores with more columns if necessary, then insert values.
      for (unsigned int r = _cv_trial_scores.size(); r < trial_score.size(); ++r)
        _cv_trial_scores.push_back(std::vector<Real>(_cv_n_trials, 0.0));
      for (auto r : make_range(trial_score.size()))
        _cv_trial_scores[r][trial] = trial_score[r];
    }

  _current_sample_size = _sampler.getNumberOfRows();
  _local_sample_size = _sampler.getNumberOfLocalRows();
  executeTraining();
}
(modules/stochastic_tools/src/trainers/SurrogateTrainer.C)
void
SurrogateTrainer::executeTraining()
{
  checkIntegrity();
  _row = _sampler.getLocalRowBegin();
  _local_row = 0;

  preTrain();

  for (_row = _sampler.getLocalRowBegin(); _row < _sampler.getLocalRowEnd(); ++_row)
  {
    // Need to do this manually in order to keep the iterators valid
    const std::vector<Real> data = _sampler.getNextLocalRow();
    for (unsigned int i = 0; i < _row_data.size(); ++i)
      _row_data[i] = data[i];

    // Set training data
    for (auto & pair : _training_data)
      pair.second->setCurrentIndex((pair.second->isDistributed() ? _local_row : _row));

    updatePredictorRow();

    if ((!_skip_unconverged || *_converged) &&
        std::find(_skip_indices.begin(), _skip_indices.end(), _row) == _skip_indices.end())
      train();

    _local_row++;
  }

  postTrain();
}
(modules/stochastic_tools/src/trainers/SurrogateTrainer.C)

The method will execute once per execution flag (see SetupInterface (execute_on)) on each processor. There are three virtual functions that derived class can and should override:

  /*
   * Setup function called before sampler loop
   */
  virtual void preTrain() {}

  /*
   * Function needed to be overried, called during sampler loop
   */
  virtual void train() {}

  /*
   * Function called after sampler loop, used for mpi communication mainly
   */
  virtual void postTrain() {}
(modules/stochastic_tools/include/trainers/SurrogateTrainer.h)

  • preTrain() is called before the sampler loop and is typically used for resizing variables for the given number of data points.

  • train() is called within the sampler loop where member variables _local_row, _row, and those declared with getTrainingData are updated.

  • postTrain() is called after the sampler loop and is typically used for MPI communication.

Gathering Training Data

In order to ease the gathering the required data needed for training, SurrogateTrainer includes API to get reporter data which takes care of the necessary size checks and distributed data indexing. The idea behind this is to emulate the element loop behavior in other MOOSE objects. For instance, in a kernel, the value of _u corresponds to the solution in an element. Here data referenced with getTrainingData will correspond to the value of the data in a sampler row. The returned reference is to be used in the train() function. There are five functions that derived classes can call to gather training data:

  /*
   * Get a reference to training data given a reporter name
   */
  template <typename T>
  const T & getTrainingData(const ReporterName & rname);

  /*
   * Get a reference to the sampler row data
   */
  const std::vector<Real> & getSamplerData() const { return _row_data; };

  /*
   * Get a reference to the predictor row data
   */
  const std::vector<Real> & getPredictorData() const { return _predictor_data; };

  /*
   * Get current sample size (this is recalculated to reflect the number of skipped samples)
   */
  unsigned int getCurrentSampleSize() const { return _current_sample_size; };

  /*
   * Get current local sample size (recalculated to reflect number of skipped samples)
   */
  unsigned int getLocalSampleSize() const { return _local_sample_size; };
(modules/stochastic_tools/include/trainers/SurrogateTrainer.h)

  • getTrainingData<T>(const ReporterName & rname) will get a vector of training data from a reporter value of type std::vector<T>, whose name is defined by rname.

  • getSamplerData() will simply return a vector of the sampler row.

  • getPredictorData() will return a vector of predictor data, including values from Reporters specified using the common "predictors" input parameter.

  • getCurrentSampleSize() and getLocalSampleSize() will return global and local sample sizes, recalculated periodically when samples are intentionally excluded (for example, during cross validation).

Accessing Common Types of Training Data

Many of the Trainers in the module share common characteristics:

  • They support specifying Reporters as predictor data using the "predictors" input parameter, and choosing column subsets from the Sampler using the "predictor_cols" input parameter.

  • They support Real and/or std::vector<Real> types for response data. The input parameters "response" and "response_type" are used to declare a Reporter, and the type of that reporter, to handle these two cases.

Although the getTrainingData API should generally be used in the constructor of the derived Trainer class to gather required training data, it is anticipated that most trainers will need similar capabilities. To facilitate these use cases, SurrogateTrainer provides input parameters and protected member variables to gather and access training data of the types defined above.

  // Common Training Data
  MooseEnum data_type("real=0 vector_real=1", "real");
  params.addRequiredParam<ReporterName>(
      "response",
      "Reporter value of response results, can be vpp with <vpp_name>/<vector_name> or sampler "
      "column with 'sampler/col_<index>'.");
  params.addParam<MooseEnum>("response_type", data_type, "Response data type.");
  params.addParam<std::vector<ReporterName>>(
      "predictors",
      std::vector<ReporterName>(),
      "Reporter values used as the independent random variables, If 'predictors' and "
      "'predictor_cols' are both empty, all sampler columns are used.");
  params.addParam<std::vector<unsigned int>>(
      "predictor_cols",
      std::vector<unsigned int>(),
      "Sampler columns used as the independent random variables, If 'predictors' and "
      "'predictor_cols' are both empty, all sampler columns are used.");
(modules/stochastic_tools/src/trainers/SurrogateTrainer.C)
  // TRAINING_DATA_MEMBERS
  ///@{
  /// Sampler being used for training
  Sampler & _sampler;
  /// During training loop, this is the row index of the data
  dof_id_type _row;
  /// During training loop, this is the local row index of the data
  dof_id_type _local_row;
  /// Response value
  const Real * _rval;
  /// Vector response value
  const std::vector<Real> * _rvecval;
  /// Predictor values from reporters
  std::vector<const Real *> _pvals;
  /// Columns from sampler for predictors
  std::vector<unsigned int> _pcols;
  /// Dimension of predictor data - either _sampler.getNumberOfCols() or _pvals.size() + _pcols.size().
  unsigned int _n_dims;
  ///@}
(modules/stochastic_tools/include/trainers/SurrogateTrainer.h)

_rval and _rvecval can be used to access Real and std::vector<Real> response data, as specified by the response and response_type input parameters. _pvals can be used to access predictor data, gather from Reporters specified using the predictors input parameter. _pcols contains the Sampler column indices specified in the predictor_cols input. For Trainers that need to gather from other types (not Real or std::vector<Real>), getTrainingData should be called in the derived classes constructor. SurrogateTrainer also provides the global and local row indices, as well as the dimensionality of the predictor data.

Declaring Training Data

Model data must be declare in the object constructor using the declareModelData methods, which are defined as follows. The desired type is provided as the template argument (T) and name to the data is the first input parameter. The second option, if provided, is the initial value for the training data. The name provided is arbitrary, but is used by the model object(s) designed to work with the training data (see Surrogates System).


#pragma once

#include "StochasticToolsApp.h"
#include "GeneralUserObject.h"
#include "LoadSurrogateDataAction.h"
#include "RestartableModelInterface.h"

#include "Sampler.h"
#include "StochasticToolsApp.h"
#include "SurrogateModelInterface.h"
#include "MooseRandom.h"

class TrainingDataBase;
template <typename T>
class TrainingData;

/**
 * This is the base trainer class whose main functionality is the API for declaring
 * model data. All trainer must at least derive from this. Unless a trainer needs
 * to perform its own loop through data, it is highly recommended to derive from
 * SurrogateTrainer.
 */
class SurrogateTrainerBase : public GeneralUserObject, public RestartableModelInterface
{
public:
  static InputParameters validParams();
  SurrogateTrainerBase(const InputParameters & parameters);

  virtual void initialize() {}                         // not required, but available
  virtual void finalize() {}                           // not required, but available
  virtual void threadJoin(const UserObject &) final {} // GeneralUserObjects are not threaded
};

/**
 * This is the main trainer base class. The main purpose is to avoid a lot of code
 * duplication from performing sampler loops and dealing with distributed data. There
 * three functions that derived trainer should override: preTrain, train, and postTrain.
 * Derived class should also use the getTrainingData functionality, which provides a
 * refernce to vector reporter data in its current state within the sampler loop.
 *
 * The idea behind this is to emulate the element loop behaiviour in other MOOSE objects.
 * For instance, in a kernel, the value of _u corresponds to the solution in an element.
 * Here data referenced with getTrainingData will correspond to the the value of the
 * data in a sampler row.
 */
class SurrogateTrainer : public SurrogateTrainerBase, public SurrogateModelInterface
{
public:
  static InputParameters validParams();
  SurrogateTrainer(const InputParameters & parameters);

  virtual void initialize() final;
  virtual void execute() final;
  virtual void finalize() final{};

protected:
  /*
   * Setup function called before sampler loop
   */
  virtual void preTrain() {}

  /*
   * Function needed to be overried, called during sampler loop
   */
  virtual void train() {}

  /*
   * Function called after sampler loop, used for mpi communication mainly
   */
  virtual void postTrain() {}

  // TRAINING_DATA_BEGIN

  /*
   * Get a reference to training data given a reporter name
   */
  template <typename T>
  const T & getTrainingData(const ReporterName & rname);

  /*
   * Get a reference to the sampler row data
   */
  const std::vector<Real> & getSamplerData() const { return _row_data; };

  /*
   * Get a reference to the predictor row data
   */
  const std::vector<Real> & getPredictorData() const { return _predictor_data; };

  /*
   * Get current sample size (this is recalculated to reflect the number of skipped samples)
   */
  unsigned int getCurrentSampleSize() const { return _current_sample_size; };

  /*
   * Get current local sample size (recalculated to reflect number of skipped samples)
   */
  unsigned int getLocalSampleSize() const { return _local_sample_size; };

  // TRAINING_DATA_END

  /*
   * Evaluate CV error using _cv_surrogate and appropriate predictor row.
   */
  virtual std::vector<Real> evaluateModelError(const SurrogateModel & surr);

  // TRAINING_DATA_MEMBERS
  ///@{
  /// Sampler being used for training
  Sampler & _sampler;
  /// During training loop, this is the row index of the data
  dof_id_type _row;
  /// During training loop, this is the local row index of the data
  dof_id_type _local_row;
  /// Response value
  const Real * _rval;
  /// Vector response value
  const std::vector<Real> * _rvecval;
  /// Predictor values from reporters
  std::vector<const Real *> _pvals;
  /// Columns from sampler for predictors
  std::vector<unsigned int> _pcols;
  /// Dimension of predictor data - either _sampler.getNumberOfCols() or _pvals.size() + _pcols.size().
  unsigned int _n_dims;
  ///@}
  // TRAINING_DATA_MEMBERS_END

private:
  /*
   * Called at the beginning of execute() to make sure values are set properly
   */
  void checkIntegrity() const;

  /*
   * Main model training method - called during crossValidate() and for final model training.
   */
  void executeTraining();

  /*
   * Call if cross-validation is turned on.
   */
  std::vector<Real> crossValidate();

  /*
   * Update predictor row (uses both Sampler and Reporter values, according to _pvals and _pcols)
   */
  void updatePredictorRow();

  /// Sampler data for the current row
  std::vector<Real> _row_data;

  /// Predictor data for current row - can be combination of Sampler and Reporter values.
  std::vector<Real> _predictor_data;

  /// Whether or not we are skipping samples that have unconverged solutions
  const bool _skip_unconverged;

  /// Whether or not the current sample has a converged solution
  const bool * _converged;

  /// Number of samples used to train the model.
  unsigned int _current_sample_size;

  /// Number of samples (locally) used to train the model.
  unsigned int _local_sample_size;

  /// Vector of reporter names and their corresponding values (to be filled by getTrainingData)
  std::unordered_map<ReporterName, std::shared_ptr<TrainingDataBase>> _training_data;

  /*
   * Variables related to cross validation.
   */
  ///@{
  /// Vector of indices to skip during executeTraining()
  std::vector<dof_id_type> _skip_indices;
  /// Type of cross validation to perform - for now, just 'none' (no CV) or 'k_fold'
  const MooseEnum & _cv_type;
  /// Number of splits (k) to split sampler data into.
  const unsigned int & _n_splits;
  /// Number of repeated trials of cross validation to perform.
  const unsigned int & _cv_n_trials;
  /// Seed used for _cv_generator.
  const unsigned int & _cv_seed;
  /// Random number generator used for shuffling sampler rows during splitting.
  MooseRandom _cv_generator;
  /// SurrogateModel used to evaluate model error relative to test points.
  const SurrogateModel * _cv_surrogate;
  /// Set to true if cross validation is being performed, controls behavior in execute().
  const bool _doing_cv;
  /// RMSE scores from each CV trial - can be grabbed by VPP or Reporter.
  std::vector<std::vector<Real>> & _cv_trial_scores;
  ///@}
};

template <typename T>
const T &
SurrogateTrainer::getTrainingData(const ReporterName & rname)
{
  auto it = _training_data.find(rname);
  if (it != _training_data.end())
  {
    auto data = std::dynamic_pointer_cast<TrainingData<T>>(it->second);
    if (!data)
      mooseError("Reporter value ", rname, " already exists but is of different type.");
    return data->get();
  }
  else
  {
    const std::vector<T> & rval = getReporterValueByName<std::vector<T>>(rname);
    _training_data[rname] = std::make_shared<TrainingData<T>>(rval);
    return std::dynamic_pointer_cast<TrainingData<T>>(_training_data[rname])->get();
  }
}

class TrainingDataBase
{
public:
  TrainingDataBase() : _is_distributed(false) {}

  virtual ~TrainingDataBase() = default;

  virtual dof_id_type size() const = 0;
  virtual void setCurrentIndex(dof_id_type index) = 0;
  bool & isDistributed() { return _is_distributed; }

protected:
  bool _is_distributed;
};

template <typename T>
class TrainingData : public TrainingDataBase
{
public:
  TrainingData(const std::vector<T> & vector) : _vector(vector) {}

  virtual dof_id_type size() const override { return _vector.size(); }
  virtual void setCurrentIndex(dof_id_type index) override { _value = _vector[index]; }

  const T & get() const { return _value; }

private:
  const std::vector<T> & _vector;
  T _value;
};
(modules/stochastic_tools/include/trainers/SurrogateTrainer.h)

These methods return a reference to the desired type that should be populated in the aforementioned train() method. For example, in the PolynomialChaosTrainer trainer object a scalar value, "order", is stored by declaring a reference to the desired type in the header.

  const unsigned int & _order;
(modules/stochastic_tools/include/trainers/PolynomialChaosTrainer.h)

Within the source the declared references are initialized with a declare method that includes data initialization.

    _order(declareModelData<unsigned int>("_order", getParam<unsigned int>("order"))),
(modules/stochastic_tools/src/trainers/PolynomialChaosTrainer.C)

The training data system leverages the Restartable within MOOSE. As such, the data store can be of an arbitrary type and is automatically used for restarting simulations.

Enabling Cross Validation

K-fold cross validation options for SurrogateTrainer are in development. The current implementation requires a Sampler, SurrogateTrainer, and SurrogateModel to be defined in the input file. At the beginning of execute(), the trainer will shuffle and partition the rows of the provided Sampler into folds. Then, in a loop over each fold, it will train the linked SurrogateModel and evaluate it for the test set. Training is performed using the same preTrain(), train(), and postTrain() methods as before. To make an existing trainer compatible with cross validation, preTrain() must reset the state of the trainer and clear any essential data related to prior training sets - for example, in PolynomialRegressionTrainer a linear system is assembled in the training loop and solved at the end in postTrain(). To enable cross validation, preTrain() was changed to reset the linear system.

void
PolynomialRegressionTrainer::preTrain()
{
  _matrix.zero();
  for (unsigned int r = 0; r < _rhs.size(); ++r)
    _rhs[r].zero();

  /// Init calculators.
  for (auto & calc : _calculators)
    calc->initializeCalculator();

  _r_sum.assign(_rhs.size(), 0.0);
}
(modules/stochastic_tools/src/trainers/PolynomialRegressionTrainer.C)

Because some indices are randomly skipped during training, insertion into pre-sized arrays by indexing operations during train() may inadvertently leave some zero values for the skipped rows. To avoid this, it may be more convenient to gather training data using push_back. If desired, memory can be reserved for these arrays using getCurrentSampleSize() or getLocalSampleSize(). An example of this approach is shown in Creating a Surrogate Model.

During cross-validation the linked SurrogateModel will be used to generate predictions for the test set. To evaluate the error for these predictions, the evaluateModelError method is called. This method is declared virtual, so that it can be overridden to meet the needs of a particular implementation (for example, evaluations requiring pre-processing of predictor data). The default implementation covers the common anticipated uses cases (Real and std::vector<Real> responses). Developers of new trainer classes should consider whether their implementation should override this class or can use the default.

std::vector<Real>
SurrogateTrainer::evaluateModelError(const SurrogateModel & surr)
{
  std::vector<Real> error(1, 0.0);

  if (_rval)
  {
    Real model_eval = surr.evaluate(_predictor_data);
    error[0] = MathUtils::pow(model_eval - (*_rval), 2);
  }
  else if (_rvecval)
  {
    error.resize(_rvecval->size());

    // Evaluate for vector response.
    std::vector<Real> model_eval(error.size());
    surr.evaluate(_predictor_data, model_eval);
    for (auto r : make_range(_rvecval->size()))
      error[r] = MathUtils::pow(model_eval[r] - (*_rvecval)[r], 2);
  }

  return error;
}
(modules/stochastic_tools/src/trainers/SurrogateTrainer.C)

For an example of the cross-validation system in use, see K-Fold Cross Validation

Output Model Data

Training model data can be output to a binary file using the SurrogateTrainerOutput object.

Example Input File Syntax

The following input file snippet adds a PolynomialChaosTrainer object for training. Please refer to the documentation on the individual models for more details.

[Trainers]
  [poly_chaos]
    type = PolynomialChaosTrainer
    execute_on = timestep_end
    order = 5
    distributions = 'D_dist S_dist'
    sampler = sample
    response = storage/data:avg:value
    regression_type = integration
  []
[]
(modules/stochastic_tools/test/tests/surrogates/poly_chaos/main_2d_mc.i)

Available Objects

Available Actions

  • Stochastic Tools App
  • AddSurrogateActionAdds SurrogateTrainer and SurrogateModel objects contained within the [Trainers] and [Surrogates] input blocks.