WARPy Developer information

This document describes the basic structure of WARPy, relevant for developers looking to modify WARPy itself. It also discusses some advanced flags for power users which generally don't need to be modified. You should be familiar with how to use WARPy already.

Directory Structure

WARPy's source code is entirely located in tools/warpy. The general directory organization for WARPy are:

• Top level files are found in the root warpy folder.
• apps contains all applications which can be added to a spatial solver or variable adjuster.
• clusters contains support for queuing a WARPXM job on clusters. It contains capabilities for running WARPXM on clusters using the PBS or Schlurm schedulers. There are default configs for these schedulers, but additional configurations may be required for certain clusters (ex.: loading modules, location of the scratch folder, etc.). These can be added by sub-classing the cluster_info class, then adding the class to clusters_map in clusters/__init__.py. The key is the name of the login node, and the value is the specific instance of a cluster_info object. Note that many clusters have multiple login nodes, and these must individually be enumerated.
• dt_calc contains all dt controllers
• functions contains all the initial condition functions, as well as the initial condition applicator.
• host_actions contains all host actions. This includes syncing, temporal solvers, etc.
• hyperapps contains old applications for the HOFVM method. These no longer work in WARPXM and should not be used.
• spatial_solvers contains all spatial solvers. Currently the only one of interest is ndg which implements Nodal DG.
• variable_adjusters contains all variable adjusters. Variable adjusters are patch processes which apply a functional transform \(q = f(u)\), where \(q\) and \(u\) are distributed variables. These include boundary conditions, calculating gradients, limiters, etc.
• xdmf contains the XDMF generator.

tools/warpy/generator.py contains the basic input file output code. It takes Python representations of input parameters and writes out the appropriate .inp format read in by WARPXM.

This is contained in the block class. It roughly represents an XML-like block in the .inp file. Note that at this level there is very little logic associated with actual logical blocks; it knows nothing about what attributes or sub blocks belong where. It's primary concern is properly formatting numbers, strings, lists, and child blocks.

The main class of interest is the warpy_obj class. This is the starting point for all higher level logic blocks. It is able to represent attributes as key-value pairs, and supports both required attributes, as well as optional attributes. An optional attribute is only written to the .inp file if it is not None.

It contains a root block which represents the warpy object itself, and child blocks can either be generated in the __init__ function, or during export by overriding the generate function. Note that if overriding the generate function this function should not modify its internal state; instead, copies should be made of anything to be modified. It is rare that a developer should have to override the generate function; most functionality can be achieved by a combination of passing in appropriate parameters to the base class constructor, or overriding the attrs member function.

The base simulation block is found in tools/warpy/simulation.py. This block represents the highest level entry in the .inp file, and the class also allows users to export the full .inp file and/or run the simulation locally or on a cluster. Configuring the simulation run parameters happens at 3 different levels:

1. Site config. This is a set of generated default parameters for WARPy. This file is generated from the template file warpy_config.in.py.
2. User config. This is a config file for user login specific preferences. It is located in $HOME/.config/warpy_user_config.py. In theory all the config settings found in warpy_config.in.py can be overridden in this file, though many of the default values should not need to be modified by the user.
3. Local config. This is a simulation-specific config preferences. It can be located anywhere and the file path is passed into simulation.run.

The configuration files are loaded in the above order. Any configs which are specified in later config files override earlier set values.

In addition to these configs, there are cluster configs found in tools/warpy/clusters. These are used for determining how to generate job files and various parameters specific to certain clusters such as the location of the scratch folder, what scheduler to use, etc. Currently WARPy supports the SLURM and PBS/Torque schedulers.

The <tt>applications_container</tt> class

The applications_container class the base class for all applications, spatial solvers, and variable adjusters. It's purpose is to simplify the process of generating blocks for internal applications.

There are a few key features hidden inside applications_container:

• The ability to map variable components to a given index. For example, ion.rho might map to component index 5. The mapping is generated by the parent, and must be unique.
• A list of contained applications. When the generate function is called, the blocks associated with these contained applications are automatically added as children.
• A list of internal_components and internal_const_components. internal_components are variable components which are expected to be read in and written out to. An example is the fluid components for an Euler flux application. internal_const_components are variable components which are only expected to be read in. An example is the electric and magnetic fields for computing the Lorenz body force contribution to a multi-fluid model.
• All the features found in the base warpy_obj class; this includes classifying attributes as required or optional. \end{itemize}

The constructor of the applications_container class is able to receive extra components and const components which are not part of any contained applications. This is useful for example if you're implementing a variable adjuster that directly operate without any applications (computing gradients, limiters, etc.).

Variable Adjusters

To implement a custom variable adjuster:

1. Create a python file in the variable_adjusters folder (or one of its subdirectories)
2. Create a class which subclasses the variable_adjuster class (which subclasses the applications_container class). Most variable adjusters should only need to call the base class constructor with appropriate parameters, and override the attrs function to add special fields for reading/writing internal components.
3. Add the python file as an argument to WARPY_FILES in the CMakeLists.txt file for that folder. This will ensure that the python file will be symbolically linked into the build directory. You should never run WARPy from inside the git directory! This will result in polluting the git repository.
4. Import your class in the __init__.py file for that directory.

The <tt>application</tt> class

In order to allow recursive nesting of applications, the application class inherits from applications_container. In addition to the concept of internal components/internal const components, the application also has the concept of components and const components. These two contain all internal components/internal const components respectively, but also contain any components which are directly used by the application.

Process for adding a new application to WARPy:

1. Create a python file in the apps folder (or one of its subdirectories)
2. Create a class which subclasses application. Most applications should only need to define the constructor, and pass appropriate parameters to the base class constructor.
3. Add the python file as an argument to WARPY_FILES in the CMakeLists.txt file for that folder. This will ensure that the python file will be symbolically linked into the build directory. You should never run WARPy from inside the git directory! This will result in polluting the git repository.
4. Import your class in the __init__.py file for that directory.

There are additional subtle points when dealing with nested applications. The components variable map given to nested applications is currently the "global" one. It is the responsability of the parent application to properly match the requested variables by the child application with the list they are given.

In order to do this, the parent application should request as inputs all variables it needs as well as any variables it's children apps need as inputs. Similarly, it should output all variables it needs as well as any it's children apps need.

class parent_app : public WmApplication
{
  std::vector<int> input_vars;
  std::vector<int> aux_vars;
  std::vector<int> output_vars;
 
  std::vector<int> our_input_vars;
  std::vector<int> our_aux_vars;
  std::vector<int> our_output_vars;
 
  std::vector<std::vector<int>> input_idcs_map;
  std::vector<std::vector<int>> aux_idcs_map;
  std::vector<std::vector<int>> output_idcs_map;
 
  std::vector<real> child_in_buf;
  std::vector<real> child_aux_buf;
  std::vector<real> child_out_buf;
 
  std::vector<std::unique_ptr<WmApplication>> child_apps;
  
public:
  const std::vector<int>& getAuxiliaryVariableIndexes(
      int flag = WMAPPLICATIONFLAG_NONE) const override
  {
    return aux_vars;
  }
 
  const std::vector<int>& getInputVariableIndexes(
      int flag = WMAPPLICATIONFLAG_NONE) const override
  {
    return input_vars;
  }
 
  const std::vector<int>& getOutputVariableIndexes(
      int flag = WMAPPLICATIONFLAG_NONE) const override
  {
    return output_vars;
  }
 
  parent_app()
  {
    // this example implements a source term
    _allowedFlags.push_back(WMAPPLICATIONFLAG_SOURCE);
  }
    
  void Setup(const WxCryptSet& wxc)
  {
    // base class setup
    WmApplication::Setup(wxc);
 
    // create and setup child applications
    auto names = wxc.getNameOfType("application");
    for(auto& name : names)
    {
      const WxCryptSet& subset = wxc.getSet(name);
      auto kind = subset.get<std::string>("Kind");
      child_apps.emplace_back(WxCreatorMap<WmApplication>::getNew(kind));
      auto& child = *child_apps.back();
      child.setup(subset);
      // add child apps inputs/aux/outputs to our inputs/aux/outputs
      // Note: this assumes an app has the same input/aux/output indexes for all flags
      input_vars.insert(input_vars.end(),
                        child.getInputVariableIndexes().begin(),
                        child.getInputVariableIndexes().end());
      aux_vars.insert(aux_vars.end(),
                      child.getAuxiliaryVariableIndexes().begin(),
                      child.getAuxiliaryVariableIndexes().end());
      output_vars.insert(output_vars.end(),
                         child.getOutputVariableIndexes().begin(),
                         child.getOutputVariableIndexes().end());
    }
    // get and add our inputs
    our_input_vars = wxc.get<std::vector<int>>("Inputs");
    our_aux_vars = wxc.get<std::vector<int>>("Aux");
    our_output_vars = wxc.get<std::vector<int>>("Outputs");
    input_vars.insert(input_vars.end(),
                      our_input_vars.begin(),
                      our_input_vars.end());
    aux_vars.insert(aux_vars.end(),
                    our_aux_vars.begin(),
                    our_aux_vars.end());
    output_vars.insert(output_vars.end(),
                       our_output_vars.begin(),
                       our_output_vars.end());
 
    // everything after this point is to handle the case if sub apps
    // only take a small portion of the inputs/outputs this app reads.
    // For example, something like the Lacuna source term shouldn't need this.
    // make inputs/aux/outputs unique
    std::sort(input_vars.begin(), input_vars.end());
    input_vars.erase(std::unique(input_vars.begin(), input_vars.end()), input_vars.end());
    std::sort(aux_vars.begin(), aux_vars.end());
    aux_vars.erase(std::unique(aux_vars.begin(), aux_vars.end()), aux_vars.end());
    std::sort(output_vars.begin(), output_vars.end());
    output_vars.erase(std::unique(output_vars.begin(), output_vars.end()), output_vars.end());
 
    // cache mappings from what we receive to what child apps expect
    for(auto& child : child_apps)
    {
      input_idcs_map.push_back(map_sub_app_indices(input_vars, child->getInputVariableIndexes()));
      aux_idcs_map.push_back(map_sub_app_indices(aux_vars, child->getAuxiliaryVariableIndexes()));
      output_idcs_map.push_back(map_sub_app_indices(output_vars, child->getOutputVariableIndexes()));
    }
    // our input/aux/outputs are the last map
    input_idcs_map.push_back(map_sub_app_indices(input_vars, our_input_vars));
    aux_idcs_map.push_back(map_sub_app_indices(aux_vars, our_aux_vars));
    output_idcs_map.push_back(map_sub_app_indices(output_vars, our_output_vars));
  }
 
  real calc_parent_source(const real* q, const real* aux, const elementGeometry_t* pEG, real* source) const;
 
  // example case where this is a source term; other application types are handled similarly
  real source(const real* q, const real* aux, const elementGeometry_t* pEG, real* source) const override
  {
    real res = INFINITY;
    // child app contributions
    for (size_t i = 0; i < child_apps.size(); ++i)
    {
      // this is the general case; you may not need to perform index re-mapping
      auto& child = *child_apps[i];
      map_parent_to_sub_app(input_idcs_map[i], q, child_in_buf);
      map_parent_to_sub_app(aux_idcs_map[i], aux, child_aux_buf);
      child_out_buf.resize(output_idcs_map[i].size());
      res = std::min(res, child.source(child_in_buf.data(), child_aux_buf.data(), pEG, child_out_buf.data()));
      map_sub_app_to_parent(output_idcs_map[i], source, child_out_buf);
    }
    
    // any additional contributions from the parent app
    map_parent_to_sub_app(input_idcs_map.back(), q, child_in_buf);
    map_parent_to_sub_app(aux_idcs_map.back(), aux, child_aux_buf);
    child_out_buf.resize(output_idcs_map.back().size());
    res = std::min(res, calc_parent_source(child_in_buf.data(), child_aux_buf.data(), pEG, child_output_buf.data()));
    map_sub_app_to_parent(output_idcs_map[i], source, child_out_buf);
    
    return res;
  }
};

initial condition functions

For implementing a new initial condition you only need to implement the function. The general applicator provided in WARPXM and WARPy should be sufficient. To implement the function:

1. Create a python file in the functions folder (or one of its subdirectories)
2. Create a class which subclasses the warpy_obj class. Most initial condition functions should only need to call the base class constructor with appropriate parameters. Note that the Type attribute should be function.
3. Add the python file as an argument to WARPY_FILES in the CMakeLists.txt file for that folder. This will ensure that the python file will be symbolically linked into the build directory. You should never run WARPy from inside the git directory! This will result in polluting the git repository.
4. Import your class in the __init__.py file for that directory.

Dataset post-processing

TODO

XDMF generator

TODO

Temporal solvers

TODO