Maintaining compatibility between pyRTE-RRTMGP and RTE-RRTMGP
When a new release of RTE-RRTMGP is published and changes are made to the functions that pyRTE-RRTMGP binds and supports, a manual update is required in pybind_interface.cpp.
Steps to Use the New Version of RTE-RRTMGP (New Source)
In CMakeLists.txt, to use a different version of RTE-RRTMGP, we need to change the GIT_TAG to the desired version.
Clone or download the source code for the version of RTE-RRTMGP you want to use. To build it, you can use the setup script provided in the source code.
The top-level folder of this repo contains a script called check_binds.py which provides general information about the state of the Pybinds and Cbinds from Fortran:
python check_binds.py --c_headers /path/to/rrtmgp_kernels.h /path/to/rte_kernels.h --pybind /path/to/pybind_interface.cpp
With the built source of RTE-RRTMGP, you can check the current Pybinds.
An example output looks like this:
Total C functions found: 41
Total Py functions bound: 41
Warning: Missing C function bindings:
- rte_net_byband_full
- rte_sum_byband
Warning: Missing Py function bindings:
- rte_lw_source_2str
- rte_sw_source_dir
This helps identify missing bindings, naming differences, or functions that have been removed.
Testing Current Bindings with the New Source
After running the script, a good rule of thumb is to build and test, checking for any errors that occur. To build, use the following commands:
python -m venv .venv_test
source .venv_test/bin/activate
pip install numpy xarray netcdf4 requests
pip install --upgrade cmake scikit-build pybind11 pytest
pip install -e .
The expected output should look something like this:
...
Successfully built pyrte_rrtmgp
Installing collected packages: pyrte_rrtmgp
Successfully installed pyrte_rrtmgp-0.0.8
To run the tests:
pytest -v
The expected output should look something like this:
======================================================================================================== test session starts ========================================================================================================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /data/vkm/code/makepath/columbia/.venv_test/bin/python
cachedir: .pytest_cache
rootdir: /data/vkm/code/makepath/columbia
configfile: pyproject.toml
testpaths: pyrte_rrtmgp/tests
collected 47 items
...
pyrte_rrtmgp/tests/test_exported_functions.py::test_rte_increment_nstream_by_nstream PASSED [ 95%]
pyrte_rrtmgp/tests/test_python_frontend/test_lw_solver.py::test_lw_solver_noscat PASSED [ 97%]
pyrte_rrtmgp/tests/test_python_frontend/test_sw_solver.py::test_sw_solver_noscat PASSED [100%]
======================================================================================================== 47 passed in 27.71s ========================================================================================================
If errors appear, there are two potential issues:
Differences in function parameters:
New parameters may have been added, or some may have been removed.
Changes in dimensions.
Changes in data types.
These changes also affect the Python implementation, which must be updated accordingly.
Differences in input and/or output data:
This can often be resolved by updating the version of the RRTMGP-DATA repository.
Example case: There is a change in the dimensions of the cl_gas parameter in rrtmgp_compute_tau_rayleigh, the build process for the wheel succeeds, but running pytest results in the following error:
pyrte_rrtmgp/kernels/rrtmgp.py:466: RuntimeError
========================================================================================= short test summary info =========================================================================================
FAILED pyrte_rrtmgp/tests/test_python_frontend/test_sw_solver.py::test_sw_solver_noscat - RuntimeError: Invalid size for input array 'col_gas'
====================================================================================== 1 failed, 46 passed in 32.04s ======================================================================================
In this situation, the solution is to check the dimensions in cbind and ensure they are correctly reflected in pybind and the Python implementation. Most error cases are similar to this.
If there is a difference in the outputs and everything appears to be correct, the problem is most likely in the input/output data. Check for any changes in the Fortran tests and replicate them accordingly.
Adding New Bindings
If a new function is introduced to the bindings, follow this procedure to add it to the Python repository:
Add the new function to
pybind_interface.cpp, ensuring proper error checking for valid dimensions and inputs.Add the new function to either
pyrte_rrtmgp/kernels/rte.pyorpyrte_rrtmgp/kernels/rrtmgp.py.Implement appropriate tests for the new functionality. If a test for this functionality exists in RTE-RRTMGP, the new test should mimic it.
Changes to functions may also require updates to input and reference data for the tests in RRTMGP-DATA, necessitating corresponding updates in PYRTE-RRTMGP’s tests.
Example Blueprint of a Bound Function: Cbind function:
void rte_sum_broadband(
const int& ncol,
const int& nlev,
const int& ngpt,
const Float* spectral_flux, // Dims: (ncol, nlev, ngpt)
Float* broadband_flux // Dims: (ncol, nlev)
);
Pybind function:
// Declaration of the Pybind function
m.def("rte_sum_broadband",
[](
// Mapping the data types with the corresponding properties for Pybind11
int ncol,
int nlev,
int ngpt,
py::array_t<Float> gpt_flux,
py::array_t<Float> flux
) {
// Error checking for dimensions
if (ncol <= 0 || nlev <= 0 || ngpt <= 0) {
throw std::runtime_error("ncol, nlev, and ngpt must be positive integers.");
}
// Error checking for array sizes
if (gpt_flux.size() != ncol * nlev * ngpt) {
throw std::runtime_error("Invalid size for input array 'gpt_flux': expected (ncol, nlev, ngpt).");
}
if (flux.size() != ncol * nlev) {
throw std::runtime_error("Invalid size for output array 'flux': expected (ncol, nlev).");
}
// Request buffer information
py::buffer_info buf_gpt_flux = gpt_flux.request();
py::buffer_info buf_flux = flux.request();
// Calling the C-bound function
fortran::rte_sum_broadband(
ncol,
nlev,
ngpt,
reinterpret_cast<Float*>(buf_gpt_flux.ptr),
reinterpret_cast<Float*>(buf_flux.ptr)
);
});