Using NeuralODEs in real life applications

NeuralODEs lead to amazing results in academic examples. But the expectations are often being disappointed as soon as one tries to adapt this concept for real life use cases. Bad convergence behavior, handling of discontinuities and/or instabilities are just some of the stumbling blocks that might pop up during the first steps. During the workshop, we want to show how to integrate real life industrial models in NeuralODEs using FMI and present sophisticated training strategies.

Despite the great potential of NeuralODEs - the structural combination of an artifical neural network and an ODE solver - they are not yet a standard tool in modeling of physical systems. We believe, there are mainly two reasons for that: First, NeuralODEs develop their full potential if paired with existing models that capture some basic physics. However, existing models typically are set up in dedicated simulation tools and thus not available in Julia. Second, training of NeuralODEs is tricky and not yet plug and play. In this tutorial, we share methods to employ models from various simulation tools in NeuralODEs and training strategies that can deal with challenges of industrial scale. Both was validated in real examples ranging from automotive to medical use cases. For us, one of the key challenges to make the jump from science to application, was to allow the re-use of existing simulation models from common modeling tools as parts of NeuralODEs. For this purpose, we picked the model exchange format FMI (https://fmi-standard.org/), which can be understood as a container format for ODEs and is supported by more than 170 simulation tools. Using our Julia package FMI.jl it is possible to handle FMUs – that’s how models compliant with FMI are called – in Julia. Further, the package connects FMUs to the common automatic differentiation packages in Julia. The library FMIFlux.jl enables the use of FMUs in machine learning applications. Using both allows for the following workflow: (1) Export your model from your favorite simulation tool as FMU, (2) Import the FMU in Julia, (3) Use the FMU as layer of an ANN in Julia, (4) Train the resulting NeuralODE, called NeuralFMU. Unfortunately, the fourth step is more complicated than it looks at first glance. NeuralFMUs or NeuralODEs in general tend to exhibit expensive gradient computation and thus very long training times. Further, they tend to converge to local minima or enter unstable regions. In order to cope with these challenges, we present strategies to • design a target-oriented topology for such a model • initialize or even pre-train such models • deal with large data and use batching • efficiently train such multi-domain models

Equipped with knowledge about typical pitfalls and their workarounds, workshop participants will have an easier time dealing with their own NeuralODE applications.