LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations

Data assimilation techniques are crucial for correcting trajectories when modeling complex physical systems. The Latent Ensemble Score Filter (Latent-EnSF), a recently developed data assimilation method, has shown great promise in high-dimensional and nonlinear data assimilation problems with sparse observations. However, this method faces the challenge of high computational cost due to the expensive forward simulation. In this paper, we introduce Latent Dynamics EnSF (LD-EnSF), a novel methodology that evolves the dynamics in a low-dimensional latent space and significantly accelerates the data assimilation process. To achieve this, we introduce a novel variant of Latent Dynamics Networks (LDNets) to effectively capture the system’s dynamics within a low-dimensional latent space. Additionally, we propose a new method for encoding sparse observations into the latent space using LSTM networks. We demonstrate the robustness, accuracy, and efficiency of the proposed method for two challenging dynamical systems with highly sparse (in both space and time) and noisy observations.