Fourier Neural Operator (FNO)

Last updated Jun 17, 2022 Edit Source

• Neural Operator - Machine learning for scientific computing
  • The classical development of neural networks has primarily focused on learning mappings between finite dimensional Euclidean spaces or finite sets. To better approximate the solution operators raised in PDEs, a generalization of neural networks to learn operators mapping between infinite dimensional function spaces is proposed
  • Formulate the approximation of operators by composition of a class of linear integral operators and nonlinear activation functions, so that the composed operator can approximate complex nonlinear operators
  • Such neural operators are resolution-invariant, and consequently more efficient compared to traditional neural networks
  • The FNO model has shown state-of-the-art performance with 1000x speedup in learning turbulent Navier-Stokes equation

• #TALK A crash course on Neural Operators (Kamyar Azizzadenesheli)
• #TALK Zongyi Li’s talk on solving PDEs from data
• #TALK Neural operator: A new paradigm for learning PDEs (Anima Anandkumar)

• #PAPER Fourier Neural Operator for Parametric Partial Differential Equations (Li 2020)
  • #CODE https://github.com/zongyi-li/fourier_neural_operator
  • #CODE https://github.com/astanziola/fourier-neural-operator-flax
  • https://zongyi-li.github.io/blog/2020/fourier-pde/
  • https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/
  • Paper explained
  • Function approximation in Fourier space instead of a the Euclidian (with conventional convolutions)
  • Fourier representation is more efficient than CNN, it’s global and continuous
  • Convolution -> pointwise multiplication (by matrix R, that is learned) in Fourier domain
  • Fourier layer.
    • Fourier transform. Filtering/truncating low frequency modes
    • Non-linearity (as in other DL models)
    • Linear transform/layer to track location information (just like a residual connection)
  • FNO captures energy spectrum
  • It can do zero-shot super-resolution: trained on a lower resolution directly evaluated on a higher resolution
• #PAPER MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework (Jiang 2020)
  • #TALK ML Cluster: Karthik Kashinath, Physics-informed Machine learning for weather and climate science
  • AI/Computer Vision/Super-resolution
• #PAPER Neural Operator: Learning Maps Between Function Spaces (Kovachki 2021)
• #PAPER Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers (Guibas 2022)
  • #CODE https://github.com/NVlabs/AFNO-transformer
  • #CODE https://github.com/DarshanDeshpande/research-paper-implementations/tree/main/tensorflow/afno (not official)
  • Adaptive FNO (AFNO) - connection between transformers and FNOs
  • Transformers are a special case of neural operators
  • AFNO is efficient for that it splits the grid (efficient token mixing)
• #PAPER FourCastNet (2022)
  • See AI4ES/Weather forecasting, nowcasting