Short Courses

Foundation models are rapidly evolving beyond generative text, emerging as powerful tools for solving complex problems in mechanics. This short course bridges the gap between Large Language Models (LLMs) and physics-aware architectures, providing a comprehensive overview of how deep learning is reshaping computational mechanics across scales.

Participants will explore how to apply foundation models to three distinct pillars of research: (1) Atomic Scale: Using equivariant GNNs (e.g., NequIP) to learn interatomic potentials with ab initio accuracy; (2) Continuum Scale: Leveraging spatiotemporal transformers and neural operators for accelerated multiphysics solvers; and (3) Research Workflows: Fine-tuning LLMs to act as reasoning agents that can generate code, configure simulations, and synthesize technical literature.

We will begin with the fundamentals of the Transformer architecture and Equivariant Graph Neural Networks. Through hands-on modules, participants will train a surrogate model for atomic interactions, deploy a spatiotemporal solver for field evolution, and configure a retrieval-augmented generation (RAG) system to assist with technical workflows.

Learning Outcomes:

• Understand the difference between invariant, equivariant, and non-geometric data representations in mechanics.

• Train/fine-tune an equivariant model for atomic property prediction.

• Apply Transformer-based architectures to spatiotemporal continuum problems.

• Utilize LLMs as “Agents” to orchestrate computational workflows and query technical databases.

Syllabus:

1. Foundations: Architectures for Physics and Language

1.1. The Transformer: Attention mechanisms, positional encodings, and why they excel at sequence data (text and time-series).

1.2. Beyond Text - Geometric Deep Learning: The necessity of symmetry in mechanics. Introduction to Graphs, Message Passing, and SE(3)-Equivariance.

1.3. The Hardware Ecosystem: Setup with PyTorch, HuggingFace, and handling GPU resources for training physics models.

2. The Atomic Scale: Foundation Models for Materials

2.1. Theory: From Density Functional Theory (DFT) to Machine Learning Potentials. Why standard Transformers fail at atomic rotation/translation.

2.2. Hands-on with NequIP / Allegro:

• Data preparation (atomic structures, forces, energies).

• Training an equivariant GNN for an example material system.

• Inference: Running a short Molecular Dynamics (MD) trajectory using the trained model.

3. The Continuum Scale: Spatiotemporal & Operator Learning

3.1. Tokenizing Physics: How to represent continuous fields (stress, strain, temperature) as tokens for a Transformer.

3.2. Architectures for Solvers: Comparing Vision Transformers (ViT) with Neural Operators (FNO/DeepONet) for spatiotemporal evolution.

3.3. Hands-on Vision Transformers Workflow:

• Using surrogate models for multiphysics problems (e.g., phase-field fracture, fluid flow).

• Training on spatiotemporal datasets.

• Evaluating extrapolation capabilities (generalizing to new boundary conditions).

4. The Agentic Layer: LLMs as Research Assistants

4.1. Modern RAG & Agents:

• Moving beyond simple Q&A.

• Using LLMs (like Llama 3 or GPT-4o) to write input files for solvers from Modules 2 & 3.

4.2. Fine-tuning for Technical Syntax: Using LoRA (Low-Rank Adaptation) to teach an LLM the specific syntax of a mechanics solver (e.g., Abaqus keywords or Python FEA scripts).

4.3. Capstone Demo: An integrated workflow where an LLM generates a simulation script, the simulation is accelerated by a foundation model, and the results are interpreted by the user.

Flow-matching models have emerged as a powerful tool for probabilistic modeling and inference with several applications spanning several science and engineering domains. The core idea behind these models is to learn the time-dependent velocity field that maps samples from a simple, known distribution to a target distribution that may be complex. The training is accomplished by using samples from these two distributions. Once the velocity field is learnt, it can be used to generate new samples from the target distribution by sampling from the source distribution and the transporting the sampled particles by solving the probability flow ode. 

This tutorial will introduce the mathematical foundations and applications of flow-matching models starting from a continuum mechanics-centric derivation. Thereafter, we will focus on solving inverse problems and performing data assimilation within the Bayesian paradigm. In this context, we will discuss both flow-matching based data-driven priors and conditional flow matching models for amortized inference for parameters in a statistical model given observation. We will also conduct a hands-on demonstration of performing Bayesian inference using flow-matching models.