Overtone introduces compute-flexible tokenization for transformer-based PDE surrogates, enabling a single model to trade speed for accuracy at inference time while also reducing long-rollout patch artifacts through cyclic patch modulation.
A universal tokenizer for spectra that directly ingests native wavelength grids without resampling, enabling seamless integration across astronomical surveys.
A billion-parameter, multimodal foundation model for astronomy that unifies heterogenous observations, telescopes, and physical processes into a single framework.
We show that latent diffusion models are robust to compression in the context of physics emulation, reducing computational cost while consistently outperforming non-generative alternatives.
100TB of cross-matched, standardized astronomy data that brings together images, spectra, and time-series data from leading surveys to accelerate machine learning breakthroughs.
We release a significant update to the AstroCLIP model, which demonstrates superior performance on all previously tested downstream tasks and introduces the capacity to tackle a host of new problems.
We introduce the Contextual Counting task, a new toy problem aimed at exploring interpretability of Transformer models in quantitative domains. We compare the performance of causal and non-causal models with different position codes and find causal models with RoPE and NoPE significantly outperform other configurations. We provide detailed explanation of how the circuits function and what makes them succeed or fail in generalization to out-of-distribution samples.
We introduce xVal, a new number encoding scheme for LLMs. Using xVal with a modified number inference method makes LLMs continuous function approximators. This makes them have a better inductive bias for data analysis in scientific domains.
We introduce Multiple Physics Pretraining, a new approach for developing large tuneable physical surrogate models. Our approach uses a built-in normalization and embedding scheme to enable learning multiple physical dynamics with a single model.
We present a self-supervised learning strategy that bridges diverse observational modalities in astrophysics. By aligning cross-modal representations of galaxies in a shared space, we are able to perform cross-modal look-up and competitive zero-shot predictions on downstream tasks.
