NLP/LLMs • Score 85
GRASP: Graph-Reasoning Aided Survey Planning for High-Fidelity Related Work Generation
arXiv:2607.03709v1 Announce Type: new
Abstract: Writing a literature review requires a deep understanding of the relationships among cited papers: how they build on, challenge, or offer alternative perspectives to one another. We present Graph-Reasoning Aided Survey Planning (GRASP), a framework combining LLM planning for related work generation with graph algorithms to extract key relationships among cited papers. Our two-layer graph structure consists of a Graph of Thoughts and an Argument-Counterargument Planning Network, representing the cited papers at different levels of granularity, and we apply topology-aware pruning via a Steiner tree to identify the core inter-paper relationships captured in our graph. Our citation analysis-based evaluation shows that GRASP generates related work sections (RWS) that closely match human-written targets in terms of the discourse roles, intents, and grouping of citations.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Additive Causal Construction for Transferable and Reconfigurable Cross-System Learning in Multi-Source Image Fusion
arXiv:2607.02572v1 Announce Type: new
Abstract: In multi-source image fusion scenarios, heterogeneous inputs are typically driven by distinct generative mechanisms and can be viewed as a composition of multiple causal systems. However, cross-system discrepancy (CSD) and cross-system entanglement (CSE) commonly arise during the fusion process, often leading to significant performance degradation under out-of-distribution (OOD) predictions. To address the CSD and CSE issues, we propose the additive causal construction (ACC) framework, which characterizes information fusion at two levels: firstly, it establishes causal "anchors" shared among multiple systems through intervention consistency to enable causal graph transferability (CGT); and secondly, it formalizes the fusion process as causal construction and models the reliability of constructed paths through uncertainty quantification to ensure causal graph reconfigurability (CGR). Building upon this, we revisit the traditional causal representation learning (CRL) with ACC and propose ACC-CRL as a learnable instantiation of the framework. The method explores joint causal content representations across systems via content-mechanism decoupling, and performs response alignment under shared anchors to mitigate CSD. Furthermore, it incorporates structural uncertainty to adaptively regulate the fusion process, thereby suppressing unstable CSE. We conduct systematic experiments on synthetic data (ColorMNIST) and real-world multi-center medical imaging tasks (microvascular invasion (MVI) prediction). The results demonstrate that the proposed method significantly improves OOD generalization while maintaining in-distribution (ID) performance, validating the effectiveness and robustness of the ACC-CRL strategy based on mechanism alignment and uncertainty modeling in open environments.
Fonte: arXiv cs.CV
RL • Score 85
Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions
arXiv:2607.03935v1 Announce Type: new
Abstract: Self-evolving frameworks usually optimize task solutions while treating the surrounding harness as fixed. We introduce Harness-Aware Self-Evolving (HASE), an agentic reinforcement-learning framework in which a single model can generate task solutions or edit selected harness components in a multi-turn action space. HASE enables a single Qwen3-8B model to match the text-classification performance of a GPT-OSS-120B model that uses Claude Code as the harness proposer. In alpha factor mining, HASE outperforms the reported GPT-OSS-120B baseline. HASE also repairs imperfect evaluation components and converges to state-of-the-art performance in circle-packing algorithm discovery. These results show that HASE improves the harness and the solution through one unified agentic process.
Fonte: arXiv cs.AI
MLOps/Systems • Score 85
SelfMem: Self-Optimizing Memory for AI Agents
arXiv:2607.03726v1 Announce Type: new
Abstract: While current AI agents support increasingly long context windows, tool use, and skill execution for long-horizon tasks, they still require memory systems to effectively leverage historical experience. Existing memory frameworks typically rely on fixed storage, retrieval, and summarization mechanisms, which can be rigid across different tasks and often require manual tuning. To address this limitation, we propose SelfMem, a self-optimizing memory framework. Inspired by prior work on self-improving AI, we follow the principle of "teaching an agent to fish rather than giving it a fish." Instead of forcing the model to follow a predefined memory strategy or format, SelfMem provides an environment with memory tools and feedback signals that allow the agent to explore, evaluate, and refine its own memory strategy. Our results show that SelfMem consistently outperforms retrieval, compression, and agent-memory baselines on BEAM across conversation scales from 100K to 1M tokens. Compared with the strongest baseline, SelfMem improves the official score by 48.7%, 40.8%, and 41.9% at 100K, 500K, and 1M, respectively. Further question-type analysis shows broad robustness across diverse memory demands, and our optimization study shows that model-guided strategy refinement further improves performance.
Fonte: arXiv cs.CL
Theory/Optimization • Score 85
msPCA: An R Package for Sparse PCA with Multiple Components
arXiv:2607.05229v1 Announce Type: new
Abstract: We present msPCA: an open-source R package for sparse principal component analysis with multiple components. It implements an alternating maximization algorithm to generate a set of sparse loading vectors that collectively explain a large fraction of the variance in a dataset, while remaining non-redundant. The algorithm supports two definitions of non-redundancy: either orthogonality of the loading vectors or zero pairwise correlation between principal components (PCs). In the reported benchmarks, msPCA solves sparse PCA problems with thousands of features, achieving competitive runtimes while producing sparse components with controlled feasibility violations and a high fraction of variance explained.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Object-Centric Environment Modeling for Agentic Tasks
arXiv:2607.02846v1 Announce Type: new
Abstract: Large language model (LLM) agents can improve through accumulated experience, but free-form textual memories become difficult to maintain, validate, and reuse as interactions grow. Recent symbolic approaches learn executable skills or programmatic world models, yet often store local procedures or assume simplified dynamics. We propose Object-Centric Environment Modeling (OCM), which organizes experience into an executable object-centric environment model. OCM maintains two connected code bases: object knowledge, which defines environment entities and mechanisms as Python classes, and procedure knowledge, which records reusable interaction patterns that must import and use the object model. OCM works in an online setting: after each episode, OCM reflects on the trajectory, updates both knowledge bases, and verifies that all procedures execute against the updated object model. During future interaction, the agent uses progressive knowledge disclosure to inspect compact code signatures first and read source code only when needed. Experiments show that OCM achieves the best average rank across benchmarks and reduces invalid actions, demonstrating that agents can benefit from building object-centric environment models.
Fonte: arXiv cs.AI
NLP/LLMs • Score 85
SiamJEPA: On the Role of Siamese Student Encoders in JEPA
arXiv:2607.04044v1 Announce Type: cross
Abstract: Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Internal Pluralism and the Limits of Pairwise Comparisons
arXiv:2607.02672v1 Announce Type: new
Abstract: Local pairwise comparisons are a standard tool for learning how people want decision rules to work, e.g., in participatory design or alignment. However, their use builds in two strong assumptions: that local comparisons are sufficient evidence about how a person wants an automated decision rule to behave, and that people can always answer those comparisons decisively. We investigate how these assumptions may be compromised under internal pluralism: the idea that an individual evaluates decision rules according to multiple authoritative priorities about how the rule should behave. We provide a formal model of such pluralistic preferences over decision rules, which then lets us identify two distinct failures of forced local pairwise comparison data. First, priorities such as proportionality, egalitarianism, and equal treatment are inherently global: what they imply in one case can depend on what happens elsewhere, so local comparisons may fail to capture them. Second, even when priorities are representable locally, tension between strongly-held priorities can generate internal conflict, producing potentially costly behavioral distortions when comparisons are forced. We then use our model to investigate the alternative -- allowing people to report indecision -- and our findings suggest that doing so can considerably reduce the number of queries needed to learn preferences accurately. We conclude by describing how our model points toward preference-learning methods that elicit these priorities directly, yielding more faithful and interpretable accounts of what people value.
Fonte: arXiv cs.AI
Theory/Optimization • Score 85
On the Convergence of Adam, Revisited
arXiv:2607.03519v1 Announce Type: cross
Abstract: We show that projected Adam for online optimization with arbitrary moment decay parameters $\beta_1,\beta_2\in[0,1)$ can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required $\beta_1<\sqrt{\beta_2}$. Similar to their result, we use a three-periodic sequence of linear functions on $[-1,1]$ with slopes $c,-1,-1$, though we use $c$ slightly larger than $2$. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.
Fonte: arXiv stat.ML
MLOps/Systems • Score 85
Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression
arXiv:2607.02562v1 Announce Type: new
Abstract: Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~$K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning.
Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ($64\!\times\!64$, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5$\times$ below H.264's practical floor and $5$-$7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using $5$-$7.6\times$ fewer bits. At $K{=}1024$, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs
arXiv:2607.03936v1 Announce Type: new
Abstract: A key challenge in Arabic NLP is the scarcity of dialectal data relative to Modern Standard Arabic (MSA), causing LLMs to overproduce MSA and struggle with dialectally accurate generation. From an interpretability perspective, this raises a fundamental question: where and how are dialectal features encoded within model internals, and can these representations be leveraged to improve dialect generation without fine-tuning? This study investigates two complementary inference-time approaches that serve simultaneously as interpretability probes and control mechanisms. First, we conduct a neuron-level analysis, identifying sparse neuron populations that encode dialect-specific features and showing that amplifying or suppressing these neurons can steer model outputs toward target dialects. Second, motivated by the entanglement of dialectal features at the single-neuron level, we apply a vector-steering approach that extracts dialect-specific activation directions and injects them during inference. Together, these methods illuminate the geometry of dialectal knowledge in Arabic LLMs and offer a principled, interpretability-grounded framework for dialect control without requiring dialect-specific fine-tuning.
Fonte: arXiv cs.CL
Vision • Score 85
Do Diabetic Foot Ulcer Segmentation Models Generalize? A Cross-Dataset Benchmark of CNN and Transformer Architectures
arXiv:2607.02555v1 Announce Type: new
Abstract: Deep learning models for diabetic foot ulcer (DFU) segmentation routinely report high accuracy, but they are almost always trained and tested on the same dataset, leaving their behaviour on data from a different clinical source largely unmeasured. We benchmark three representative segmentation architectures -- U-Net and DeepLabV3+ (convolutional) and SegFormer-B2 (Transformer) -- under an identical, leakage-screened protocol: training on the combined FUSeg/AZH wound data and evaluating, without fine-tuning, on two independent external datasets (DFUC2022 and Medetec). All models achieve strong in-domain performance (Dice 0.80--0.83) but degrade substantially across datasets. The degradation is, however, architecture-dependent: SegFormer-B2 generalizes best on both external sets (DFUC2022 Dice 0.557, Medetec Dice 0.786), outperforming both convolutional models, while the more complex DeepLabV3+ generalizes worse than the simpler U-Net. Per-image failure analysis on 2,160 images across both external test sets confirms that SegFormer-B2 produces the fewest catastrophic failures on DFUC2022 (31.1%), compared with U-Net (38.5%) and DeepLabV3+ (43.0%). The consistent ranking across two independent external sources, confirmed by Wilcoxon signed-rank tests (p < 0.001 on both datasets), indicates that architecture family, not model complexity, drives cross-hospital generalization.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
Separating Representation from Reconstruction Enables Scalable Text Encoders
arXiv:2607.04011v1 Announce Type: new
Abstract: While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose $\textbf{CrossBERT}$, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all tokens via a $\textit{Complementary Masking Strategy}$, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.
Fonte: arXiv cs.CL
Theory/Optimization • Score 85
In-span learning: adapting reduced-order models using their own predictions
arXiv:2607.02937v1 Announce Type: new
Abstract: Reduced-order models compress high-dimensional dynamics into low-dimensional representations that can be evaluated rapidly, but they lose accuracy when online dynamics drift beyond the training data. Adaptive methods address this by updating the subspace online with external, out-of-span information, such as full-order corrections or sensor snapshots. We discovered that a complementary and previously unexploited in-span adaptation channel exists within the current reduced subspace. By streaming the model's own predictions through an incremental singular-value decomposition with forgetting, we obtain a trajectory-informed spectral preconditioner, in which the subspace is unchanged but the basis is reweighted and realigned toward the modes visited by the dynamics. This enables the model to absorb future out-of-span corrections more effectively. We expose aspects of this mechanism on a three-dimensional spiral and confirm it on viscous Burgers and Fisher-KPP dynamics. We also discuss how in-span learning can be viewed as a dynamical-systems analogue of in-context learning. More broadly, in-span learning suggests a new principle for computational science, revealing that model-generated trajectories contain more usable information than previously recognized.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Robustness Meets Uncertainty: Evidential Adversarial Training for Robust Selective Classification
arXiv:2607.03075v1 Announce Type: new
Abstract: Safety-critical applications require classifiers that are both robust and reliable. Adversarial training is a widely adopted defense for improving robustness in deep neural networks; however, its effect on the reliability of predictive uncertainty remains underexplored. We investigate this gap through the lens of selective classification, which has rarely been systematically analyzed alongside adversarial robustness. We introduce a unified benchmark for the robustness-uncertainty trade-off. It standardizes architectures, augmentations, threat models, and evaluation metrics across clean, adversarial, and common-corruption settings. Across a wide range of state-of-the-art adversarial training methods, we uncover a recurring failure mode: several approaches improve robust accuracy while degrading uncertainty ranking, leading to poorer selective behavior. To address this, we propose Evidential Adversarial Training (EV-AT), which models uncertainty through a Dirichlet distribution and combines (i) an evidence-based loss promoting clean accuracy and reliable uncertainty with (ii) a robust evidence-alignment loss matching clean and adversarial predictions in log Dirichlet-parameter space. Extensive experiments show that EV-AT shifts the Pareto frontier of robustness-uncertainty trade-offs beyond prior state-of-the-art adversarial training methods. Our source code is publicly available at https://github.com/NicolasSournac/Robustness_Meets_Uncertainty.EV-AT.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
arXiv:2607.02964v1 Announce Type: new
Abstract: A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.
Fonte: arXiv cs.LG
MLOps/Systems • Score 85
Can Model Merging Improve Aggregation in DiLoCo?
arXiv:2607.03011v1 Announce Type: new
Abstract: Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a broad array of methods having been proposed to tackle this problem. Simultaneously, an emerging trend in distributed learning has been the use of methods such as local SGD and DiLoCo, which greatly reduce communication costs by periodically aggregating the independently trained local models. However, these communication-efficient methods have been shown to degrade in performance relative to the FLOP-matched data-parallel gold standard as the number of independent local models grows and as the number of local training steps before global communication is increased. In this work, we draw an explicit analogy between the pseudo-gradient aggregation step in local SGD/DiLoCo and task arithmetic-based model merging, establishing a straightforward way to utilize merging methods in the context of distributed optimization. We then evaluate multiple state-of-the-art model merging methods in this setting and identify one method in particular, Iso-C, as a promising approach for improving DiLoCo. We find that DiLoCo SGD with Iso-C aggregation outperforms not only simple pseudo-gradient averaging but even the momentum-based DiLoCo, despite lacking a momentum mechanism itself. Building on this finding, we propose IsoLoCo, which adapts Iso-C for distributed training by equipping it with Nesterov momentum. Our empirical evaluations on language model pre-training across varying numbers of local workers show that IsoLoCo significantly outperforms DiLoCo, with the gap between them widening as the number of workers increases. This advantage remains present across model sizes and inner step counts, confirming that merging-inspired aggregation is an effective strategy for low-communication distributed training.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
arXiv:2607.03057v1 Announce Type: new
Abstract: The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).
Fonte: arXiv cs.LG
RL • Score 85
Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
arXiv:2607.03125v1 Announce Type: new
Abstract: Deep reinforcement learning (DRL) offers powerful control for industrial cyber-physical systems (ICPSs), but its "black-box" exploration risks violating strict hardware safety limits. Typically, these constraints are managed through complex reward shaping. In this work-in-progress paper, we embed a differentiable physics model directly into the proximal policy optimization (PPO) actor loss function. By simulating short-horizon future trajectories during training, the policy is penalized for anticipated safety violations independent of the task-reward signal. Evaluated on a simulated 1-degree-of-freedom helicopter testbed with strict pitch constraints, our physics-informed soft regularizations substantially reduce constraint violations while maintaining reliable target tracking.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Out-of-distribution Neural Inference in Dynamical Ising Models
arXiv:2607.03039v1 Announce Type: new
Abstract: Neural networks are increasingly used to infer hidden physical structure from dynamical observations, yet it remains unclear whether their out-of-distribution performance reflects transferable physical rule learning. We address this question in a controlled inverse problem: reconstructing interaction graphs of a kinetic Ising model from Glauber magnetization trajectories. Across convolutional, graph, Transformer, and hybrid architectures, we find that data-driven training produces distinct and reproducible statistical strategies under topology and temperature shifts. Edge-population diagnostics reveal that Transformer-based models tend to preserve the link density of the training ensemble, whereas convolutional models can collapse toward sparse- or no-link predictions that appear out-of-distribution stable by exploiting the majority no-link class. Thus, high in-distribution accuracy and apparent out-of-distribution robustness do not necessarily imply a learned dynamics-to-structure rule. Instead, neural reconstruction can be governed by architecture-dependent statistical priors. Our results identify a concrete failure mode of standard data-driven learning in physical inverse problems and motivate rule-guided principles for machine-learning-assisted scientific discovery.
Fonte: arXiv cs.LG
Theory/Optimization • Score 85
Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weights
arXiv:2607.03108v1 Announce Type: new
Abstract: Post-hoc analysis of trained neural network weights often seeks to recover geometric structure directly from the parameters. We show that, for positional-encoding-equipped neural fields, the symmetry visible from weights is not the true symmetry group itself, but an observable symmetry set determined by the trained parameters, the positional encoding (PE), and readout observable. We formulate this dependence through an exact observability hierarchy, $G_{\mathrm{obs}}^{\mathrm{exact}} \subseteq G_{\mathrm{lift}}^{\mathrm{exact}}(\phi) \cap G_{\mathrm{true}}$, where $G_{\mathrm{lift}}^{\mathrm{exact}}(\phi)$ is the set of input transformations that the PE can exactly lift to the feature space. The hierarchy implies that even when a target function has a geometric symmetry, that symmetry may be structurally invisible to weight-level observables if the PE does not represent the corresponding transformation. We test this prediction using MLPs trained on two-dimensional signed distance functions with multiple shape symmetry groups, positional encodings, and Gram-based observables. The results show a consistent PE-dependent pattern: DyadicAxisPE supports $D_4$-sensitive readout but structurally suppresses $D_3$ rotations, TriAxisPE yields lower $D_3$ / $D_6$ readout scores under the tested Gram observables by replacing coordinate axes with three 120-degree-separated axes, and random Fourier features mainly exhibit a $\pi$-rotation response under these readouts. These findings show that PE design affects not only approximation behavior but also which structures are accessible to post-hoc weight-level readouts. This provides a basis for a principled observable-dependent symmetry readout.
Fonte: arXiv cs.LG
Theory/Optimization • Score 85
Rethinking Neural Nonlinearity as Gating
arXiv:2607.03148v1 Announce Type: new
Abstract: Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Spectral Signatures of Large Language Models
arXiv:2607.03377v1 Announce Type: new
Abstract: The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape-based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections.
Fonte: arXiv cs.CL
RL • Score 85
Fitted Occupancy-Ratio Evaluation without Bellman Completeness
arXiv:2607.05375v1 Announce Type: new
Abstract: Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Fixed-Confidence Best-Arm Identification for Causal Mediation Analysis
arXiv:2607.04315v1 Announce Type: new
Abstract: This paper studies the problem of identifying the treatment that maximizes the expected natural direct potential outcome (NDPO), which captures the potential outcome of an intervention while excluding the pathway transmitted through a mediator that researchers may wish to remove from evaluation. We first establish population-level identification of the expected NDPO in a causal bandit setting using observable interventional distributions. We then develop a fixed-confidence best-arm identification (BAI) algorithm based on the Track-and-Stop (TaS) framework, employing a cutting-set method to solve the resulting semi-infinite optimization problem. The proposed algorithm achieves sample-efficient identification with a high-probability correctness guarantee. We prove that it satisfies $\delta$-correctness and asymptotic optimality. Finally, we validate the approach through empirical evaluations on a large-scale real-world advertising dataset (IPinYou).
Fonte: arXiv stat.ML
RL • Score 85
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
arXiv:2607.03126v1 Announce Type: new
Abstract: Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.
Fonte: arXiv cs.LG
Theory/Optimization • Score 85
Missing Data Imputation under Manifold Hypothesis
arXiv:2607.03641v1 Announce Type: new
Abstract: The manifold hypothesis posits that high-dimensional data are concentrated near a low-dimensional embedded manifold. Recent advances in mixture variational autoencoders (VAEs) provide a powerful tool for extracting such underlying structure in a faithful manner. The resulting geometric structure naturally introduces local and global relationships among variables, thereby providing a systematic way of imputing missing data. We propose a model-based imputation method that enables sampling from \( p(\bm{x}_{\mathrm{mis}} \mid \bm{x}_{\mathrm{obs}}) \) via a sampling-importance-resampling (SIR) procedure, which can be further augmented with a joint diffusion model in the latent space. Our method imputes missing data while respecting the underlying geometry, achieves competitive performance compared to state-of-the-art procedures, quantifies uncertainty in the imputations, and is model-based, thereby enabling on-the-fly imputation without rerunning the entire procedure.
Fonte: arXiv stat.ML
RL • Score 85
Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization
arXiv:2607.03140v1 Announce Type: new
Abstract: Industrial automation increasingly demands control strategies that balance operational performance with strict energy efficiency requirements. A common approach to solving this multi-objective problem, particularly within the framework of reinforcement learning (RL), is to formulate a single, scalar reward function that linearly combines the competing objectives. However, the manual weighting of these different objectives is heavily reliant on domain intuition, incredibly time-consuming, prone to human bias, and frequently fails to uncover optimal trade-off solutions. This work addresses the critical challenge of automating the weight selection process to systematically and efficiently discover the Pareto front of optimal trade-off policies. We formulate the weight selection process as a multi-objective Bayesian optimization (MOBO) problem and evaluate its sample efficiency against a standard uniform grid search baseline. Using a physical Quanser Aero 2 testbed configured for 1-DoF pitch control, our results demonstrate that the MOBO approach, utilizing the expected hypervolume improvement (qEHVI) acquisition function, consistently outperforms uniform grid sampling. MOBO achieves superior hypervolume and maximum spread, successfully identifying high-quality, diverse trade-off policies with a reduced evaluation budget, thereby enabling highly efficient energy-aware control in complex mechatronic systems.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Conditional Diffusion Guided Knowledge Transfer for Multi-Domain Knowledge Graph Completion
arXiv:2607.03154v1 Announce Type: new
Abstract: Multi-domain knowledge graph completion (MKGC) aims to improve missing triple prediction in a target KG by transferring knowledge from other support KGs. Existing methods typically enforce consistency constraints on equivalent entities across KGs to transfer knowledge, which risks suppressing domain-specific contextual information of entities. This design can also compromise entity representation information from all KG domains, impeding performance improvements, especially in low-resource data scenarios. To address this, we pioneer a generation-based paradigm for MKGC and propose DMKGC, a conditional diffusion-guided knowledge transfer framework. Our key insight is to treat each KG as a partial view of the entity entire information, and generate informative domain-general entity embeddings through diffusion models conditioned on support KGs. Particularly, we first initialize domain-agnostic entity embeddings as prior entity embeddings, and then encode them within individual KGs. Afterward, we fuse equivalent entities from support KGs as the conditional diffusion generation guidance. We leverage the prior entity embeddings as the proxy generation objective, which ensures this conditional generation to be unbiased towards any conditioned KGs. Simultaneously, we also train the generated embeddings to be predictive across KGs, thus preserving domain-specific information. Extensive experiments on 14 KGs in 3 benchmarks demonstrate a 4.3\% average MRR improvement in tail entity prediction over state-of-the-art methods, with sustained gains in low-resource data settings.
Fonte: arXiv cs.CL
Theory/Optimization • Score 85
Dimension Reduction for Curves: Simplified and Generalized
arXiv:2607.03112v1 Announce Type: cross
Abstract: We revisit random projections for reducing the dimension of high-dimensional polygonal curves. Drawing from the toolbox of randomized linear algebra, we give a considerably simplified proof of the known $O(\varepsilon^{-2}\log(nm))$ bound on the target dimension of a random projection that preserves the continuous Fr\'echet distance of polygonal curves up to a factor $(1\pm\varepsilon)$. Our proof is based on the concept of sparse oblivious subspace embeddings. While previous techniques were limited to the case of the Fr\'echet distance, our techniques are fairly general and extend to all possible distance measures that involve the maximum, a sum or an integral over Euclidean distances between pairs of points on both input curves. We define a generalized dissimilarity measure for curves that includes several popular measures such as Fr\'echet, $q$-DTW, Hausdorff, etc. as special cases and show that the same dimension reduction technique works for this generalized dissimilarity measure. Finally, we apply the same framework for dimension reduction to piecewise linear surfaces, after extending the distance measure suitably to such surfaces.
Fonte: arXiv stat.ML
Multimodal • Score 85
Integrating Neural Encoders in Bayesian Generalized Linear Mixed Models for Multimodal Data
arXiv:2607.04647v1 Announce Type: new
Abstract: Scalable Bayesian inference for generalized linear mixed models (GLMMs) provides uncertainty-aware analysis of correlated longitudinal data, but existing scalable approaches largely assume low-dimensional tabular predictors and do not directly accommodate high-dimensional modalities such as images and text. We address this limitation by learning one or more modality-specific neural encoders jointly with a GLMM objective, then performing variance-corrected stochasticgradient MCMC for the GLMM parameters conditional on the learned representation. This conditional-Bayes design combines supervised representation learning with posterior uncertainty quantification for population-level effects, subjectspecific heterogeneity, and modality-level random slopes. The resulting model preserves interpretable fixed and random effects for structured covariates and learned modalities while scaling gracefully to large longitudinal datasets. In simulation studies, our method recovers posterior means and variance estimates from full-data MCMC benchmarks after covariance correction. We further evaluate uncertainty through parameter-level interval coverage in simulations and predictive calibration on held-out data. Applications to glaucoma progression and adolescent mental health demonstrate that the framework allows nuanced assessment of the relative importance of each modality on both individual and population levels without sacrificing predictive performance.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation
arXiv:2607.04809v1 Announce Type: new
Abstract: Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Physics-Informed Domain-Invariant Feature Learning with Autoencoder-Driven Gaussian Clustering for Robust Non-line-of-Sight Scenarios
arXiv:2607.02537v1 Announce Type: cross
Abstract: Jamming and spoofing pose significant threats to wireless and satellite navigation by disrupting radio-frequency (RF) signals and compromising availability and integrity. Robust RF interference direction finding through angle-of-arrival (AoA) estimation is therefore essential for detecting and localizing anomalous signals. Although data-driven methods perform well under line-of-sight (LoS) conditions, their performance degrades in practical environments due to non-line-of-sight (NLoS) multipath propagation. In this work, we propose a hybrid learning framework that incorporates physics-informed constraints into deep neural networks to improve the robustness of AoA estimation. A neural network is trained to estimate the azimuth and elevation of incoming signals received by a four-element antenna array, while a physics-informed loss enforces consistency between the predicted angles and inter-antenna phase differences under a plane-wave model. We further introduce a latent-space classifier to distinguish LoS from NLoS samples. Since inter-antenna phase differences under LoS propagation exhibit domain-invariant structure across environments, the physics-based loss is applied only to LoS samples, promoting physically consistent and domain-invariant representations without over-constraining the model in NLoS scenarios. In addition, domain-incremental learning (DIL) across NLoS environments with varying scatterer distributions improves cross-domain generalization. Evaluations on real-world datasets show that the proposed method reduces AoA estimation error by up to 6{\deg} in low-exemplar settings compared with DIL baselines.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
On Pairwise Quantile Regression -- Statistical Guarantees and Applications
arXiv:2607.04431v1 Announce Type: new
Abstract: Quantile regression provides a powerful tool for summarizing the conditional distribution of a real valued random variable (r.v.) of interest $Y$ as a function of covariates $Z$ in cases where it shows a large dispersion with high probability, going beyond the situation where standard least square regression is informative/predictive. This article aims to extend this methodology to the pairwise case, when the variable to be explained takes the form of a similarity function between two independent observations, such as pixelated ID photos, as input data of biometric systems) and the explanatory variables take the form of a pair of covariates of the observations, such as the age or the hair color. We establish theoretical guarantees for solutions of this statistical learning problem, considered here as empirical minimizers of a pairwise version of the pinball loss. Leveraging sharp concentration results for $U$-processes, we prove generalization bounds and identify mild conditions under which fast learning rates can be achieved. Confirming the probabilistic analysis, experiments based on simulation data also provide solid empirical evidence of the validity of the methodology promoted here for pairwise quantile regression. Finally, its usefulness from an application perspective is demonstrated by a detailed study aimed at analyzing errors in similarity scoring for facial recognition.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
A Gradient Flow Perspective on Minimum MMD Estimation
arXiv:2607.03871v1 Announce Type: cross
Abstract: Minimum maximum mean discrepancy (MMD) estimation has emerged as a robust and likelihood-free alternative to maximum likelihood estimation for parameter estimation. Yet, despite its practical success, the associated optimization problem remains poorly understood, with theoretical guarantees for existing algorithms hinging on convexity assumptions that rarely hold in practice. We address this gap by proposing a preconditioned gradient descent (PGD) scheme, establishing its asymptotic \emph{global} convergence under explicit gradient-dominance and projection-residual conditions. Our approach is inspired by recent progress on MMD gradient flows, a nonparametric descent scheme on the space of probability measures. We provide extensive empirical evidence that our PGD scheme outperforms standard gradient descent across a range of challenging parameter estimation and composite hypothesis testing problems.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens
arXiv:2607.03502v1 Announce Type: new
Abstract: Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT). This is a limit case for behavioral oversight, where surface tokens carry no information about the underlying reasoning. But hidden from the output is not the same as hidden from us. On four task families (fact retrieval, parallel numeric composition, string manipulation, and in-context computation), two open-weights frontier models (DeepSeek V3, Kimi K2) compute over filler tokens in a structured, legible way: attention routes the question through the filler region to the answer, logit-lens readouts show retrieved facts emerging early and their composition crystallizing in late layers, and KV-cache transplants at filler positions causally swap outputs between examples. We introduce an unsupervised decoding pipeline that takes only hidden states as input and recovers intermediate values with 80-95% accuracy (best LLM judge) across both models and all four tasks, without ground-truth labels or training. Hidden computation that defeats behavioral CoT monitoring is, on these tasks, directly readable from the residual stream, suggesting monitorability is a property of the model's full computational trace, not just its surface tokens.
Fonte: arXiv cs.CL
RL • Score 85
A Hierarchy of Policy Learning Problems
arXiv:2607.03385v1 Announce Type: new
Abstract: Policy learning has received substantial attention with the goal of learning policies from observational data for decision-making. A majority of work in this space has focused on developing algorithms for computing policies that minimize regret compared to the optimal policy. However, in many practical settings, there is insufficient data to obtain low regret. As a result, recent work has shifted attention to alternative objectives, most notably, studying whether it is possible to learn an improving policy that statistically significantly outperforms baseline policies. We argue that there is substantial merit in studying a broader range of policy learning problems. When there is insufficient data to learn an improving policy, there may still be useful questions that can be answered. To this end, we provide a mathematical framework for studying the relationships between policy learning problems. We formalize three problems within our framework: beyond the optimal policy problem and the improving policy problem, we also propose the policy existence problem, which aims to determine if an improving policy exists. Within our framework, we show that the policy existence problem reduces to the improving policy problem, which in turn reduces to the optimal policy problem; these reductions prove that each problem is at least as easy as the next one (in sample complexity). A key question remains: is this hardness strict? We provide partial answers. First, the gap between the optimal policy and improving policy problems is strict. For the improving policy and policy existence problems, we prove that a sublinear polynomial gap exists under natural conditions on improving policy learning algorithms. Thus, we may be able to answer questions about the existence of an improving policy even when we cannot find one. These results highlight the value in studying a broader range of policy learning problems.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Benign Overfitting Does Not Occur in Diffusion Models
arXiv:2607.02671v1 Announce Type: new
Abstract: Benign overfitting and double descent have come to shape our understanding of generalization in deep learning, establishing that overfitting is not only compatible with good generalization but can actively benefit it. Diffusion models share much of the machinery of standard deep learning, so it is natural to assume that they also exhibit these properties. In this work, we show that this assumption is largely incorrect. We first establish fundamental impossibility results showing that, unless the sample size grows exponentially with the data dimension, overfitting and good generalization cannot occur simultaneously. Consequently, the population loss follows a classical U-shaped curve in model complexity rather than exhibiting double descent. Analyzing a simplified setting, we identify a key difference between regression and score matching: regression benefits from an alignment between the target and the empirical covariance; score matching admits no such alignment, leaving overfitting irreparably harmful. We further identify implicit regularization stemming from time-smoothness of the score and early stopping during training as mechanisms that prevent such overfitting and verify our findings with high-dimensional image generation experiments. Our results reveal that generalization in diffusion models is governed by mechanisms distinct from those of traditional regression, motivating the development of new theory.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models
arXiv:2607.04360v1 Announce Type: new
Abstract: Conditional generative models have emerged as powerful tools for sampling from target conditional distributions, driving substantial advances across a wide range of scientific and applied domains. As these models proliferate, practitioners often face multiple plausible generators whose performance can vary with the task, data, or input condition. We propose an optimal model averaging framework for conditional generative models, allowing candidate generators to be combined even when they are accessible only through conditional samples without tractable densities. Specifically, we use a sample-based maximum mean discrepancy between conditional distributions, which first leads to a static model averaging method, StaticMA, assigning fixed weights to different candidates. In addition, we develop MoEMA (mixture-of-experts model averaging), an input-adaptive method that parameterizes covariate-dependent weights through a softmax neural-network gate. We establish in-sample and out-of-sample asymptotic optimality for the proposed methods, together with consistency of the estimated adaptive weight function under regularity conditions. The framework applies directly to Euclidean responses and extends to unstructured data by combining our formulation with fixed representation maps. Across a broad set of simulations and real-data studies spanning tabular, image, and text modalities, MoEMA generally improves over competing baselines, demonstrating the effectiveness of our proposed methods.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Tightening the Score Matching Gap for Diffusion Models
arXiv:2607.04442v1 Announce Type: new
Abstract: Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation primarily rely on an Evidence Lower Bound (ELBO), which relates the Kullback-Leibler (KL) divergence of model samples to the score matching loss along the path, which serves as a tractable surrogate. The difference between sample quality and the score matching loss produced by this bound leads to the \emph{score matching gap}, which is known to be tight in the worst-case but not descriptive of sample quality in general. In this work, we provide a theoretical analysis of this gap, developing tighter bounds for three metrics: KL divergence, reverse KL divergence, and Wasserstein distance, effectively exploiting the regularity of the class of score estimators. Our results suggest that the quality of the score approximation has more impact on closing the score matching gap for low noise scales. To obtain these bounds, our key technical insight is to exploit the contraction properties of the backward processes. In particular, we rely on entropy flows, logarithmic Sobolev inequalities and reflection couplings, rigorously linking the ergodicity of the Langevin diffusion to the score matching gap problem.
Fonte: arXiv stat.ML
MLOps/Systems • Score 85
Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities
arXiv:2607.03171v1 Announce Type: new
Abstract: Decentralised federated learning, based on peer-to-peer communication, is increasingly proposed for on-device training of machine learning models, promising a privacy-preserving, communication-efficient training process with no risk of single-point failure. However, the role of structural and temporal inhomogeneities in such fully decentralised settings remains poorly understood. Here, we investigate their effects when model parameters are locally averaged during aggregation. We show that the decentralised federated learning process is governed, both in the early phase and the late, stationary limit, by the same dynamics as a lazy random-walk diffusion process on temporal networks. Based on this mapping, we demonstrate that the typical experimental scenario used in decentralised federated learning leads to unrealistically rapid convergence because of ignoring the temporal and structural inhomogeneities inherent in the communication network. We analyse real-world temporal networks and find that inhomogeneities most often dramatically slow down diffusion, hence the convergence process.
Fonte: arXiv cs.LG
Privacy/Security/Fairness • Score 85
Locally Private Online Quantile Regression: Estimation and Inference
arXiv:2607.05312v1 Announce Type: new
Abstract: We study estimation and inference for online quantile regression under a one-report user-level $\eps$-locally differentially private ($\eps$-LDP) protocol. The main difficulty is that the standard quantile-regression estimating-equation contribution couples covariates with a residual comparison, so a server that receives only privatized reports cannot form the usual online update. We address this by developing a finite-alphabet channel in which each user computes the contribution locally, applies support-aware stochastic quantization and randomized response to one selected-block category, and sends one report. A public decoder corrects the randomized-response distortion and reconstructs a server-side estimating-equation input with the correct conditional mean. These decoded inputs are then used in projected Polyak-Ruppert averaging. For fixed finite channel designs, we establish local privacy, decoder unbiasedness, consistency, asymptotic normality, and Hessian-free self-normalized inference for prespecified scalar contrasts. Simulations and a New York City taxi-trip illustration show that the private trajectory approaches the nonprivate online reference as the privacy budget grows and outperforms direct Laplace and face-exponential geometric releases in the reported regimes.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Back to Basics: Improving Molecular Understanding in LLMs via SMILES-Graph Translation
arXiv:2607.03007v1 Announce Type: new
Abstract: Recent advances in molecular large language models have led to strong performance on molecular understanding and generation tasks, yet these gains often come without reliable structural grounding. In particular, existing approaches conflict with the chemistry principle that structure determines function: despite their downstream success, current molecular LLMs perform poorly on basic structure recognition, suggesting that they fail to capture molecular graphs from canonical SMILES. To remedy this, we propose MolBasic, a structure-first framework that strengthens structural comprehension via SMILES-Graph translation. MolBasic is built around a multi-level structure perception benchmark, where bidirectional SMILES-Graph conversion serves as the core task to align sequential and topological representations. On top of this foundation, we employ a progressive learning scheme with a standardized Chain-of-Thought (CoT) to steer models from structure acquisition toward higher-level molecular reasoning. Experiments show that MolBasic substantially improves structural understanding and yields robust gains on downstream tasks, including property prediction and objective optimization, supporting our structure-first paradigm.
Fonte: arXiv cs.LG
Theory/Optimization • Score 85
Outcome-adapted Automatic Debiased Machine Learning
arXiv:2607.03351v1 Announce Type: cross
Abstract: Parameters of interest in causal inference, such as treatment or policy effects, can often be expressed as linear functionals of an outcome regression function. Automatic debiased machine learning (AutoDML) is a unified framework for obtaining asymptotically normal estimators of such parameters, which requires estimation of both a regression function and a Riesz representer. Existing AutoDML neural network architectures, such as RieszNet and MADNet, use a shared intermediate covariate representation. However, it remains unclear whether this shared representation should be predictive of the Riesz representer or the outcome.
We show that a shared representation of the covariates that preserves predictive power of the outcome while discarding information about the Riesz representer is asymptotically more efficient than the baseline AutoDML estimator that uses all covariates. Motivated by these results, we propose the outcome-adapted AutoDML estimator and establish its asymptotic behavior in a sample splitting framework. We provide a neural network implementation of the estimator that learns a sparse representation of the covariates that is predictive of the outcome but not predictive of the Riesz representer. We demonstrate the efficiency gains of our estimator over existing alternatives on synthetic data and achieve state-of-the-art estimation accuracy on the semi-synthetic IHDP benchmark dataset.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Significance-First Splitting: Aligning Treatment Heterogeneity Detection with Honest Estimation
arXiv:2607.03999v1 Announce Type: cross
Abstract: Estimating heterogeneous treatment effects (CATE) requires simultaneously detecting effect modification and quantifying estimation uncertainty. Existing tree-based methods make an uneasy trade-off: significance-based approaches (Radcliffe and Surry 2011) identify subgroup interactions directly but lack valid inference; honest causal trees (Athey and Imbens 2016) deliver nominal confidence interval coverage but use outcome-agnostic splitting criteria that sacrifice interaction sensitivity. We introduce a hybrid algorithm that fuses significance-based splitting with honest sample-splitting and cross-validation. Our splitting criterion uses the squared $t$-statistic for the treatment $\times$ side interaction ($t^2$), which is shown to be directly aligned with the honest $\text{EMSE}_\tau$ criterion when the interaction is strong. Post-hoc honest cross-validation selects the cost-complexity penalty, giving a single principled estimator with nominal CI coverage at the leaf level. For forests, we retain bootstrap count vectors to enable an infinitesimal jackknife (IJ) variance estimate of Monte-Carlo convergence rather than formal pointwise inference. On the three synthetic designs from (Athey and Imbens 2016) the single tree achieves approximately 90\% leaf-average CI coverage at the 90\% nominal level across all three designs (200 replications each); on the Criteo and Starbucks uplift datasets we match Qini coefficient performance of S- and T-learner baselines. An open-source Python package with reproducible seeds, sklearn-compatible API, and full test coverage accompanies this work (https://codeberg.org/hadjipantelis/rattus).
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
OpFlow: Learning Opportunity-Conditioned Choice Potentials for Robust OD Flow Prediction
arXiv:2607.03200v1 Announce Type: cross
Abstract: Origin-destination (OD) flow prediction is central to urban analytics, yet deep models trained on raw counts remain vulnerable to distribution shift. The core problem is that raw count supervision cannot distinguish transferable choice mechanisms from environment-specific shortcuts. Raw OD count mixes two objects: how much demand an origin produces and how that demand is allocated across destinations. We argue that the transferable object is the exposure-to-choice law that maps spatial conditions to relative destination preferences. We propose OpFlow, a mechanism-constrained framework that learns row-centered choice potentials and reconstructs flows by combining the induced allocation with a separately calibrated origin scale. Under distribution shift, spatial exposures and the induced allocations are allowed to vary; what transfers is the conditional map from exposure states to relative choice potentials. Theoretically, we characterize the identifiable row-centered potential and show that classical spatial interaction laws are restricted log-potential cases. Controlled synthetic shifts and a real-world experiment show OpFlow improves robustness under environment shifts.
Fonte: arXiv stat.ML
MLOps/Systems • Score 85
PLACEMEM: Toward a Compute-Aware Memory Plane for Lifelong Agents
arXiv:2607.04089v1 Announce Type: new
Abstract: Lifelong agents need more than larger context windows and better retrieval. They need memories that can persist, evolve, and be corrected without forcing the serving stack to recompute the same history on every turn or silently reuse stale runtime state. We present PLACEMEM as a systems position on lifelong-agent memory, instantiated by an executable control-plane prototype. The central claim is that agent memory should be represented as versioned capsules that unify semantics, provenance, validity, and reusable runtime state under one correction-aware identity. In the current prototype, capsules drive prompt-level text retrieval, KV-aware routing, and cascading invalidation over live streamed backends; prospective layer-frontier replay is intentionally framed as a deeper integration agenda rather than a claimed engine feature. We describe a vLLM-first prototype with persistent capsule state, concurrency-safe invalidation, an OpenAI-compatible routing sidecar, a typed metadata contract, and a benchmark harness that measures live first-token latency, reuse, and post-correction behavior. The result is both an executable artifact that demonstrates correction-aware control-plane behavior today and a concrete roadmap for replay-aware serving integration in future lifelong-agent systems.
Fonte: arXiv cs.AI
Theory/Optimization • Score 85
Tightening Control in Neyman--Pearson Linear Classification
arXiv:2607.03590v1 Announce Type: cross
Abstract: Neyman--Pearson classification prioritizes one class by constraining its accuracy above a prespecified level, and then takes the accuracy of the other class as the utility objective. This paradigm is well suited for disease screening and diagnosis, among other applications. Statistical learning under this framework is complicated since classifier performance determines its acceptability. Furthermore, no learned classifier that is consistent for the oracle classifier can guarantee satisfaction of the control constraint in finite samples. Classical learning theory targets a control-relaxed empirical utility maximization (EUM) classifier. However, even the EUM classifier fails to achieve the desired control level on average. We conjecture that this under-control phenomenon is a manifestation of the over-optimism bias well known in standard statistical learning, and develop asymptotic theory to confirm it. Motivated by this insight, we propose refined learning procedures under two accuracy control strategies for the prioritized class: one controlling accuracy in expectation and the other with high probability. We further develop training-data-based methods to predict and infer class-specific accuracies of the resulting classifiers. Simulation studies demonstrate favorable finite-sample performance, and we illustrate the proposed methods with an application to cancer detection.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG
arXiv:2607.03929v1 Announce Type: new
Abstract: Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata (the news source) that it extracts from each query by prompting gpt-3.5-turbo. We show this proprietary, free-form extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model. On all 2556 MultiHop-RAG queries the probe reaches 90.9% set-exact accuracy against 88.0% for a model-free substring baseline and 80.9% for GPT-3.5, a margin that comes entirely from null queries, on which GPT-3.5 never abstains; on non-null queries all three stay within about a point. Because the probe's output space is exactly the fixed 49-source vocabulary, it cannot drift outside the allow-list as the prompted model does. Three design choices make it work: selecting a shallow layer, mean pooling, and class-imbalance-aware multi-label training over the long tail of sources. A 135M-parameter model lands within ~1.5 points of a 1.5B one, so the filter is cheap to output: a partial forward pass through the first few layers plus one linear head, with no API. The code is available at https://github.com/mxpoliakov/Multi-Meta-RAG.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
arXiv:2607.03809v1 Announce Type: cross
Abstract: Normalising flows provide a powerful variational family for approximate inference, yet individual architectures often fail to generalise across heterogeneous posterior geometries. We revisit mixture-based flow formulations and introduce \emph{AMF\mbox{-}VI\mbox{-}sEMA}, a two-stage framework featuring a \emph{stable global weighting} mechanism based on a \emph{Simplex Exponential Moving Average} (sEMA) update. In Stage~1, a heterogeneous set of experts (\textsc{RealNVP}, \textsc{MAF}, \textsc{RBIG}) are trained independently to specialise in distinct structural regimes. In Stage~2, expert parameters are frozen and global mixture weights are learned through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update on the probability simplex. This design produces a tractable, data-agnostic gating mechanism (without per-sample gating or gradient backpropagation through weights) that adaptively reallocates capacity while avoiding component collapse. We evaluate the framework on ten posterior benchmarks: six canonical 2D synthetic families (Banana, X-Shaped, Bimodal, Multimodal, Two-moons, Rings) and four real/low-dimensional Bayesian targets (BLR, BPR, Weibull, Real-GMM2), with stronger baselines (\textsc{NICE}, \textsc{ResFlow}, and EM-Mixing). Comprehensive evaluation covers NLL, KL divergence, Wasserstein-2 distance, and MMD, together with diagnostics of mixture dynamics, hyperparameter sensitivity, and cross-seed robustness. Empirically, \emph{AMF\mbox{-}VI\mbox{-}sEMA} achieves consistent NLL improvements over its predecessor \emph{AMF\mbox{-}VI} and avoids the catastrophic transport failures of single-flow baselines, while maintaining stable weight trajectories ($N_{\mathrm{eff}}{>}1.4$ on all datasets) with minimal computational overhead.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Rethinking Scientific Discovery in an Agentic Era
arXiv:2607.03863v1 Announce Type: new
Abstract: Artificial intelligence has advanced scientific discovery, but most AI4Science systems remain fragmented tools that rely on humans to coordinate problem formulation, literature grounding, model use, simulation, validation, and knowledge reuse. This paper presents \textbf{SCION (Scientific Collaborative Innovation with Agentic Organizational Nexus)}, an agentic scientific operating system that acts as an \textbf{organizational nexus}. Through a Science Agent serving as a \textbf{Meta-Harness}, SCION connects scientific tasks, tools, agents, artifacts, and memory, transforming research into an executable, auditable, and reusable operational process. At its core is the \textbf{Research Execution Plan (REP)}, which compiles high-level scientific intent into staged objectives, dependencies, verification checkpoints, tool requirements, expected artifacts, and fallback conditions. SCION further integrates hierarchical multi-agent execution, profile-driven specialization, selective context construction, governed delegation, and layered epistemic memory to support long-horizon scientific work. We formulate discovery under SCION as \textbf{Target-conditioned Inverse Search} and extend it to hidden-target settings through batch active search under finite experimental budgets. Applications in materials analysis, molecule design, and protein or antibody screening, together with experiments on scientific reading, idea generation, molecule generation, and antibody screening, show that SCION outperforms existing autonomous research-agent baselines, especially in decomposition, verification, refinement, and memory reuse. Overall, SCION shifts AI from isolated tools toward a coordinated operational layer for traceable and reusable scientific innovation.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO
arXiv:2607.03957v1 Announce Type: new
Abstract: Language models can reach the right normative verdict for the wrong reason. We introduce NormWorlds-CF, a solver-verified environment for counterfactual normative reasoning in executable rule worlds. Its deterministic solver produces final answers, proof and falsification certificates, argument statuses, support sets, and paired-world change labels, enabling supervision and evaluation without LLM judges. The benchmark contains staged SFT diagnostics and a compact paired-world task with 270 root families and 1080 canonical-to-variant pairs. The SFT diagnostics show that final-answer supervision is an unsafe proxy: answer-only SFT reaches perfect accuracy on answer tasks but scores zero on falsification, while proof-plus-falsification training with targeted replay reaches strong all-task accuracy. For the structured-change task, we introduce metamorphic-relation GRPO (MR-GRPO), a class-conditioned reward for GRPO that gives partial credit for relation families and solver-visible change fields. In matched 1.7B continuation experiments, MR-GRPO improves held-out relation accuracy and relation-family correctness, and reduces wrong-family error, compared to sparse and answer-only GRPO. In Qwen3-4B three-seed validation, answer-only reward improves answer-change fields but weakens relation-family structure, sparse reward preserves coarse relation labels best, and MR-GRPO delivers the strongest balanced performance across answer-change, support-change, status-change, and soft root-level metamorphic-relation metrics. These results show that verified counterfactual structure can shape post-training beyond final answers, while exact full change-record generation, invariant subtype recognition, and out-of-distribution (OOD) transfer remain open problems.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
arXiv:2607.04061v1 Announce Type: cross
Abstract: Distinguishing Large Language Model (LLM) generated text from human writing is a critical and difficult challenge. While LLMs are trained to write like humans, we hypothesize that this training leaves an indelible mark. LLMs develop a particularly strong aversion to token repetition very early in training. This bias persists as a ''Vestigial Heuristic'' (a developmental artifact) that is activated in LLM-generated text, separating LLM from human writing. To probe this phenomenon, we introduce Telescope Perplexity, a metric that evaluates the token repetition of the model, $P(s_i | s_{1:i})$ . Our empirical investigation reveals that the Telescope Perplexity signature emerges early in pre-training, and Telescope Perplexity empirically enables highly effective zero-shot LLM detection. We show state-of-the-art or competitive performance across diverse datasets (including modern evaluation sets we introduce), reference models, and perturbation schemes with greater efficiency than other methods.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It
arXiv:2607.03598v1 Announce Type: new
Abstract: When a person shares something with a language model, the model often answers the surface of the message rather than what the sender was doing by sending it: share a finished project and it critiques the code; share a raw late-night line and it runs a wellness check. We treat the sender's communicative intent, the Gricean what-was-meant, as a first-class interpretability object, and show the failure is one of readout on top of a robust representation. A linear probe decodes the sender's intent, whether they want a thing recognized or evaluated, from a model's default-pass hidden states, cleanly and surface-independently, across six models and four families and in the base checkpoints. The representation generalizes further, to intent that is only pragmatically inferred, and to a second, lexically clean intent (support versus help). The behavioral half of the story, and every causal test, is established on the recognize/evaluate contrast, where what varies is whether the default output acts on the intent. The readout lags the representation in depth within a model (the intent is decodable several layers before it drives the output); across models, which ones act on it by default is model-specific, an observed stratification (three of six show the failure) that we do not read as a scaling law. Where the gap is open, a direction closely tied to the representation, the discriminative direction at a searched-for layer, is a causal handle: steering it recovers the intended behavior, as well as an explicit instruction does and with no prompt at all. This direction is near-orthogonal to the feedback-offering axis, so it routes a represented intent rather than a generic feedback knob, though at the recovery dose the routed intent can override an explicit request. We support each link with controls against obvious deflations and report the nulls as plainly as the confirmations.
Fonte: arXiv cs.CL
Vision • Score 85
Coordinate Singularities Break Conformal Coverage for Gaze and Head Pose
arXiv:2607.02565v1 Announce Type: new
Abstract: Conformal prediction provides distribution-free reliability guarantees for vision systems, but these guarantees depend on how prediction errors are measured in the output space. Many vision tasks produce outputs on curved spaces (e.g. gaze directions on the sphere or 3D head rotations), yet intermediate prediction heads, residuals, uncertainty estimates, or conformal scores are often defined in flat coordinate charts such as yaw-pitch or Euler angles. We show that this scoring choice introduces systematic geometric distortion near coordinate singularities (large pitch angles on the sphere and poses approaching gimbal lock in 3D rotations). Across four datasets (ETH-XGaze, Gaze360, BIWI, AFLW2000-3D), slice-conditional coverage at a nominal 90% target drops by 30-50 percentage points in these regions, falling to 38.9% on ETH-XGaze and 42.0% on Gaze360 at gaze pitch above 70 degrees, and to 57.5% on BIWI and 55.2% on AFLW2000-3D at head pose pitch above 60 degrees near gimbal lock, despite marginal coverage remaining near 90%. We prove that this is structural. Scalar thresholding changes the size of chart-coordinate prediction sets but leaves their distorted axis ratios unchanged. To diagnose this hidden failure mode, we show that a simple geometric quantity, the Riemannian volume density, strongly correlates with where coverage collapse occurs. Finally, we show that coordinate-free geodesic scoring removes this distortion. It requires no retraining and adds negligible computational cost.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
Symmetry-Structured Neural Completion of Islamic Geometric Patterns from Sparse Control Geometry
arXiv:2607.02573v1 Announce Type: new
Abstract: Islamic geometric patterns are governed by exact rotational symmetry and strict construction rules. This paper treats these rules as formal geometric knowledge and embeds them in a neural completion framework, rather than leaving them to be learned statistically from data. Given sparse control geometry and a target symmetry order, the system completes the pattern as a vector graph by predicting edges and refinements of bounded curves over a candidate lattice whose edges are organised into rotational orbits under the cyclic group. Symmetry is enforced either by constraining predictions within these orbits or by projecting them onto them during inference. The orbit-tied variant provides a constructive guarantee: for any input and any orbit-level selection rule, it produces exact N-fold symmetry, preserves anchor points, and keeps all refinements within prescribed bounds. These properties are verified numerically. The study focuses on rotational symmetry, and all quantitative results are obtained from procedurally generated graphs inspired by Islamic geometric design rather than from a historical corpus. On clean inputs, enforcing exact validity produces no measurable loss in fidelity. When control geometry is missing, an unstructured decoder loses fidelity and breaks symmetry; retraining on corrupted inputs recovers much of the fidelity but not exact validity. Symmetry-structured inference, by contrast, keeps violations at zero throughout. The results show that augmentation and symmetry structure address distinct failure modes: augmentation improves fidelity under corruption, while symmetry structure guarantees validity. The framework therefore provides a knowledge-constrained, guarantee-backed approach to neural completion for scalable vector ornaments whose validity depends on exact geometric structure.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding
arXiv:2607.03236v1 Announce Type: new
Abstract: Diffusion language models (DLLMs) generate text by iteratively denoising masked positions, exposing a trajectory of predictive distributions rather than a single instantaneous belief. Most existing decoders ignore this trajectory and commit tokens from the current snapshot alone, conflating confidence with commitment readiness: a transient top-1 peak under incomplete context can be locked in, while candidates with consistent cross-step support are delayed. We propose Trajectory-Aware Commit Gating (TACG), a training-free gate-level decoder that anchors token identities to the base posterior and uses trajectory-aware signals only to decide whether the current proposal is ready to commit. TACG combines Temporal Implicit Logits Guidance (TILG), which keeps an exponential moving average of past logits as a self-reference and contrasts the current logits against this reference in natural-parameter space, with a History Gate (HG) that enforces short-term proposal persistence before commitment. Together with a capped extra-promotion budget, these components yield a stability-constrained commit rule without auxiliary networks or extra forward passes. We evaluate TACG on LLaDA, Dream, and LLaDA2-Mini across code (HumanEval, MBPP) and math (GSM8K, MATH500) benchmarks; it typically improves or preserves accuracy while reducing denoising steps and increasing tokens per forward (TPF). The code is publicly available at https://github.com/Clarence-CV/TACG-DLLM.
Fonte: arXiv cs.CL
Evaluation/Benchmarks • Score 85
Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits
arXiv:2607.02586v1 Announce Type: new
Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence. We argue the audits are themselves fragile: their conclusions can be silently manufactured by implementation details that readers cannot see in the reported numbers. We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models. Under a unified six-point due-diligence gate, every cell lands in a non-confirmatory bucket, and no cell reaches confirmatory. The evidence here is a single two-model, five-benchmark case study, and F1--F5 is an illustrative, deliberately non-exhaustive starting taxonomy -- not a comprehensive partition of audit failures. We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence, and not as a route to benchmark-validity verdicts.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery
arXiv:2607.02807v1 Announce Type: new
Abstract: Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems. However, they tend to converge onto a single high-level approach, then proceed with low-level edits while missing other superior approaches to the problem. We hypothesize two harness-level design choices contribute to this behavior: accumulating context in a single long-running agent and only exposing a single program state to edit. We introduce SwarmResearch, an orchestrator-subagent harness in which a Shepherd Agent uses global context to steer a population of Search Agents, each operating with local context in their respective git branch. On open-ended optimization tasks, SwarmResearch discovers better or comparable solutions to state-of-the-art LLM-guided evolution and multi-agent techniques on 13/15 tasks, driven by higher-level exploration. Compared with fixed scaling of serial and parallel agents, SwarmResearch's orchestrator-guided scaling discovers better-performing solutions by adapting parallelism at different search depths.
Fonte: arXiv cs.AI
NLP/LLMs • Score 85
A Unified Framework for In-Context Learning with Causal and Masked Language Models
arXiv:2607.04081v1 Announce Type: cross
Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models. Masked language models instead recover masked tokens from bidirectional context, and their role in ICL remains less understood. We develop a statistical learning framework that represents the context examples by their empirical measure and models prediction as a function of the context and the query. This formulation places autoregressive and masked pretraining objectives within a common excess-risk analysis. Under Wasserstein-type regularity conditions, we relate pretraining with T tasks and N samples per task to k-shot excess risk at inference, obtaining same-order upper bounds for masked and autoregressive objectives. We also study task-distribution shift, where pretraining tasks are sampled from P and inference tasks from Q; the resulting bound contains an additional term controlled by the lifted Wasserstein distance between P and Q. The bounds further imply an order-optimal allocation under a fixed pretraining data budget and refined rates under intrinsic low-dimensional structure. Experiments on controlled function-learning tasks show that the Masked Pair Encoder (MPE) can achieve performance comparable to GPT-2-style causal Transformers, suggesting that ICL behavior is not specific to causal language models.
Fonte: arXiv stat.ML