NLP/LLMs • Score 85
Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG
arXiv:2607.03929v1 Announce Type: new
Abstract: Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata (the news source) that it extracts from each query by prompting gpt-3.5-turbo. We show this proprietary, free-form extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model. On all 2556 MultiHop-RAG queries the probe reaches 90.9% set-exact accuracy against 88.0% for a model-free substring baseline and 80.9% for GPT-3.5, a margin that comes entirely from null queries, on which GPT-3.5 never abstains; on non-null queries all three stay within about a point. Because the probe's output space is exactly the fixed 49-source vocabulary, it cannot drift outside the allow-list as the prompted model does. Three design choices make it work: selecting a shallow layer, mean pooling, and class-imbalance-aware multi-label training over the long tail of sources. A 135M-parameter model lands within ~1.5 points of a 1.5B one, so the filter is cheap to output: a partial forward pass through the first few layers plus one linear head, with no API. The code is available at https://github.com/mxpoliakov/Multi-Meta-RAG.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation
arXiv:2607.04809v1 Announce Type: new
Abstract: Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Physics-Informed Domain-Invariant Feature Learning with Autoencoder-Driven Gaussian Clustering for Robust Non-line-of-Sight Scenarios
arXiv:2607.02537v1 Announce Type: cross
Abstract: Jamming and spoofing pose significant threats to wireless and satellite navigation by disrupting radio-frequency (RF) signals and compromising availability and integrity. Robust RF interference direction finding through angle-of-arrival (AoA) estimation is therefore essential for detecting and localizing anomalous signals. Although data-driven methods perform well under line-of-sight (LoS) conditions, their performance degrades in practical environments due to non-line-of-sight (NLoS) multipath propagation. In this work, we propose a hybrid learning framework that incorporates physics-informed constraints into deep neural networks to improve the robustness of AoA estimation. A neural network is trained to estimate the azimuth and elevation of incoming signals received by a four-element antenna array, while a physics-informed loss enforces consistency between the predicted angles and inter-antenna phase differences under a plane-wave model. We further introduce a latent-space classifier to distinguish LoS from NLoS samples. Since inter-antenna phase differences under LoS propagation exhibit domain-invariant structure across environments, the physics-based loss is applied only to LoS samples, promoting physically consistent and domain-invariant representations without over-constraining the model in NLoS scenarios. In addition, domain-incremental learning (DIL) across NLoS environments with varying scatterer distributions improves cross-domain generalization. Evaluations on real-world datasets show that the proposed method reduces AoA estimation error by up to 6{\deg} in low-exemplar settings compared with DIL baselines.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
On Pairwise Quantile Regression -- Statistical Guarantees and Applications
arXiv:2607.04431v1 Announce Type: new
Abstract: Quantile regression provides a powerful tool for summarizing the conditional distribution of a real valued random variable (r.v.) of interest $Y$ as a function of covariates $Z$ in cases where it shows a large dispersion with high probability, going beyond the situation where standard least square regression is informative/predictive. This article aims to extend this methodology to the pairwise case, when the variable to be explained takes the form of a similarity function between two independent observations, such as pixelated ID photos, as input data of biometric systems) and the explanatory variables take the form of a pair of covariates of the observations, such as the age or the hair color. We establish theoretical guarantees for solutions of this statistical learning problem, considered here as empirical minimizers of a pairwise version of the pinball loss. Leveraging sharp concentration results for $U$-processes, we prove generalization bounds and identify mild conditions under which fast learning rates can be achieved. Confirming the probabilistic analysis, experiments based on simulation data also provide solid empirical evidence of the validity of the methodology promoted here for pairwise quantile regression. Finally, its usefulness from an application perspective is demonstrated by a detailed study aimed at analyzing errors in similarity scoring for facial recognition.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Benign Overfitting Does Not Occur in Diffusion Models
arXiv:2607.02671v1 Announce Type: new
Abstract: Benign overfitting and double descent have come to shape our understanding of generalization in deep learning, establishing that overfitting is not only compatible with good generalization but can actively benefit it. Diffusion models share much of the machinery of standard deep learning, so it is natural to assume that they also exhibit these properties. In this work, we show that this assumption is largely incorrect. We first establish fundamental impossibility results showing that, unless the sample size grows exponentially with the data dimension, overfitting and good generalization cannot occur simultaneously. Consequently, the population loss follows a classical U-shaped curve in model complexity rather than exhibiting double descent. Analyzing a simplified setting, we identify a key difference between regression and score matching: regression benefits from an alignment between the target and the empirical covariance; score matching admits no such alignment, leaving overfitting irreparably harmful. We further identify implicit regularization stemming from time-smoothness of the score and early stopping during training as mechanisms that prevent such overfitting and verify our findings with high-dimensional image generation experiments. Our results reveal that generalization in diffusion models is governed by mechanisms distinct from those of traditional regression, motivating the development of new theory.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models
arXiv:2607.04360v1 Announce Type: new
Abstract: Conditional generative models have emerged as powerful tools for sampling from target conditional distributions, driving substantial advances across a wide range of scientific and applied domains. As these models proliferate, practitioners often face multiple plausible generators whose performance can vary with the task, data, or input condition. We propose an optimal model averaging framework for conditional generative models, allowing candidate generators to be combined even when they are accessible only through conditional samples without tractable densities. Specifically, we use a sample-based maximum mean discrepancy between conditional distributions, which first leads to a static model averaging method, StaticMA, assigning fixed weights to different candidates. In addition, we develop MoEMA (mixture-of-experts model averaging), an input-adaptive method that parameterizes covariate-dependent weights through a softmax neural-network gate. We establish in-sample and out-of-sample asymptotic optimality for the proposed methods, together with consistency of the estimated adaptive weight function under regularity conditions. The framework applies directly to Euclidean responses and extends to unstructured data by combining our formulation with fixed representation maps. Across a broad set of simulations and real-data studies spanning tabular, image, and text modalities, MoEMA generally improves over competing baselines, demonstrating the effectiveness of our proposed methods.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Tightening the Score Matching Gap for Diffusion Models
arXiv:2607.04442v1 Announce Type: new
Abstract: Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation primarily rely on an Evidence Lower Bound (ELBO), which relates the Kullback-Leibler (KL) divergence of model samples to the score matching loss along the path, which serves as a tractable surrogate. The difference between sample quality and the score matching loss produced by this bound leads to the \emph{score matching gap}, which is known to be tight in the worst-case but not descriptive of sample quality in general. In this work, we provide a theoretical analysis of this gap, developing tighter bounds for three metrics: KL divergence, reverse KL divergence, and Wasserstein distance, effectively exploiting the regularity of the class of score estimators. Our results suggest that the quality of the score approximation has more impact on closing the score matching gap for low noise scales. To obtain these bounds, our key technical insight is to exploit the contraction properties of the backward processes. In particular, we rely on entropy flows, logarithmic Sobolev inequalities and reflection couplings, rigorously linking the ergodicity of the Langevin diffusion to the score matching gap problem.
Fonte: arXiv stat.ML
Privacy/Security/Fairness • Score 85
Locally Private Online Quantile Regression: Estimation and Inference
arXiv:2607.05312v1 Announce Type: new
Abstract: We study estimation and inference for online quantile regression under a one-report user-level $\eps$-locally differentially private ($\eps$-LDP) protocol. The main difficulty is that the standard quantile-regression estimating-equation contribution couples covariates with a residual comparison, so a server that receives only privatized reports cannot form the usual online update. We address this by developing a finite-alphabet channel in which each user computes the contribution locally, applies support-aware stochastic quantization and randomized response to one selected-block category, and sends one report. A public decoder corrects the randomized-response distortion and reconstructs a server-side estimating-equation input with the correct conditional mean. These decoded inputs are then used in projected Polyak-Ruppert averaging. For fixed finite channel designs, we establish local privacy, decoder unbiasedness, consistency, asymptotic normality, and Hessian-free self-normalized inference for prespecified scalar contrasts. Simulations and a New York City taxi-trip illustration show that the private trajectory approaches the nonprivate online reference as the privacy budget grows and outperforms direct Laplace and face-exponential geometric releases in the reported regimes.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Outcome-adapted Automatic Debiased Machine Learning
arXiv:2607.03351v1 Announce Type: cross
Abstract: Parameters of interest in causal inference, such as treatment or policy effects, can often be expressed as linear functionals of an outcome regression function. Automatic debiased machine learning (AutoDML) is a unified framework for obtaining asymptotically normal estimators of such parameters, which requires estimation of both a regression function and a Riesz representer. Existing AutoDML neural network architectures, such as RieszNet and MADNet, use a shared intermediate covariate representation. However, it remains unclear whether this shared representation should be predictive of the Riesz representer or the outcome.
We show that a shared representation of the covariates that preserves predictive power of the outcome while discarding information about the Riesz representer is asymptotically more efficient than the baseline AutoDML estimator that uses all covariates. Motivated by these results, we propose the outcome-adapted AutoDML estimator and establish its asymptotic behavior in a sample splitting framework. We provide a neural network implementation of the estimator that learns a sparse representation of the covariates that is predictive of the outcome but not predictive of the Riesz representer. We demonstrate the efficiency gains of our estimator over existing alternatives on synthetic data and achieve state-of-the-art estimation accuracy on the semi-synthetic IHDP benchmark dataset.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Tightening Control in Neyman--Pearson Linear Classification
arXiv:2607.03590v1 Announce Type: cross
Abstract: Neyman--Pearson classification prioritizes one class by constraining its accuracy above a prespecified level, and then takes the accuracy of the other class as the utility objective. This paradigm is well suited for disease screening and diagnosis, among other applications. Statistical learning under this framework is complicated since classifier performance determines its acceptability. Furthermore, no learned classifier that is consistent for the oracle classifier can guarantee satisfaction of the control constraint in finite samples. Classical learning theory targets a control-relaxed empirical utility maximization (EUM) classifier. However, even the EUM classifier fails to achieve the desired control level on average. We conjecture that this under-control phenomenon is a manifestation of the over-optimism bias well known in standard statistical learning, and develop asymptotic theory to confirm it. Motivated by this insight, we propose refined learning procedures under two accuracy control strategies for the prioritized class: one controlling accuracy in expectation and the other with high probability. We further develop training-data-based methods to predict and infer class-specific accuracies of the resulting classifiers. Simulation studies demonstrate favorable finite-sample performance, and we illustrate the proposed methods with an application to cancer detection.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
SiamJEPA: On the Role of Siamese Student Encoders in JEPA
arXiv:2607.04044v1 Announce Type: cross
Abstract: Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
arXiv:2607.03809v1 Announce Type: cross
Abstract: Normalising flows provide a powerful variational family for approximate inference, yet individual architectures often fail to generalise across heterogeneous posterior geometries. We revisit mixture-based flow formulations and introduce \emph{AMF\mbox{-}VI\mbox{-}sEMA}, a two-stage framework featuring a \emph{stable global weighting} mechanism based on a \emph{Simplex Exponential Moving Average} (sEMA) update. In Stage~1, a heterogeneous set of experts (\textsc{RealNVP}, \textsc{MAF}, \textsc{RBIG}) are trained independently to specialise in distinct structural regimes. In Stage~2, expert parameters are frozen and global mixture weights are learned through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update on the probability simplex. This design produces a tractable, data-agnostic gating mechanism (without per-sample gating or gradient backpropagation through weights) that adaptively reallocates capacity while avoiding component collapse. We evaluate the framework on ten posterior benchmarks: six canonical 2D synthetic families (Banana, X-Shaped, Bimodal, Multimodal, Two-moons, Rings) and four real/low-dimensional Bayesian targets (BLR, BPR, Weibull, Real-GMM2), with stronger baselines (\textsc{NICE}, \textsc{ResFlow}, and EM-Mixing). Comprehensive evaluation covers NLL, KL divergence, Wasserstein-2 distance, and MMD, together with diagnostics of mixture dynamics, hyperparameter sensitivity, and cross-seed robustness. Empirically, \emph{AMF\mbox{-}VI\mbox{-}sEMA} achieves consistent NLL improvements over its predecessor \emph{AMF\mbox{-}VI} and avoids the catastrophic transport failures of single-flow baselines, while maintaining stable weight trajectories ($N_{\mathrm{eff}}{>}1.4$ on all datasets) with minimal computational overhead.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Significance-First Splitting: Aligning Treatment Heterogeneity Detection with Honest Estimation
arXiv:2607.03999v1 Announce Type: cross
Abstract: Estimating heterogeneous treatment effects (CATE) requires simultaneously detecting effect modification and quantifying estimation uncertainty. Existing tree-based methods make an uneasy trade-off: significance-based approaches (Radcliffe and Surry 2011) identify subgroup interactions directly but lack valid inference; honest causal trees (Athey and Imbens 2016) deliver nominal confidence interval coverage but use outcome-agnostic splitting criteria that sacrifice interaction sensitivity. We introduce a hybrid algorithm that fuses significance-based splitting with honest sample-splitting and cross-validation. Our splitting criterion uses the squared $t$-statistic for the treatment $\times$ side interaction ($t^2$), which is shown to be directly aligned with the honest $\text{EMSE}_\tau$ criterion when the interaction is strong. Post-hoc honest cross-validation selects the cost-complexity penalty, giving a single principled estimator with nominal CI coverage at the leaf level. For forests, we retain bootstrap count vectors to enable an infinitesimal jackknife (IJ) variance estimate of Monte-Carlo convergence rather than formal pointwise inference. On the three synthetic designs from (Athey and Imbens 2016) the single tree achieves approximately 90\% leaf-average CI coverage at the 90\% nominal level across all three designs (200 replications each); on the Criteo and Starbucks uplift datasets we match Qini coefficient performance of S- and T-learner baselines. An open-source Python package with reproducible seeds, sklearn-compatible API, and full test coverage accompanies this work (https://codeberg.org/hadjipantelis/rattus).
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Rethinking Neural Nonlinearity as Gating
arXiv:2607.03148v1 Announce Type: new
Abstract: Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.
Fonte: arXiv cs.LG
RL • Score 85
Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
arXiv:2607.03125v1 Announce Type: new
Abstract: Deep reinforcement learning (DRL) offers powerful control for industrial cyber-physical systems (ICPSs), but its "black-box" exploration risks violating strict hardware safety limits. Typically, these constraints are managed through complex reward shaping. In this work-in-progress paper, we embed a differentiable physics model directly into the proximal policy optimization (PPO) actor loss function. By simulating short-horizon future trajectories during training, the policy is penalized for anticipated safety violations independent of the task-reward signal. Evaluated on a simulated 1-degree-of-freedom helicopter testbed with strict pitch constraints, our physics-informed soft regularizations substantially reduce constraint violations while maintaining reliable target tracking.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Heterogeneous Graph Condensation via Role-Aware Clustering
arXiv:2607.03097v1 Announce Type: new
Abstract: Heterogeneous Graph Neural Networks (HGNNs) have exhibited remarkable efficacy in modeling complex systems with multiple types of nodes and relations, yet their training on large-scale heterogeneous graphs remains computationally prohibitive. Although graph condensation methods can effectively improve learning efficiency on large-scale graphs, existing condensation processes are mainly designed for homogeneous graphs and typically rely on computationally expensive gradient matching or bilevel optimization paradigms, rendering them impractical for heterogeneous settings. To address these limitations, we propose HGC-RC, a simple yet effective role-aware heterogeneous graph condensation framework. Specifically, HGC-RC first extracts semantically enhanced node embeddings via lightweight propagation. It then introduces a role-aware hybrid clustering strategy consisting of class-partitioned clustering for labeled target nodes to preserve class distributions and unsupervised type-wise clustering for non-target nodes to retain critical cross-type connectivity. Finally, a compact heterogeneous graph is efficiently reconstructed based on the resulting cluster assignments. Extensive experiments demonstrate that HGC-RC outperforms state-of-the-art baselines, offering a practical pathway to accelerate HGNN training on large-scale heterogeneous graphs without sacrificing task performance
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
arXiv:2607.02964v1 Announce Type: new
Abstract: A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
arXiv:2607.03660v1 Announce Type: new
Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
What is Left for Us? Second Scholarship Against the Degradation of Research by AI
arXiv:2607.04049v1 Announce Type: new
Abstract: We argue that generative AI can degrade research by eroding the very practices through which scholarly judgement is formed and academic trust is built. As constitutive conditions for the production and validation of knowledge, these practices cannot be reduced to the final outputs of research, which is what AI so effectively simulate. Accordingly, when researchers delegate central tasks of inquiry to systems like Large Language Models, they may stop enacting these practices and, with them, lose access to the formation they provide. An individual research output generated by AI may even appear improved but the researcher behind it fails to develop. Against this risk, merely keeping humans in the loop as prompters or quality checkers of AI outputs is insufficient to preserve research as a site of intellectual formation. What is needed instead is a renewed commitment to research as a lived practice in which judgement is formed gradually, often through frictions, and participation in a scholarly community. We defend it because it rests on four sources and warrants of research that cannot be automated: tacit knowledge, personal commitment, socialisation, and deep reading. This practice enacts what we call second scholarship, by which we understand the reappropriation of scholarly craft, chosen out of a critical experience of what generative AI can and cannot do. What cannot and should not be delegated becomes what research communities must value and answer for. This is what is left for us.
Fonte: arXiv cs.AI
NLP/LLMs • Score 85
Labeled-Data-Free Meta-Learning: Efficient Task Generation Using Pre-trained Models and Unlabeled Data
arXiv:2607.02850v1 Announce Type: new
Abstract: Meta-learning without labeled data is crucial for real-world applications, where obtaining labeled datasets can be expensive or restricted due to privacy concerns. Data-Free Meta-Learning (DFML) addresses this challenge by leveraging pre-trained models without access to training data. However, existing DFML methods rely on model inversion to generate training data, a process that is generally difficult and computationally expensive due to the need to generate high-dimensional data matching the original distribution. To address this limitation, we propose a novel meta-learning setting that avoids model inversion by jointly leveraging pre-trained models and unlabeled data. Our method generates meta-training tasks by assigning soft labels from pre-trained models to unlabeled data. Since the quality of these tasks can vary, we introduce a task-weighting mechanism based on task confidence and class distribution balance to ensure effective meta-learning. Extensive experiments demonstrate that our approach substantially reduces computational cost and improves generalization, achieving up to 104-fold speedup and 8.4 percent to 36.4 percent improvements in few-shot classification accuracy compared to state-of-the-art DFML methods.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
arXiv:2607.04293v1 Announce Type: cross
Abstract: Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally relies on uncovering causal relationships from observations, the capability of causal thinking, i.e., distinguishing causation from correlation and recognizing hidden biases, is essential to LLM agents. Although a number of benchmarks exist for AI Scientists, none explicitly incorporate challenges from selection bias, measurement error, and hidden confounders that widely exist in real-world scientific discovery. To this end, we present CausalGame, a benchmark that evaluates the causal thinking capabilities of LLM agents through interactive games. CausalGame asks LLM agents to actively design experimental protocols, collect observation data, and derive a final solution with an explanation report. To emulate realistic scientific discovery challenges, we design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders. Across 30 LLM agents, none demonstrates reliable causal thinking: the best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics. CausalGame provides a scalable and controlled testbed for evaluating the causal thinking of AI Scientist agents.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
CuBAS: Information Geometric Curvature-Based Adaptive Sampling for Supervised Classification
arXiv:2607.03145v1 Announce Type: new
Abstract: The informativeness of a training set is as consequential as its size, yet most sampling strategies remain agnostic to the intrinsic geometry of the data distribution. We introduce CuBAS (Curvature-Based Adaptive Sampling), an information-geometric framework for adaptive data selection in supervised classification, grounded in the q-state Potts Markov random field (MRF) model. The central insight is that a labeled dataset can be viewed as a statistical manifold, on which local curvature, estimated via the ratio of second to first-order observed Fisher information, faithfully encodes the geometric complexity of the data distribution. We construct a k-nearest-neighbor graph over the labeled data and derive a closed-form curvature score at each vertex from the Potts sufficient statistics. This curvature signal partitions the graph into two complementary regimes: low-curvature regions, corresponding to smooth, homogeneous clusters, and high-curvature regions, concentrated around decision boundaries that are disproportionately informative for classification. By selecting nodes from both regimes, CuBAS constructs compact yet maximally informative training subsets. Empirical evaluation across more than 60 benchmark datasets demonstrates consistent and statistically significant improvements over random sampling and uncertainty-based baselines, across a wide range of labeling budgets and classifier architectures. CuBAS is computationally efficient (linear in the number of k-NN graph edges), theoretically grounded in the differential geometry of statistical manifolds, and interpretable in terms of the local shape operator of the data manifold.
Fonte: arXiv cs.LG
Theory/Optimization • Score 85
ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum
arXiv:2607.04535v1 Announce Type: cross
Abstract: Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonormal basis still need direction-dependent attenuation or amplification. We introduce ManifoldFlow, a minimal relaxation of a fixed-spectrum Stiefel layer that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through W = Q S^{1/2}, with Q^T Q = I and S positive definite. Since W^T W = S, the eigenvalues of S are exactly the squared singular values of the realized weight, making eigenvalue clipping a direct singular-value control mechanism. Across paired sequence, tabular, and image experiments, the learnable SPD spectrum improves the fixed-spectrum Stiefel counterpart in the reported settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases in convolutional classifier heads clarify the intended scope: ManifoldFlow is not a universal dense-layer replacement, but a spectrum-learnable Stiefel relaxation for settings where an orthonormal basis is a useful prior. When the basis should be orthonormal, its spectrum need not be frozen. Code available at https://github.com/Hik289/manifold_flow
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Open Problem: Is Interaction Necessary for Order-Optimal 1-bit Mean Estimation?
arXiv:2607.02896v1 Announce Type: cross
Abstract: We ask whether interaction is necessary for order-optimal 1-bit mean estimation over nonparametric finite-moment classes. Adaptive threshold-query protocols achieve the order-optimal 1-bit minimax rate, and the same rate is attainable with general 1-bit queries using only one adaptive transition (i.e., two stages of querying). In the non-adaptive setting, threshold and interval queries are known to be highly suboptimal, but the case of arbitrary non-adaptive quantizers remains unresolved. Can such quantizers match the adaptive rate, yielding an optimal one-shot protocol? Or is the known two-stage estimator stage-optimal, with a single adaptive transition being necessary and sufficient?
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Fast SDP certification of neural networks : towards large multi-class datasets
arXiv:2607.03232v1 Announce Type: cross
Abstract: We present a new quadratic model for the certification problem in adversarial robustness, which simultaneously accounts for all possible target classes. Building on this model, we propose a novel semidefinite programming (SDP) relaxation for incomplete verification. A key advantage of our approach is that it certifies robustness in a single optimization, avoiding the need for a separate resolution per class. This yields a significant computational speed-up and enables scalability to large datasets with many classes. To further improve efficiency, we also propose an effective pruning strategy of active neurons, thus reducing the problem dimensionality and accelerating convergence.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Reflected Schr\"odinger Bridge Matching
arXiv:2607.03626v1 Announce Type: cross
Abstract: Recent advances in generative modeling have enabled the efficient computation of Schr\"odinger bridges (SB) in high-dimensional settings by leveraging partially simulation-free training methods inspired by flow matching. However, these have not covered SBs with reflecting dynamics, a useful model choice with built-in guarantees that generated samples stay in the data domain. Existing alternatives for reflected SBs instead rely on more complex training based on forward--backward SDE theory, requiring expensive higher-order derivatives and sampling entire paths during training. In this article, we introduce a partially simulation-free framework that allows reflected SBs to be trained similarly to flow matching, using a new sampling method and regression target. We demonstrate our results by coupling pairs of well-known high-dimensional image datasets. Using reflected dynamics incurs negligible additional wall-clock time during both training and inference while maintaining or slightly improving generative performance.
Fonte: arXiv stat.ML
MLOps/Systems • Score 85
Efficient Decentralized Multi-task Dataset Valuation via Model Merging
arXiv:2607.03346v1 Announce Type: new
Abstract: Accurate and efficient dataset valuation is essential for enabling fair and transparent data marketplaces, especially when multiple contributors provide data for training multi-task models. Most existing valuation methods, however, are limited to single-task settings, overlooking scenarios where a buyer aims to optimize performance across multiple downstream tasks. Moreover, traditional valuation approaches, such as Shapley-based or retraining-based methods, are computationally expensive and poorly suited for decentralized environments without a trusted central coordinator and with strict privacy constraints. We propose DMVM (Decentralized Multi-task Valuation via Model Merging), a novel framework that bypasses retraining and data sharing by leveraging task arithmetic to infer dataset contributions directly from model combinations. Instead of retraining or sharing raw data, DMVM quantifies how models trained on different datasets combine in parameter space to infer each dataset's marginal utility across multiple tasks. This formulation yields a valuation process that is scalable, computationally efficient, and explicitly aligned with multi-task generalization behavior. To support decentralized deployment, we introduce a secure aggregation protocol that enables collaborative valuation without revealing individual model parameters or private data. We also provide theoretical error bounds characterizing the approximation quality of DMVM and validate our framework through comprehensive experiments on computer vision and natural language processing tasks.
Fonte: arXiv cs.CL
Theory/Optimization • Score 85
Deep Learning for Dynamic Programming with Recursive Utility
arXiv:2607.04278v1 Announce Type: cross
Abstract: We propose the first deep learning algorithm, the Certainty Equivalent Learning (CEL) algorithm, for solving high-dimensional discrete-time dynamic programming problems with recursive utility. Dynamic programming with recursive utility is numerically challenging because the recursive utility does not have an explicit representation and the Bellman equation contains a certainty equivalent that is difficult to evaluate. The CEL algorithm learns this certainty-equivalent value directly with neural networks and jointly approximates value functions, policy functions, and certainty-equivalent functions. The CEL algorithm is mesh-free and simulation-based, allowing high-dimensional state and control spaces, and does not rely on Euler equations, first-order conditions, or differentiability of the state transition function. The CEL algorithm also works for dynamic programming problems with expected utility as expected utility is a special case of recursive utility. We apply the CEL to discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein-Zin DSGE, and multivariate strategic asset allocation problems. Compared with closed-form and VFI-based benchmarks, the CEL delivers accurate value and policy approximations, remains effective in high-dimensional problems, achieves accuracy comparable to VFI in the small-noise robust-control case, and produces out-of-sample Bellman errors and Euler or first-order residuals that are in the range from 1.0e-4 to 1.0e-3 for most problems.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Targeted Highly Adaptive Lasso Minimum Loss Estimation of Target Functions
arXiv:2607.03824v1 Announce Type: cross
Abstract: We propose a Targeted Highly Adaptive Lasso for estimation of non-pathwise differentiable functional parameters such as the dose-response curve (DRC) for continuous exposure. We assume the target function lies in the $k$-th order smoothness class used to define the $k$-th order Highly Adaptive Lasso (HAL), which can be well approximated by linear spans of $k$-th order spline basis functions. We construct a projection of the true target function onto a large finite dimensional working model spanned by an initial set of $k$-th order spline basis functions, which defines a pathwise differentiable approximation of the target functional parameter. A standard TMLE is then applied with a data-adaptive initial fit, replacing the MLE targeting step with a LASSO step over HAL spline basis functions that span the target function. We prove that the resulting Targeted HAL-MLE is pointwise asymptotically normally distributed and achieves a convergence rate determined solely by the dimension and smoothness of the target function, giving dimension free rates up till $\log n$-factors. Through a simulation study for the DRC, we show that the Targeted HAL outperforms a HAL plug-in estimator in terms of bias and mean squared error. Targeted HAL offers a fully data-adaptive approach to inference on functional parameters without requiring sieve specification or parametric assumptions.
Fonte: arXiv stat.ML
NLP/LLMs • Score 85
Where do LLMs Fall Short in CBT-Guided Affective Reasoning?
arXiv:2607.02885v1 Announce Type: new
Abstract: Cognitive Behavioral Therapy (CBT) provides a structured framework for understanding a user's mental state by examining the interaction between cognitive and behavioral factors. However, out-of-the-box LLMs respond fluently and empathetically, yet collapse into validation & reflection, regardless of what the user actually needs. They know theoretical CBT (scoring up to 96% accuracy on licensing exam questions) but fail to apply it effectively. We explore this gap with a knowledge-guided framework that treats CBT dialogue as controlled affective reasoning: user narratives are decomposed into Beck's Cognitive Conceptualization structure, grounded in clinical SNOMED CT concepts validated via Natural Language Inference, and a Multiple Chain-of-Thought (MCoT) strategy selection between Validation & Reflection, Socratic Questioning, or Alternative Perspectives. To measure whether such guidance actually changes behavior, we introduce the Protocol Leverage Force (F), a behavior-level metric that captures how far an intervention shifts a model away from its default response. Across three open-weight LLMs and 14 RealCBT-derived case studies, evaluated with human experts, valence-arousal trajectories, and linguistic entrainment, F shows that simply introducing protocol definitions via single chain-of-thought prompting fails to change LLM behavior, while MCoT on these definitions guides strategy selection better. Still, the effect stays within 1% (approx. 1.2-1.3%), and all models remain biased toward Validation & Reflection. These results show CBT knowledge alone does not ensure effective application, giving the affective-computing community instrumentation to measure where LLMs fall short.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Lacuna Inc. at SemEval-2026 Task 4: Structurally Gated State-Space Models for Disentangling Narrative Similarity
arXiv:2607.03482v1 Announce Type: new
Abstract: In this paper, we present the Invariant-Variant Disentangled State-Space Model (IVD-SSM), our submission to SemEval-2026 Task 4 on Narrative Story Similarity and Narrative Representation Learning. Evaluating narrative similarity is a profound computational challenge that requires models to look past concrete, superficial elements such as specific names, actors, objects, or settings to isolate and compare abstract patterns of causality and plot progression. To model these extended causal chains without the quadratic bottlenecks of standard Transformers, we leverage a hybrid State-Space Model (Jamba-1.5-Mini). Building upon this backbone, we introduce the Structurally Gated Alignment (SGA) head, a novel, differentiable algorithmic architecture. The SGA head operates on two scales: a heavily strided Macro-path maps the coarse structural skeleton of a story, which then acts as a gating mechanism to filter a full-resolution Micro-path, actively suppressing semantic noise and superficial keyword overlaps. Evaluated on both pairwise comparative judgments (Track A) and dense representation learning (Track B), our approach demonstrates that explicitly disentangling structural invariants from lexical variants provides a robust, principled framework for deep narrative understanding.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding
arXiv:2607.03236v1 Announce Type: new
Abstract: Diffusion language models (DLLMs) generate text by iteratively denoising masked positions, exposing a trajectory of predictive distributions rather than a single instantaneous belief. Most existing decoders ignore this trajectory and commit tokens from the current snapshot alone, conflating confidence with commitment readiness: a transient top-1 peak under incomplete context can be locked in, while candidates with consistent cross-step support are delayed. We propose Trajectory-Aware Commit Gating (TACG), a training-free gate-level decoder that anchors token identities to the base posterior and uses trajectory-aware signals only to decide whether the current proposal is ready to commit. TACG combines Temporal Implicit Logits Guidance (TILG), which keeps an exponential moving average of past logits as a self-reference and contrasts the current logits against this reference in natural-parameter space, with a History Gate (HG) that enforces short-term proposal persistence before commitment. Together with a capped extra-promotion budget, these components yield a stability-constrained commit rule without auxiliary networks or extra forward passes. We evaluate TACG on LLaDA, Dream, and LLaDA2-Mini across code (HumanEval, MBPP) and math (GSM8K, MATH500) benchmarks; it typically improves or preserves accuracy while reducing denoising steps and increasing tokens per forward (TPF). The code is publicly available at https://github.com/Clarence-CV/TACG-DLLM.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It
arXiv:2607.03598v1 Announce Type: new
Abstract: When a person shares something with a language model, the model often answers the surface of the message rather than what the sender was doing by sending it: share a finished project and it critiques the code; share a raw late-night line and it runs a wellness check. We treat the sender's communicative intent, the Gricean what-was-meant, as a first-class interpretability object, and show the failure is one of readout on top of a robust representation. A linear probe decodes the sender's intent, whether they want a thing recognized or evaluated, from a model's default-pass hidden states, cleanly and surface-independently, across six models and four families and in the base checkpoints. The representation generalizes further, to intent that is only pragmatically inferred, and to a second, lexically clean intent (support versus help). The behavioral half of the story, and every causal test, is established on the recognize/evaluate contrast, where what varies is whether the default output acts on the intent. The readout lags the representation in depth within a model (the intent is decodable several layers before it drives the output); across models, which ones act on it by default is model-specific, an observed stratification (three of six show the failure) that we do not read as a scaling law. Where the gap is open, a direction closely tied to the representation, the discriminative direction at a searched-for layer, is a causal handle: steering it recovers the intended behavior, as well as an explicit instruction does and with no prompt at all. This direction is near-orthogonal to the feedback-offering axis, so it routes a represented intent rather than a generic feedback knob, though at the recovery dose the routed intent can override an explicit request. We support each link with controls against obvious deflations and report the nulls as plainly as the confirmations.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO
arXiv:2607.03957v1 Announce Type: new
Abstract: Language models can reach the right normative verdict for the wrong reason. We introduce NormWorlds-CF, a solver-verified environment for counterfactual normative reasoning in executable rule worlds. Its deterministic solver produces final answers, proof and falsification certificates, argument statuses, support sets, and paired-world change labels, enabling supervision and evaluation without LLM judges. The benchmark contains staged SFT diagnostics and a compact paired-world task with 270 root families and 1080 canonical-to-variant pairs. The SFT diagnostics show that final-answer supervision is an unsafe proxy: answer-only SFT reaches perfect accuracy on answer tasks but scores zero on falsification, while proof-plus-falsification training with targeted replay reaches strong all-task accuracy. For the structured-change task, we introduce metamorphic-relation GRPO (MR-GRPO), a class-conditioned reward for GRPO that gives partial credit for relation families and solver-visible change fields. In matched 1.7B continuation experiments, MR-GRPO improves held-out relation accuracy and relation-family correctness, and reduces wrong-family error, compared to sparse and answer-only GRPO. In Qwen3-4B three-seed validation, answer-only reward improves answer-change fields but weakens relation-family structure, sparse reward preserves coarse relation labels best, and MR-GRPO delivers the strongest balanced performance across answer-change, support-change, status-change, and soft root-level metamorphic-relation metrics. These results show that verified counterfactual structure can shape post-training beyond final answers, while exact full change-record generation, invariant subtype recognition, and out-of-distribution (OOD) transfer remain open problems.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Rethinking Scientific Discovery in an Agentic Era
arXiv:2607.03863v1 Announce Type: new
Abstract: Artificial intelligence has advanced scientific discovery, but most AI4Science systems remain fragmented tools that rely on humans to coordinate problem formulation, literature grounding, model use, simulation, validation, and knowledge reuse. This paper presents \textbf{SCION (Scientific Collaborative Innovation with Agentic Organizational Nexus)}, an agentic scientific operating system that acts as an \textbf{organizational nexus}. Through a Science Agent serving as a \textbf{Meta-Harness}, SCION connects scientific tasks, tools, agents, artifacts, and memory, transforming research into an executable, auditable, and reusable operational process. At its core is the \textbf{Research Execution Plan (REP)}, which compiles high-level scientific intent into staged objectives, dependencies, verification checkpoints, tool requirements, expected artifacts, and fallback conditions. SCION further integrates hierarchical multi-agent execution, profile-driven specialization, selective context construction, governed delegation, and layered epistemic memory to support long-horizon scientific work. We formulate discovery under SCION as \textbf{Target-conditioned Inverse Search} and extend it to hidden-target settings through batch active search under finite experimental budgets. Applications in materials analysis, molecule design, and protein or antibody screening, together with experiments on scientific reading, idea generation, molecule generation, and antibody screening, show that SCION outperforms existing autonomous research-agent baselines, especially in decomposition, verification, refinement, and memory reuse. Overall, SCION shifts AI from isolated tools toward a coordinated operational layer for traceable and reusable scientific innovation.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens
arXiv:2607.03502v1 Announce Type: new
Abstract: Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT). This is a limit case for behavioral oversight, where surface tokens carry no information about the underlying reasoning. But hidden from the output is not the same as hidden from us. On four task families (fact retrieval, parallel numeric composition, string manipulation, and in-context computation), two open-weights frontier models (DeepSeek V3, Kimi K2) compute over filler tokens in a structured, legible way: attention routes the question through the filler region to the answer, logit-lens readouts show retrieved facts emerging early and their composition crystallizing in late layers, and KV-cache transplants at filler positions causally swap outputs between examples. We introduce an unsupervised decoding pipeline that takes only hidden states as input and recovers intermediate values with 80-95% accuracy (best LLM judge) across both models and all four tasks, without ground-truth labels or training. Hidden computation that defeats behavioral CoT monitoring is, on these tasks, directly readable from the residual stream, suggesting monitorability is a property of the model's full computational trace, not just its surface tokens.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
GRASP: Graph-Reasoning Aided Survey Planning for High-Fidelity Related Work Generation
arXiv:2607.03709v1 Announce Type: new
Abstract: Writing a literature review requires a deep understanding of the relationships among cited papers: how they build on, challenge, or offer alternative perspectives to one another. We present Graph-Reasoning Aided Survey Planning (GRASP), a framework combining LLM planning for related work generation with graph algorithms to extract key relationships among cited papers. Our two-layer graph structure consists of a Graph of Thoughts and an Argument-Counterargument Planning Network, representing the cited papers at different levels of granularity, and we apply topology-aware pruning via a Steiner tree to identify the core inter-paper relationships captured in our graph. Our citation analysis-based evaluation shows that GRASP generates related work sections (RWS) that closely match human-written targets in terms of the discourse roles, intents, and grouping of citations.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Additive Causal Construction for Transferable and Reconfigurable Cross-System Learning in Multi-Source Image Fusion
arXiv:2607.02572v1 Announce Type: new
Abstract: In multi-source image fusion scenarios, heterogeneous inputs are typically driven by distinct generative mechanisms and can be viewed as a composition of multiple causal systems. However, cross-system discrepancy (CSD) and cross-system entanglement (CSE) commonly arise during the fusion process, often leading to significant performance degradation under out-of-distribution (OOD) predictions. To address the CSD and CSE issues, we propose the additive causal construction (ACC) framework, which characterizes information fusion at two levels: firstly, it establishes causal "anchors" shared among multiple systems through intervention consistency to enable causal graph transferability (CGT); and secondly, it formalizes the fusion process as causal construction and models the reliability of constructed paths through uncertainty quantification to ensure causal graph reconfigurability (CGR). Building upon this, we revisit the traditional causal representation learning (CRL) with ACC and propose ACC-CRL as a learnable instantiation of the framework. The method explores joint causal content representations across systems via content-mechanism decoupling, and performs response alignment under shared anchors to mitigate CSD. Furthermore, it incorporates structural uncertainty to adaptively regulate the fusion process, thereby suppressing unstable CSE. We conduct systematic experiments on synthetic data (ColorMNIST) and real-world multi-center medical imaging tasks (microvascular invasion (MVI) prediction). The results demonstrate that the proposed method significantly improves OOD generalization while maintaining in-distribution (ID) performance, validating the effectiveness and robustness of the ACC-CRL strategy based on mechanism alignment and uncertainty modeling in open environments.
Fonte: arXiv cs.CV
MLOps/Systems • Score 85
SelfMem: Self-Optimizing Memory for AI Agents
arXiv:2607.03726v1 Announce Type: new
Abstract: While current AI agents support increasingly long context windows, tool use, and skill execution for long-horizon tasks, they still require memory systems to effectively leverage historical experience. Existing memory frameworks typically rely on fixed storage, retrieval, and summarization mechanisms, which can be rigid across different tasks and often require manual tuning. To address this limitation, we propose SelfMem, a self-optimizing memory framework. Inspired by prior work on self-improving AI, we follow the principle of "teaching an agent to fish rather than giving it a fish." Instead of forcing the model to follow a predefined memory strategy or format, SelfMem provides an environment with memory tools and feedback signals that allow the agent to explore, evaluate, and refine its own memory strategy. Our results show that SelfMem consistently outperforms retrieval, compression, and agent-memory baselines on BEAM across conversation scales from 100K to 1M tokens. Compared with the strongest baseline, SelfMem improves the official score by 48.7%, 40.8%, and 41.9% at 100K, 500K, and 1M, respectively. Further question-type analysis shows broad robustness across diverse memory demands, and our optimization study shows that model-guided strategy refinement further improves performance.
Fonte: arXiv cs.CL
MLOps/Systems • Score 85
Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression
arXiv:2607.02562v1 Announce Type: new
Abstract: Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~$K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning.
Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ($64\!\times\!64$, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5$\times$ below H.264's practical floor and $5$-$7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using $5$-$7.6\times$ fewer bits. At $K{=}1024$, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.
Fonte: arXiv cs.CV
Vision • Score 85
Do Diabetic Foot Ulcer Segmentation Models Generalize? A Cross-Dataset Benchmark of CNN and Transformer Architectures
arXiv:2607.02555v1 Announce Type: new
Abstract: Deep learning models for diabetic foot ulcer (DFU) segmentation routinely report high accuracy, but they are almost always trained and tested on the same dataset, leaving their behaviour on data from a different clinical source largely unmeasured. We benchmark three representative segmentation architectures -- U-Net and DeepLabV3+ (convolutional) and SegFormer-B2 (Transformer) -- under an identical, leakage-screened protocol: training on the combined FUSeg/AZH wound data and evaluating, without fine-tuning, on two independent external datasets (DFUC2022 and Medetec). All models achieve strong in-domain performance (Dice 0.80--0.83) but degrade substantially across datasets. The degradation is, however, architecture-dependent: SegFormer-B2 generalizes best on both external sets (DFUC2022 Dice 0.557, Medetec Dice 0.786), outperforming both convolutional models, while the more complex DeepLabV3+ generalizes worse than the simpler U-Net. Per-image failure analysis on 2,160 images across both external test sets confirms that SegFormer-B2 produces the fewest catastrophic failures on DFUC2022 (31.1%), compared with U-Net (38.5%) and DeepLabV3+ (43.0%). The consistent ranking across two independent external sources, confirmed by Wilcoxon signed-rank tests (p < 0.001 on both datasets), indicates that architecture family, not model complexity, drives cross-hospital generalization.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs
arXiv:2607.03936v1 Announce Type: new
Abstract: A key challenge in Arabic NLP is the scarcity of dialectal data relative to Modern Standard Arabic (MSA), causing LLMs to overproduce MSA and struggle with dialectally accurate generation. From an interpretability perspective, this raises a fundamental question: where and how are dialectal features encoded within model internals, and can these representations be leveraged to improve dialect generation without fine-tuning? This study investigates two complementary inference-time approaches that serve simultaneously as interpretability probes and control mechanisms. First, we conduct a neuron-level analysis, identifying sparse neuron populations that encode dialect-specific features and showing that amplifying or suppressing these neurons can steer model outputs toward target dialects. Second, motivated by the entanglement of dialectal features at the single-neuron level, we apply a vector-steering approach that extracts dialect-specific activation directions and injects them during inference. Together, these methods illuminate the geometry of dialectal knowledge in Arabic LLMs and offer a principled, interpretability-grounded framework for dialect control without requiring dialect-specific fine-tuning.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Separating Representation from Reconstruction Enables Scalable Text Encoders
arXiv:2607.04011v1 Announce Type: new
Abstract: While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose $\textbf{CrossBERT}$, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all tokens via a $\textit{Complementary Masking Strategy}$, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
arXiv:2607.04061v1 Announce Type: cross
Abstract: Distinguishing Large Language Model (LLM) generated text from human writing is a critical and difficult challenge. While LLMs are trained to write like humans, we hypothesize that this training leaves an indelible mark. LLMs develop a particularly strong aversion to token repetition very early in training. This bias persists as a ''Vestigial Heuristic'' (a developmental artifact) that is activated in LLM-generated text, separating LLM from human writing. To probe this phenomenon, we introduce Telescope Perplexity, a metric that evaluates the token repetition of the model, $P(s_i | s_{1:i})$ . Our empirical investigation reveals that the Telescope Perplexity signature emerges early in pre-training, and Telescope Perplexity empirically enables highly effective zero-shot LLM detection. We show state-of-the-art or competitive performance across diverse datasets (including modern evaluation sets we introduce), reference models, and perturbation schemes with greater efficiency than other methods.
Fonte: arXiv stat.ML
Vision • Score 85
Coordinate Singularities Break Conformal Coverage for Gaze and Head Pose
arXiv:2607.02565v1 Announce Type: new
Abstract: Conformal prediction provides distribution-free reliability guarantees for vision systems, but these guarantees depend on how prediction errors are measured in the output space. Many vision tasks produce outputs on curved spaces (e.g. gaze directions on the sphere or 3D head rotations), yet intermediate prediction heads, residuals, uncertainty estimates, or conformal scores are often defined in flat coordinate charts such as yaw-pitch or Euler angles. We show that this scoring choice introduces systematic geometric distortion near coordinate singularities (large pitch angles on the sphere and poses approaching gimbal lock in 3D rotations). Across four datasets (ETH-XGaze, Gaze360, BIWI, AFLW2000-3D), slice-conditional coverage at a nominal 90% target drops by 30-50 percentage points in these regions, falling to 38.9% on ETH-XGaze and 42.0% on Gaze360 at gaze pitch above 70 degrees, and to 57.5% on BIWI and 55.2% on AFLW2000-3D at head pose pitch above 60 degrees near gimbal lock, despite marginal coverage remaining near 90%. We prove that this is structural. Scalar thresholding changes the size of chart-coordinate prediction sets but leaves their distorted axis ratios unchanged. To diagnose this hidden failure mode, we show that a simple geometric quantity, the Riemannian volume density, strongly correlates with where coverage collapse occurs. Finally, we show that coordinate-free geodesic scoring removes this distortion. It requires no retraining and adds negligible computational cost.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
Symmetry-Structured Neural Completion of Islamic Geometric Patterns from Sparse Control Geometry
arXiv:2607.02573v1 Announce Type: new
Abstract: Islamic geometric patterns are governed by exact rotational symmetry and strict construction rules. This paper treats these rules as formal geometric knowledge and embeds them in a neural completion framework, rather than leaving them to be learned statistically from data. Given sparse control geometry and a target symmetry order, the system completes the pattern as a vector graph by predicting edges and refinements of bounded curves over a candidate lattice whose edges are organised into rotational orbits under the cyclic group. Symmetry is enforced either by constraining predictions within these orbits or by projecting them onto them during inference. The orbit-tied variant provides a constructive guarantee: for any input and any orbit-level selection rule, it produces exact N-fold symmetry, preserves anchor points, and keeps all refinements within prescribed bounds. These properties are verified numerically. The study focuses on rotational symmetry, and all quantitative results are obtained from procedurally generated graphs inspired by Islamic geometric design rather than from a historical corpus. On clean inputs, enforcing exact validity produces no measurable loss in fidelity. When control geometry is missing, an unstructured decoder loses fidelity and breaks symmetry; retraining on corrupted inputs recovers much of the fidelity but not exact validity. Symmetry-structured inference, by contrast, keeps violations at zero throughout. The results show that augmentation and symmetry structure address distinct failure modes: augmentation improves fidelity under corruption, while symmetry structure guarantees validity. The framework therefore provides a knowledge-constrained, guarantee-backed approach to neural completion for scalable vector ornaments whose validity depends on exact geometric structure.
Fonte: arXiv cs.CV
Evaluation/Benchmarks • Score 85
Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits
arXiv:2607.02586v1 Announce Type: new
Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence. We argue the audits are themselves fragile: their conclusions can be silently manufactured by implementation details that readers cannot see in the reported numbers. We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models. Under a unified six-point due-diligence gate, every cell lands in a non-confirmatory bucket, and no cell reaches confirmatory. The evidence here is a single two-model, five-benchmark case study, and F1--F5 is an illustrative, deliberately non-exhaustive starting taxonomy -- not a comprehensive partition of audit failures. We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence, and not as a route to benchmark-validity verdicts.
Fonte: arXiv cs.LG
NLP/LLMs • Score 85
SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery
arXiv:2607.02807v1 Announce Type: new
Abstract: Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems. However, they tend to converge onto a single high-level approach, then proceed with low-level edits while missing other superior approaches to the problem. We hypothesize two harness-level design choices contribute to this behavior: accumulating context in a single long-running agent and only exposing a single program state to edit. We introduce SwarmResearch, an orchestrator-subagent harness in which a Shepherd Agent uses global context to steer a population of Search Agents, each operating with local context in their respective git branch. On open-ended optimization tasks, SwarmResearch discovers better or comparable solutions to state-of-the-art LLM-guided evolution and multi-agent techniques on 13/15 tasks, driven by higher-level exploration. Compared with fixed scaling of serial and parallel agents, SwarmResearch's orchestrator-guided scaling discovers better-performing solutions by adapting parallelism at different search depths.
Fonte: arXiv cs.AI
Theory/Optimization • Score 85
Robust Feasible Route Construction through Collaborative Partition Optimization
arXiv:2607.03694v1 Announce Type: new
Abstract: Large-scale Capacitated Vehicle Routing Problems (CVRPs) are commonly solved by partitioning customers into smaller routing problems that can be optimized independently. While this substantially reduces computational complexity, independently constructed routing solutions may leave some customer demand unserved even when sufficient resources exist elsewhere in the fleet. We present Collaborative Routing Constructors (CoRC), a routing framework that enables independently solved subproblems to exchange customers and vehicles during optimization rather than relying solely on a fixed partition or a subsequent global re-optimization stage. Computational experiments on AGS benchmark instances and synthetic instances containing up to 200,000 customers compare CoRC against independent routing, post-routing global re-optimization, and state-of-the-art, end-to-end routing frameworks. Across all evaluated partitioning strategies, CoRC consistently constructs feasible routing solutions where competing partition-based methods do not. Furthermore, it remains effective on problem instances for which the evaluated end-to-end routing frameworks did not produce solutions under the same computational budget. These results demonstrate that collaboration between routing subproblems provides a robust and scalable approach for feasible large-scale route construction.
Fonte: arXiv cs.AI
NLP/LLMs • Score 85
A Unified Framework for In-Context Learning with Causal and Masked Language Models
arXiv:2607.04081v1 Announce Type: cross
Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models. Masked language models instead recover masked tokens from bidirectional context, and their role in ICL remains less understood. We develop a statistical learning framework that represents the context examples by their empirical measure and models prediction as a function of the context and the query. This formulation places autoregressive and masked pretraining objectives within a common excess-risk analysis. Under Wasserstein-type regularity conditions, we relate pretraining with T tasks and N samples per task to k-shot excess risk at inference, obtaining same-order upper bounds for masked and autoregressive objectives. We also study task-distribution shift, where pretraining tasks are sampled from P and inference tasks from Q; the resulting bound contains an additional term controlled by the lifted Wasserstein distance between P and Q. The bounds further imply an order-optimal allocation under a fixed pretraining data budget and refined rates under intrinsic low-dimensional structure. Experiments on controlled function-learning tasks show that the Masked Pair Encoder (MPE) can achieve performance comparable to GPT-2-style causal Transformers, suggesting that ICL behavior is not specific to causal language models.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Applying Answer Set Programming with Fuzzy Membership Functions: a Case Study
arXiv:2607.03550v1 Announce Type: new
Abstract: Human reasoning often operates through qualitative concepts expressed by linguistic labels such as high, low, expensive, or cheap, whose interpretation depends on context and is usually vague, despite being rooted in numerical data. This paper explores a novel fuzzy-logic-based qualitative extension of Answer Set Programming (ASP) to bridge numerical information and qualitative reasoning. The underlying language, formally introduced in a separate work, provides a principled framework that avoids rigid thresholds and supports robust reasoning under vagueness. Focusing on a representative use case, we illustrate how the framework integrates numerically grounded inputs (such as outputs of machine learning models) with symbolic reasoning over qualitative labels. Key features, including learning-based membership functions and semantically enriched predicates, enable the combination of expert knowledge, contextual factors, and subjective interpretations within a unified declarative setting.
Fonte: arXiv cs.AI
NLP/LLMs • Score 85
CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?
arXiv:2607.04029v1 Announce Type: new
Abstract: Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations and training a classifier on these features to detect hallucinations, demonstrating promising results. Notwithstanding this advancement, most internal-state hallucination detection techniques have been explored predominantly in English, raising the question of whether such internal signals generalize across different languages and domains. To address this gap, we present CrossHallu, the first study to evaluate the cross-lingual and cross-domain generalization of hallucination detection using internal representations from six LLMs on the generative question-answering task. We conduct a systematic Arabic English evaluation using TruthfulQA, an Arabic translated version of TruthfulQA, and HalluScore. This evaluation encompasses monolingual training and testing, cross-lingual transfer, cross-domain transfer, and combined cross-lingual and cross-domain transfer. The results reveal that internal-state hallucination signals in LLMs transfer across languages and domains for most models, with cross-lingual performance highly dependent on both class separability and language alignment in the feature space, whereas cross-domain transfer within Arabic varies depending on the training and testing datasets used for the hallucination detector. The code is publicly available at https://github.com/aishaalansari57/CrossHal.
Fonte: arXiv cs.CL
Theory/Optimization • Score 85
Knowledge-Informed Local Causal Discovery of Optimal Adjustment Sets
arXiv:2607.04447v1 Announce Type: cross
Abstract: Local causal discovery is a scalable alternative to global structure learning. However, it can struggle to identify valid adjustment sets in data-scarce settings because of finite-sample uncertainty, incomplete local neighborhoods, and unresolved Markov equivalence. Although many application domains provide structured background knowledge, its integration into local causal discovery remains limited. We propose b-LOAD, a knowledge-informed extension of the LOAD algorithm for local discovery of optimal adjustment sets. b-LOAD incorporates prior edge constraints directly into the local structure-learning procedure and uses Meek's rules to expand the discovery frontier dynamically, yielding a knowledge-constrained partially directed graph over the relevant local subgraph. This strategy helps prevent structurally relevant nodes introduced by prior knowledge from being excluded by local search. We prove that, under sound background knowledge, the procedure monotonically refines the admissible equivalence class and can enlarge the set of identifiable causal queries, enabling recovery of optimal adjustment sets that are not identifiable from observational conditional-independence information alone. Empirically, b-LOAD improves downstream causal effect estimation relative to purely data-driven and standard knowledge-augmented baselines, particularly in data-scarce and structurally complex regimes. Results on real-world biological networks show that locally targeted prior knowledge provides the largest gains and remains beneficial under moderate structural noise. These findings position b-LOAD as a scalable approach for converting fragmented domain knowledge into more reliable causal-effect estimation.
Fonte: arXiv stat.ML
RL • Score 85
Dynamic Regret for Non-Stationary Linear Bandits via Misspecification Reductions
arXiv:2607.02891v1 Announce Type: new
Abstract: Many online decision-making problems involve both round-specific feasible actions and drifting reward models: eligible ad impressions, feasible prices, and available treatments can change over time, while user preferences, demand curves, and patient responses may evolve. Motivated by these applications, we study non-stationary linear bandits with round-specific feasible decision sets. Existing methods that obtain the optimal \(\widetilde O(T^{2/3}P_T^{1/3})\) dependence, where \(P_T\) is the path length of the reward-parameter sequence, impose an orthogonal-structure assumption on round-specific decision sets, which can be restrictive in contextual applications. We address this gap through a unified misspecification-reduction viewpoint: after partitioning the horizon into blocks, we relate each block's dynamic regret to regret against a fixed-parameter linear bandit benchmark, with the within-block parameter drift entering as bounded misspecification. Restarting algorithms with misspecification-dependent regret guarantees then yields the optimal \(T^{2/3}P_T^{1/3}\) dynamic-regret dependence for both linear bandits with general compact decision sets and \(K\)-armed contextual linear bandits.
Fonte: arXiv cs.LG
Theory/Optimization • Score 85
A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function
arXiv:2607.03664v1 Announce Type: cross
Abstract: The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent stochastic Bernoulli gate. This work gives a complementary interpretation based on the Gaussian complementary first-order loss function: GELU is the signal-transmission term of the expected surplus of a hard linear gate with a Gaussian random threshold. This view separates loss accounting from forward signal transmission and generalises to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. The uniform-threshold case recovers a hard-swish-like compact piecewise-polynomial gate with an explicit threshold-width parameter, yielding fixed- and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniform-threshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Geometric Causal Models
arXiv:2607.05153v1 Announce Type: new
Abstract: Scientists often seek to draw causal inferences from structured data that is not independently and identically distributed, such as spatial data, network data, or molecular data. We develop geometric causal models (GCMs), a framework for causal inference from dependent data that exploits underlying symmetries of the data generating process. For example, in spatial data, we consider processes that are symmetric under translations, or in graph data, symmetric under permutations of the nodes. We show how symmetries, formalized with group theory, can enable causal identification and estimation. We deploy ergodic theory for amenable groups to establish identification, and combine geometric deep learning with scalable Bayesian inference for estimation. We recover i.i.d. causal models and do-calculus when the data is a sequence and the symmetry is permutation equivariance, and find novel types of causal models when we use alternate structures and symmetries. As an example, we construct a causal model that satisfies the symmetries of DNA. This GCM enables new estimators for the effects of genetic variation, combining deep functional genomics models to describe outcomes and DNA language models to describe propensities. We illustrate on semisynthetic data.
Fonte: arXiv stat.ML
Theory/Optimization • Score 85
Minimax Estimation of Kernel Stein Discrepancy: Trace versus Hilbert-Schmidt Scales
arXiv:2607.03367v1 Announce Type: cross
Abstract: Kernel Stein Discrepancy (KSD) compares a sample to a fixed target distribution known only through its score, and is widely used for goodness-of-fit testing, sample quality assessment, and approximate inference. We study the estimation of $\operatorname{KSD}(P_0,P)$ from $n$ independent observations and identify the sharp spectral constant governing the minimax risk: it is the Hilbert-Schmidt norm of the Stein covariance operator $C_\star$, giving the minimax scale $\sqrt{\|C_\star\|_{\mathrm{HS}}/n}$. This scale is attained by the positive-part square-root U-statistic, whereas the standard plug-in V-statistic remains at the trace scale $\sqrt{\operatorname{tr}(C_\star)/n}$ and is therefore suboptimal by the fourth root of the effective rank of $C_\star$; for a Gaussian target with a fixed-bandwidth Gaussian kernel this factor is exponential in the dimension.
Fonte: arXiv stat.ML
NLP/LLMs • Score 92
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
arXiv:2607.02980v1 Announce Type: new
Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than $64\times$ the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.
Fonte: arXiv cs.CL
NLP/LLMs • Score 85
CV-DCLR: Causal-Visual Dynamic Label Refinement for Robust Zero-Shot Learning
arXiv:2607.02601v1 Announce Type: new
Abstract: Zero-Shot Learning (ZSL) facilitates knowledge transfer via shared semantic spaces. However, a critical bottleneck in this paradigm is Semantic Entanglement, where visual representations are inevitably conflated with visually similar semantic concepts, such as distinguishing the intrinsic traits of a Wolf from the shared features of a Husky. Existing global alignment methods often indiscriminately maximize correlations between visual and semantic modalities, leading models to overfit spurious similarities rather than capturing distinctive class identities. To address this fundamental limitation, we propose the Causal-Visual Dynamic Label Refinement (CV-DCLR) framework. Unlike traditional approaches that rely on superficial visual statistics, CV-DCLR recalibrates visual-semantic associations via a Dual-Stream Mutual Correction Mechanism. This includes a Visual Likelihood Stream to model observational patterns and a Causal Importance Stream that verifies the structural necessity of candidate prototypes through Counterfactual Intervention. Acting as a logical filter, our adaptive gating mechanism dynamically modulates feature responses to amplify genuine causal traits while suppressing visually plausible but structurally irrelevant distractors. Extensive experiments on the CUB, SUN, and AWA2 benchmarks under a rigorous Semantic Entanglement Injection protocol demonstrate that CV-DCLR significantly outperforms state-of-the-art methods in high-ambiguity scenarios. Specifically, while existing models suffer catastrophic degradation under entanglement, our framework maintains robust performance, effectively disentangling true class identities from semantic confounders.
Fonte: arXiv cs.CV
NLP/LLMs • Score 85
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
arXiv:2607.02584v1 Announce Type: new
Abstract: In \textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized \textbf{FlashAttention} offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: \textbf{1)} applying online rotation matrices -- a widely used technique for mitigating outliers in Queries ($Q$) and Keys ($K$) -- is difficult to reconcile with \textbf{RoPE}; and \textbf{2)} the non-negative attention matrix $P = \exp(QK - \max(QK))$ makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of $Q$ and $K$ are strongly affected by the dimensional partitioning of \textbf{3D RoPE}. Based on this finding, we propose \textbf{RotateAttention}, an efficient \textbf{mixed-precision INT4 FlashAttention} framework tailored for \textbf{DiT-based video generation models with 3D RoPE}, using selective \textbf{FP16 fallback} for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: \textbf{1) RoPE-aware Rotation}, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in $Q$ and $K$; and \textbf{2) Range-optimized $P$ Quantization}, which uses fixed scales and zero-points to fully exploit the \textbf{INT4 numerical range} with minimal computational overhead. Experiments show that \textbf{RotateAttention} preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$\times$ end-to-end speedup and 2.2$\times$ kernel-level acceleration.
Fonte: arXiv cs.CV