PLE: Physical-Engineering's Last Exam
Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work. Yet no existing benchmark tests…
nlp
MLINFO COLLECTION
LLM Fine-tuning
LoRA、RLHF、SFT、軽量ファインチューニングなど、LLMを用途に合わせる技術を扱います。
記事一覧
30 / 36IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval
Industry Foundation Classes (IFC), the standard for building information exchange in architecture, engineering, and construction (AEC), is increasingly queried through large language models (LLMs)…
nlp
Beyond End-to-End Recovery: Towards a Better Understanding of LLM Steganography via Rolewise Decomposition
In large language model (LLM) steganography, an encoder hides a private message in an otherwise ordinary response, as text transmitted to a decoder, who attempts to recover it. The success of this…
nlp
Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals
LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions.
deep-learning
Rethinking the Evaluation of LLM Fact-Checkers
Large language models (LLMs) are widely used to answer factual questions, but their answers can contain errors and must be fact-checked claim by claim. Yet the resulting score depends on three design…
nlp
From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science
AI agents now run experiments and report findings, so research produces claims faster than researchers can check them. Each empirical claim implies a property that must hold over its cases, but it…
nlp
What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation
LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers,…
nlp
From Single-Actor Environments to Multi-Actor Event Worlds: Can LLMs Be Event World Models?
World models help agents assess the consequences of their actions before acting. Many real systems, however, involve multiple actors who make their own decisions, can evolve even when the agent takes…
nlp
Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences
Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction…
deep-learning
FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization
Weight-only quantization keeps LLM activations in floating point and stores weights as low-bit integers, turning the dominant workload into floating-point-integer (FP-INT) general matrix…
deep-learning
Externalise the Memory: A Pre-Registered Study of Where an Embodied Agent's Experience Should Live
Should an embodied agent's experience be baked into its policy weights, or kept external and retrieved at inference? A memory-augmented teacher (a VLM that retrieves lessons, with their episode…
deep-learning
Same Beliefs, Different Policies: Learning Reusable Representations for Early Stopping in LLM Reasoning
Reasoning LLMs often continue reasoning after reaching a correct answer, wasting computation and sometimes replacing it with an incorrect one. Deciding when to stop is a metareasoning problem: the…
nlp
RoleFlow: Self-Evolving Multi-Role Agentic Collaboration via Structural Adaptation
Large language model (LLM) based multi-agent systems often rely on predefined collaboration structures, limiting their adaptability to evolving task requirements. Although recent approaches enable…
nlp
X-pressing Intimacy: Mental Health Disclosure After Twitter Acquisition
Social media platforms provide spaces for individuals to disclose mental health experiences and seek support, yet how these disclosures evolve during major shifts in platform governance remains…
nlp
SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots
LLM-based chat agents are increasingly used for complex, multi-turn, multilingual medical triage in low- and middle-income countries (LMICs), yet evaluation methods focus on single-turn, English-only…
nlp
A Transformer-based Approach for Real-time Multi-agent Path Planning with Varying Initial/Final Conditions
We address the problem of real-time multi-agent path planning under prescribed initial and final configurations, formulated as a high-dimensional optimal control problem subject to inter-agent…
deep-learning
Text-Only LLM Correctors Break Correct Transcripts: A Repair-Damage Analysis of Audio Grounding in ASR Error Correction
LLM correctors of ASR output are judged by aggregate WER, which hides what they fix and what they break. We decompose every WER change exactly into repaired tokens, damaged tokens and insertions, and…
nlp
Climbing the Ladder: Automating Data Pipelines and Reproducing Baselines in Low-Resource Settings
The massively multilingual translation systems that now include Sicilian learned it from noisy, non-standardised web text. To obtain better translation quality, we redevelop the small neural machine…
deep-learning
Probing and Steering for Safety-Relevant Behaviours in Large Language Models
Activation probes are computed directions in a language model's residual stream, used both to detect safety-relevant behaviours and, by adding their direction back into the stream, to steer them. We…
nlp
Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable
LLMs are increasingly consulted for moral judgment. We aim to improve performance on such tasks by rewarding good arguments, on the premise that arguing well increases the likelihood of judging well.
nlp
AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining
Training Arabic large language models faces challenges from noisy web data, dialectal variation, and the high cost of continual pretraining (CPT). We introduce AQL-Lite (AQL: Arabic Quality…
deep-learning
R4DGS: Referring Segmentation in 4D Gaussian Splatting
Dynamic representations based on 4D Gaussian Splatting (4DGS) have shown strong capabilities in reconstruction and rendering, yet current methods remain limited in supporting complex natural-language…
nlp
Measuring Research Progress and Linguistic Inclusion in Low-Resource Languages
Large Language Models (LLMs) have accelerated research on low-resource and multilingual NLP, but their effectiveness in creating an equitable landscape remains unmapped. We analyze 1,158…
nlp
Alpha: Systematic Fault Injection and Robustness Evaluation for LLM-Driven Embodied Manipulation
Large language models (LLMs) are increasingly integrated into embodied robotic planning, yet standard benchmarks primarily evaluate end-to-end task completion in nominal environments, obscuring…
deep-learning