Does a Norwegian further-trained model give more outdated legal values than its base model? A preregistered paired comparison of Borealis 2 and Gemma 4

概要

We compare how often two released language models give outdated numeric values from Norwegian statutes and regulations: NbAiLab/borealis2-26b-a4b-preview (Borealis 2), which the National Library of Norway further trained from google/gemma-4-26B-A4B-it (Gemma 4), and Gemma 4 itself. The study was preregistered (doi:10.5281/zenodo.23858965) and uses the closed norregel benchmark, scored deterministically against a value history. On the 411 of 503 primary items where both models gave an interpretable answer, Borealis 2 alone was outdated on 10 items and Gemma 4 alone on 11 (d = -0.2 percentage points, 95% cluster bootstrap interval [-2.7, +2.5], exact one-sided McNemar p = 0.668). By the preregistered rule this is no effect of interest: the whole interval lies within the smallest effect of interest of +/-10 percentage points. Among the secondary measures, only the share of answers giving a value that took effect after the question date differs after Holm correction; Gemma 4 gave such answers more often. An exploratory comparison of two smaller Borealis models had no discordant pairs and is not informative. The design compares released models and does not isolate the effect of any part of their training.

参考リンク

• Paper ↗

リンク先のライセンス・再現性・データ条件をご確認ください。自動分類や実装の目安は保証ではありません。

実装を始めるプロンプト

参考リンクを確認したうえで、次の内容をコピーして利用できます。

# 論文・記事
Does a Norwegian further-trained model give more outdated legal values than its base model? A preregistered paired comparison of Borealis 2 and Gemma 4

# 概要
We compare how often two released language models give outdated numeric values from Norwegian statutes and regulations: NbAiLab/borealis2-26b-a4b-preview (Borealis 2), which the National Library of…

# 原典
https://arxiv.org/abs/2611.02334

上記の手法を実装してください。実行前に依存関係とライセンスを確認してください。