Blog
-
LeRoPE: letting RoPE learn its own frequencies
LeRoPE learns RoPE's rotation frequencies during pretraining and improves long-context performance. It's now used in Mistral Large 4.
LeRoPE learns RoPE's rotation frequencies during pretraining and improves long-context performance. It's now used in Mistral Large 4.