Refactor KTO coordinated with DPO [a/N]: Remove encoder-decoder support

[albertvillanova]

Refactor KTO coordinated with DPO [a/N]: Remove encoder-decoder support.

This cleanup significantly simplifies the KTO trainer and makes subsequent refactoring much easier.

Part of:

• KTO refactoring #4786

Coordinated with:

• [WIP] Refactor DPO #3906

Key Changes:

1. KTOConfig

• Removed is_encoder_decoder parameter and documentation
• Removed max_completion_length parameter (because it is specific to encoder-decoder models) and documentation

2. KTOTrainer

   Initialization:

   • Added clear error message when user tries to use encoder-decoder model
   • Removed self.is_encoder_decoder attribute initialization
   • Removed self.max_completion_length attribute setup
   • Hardcoded is_encoder_decoder=False in DPODataCollatorWithPadding call

   Data Processing:

   • Simplified _process_tokens() function - removed entire encoder-decoder branch (~90 lines)
   • Kept only causal LM tokenization logic

   Model Forward Pass:

   • Simplified get_batch_logps(): removed is_encoder_decoder parameter
   • Always shift labels/logits by one position (causal LM only)
   • Updated all 4 calling sites to remove the parameter

   Reference Model Computation:

   • Simplified compute_reference_log_probs() - removed encoder-decoder branches
   • Simplified _compute_kl_logps() - removed encoder-decoder conditional
   • Simplified forward() - removed encoder-decoder model_kwargs
   • Simplified _compute_loss_liger() - removed encoder-decoder branches for hidden states

3. Error Handling:

• Users attempting to use encoder-decoder models will now receive a clear error

4. Tests

• Test updated
• Remove commented encoder-decoder tests

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Yes, as @albertvillanova says, I advocated for this change in #3906. It is important change, so I would be interested to hear the opinions of @lewtun, @edbeeching, and @kashif, and ensure we are aligned on this decision for both KTO and DPO.

[qgallouedec]

you could probably remove this

[albertvillanova]

Done here and in other places.

[edbeeching]

Yes, as @albertvillanova says, I advocated for this change in #3906. It is important change, so I would be interested to hear the opinions of @lewtun, @edbeeching, and @kashif, and ensure we are aligned on this decision for both KTO and DPO.

Yes I agree it is best to streamline the repo and cut features which (presumably) are not widely used.

[qgallouedec]

a few suggestions, otherwise lgtm!

[qgallouedec]

Suggested change

"max_completion_length": None,

[qgallouedec]

Suggested change

"max_completion_length": None,

[qgallouedec]

and remove all kwargs["max_completion_length"] in _process_tokens

EDIT: ignore the above, none remain