Skip to content

Commit b41e2a8

Browse files
committed
fixed config error
1 parent e5e873e commit b41e2a8

File tree

1 file changed

+5
-4
lines changed

1 file changed

+5
-4
lines changed

examples/configs/recipes/llm/sft-llama3.1-8b-1n8g-fsdp2tp1-lora-megatron.yaml

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,11 @@ policy:
2121
dim: 128
2222
alpha: 128
2323
optimizer:
24-
kwargs:
25-
lr: 2.0e-05
26-
weight_decay: 0.01
27-
eps: 1.0e-08
24+
optimizer: "adam"
25+
lr: 2.0e-05
26+
min_lr: 2.0e-05
27+
weight_decay: 0.01 # When weight decay is set, it actually uses AdamW
28+
adam_eps: 1.0e-08
2829
train_global_batch_size: 128
2930
max_total_sequence_length: 4096
3031
make_sequence_length_divisible_by: 2

0 commit comments

Comments
 (0)