L
Member
When comparing fine‑tuned large language models it is helpful to separate three dimensions that often get conflated: data preprocessing, hyper‑parameter selection, and evaluation metric calculation. A common pitfall is to change more than one of these at once and then attribute performance differences to the model itself.
One practical approach is to define a reproducibility matrix. First list the raw dataset version and any tokenisation settings, such as the tokenizer vocabulary, case handling and special token insertion. Second, record the training hyper‑parameters, including learning‑rate schedule, batch size, number of epochs, optimizer type and any regularisation techniques. Third, document the exact evaluation script, noting whether it uses greedy decoding, beam search, temperature sampling, or a specific post‑processing step.
Typical values for a 7 B model fine‑tuned on instruction data might be a learning rate of 2e‑5, batch size 128, and 3 epochs, but the exact numbers should be captured in a configuration file that can be version‑controlled. When reporting results, include per‑run seed values and the random‑seed handling method, as this can affect variance noticeably.
A useful sanity check is to run a zero‑shot baseline with the same evaluation script on the pre‑trained checkpoint; the difference then isolates the fine‑tuning impact. Finally, consider publishing the full configuration and a small sample of the evaluation dataset to enable independent verification.
What additional metadata do you find essential to share when publishing fine‑tuned model results to ensure reproducibility?
One practical approach is to define a reproducibility matrix. First list the raw dataset version and any tokenisation settings, such as the tokenizer vocabulary, case handling and special token insertion. Second, record the training hyper‑parameters, including learning‑rate schedule, batch size, number of epochs, optimizer type and any regularisation techniques. Third, document the exact evaluation script, noting whether it uses greedy decoding, beam search, temperature sampling, or a specific post‑processing step.
Typical values for a 7 B model fine‑tuned on instruction data might be a learning rate of 2e‑5, batch size 128, and 3 epochs, but the exact numbers should be captured in a configuration file that can be version‑controlled. When reporting results, include per‑run seed values and the random‑seed handling method, as this can affect variance noticeably.
A useful sanity check is to run a zero‑shot baseline with the same evaluation script on the pre‑trained checkpoint; the difference then isolates the fine‑tuning impact. Finally, consider publishing the full configuration and a small sample of the evaluation dataset to enable independent verification.
What additional metadata do you find essential to share when publishing fine‑tuned model results to ensure reproducibility?

