Welcome to the LaMP benchmark leaderboard! To assess your results on the test set or submit your model for inclusion on the leaderboard, please fill this form.


Name ↕Accuracy ↕Accuracy ↕F1 ↕MAE ↕RMSE ↕Rouge-1 ↕Rouge-L ↕Rouge-1 ↕Rouge-L ↕Rouge-1 ↕Rouge-L ↕Rouge-1 ↕Rouge-L ↕Accuracy ↕Accuracy ↕F1 ↕MAE ↕RMSE ↕Rouge-1 ↕Rouge-L ↕Rouge-1 ↕Rouge-L ↕Rouge-1 ↕Rouge-L ↕Rouge-1 ↕Rouge-L
FlanT5-base-finetuned + Contriever 0.734 FlanT5-base-finetuned + Contriever 0.556 0.519 FlanT5-base-finetuned + Contriever 0.245 0.56 FlanT5-base-finetuned + Contriever 0.186 0.171 FlanT5-base-finetuned + BM25 0.45 0.409 FlanT5-base-finetuned + BM25 0.587 0.575 FlanT5-base-finetuned + Contriever 0.528 0.475 FlanT5-base-finetuned + Contriever 0.714 FlanT5-base-finetuned + Contriever 0.564 0.519 FlanT5-base-finetuned + Recency 0.266 0.598 FlanT5-base-finetuned + Recency 0.177 0.162 FlanT5-base-finetuned + Contriever 0.479 0.431 FlanT5-base-finetuned + Contriever 0.547 0.533 FlanT5-base-finetuned + Contriever 0.516 0.465
FlanT5-XXL-zeroshot + Contriever 0.699 FlanT5-XXL-zeroshot + Contriever 0.414 0.364 FlanT5-XXL-zeroshot + Contriever 0.267 0.552 FlanT5-XXL-zeroshot + Contriever 0.182 0.167 FlanT5-XXL-zeroshot + BM25 0.45 0.411 FlanT5-XXL-zeroshot + BM25 0.482 0.471 FlanT5-XXL-zeroshot + Contriever 0.448 0.394 FlanT5-XXL-zeroshot + Contriever 0.636 FlanT5-XXL-zeroshot + Contriever 0.396 0.304 FlanT5-XXL-zeroshot + Recency 0.299 0.616 FlanT5-XXL-zeroshot + Recency 0.188 0.172 FlanT5-XXL-zeroshot + Contriever 0.483 0.433 FlanT5-XXL-zeroshot + Contriever 0.401 0.387 FlanT5-XXL-zeroshot + Contriever 0.44 0.389
GPT-3.5-zeroshot + Contriever 0.695 GPT-3.5-zeroshot + Contriever 0.508 0.457 GPT-3.5-zeroshot + Contriever 0.62 1.049 GPT-3.5-zeroshot + Contriever 0.15 0.133 GPT-3.5-zeroshot + BM25 0.39 0.329 0 0 GPT-3.5-zeroshot + Contriever 0.39 0.322 GPT-3.5-zeroshot + Contriever 0.634 GPT-3.5-zeroshot + Contriever 0.466 0.418 GPT-3.5-zeroshot + Recency 0.603 1.002 GPT-3.5-zeroshot + Recency 0.158 0.14 GPT-3.5-zeroshot + Contriever 0.425 0.351 0 0 GPT-3.5-zeroshot + Contriever 0.382 0.318
GPT-3.5-zeroshot 0.541 GPT-3.5-zeroshot 0.408 0.314 GPT-3.5-zeroshot 0.706 0.972 GPT-3.5-zeroshot 0.136 0.119 GPT-3.5-zeroshot 0.387 0.329 0 0 GPT-3.5-zeroshot 0.399 0.336 GPT-3.5-zeroshot 0.508 GPT-3.5-zeroshot 0.382 0.299 GPT-3.5-zeroshot 0.677 0.948 GPT-3.5-zeroshot 0.146 0.128 GPT-3.5-zeroshot 0.424 0.355 0 0 GPT-3.5-zeroshot 0.39 0.33
FlanT5-XXL-zeroshot 0.52 FlanT5-XXL-zeroshot 0.365 0.308 FlanT5-XXL-zeroshot 0.344 0.65 FlanT5-XXL-zeroshot 0.163 0.147 FlanT5-XXL-zeroshot 0.442 0.4 FlanT5-XXL-zeroshot 0.362 0.343 FlanT5-XXL-zeroshot 0.453 0.395 FlanT5-XXL-zeroshot 0.502 FlanT5-XXL-zeroshot 0.36 0.276 FlanT5-XXL-zeroshot 0.333 0.65 FlanT5-XXL-zeroshot 0.176 0.16 FlanT5-XXL-zeroshot 0.471 0.422 FlanT5-XXL-zeroshot 0.335 0.319 FlanT5-XXL-zeroshot 0.448 0.396
Learning To Prompt 0.782 0 0 Learning To Prompt 0.193 0.466 Learning To Prompt 0.221 0.202 Learning To Prompt 0.472 0.432 0 0 0 0 0 0 0 10000 10000 0 0 0 0 0 0 0 0
0 0 0 10000 10000 0 0 0 0 0 0 0 0 ROPG-RSPG (FlanT5-XXL zero-shot) 0.672 ROPG-RSPG (FlanT5-XXL zero-shot) 0.43 0.339 ROPG-RSPG (FlanT5-XXL zero-shot) 0.264 0.568 ROPG-RSPG (FlanT5-XXL zero-shot) 0.203 0.186 ROPG-RSPG (FlanT5-XXL zero-shot) 0.483 0.431 ROPG-RSPG (FlanT5-XXL zero-shot) 0.433 0.418 ROPG-RSPG (FlanT5-XXL zero-shot) 0.461 0.409