RepoExec Leaderboard

RepoExec: Evaluate Code Generation with a Repository-Level Executable Benchmark

blog leaderboard data
github paper

📝 Notes

  1. Evaluated using RepoExec
  2. Models are ranked according to Pass@1 using greedy decoding.
  3. ✨ marks models evaluated using a chat setting, while others perform direct code completion. We note that some instruction-tuned models miss the chat template in their tokenizer configuration.
  4. Model providers have the responsibility to avoid data contamination. Models trained on close data can be affected by contamination.
  5. 💚 means open weights and open data. 💙 means open weights and open SFT data, but the base model is not data-open. What does this imply? 💚💙 models open-source the data such that one can concretely reason about contamination.
  6. "Size" here is the amount of activated model weight during inference.

🤗 More Leaderboards

In addition to RepoExec leaderboards, it is recommended to comprehensively understand LLM coding ability through a diverse set of benchmarks and leaderboards, such as:
  1. Bigcode-Bench Leaderboard
  2. EvalPlus Leaderboard
  3. Big Code Models Leaderboard
  4. Chatbot Arena Leaderboard
  5. CrossCodeEval
  6. ClassEval
  7. CRUXEval
  8. Code Lingua
  9. Evo-Eval
  10. HumanEval.jl - Julia version HumanEval with EvalPlus test cases
  11. InfiCoder-Eval
  12. LiveCodeBench
  13. NaturalCodeBench
  14. RepoBench
  15. SWE-bench
  16. TabbyML Leaderboard
  17. OOP

🙏 Acknowledgements

  • We thank the EvalPlus and BigCode teams for providing the leaderboard template.