GoEx and Berkeley Function Calling Leaderboard Updates

[ShishirPatil]

😍 v0.3 release 🚀

Highlights

⚡️ Released GoEx: A runtime that presents abstractions for safe execution of LLM generated code, APIs, actions, etc

🏆 Updates to Berkeley Function Calling Leaderboard (aka Berkeley Tool Calling Leaderboard) : Newer models including GPT-4o, gemini-flash and 1.5-pro, Hermes-2-Pro, etc. All measured along P95 and P99 latency, and costs besides accuracy.

What's Changed

• Fix Typos in Evaluation Script and System Prompt. Identify Errors in a Dataset by @zuxin666 in Fix Typos in Evaluation Script and System Prompt. Identify Errors in a Dataset #335
• BFCL April 8th Release by @HuanzhiMao in BFCL April 8th Release #330
• Initial goex commit by @ShishirPatil in Initial goex commit #336
• BFCL April 9th Release (Dataset Bug Fix) by @HuanzhiMao in BFCL April 9th Release (Dataset Bug Fix) #338
• BFCL April 10th Release (API Sanity Check) by @HuanzhiMao in BFCL April 10th Release (API Sanity Check) #339
• Add Support for NousResearch/Hermes-2-Pro-Mistral-7B Function Calling by @Fanjia-Yan in Add Support for NousResearch/Hermes-2-Pro-Mistral-7B Function Calling #327
• Update raft.py with default p to match paper by @ShishirPatil in Update raft.py with default p to match paper #353
• GoEx Import Issues by @royh02 in GoEx Import Issues #354
• BFCL April 11th Patch. Add Latency Statistics. by @HuanzhiMao in BFCL April 11th Patch. Add Latency Statistics. #347
• GoEx Gitignore User Credentials by @royh02 in GoEx Gitignore User Credentials #344
• Fix Circular Import Issue for BFCL evluation pipeline by @HuanzhiMao in Fix Circular Import Issue for BFCL evluation pipeline #356
• Added Docker to README by @Noppapon in Added Docker to README #355
• [Bug fix] Add Hermes-2-Pro-Mistral-7B model to UNDERSCORE_TO_DOT to parse API properly by @JasonZhu1313 in [Bug fix] Add Hermes-2-Pro-Mistral-7B model to UNDERSCORE_TO_DOT to parse API properly  #364
• Update requirements.txt by @viniciuslazzari in Update requirements.txt #343
• Fix script argument by @ricklamers in Fix script argument #367
• BFCL April 16th Release by @HuanzhiMao in BFCL April 16th Release #366
• Log error messages from API validation by @eitanturok in Log error messages from API validation #369
• Update .gitignore by @eitanturok in Update .gitignore #370
• BFCL April 18th Release (Pipeline only) by @HuanzhiMao in BFCL April 18th Release (Pipeline only) #375
• Add missing argument to OSSHandler's _format_prompt function by @eitanturok in Add missing argument to OSSHandler's _format_prompt function #373
• Add FC + Prompt for Cohere command-r-plus by @harry-cohere in Add FC + Prompt for Cohere command-r-plus #350
• BFCL April 19th Release (Dataset & Pipeline) by @HuanzhiMao in BFCL April 19th Release (Dataset & Pipeline) #377
• Azure OpenAI support in raft.py by @cedricvidal in Azure OpenAI support in raft.py #381
• BFCL April 25th Release (New Models) by @HuanzhiMao in BFCL April 25th Release (New Models) #386
• Colored logging configuration + displaying progress in logs by @cedricvidal in Colored logging configuration + displaying progress in logs #384
• BFCL April 27th Release (Bug Fix in Cost/Latency Calculation) by @HuanzhiMao in BFCL April 27th Release (Bug Fix in Cost/Latency Calculation) #390
• BFCL April 28th Release (New Model: snowflake/arctic) by @Fanjia-Yan in BFCL April 28th Release (New Model: snowflake/arctic) #397
• RAFT Recovery Mode for interruptions by @kaiwen129 in RAFT Recovery Mode for interruptions  #410
• Small corrections to possible_answers for simple test category by @aastroza in Small corrections to possible_answers for simple test category #405
• BFCL May 6th Release (Dataset Bug Fix) by @HuanzhiMao in BFCL May 6th Release (Dataset Bug Fix) #412
• RAFT DevContainer for GitHub Codespaces by @cedricvidal in RAFT DevContainer for GitHub Codespaces #379
• RAFT Add support for configuring separate completion and embedding endpoints + pytest by @cedricvidal in RAFT Add support for configuring separate completion and embedding endpoints + pytest #396
• RAFT Fix arbitrary code execution vulnerability in checkpoint feature by @cedricvidal in RAFT Fix arbitrary code execution vulnerability in checkpoint feature #415
• handle parallel function calls from gemini by @vandyxiaowei in handle parallel function calls from gemini #406
• RAFT Support for chat and completion model formats by @cedricvidal in RAFT Support for chat and completion model formats #417
• [RAFT] Edit encode prompt to include <ANSWER>: tag in label by @kaiwen129 in [RAFT] Edit encode prompt to include <ANSWER>: tag in label #422
• [BFCL] Patch Gemini Handler by @HuanzhiMao in [BFCL] Patch Gemini Handler  #421
• BFCL May 14th Release (GPT-4o and Gemini) by @Fanjia-Yan in BFCL May 14th Release (GPT-4o and Gemini) #426
• [BFCL] update tree_sitter version in requirements.txt by @justinwangx in [BFCL] update tree_sitter version in requirements.txt #433
• Fix indentation in leaderboard README by @polm-stability in Fix indentation in leaderboard README #449
• Fix breaking changes due to updated Anthropic SDK by @eitanturok in Fix breaking changes due to updated Anthropic SDK #452

New Contributors

• @zuxin666 made their first contribution in Fix Typos in Evaluation Script and System Prompt. Identify Errors in a Dataset #335
• @JasonZhu1313 made their first contribution in [Bug fix] Add Hermes-2-Pro-Mistral-7B model to UNDERSCORE_TO_DOT to parse API properly  #364
• @ricklamers made their first contribution in Fix script argument #367
• @eitanturok made their first contribution in Log error messages from API validation #369
• @harry-cohere made their first contribution in Add FC + Prompt for Cohere command-r-plus #350
• @cedricvidal made their first contribution in Azure OpenAI support in raft.py #381
• @aastroza made their first contribution in Small corrections to possible_answers for simple test category #405
• @vandyxiaowei made their first contribution in handle parallel function calls from gemini #406
• @justinwangx made their first contribution in [BFCL] update tree_sitter version in requirements.txt #433
• @polm-stability made their first contribution in Fix indentation in leaderboard README #449

Full Changelog: v0.2...v0.3

---

This discussion was created from the release GoEx and Berkeley Function Calling Leaderboard Updates.