Search results

[BUG] Accelerate backend with dp fails to load cache

bug

Status: Open.

#1102 In huggingface/lighteval;

· Aceticia opened on Dec 10, 2025

[BUG] concurrent_requests in litellm backend limited to 100

bug

Status: Open.

#1100 In huggingface/lighteval;

· yt0428 opened on Dec 8, 2025

[FT] I am unable to use microsoft/Phi-4-multimodal-instruct

Status: Open.

#1096 In huggingface/lighteval;

· ryuzakace opened on Dec 3, 2025

[BUG] Get low performance when evaluate Qwen3-8B on MMLU-Redux using the thinking mode.

bug

Status: Open.

#1094 In huggingface/lighteval;

· StarLooo opened on Dec 3, 2025

[FT] Log-likelihood metrics support for API-based evaluation (LiteLLM / completions endpoint)

Status: Open.

#1093 In huggingface/lighteval;

· DenysYurchenko24 opened on Dec 2, 2025

[BUG] Breaking change for Hellaswag/Piqa

bug

Status: Open.

#1090 In huggingface/lighteval;

· Giuseppe5 opened on Nov 28, 2025

[BUG] Cannot run math_500/mmlu_pro on models without chat template

bug

Status: Open.

#1089 In huggingface/lighteval;

· Giuseppe5 opened on Nov 28, 2025

Improving input token throughput?

Status: Open.

#1063 In huggingface/lighteval;

· carlesoctav opened on Nov 17, 2025

[EVAL] Long Horizon Execution

good first issue

help wanted

new-task

Status: Open.

#1056 In huggingface/lighteval;

· NathanHB opened on Nov 13, 2025

[BUG] git clone issues due to filenames with special characters

bug

Status: Open.

#1055 In huggingface/lighteval;

· amosyou opened on Nov 13, 2025

[FT] Prompt Level Caching

Status: Open.

#1053 In huggingface/lighteval;

· f14-bertolotti opened on Nov 12, 2025

[BUG] Got Zero em on MMLU using Deepseek distilled Qwen models

bug

Status: Open.

#1047 In huggingface/lighteval;

· Zoeyyao27 opened on Nov 11, 2025