All pull requests

Pull requests

• fix(livecodebench): support sys.stdout.buffer in output capture

  no_review
  #581·mantuoluozk opened Sep 29, 2026·

  ·
  1

  1

• fix(livecodebench): judge five v6 tasks by problem rules

  no_review
  #578·mantuoluozk opened Sep 29, 2026·

  ·
  1

  2

• [feature]add MME benchmark support

  featureno_review
  #577·jschen069 opened Sep 29, 2026Collaborator·1 of 15 tasks·

  ·
  1

  6

• add agent doc

  no_issueno_review
  #575·jschen069 opened Sep 24, 2026Collaborator·15 tasks·

  ·
  1

• fix: avoid unnecessary tokenizer loading in perf summary

  no_issueno_review
  #574·jschen069 opened Sep 23, 2026Collaborator·15 tasks·

  ·
  1

• [fix][datasets] Support \boxed{X} answer extraction in GPQA_Simple_Eval_postprocess

  bugfixno_review
  #546·tunglambk opened Sep 15, 2026·4 of 15 tasks·

  ·
  1

  2

• Escape backslash in HLE judge prompt config

  no_issueno_review
  #543·MBK-fr opened Sep 14, 2026·

  ·
  1

• fix(livecodebench): expose .buffer on mocked stdin/stdout

  no_issuereview_opened
  #518·yqy003 opened Sep 6, 2026·15 tasks·

  ·
  6

• fix: skip id: lines in streaming response parsing

  bugfixno_issueno_review
  #481·navtrade opened Aug 21, 2026·1 of 9 tasks·

  ·
  1

  1

• 0819 fix itl

  no_issueno_review
  #479·jschen069 opened Aug 19, 2026Collaborator·15 tasks·

  ·
  1

• Add din_integration/: SWE-bench DinD evaluation pipeline integration

  large_prno_issueno_reviewother
  #456·Huafubing opened Aug 12, 2026Contributor·1 of 2 tasks·

  ·
  4

• feat(acc): add gpqa 5-shot-prompt

  feature
  #450·ivanbao9783 opened Aug 11, 2026Collaborator·1 of 15 tasks·

  ·

• feat: add aa_lcr unit test

  #420·jschen069 opened Jul 24, 2026Collaborator·15 tasks·

  ·
  1

• fix: add cross-platform file locking and background progress thread

  bugfix
  #414·jschen069 opened Jul 23, 2026Collaborator·1 of 15 tasks·

  ·
  9

• feat: add ifbench unit test

  test-cases
  #413·jschen069 opened Jul 23, 2026Collaborator·1 of 15 tasks·

  ·
  1

• [Feature][CI/CD] Support agent runtime env auto deploy

  feature
  #410·SJTUyh opened Jul 23, 2026Collaborator·2 of 15 tasks·

  ·
  1

  1

• 【WIP】 control Prefill con in aisbench

  #396·vasede opened Jul 14, 2026·15 tasks·

  ·
  5

• bugfix: Support for extracting answers expressed in thousand separato…

  bugfix
  #360·yejj710 opened Jun 24, 2026Collaborator·1 of 9 tasks·

  ·
  2

• test ai

  #352·wenba0 opened Jun 23, 2026Collaborator·15 tasks·

  ·
  3

• Update answer pattern for COT chat prompt

  #347·F0undLinks opened Jun 18, 2026·7 tasks·

  ·
  4

• Update cmmlu_gen_5_shot_cot_chat_prompt.py

  #346·F0undLinks opened Jun 18, 2026·7 tasks·

  ·
  4

• 增加用户自定义mcq数据0-shot&5-shot的语言模型精度评测功能，增加README

  #345·F0undLinks opened Jun 16, 2026·14 tasks·

  ·
  8

• 【Feature】Supports rerunning specified use cases on SWE dataset

  #331·yejj710 opened Jun 8, 2026Collaborator·6 tasks·

  ·
  4

• [bugfix]: fix answer extraction regex and evaluator bugs in MMLU-Pro

  bugfix
  #269·lvhua6352 opened May 2, 2026·1 of 15 tasks·

  ·
  1

  3

• [Bugfix] Add pred and choices parsing to fix the issue of score=0 for…

  bugfixno_UTreview_opened
  #258·Yanguan619 opened Apr 20, 2026·1 of 15 tasks·

  ·
  1

  13