From 3bfb1be960ade04c0cb7233056397a861f5d1c86 Mon Sep 17 00:00:00 2001 From: Russell Bryant Date: Mon, 11 Mar 2024 19:44:57 -0400 Subject: [PATCH] skill: Add reasoning skill about siblings This skill was inspired by one of the tests included in mt-bench. It included some questions that the model answered incorrectly. When given the same question manually, I got the following incorrect answer: > Q: David has three sisters. Each of them has one brother. How many > brothers does David have? > A: David has three sisters, and each of them has one brother, so it > must be the same brother in all cases. Therefore, David has 1 brother. Signed-off-by: Russell Bryant Signed-off-by: BJ Hargrave --- .../logic/induction/siblings/attribution.txt | 1 + .../logic/induction/siblings/qna.yaml | 29 +++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 compositional_skills/philosophy/logic/induction/siblings/attribution.txt create mode 100644 compositional_skills/philosophy/logic/induction/siblings/qna.yaml diff --git a/compositional_skills/philosophy/logic/induction/siblings/attribution.txt b/compositional_skills/philosophy/logic/induction/siblings/attribution.txt new file mode 100644 index 000000000..1e9ffcd07 --- /dev/null +++ b/compositional_skills/philosophy/logic/induction/siblings/attribution.txt @@ -0,0 +1 @@ +Self-authored diff --git a/compositional_skills/philosophy/logic/induction/siblings/qna.yaml b/compositional_skills/philosophy/logic/induction/siblings/qna.yaml new file mode 100644 index 000000000..2ee792fbc --- /dev/null +++ b/compositional_skills/philosophy/logic/induction/siblings/qna.yaml @@ -0,0 +1,29 @@ +created_by: russellb +version: 3 +task_description: Inductive reasoning about sibling relationships +seed_examples: + - question: | + David has three sisters. Each of them has one brother. How many brothers does David have? + answer: | + David has three sisters, and each of them has one brother. + David is their one brother. So, David has no brothers. + - question: | + David has three sisters. Each of them has two brothers. How many brothers does David have? + answer: | + David has three sisters, and each of them has two brothers. + David is one of their brothers. So, David has one brother. + - question: | + Sally has three brothers. Each of them has one sister. How many sisters does Sally have? + answer: | + Sally has three brothers, and each of them has one sister. + Sally is their one sister. So, Sally has no sisters. + - question: | + Sally has three brothers. Each of them has two sisters. How many sisters does Sally have? + answer: | + Sally has three brothers, and each of them has two sisters. + Sally is their one sister. So, Sally has one sister. + - question: | + Sally has one brother and one sister. How many sisters does Sally's brother have? + answer: | + Sally has one brother and one sister. There are two sisters and one brother. + Sally's brother has two sisters.