skill: Add reasoning skill about siblings

This skill was inspired by one of the tests included in mt-bench. It included some questions that the model answered incorrectly. When given the same question manually, I got the following incorrect answer: > Q: David has three sisters. Each of them has one brother. How many > brothers does David have? > A: David has three sisters, and each of them has one brother, so it > must be the same brother in all cases. Therefore, David has 1 brother. Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: BJ Hargrave <[email protected]>
instructlab · Aug 27, 2024 · 3bfb1be · 3bfb1be
1 parent 2631714
commit 3bfb1be
Show file tree

Hide file tree

Showing 2 changed files with 30 additions and 0 deletions.
diff --git a/compositional_skills/philosophy/logic/induction/siblings/attribution.txt b/compositional_skills/philosophy/logic/induction/siblings/attribution.txt
@@ -0,0 +1 @@
+Self-authored
diff --git a/compositional_skills/philosophy/logic/induction/siblings/qna.yaml b/compositional_skills/philosophy/logic/induction/siblings/qna.yaml
@@ -0,0 +1,29 @@
+created_by: russellb
+version: 3
+task_description: Inductive reasoning about sibling relationships
+seed_examples:
+  - question: |
+      David has three sisters. Each of them has one brother. How many brothers does David have?
+    answer: |
+      David has three sisters, and each of them has one brother.
+      David is their one brother. So, David has no brothers.
+  - question: |
+      David has three sisters. Each of them has two brothers. How many brothers does David have?
+    answer: |
+      David has three sisters, and each of them has two brothers.
+      David is one of their brothers. So, David has one brother.
+  - question: |
+      Sally has three brothers. Each of them has one sister. How many sisters does Sally have?
+    answer: |
+      Sally has three brothers, and each of them has one sister.
+      Sally is their one sister. So, Sally has no sisters.
+  - question: |
+      Sally has three brothers. Each of them has two sisters. How many sisters does Sally have?
+    answer: |
+      Sally has three brothers, and each of them has two sisters.
+      Sally is their one sister. So, Sally has one sister.
+  - question: |
+      Sally has one brother and one sister. How many sisters does Sally's brother have?
+    answer: |
+      Sally has one brother and one sister. There are two sisters and one brother.
+      Sally's brother has two sisters.