Tests using WatsonX (#12)

* default params for liteLLM, tests use watsonx Signed-off-by: Mandana Vaziri <[email protected]> * Bug fix in result recording (#6) Signed-off-by: Mandana Vaziri <[email protected]> * Add pyproject.toml (#8) Signed-off-by: Mandana Vaziri <[email protected]> * Use pyproject.toml instead of setup.py (#9) Signed-off-by: Mandana Vaziri <[email protected]> * Update mkdocs configuration (#10) Signed-off-by: Mandana Vaziri <[email protected]> * cleanup Signed-off-by: Mandana Vaziri <[email protected]> * cleanup Signed-off-by: Mandana Vaziri <[email protected]> * cleanup Signed-off-by: Mandana Vaziri <[email protected]> * cleanup Signed-off-by: Mandana Vaziri <[email protected]> * removed travis.yml Signed-off-by: Mandana Vaziri <[email protected]> * removing bam dependence from line number tests Signed-off-by: Mandana Vaziri <[email protected]> --------- Signed-off-by: Mandana Vaziri <[email protected]> Co-authored-by: Louis Mandel <[email protected]>
IBM · Aug 30, 2024 · e6aa4e3 · e6aa4e3
1 parent 6f12785
commit e6aa4e3
Show file tree

Hide file tree

Showing 29 changed files with 1,002 additions and 785 deletions.
diff --git a/.travis.yml b/.travis.yml
diff --git a/README.md b/README.md
@@ -21,8 +21,8 @@ The PDL interpreter (`pdl/pdl.py`) takes a PDL program as input and renders it i
 
 See below for installation notes, followed by an [overview](#overview) of the language. A more detailed description of the language features can be found in this [tutorial](https://ibm.github.io/prompt-declaration-language/tutorial).
 
-## Demo Video
-
+## Demo Video 
+ 
 https://github.com/user-attachments/assets/2629bf1e-bc54-4c45-b598-47914ab05a45
 
 
@@ -46,7 +46,7 @@ In order to run the examples that use foundation models hosted on IBM's [watsonx
 - `WATSONX_APIKEY`, the API key (see information on [key creation](https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui#create_user_key))
 - `WATSONX_PROJECT_ID`, the project hosting the resources (see information about [project creation](https://www.ibm.com/docs/en/watsonx/saas?topic=projects-creating-project) and [finding project ID](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-project-id.html?context=wx)).
 
-
+ 
 To run the interpreter:
 
 ```

diff --git a/examples/arith/Arith-litellm.pdl b/examples/arith/Arith-litellm.pdl
@@ -0,0 +1,68 @@
+description: Arithmetic Expressions
+document:
+- |
+  Question: Noah charges $60 for a large painting and $30 for a small painting.
+  Last month he sold eight large paintings and four small paintings.
+  If he sold twice as much this month, how much is his sales for this month?
+
+  Answer: Let's think step by step.
+  He sold 8 large paintings and 4 small paintings last month.
+  He sold twice as many this month.
+  8 large paintings x $60 = << 8*60= 480 >> 480
+  4 small paintings x $30 = << 4*30= 120 >> 120
+  So he sold << 480+120= 600 >> 600 paintings last month.
+  Therefore he sold << 600*2= 1200 >> this month.
+  The answer is $1200.
+
+  Question: Noah charges $30 for a large vases and $10 for a small vases.
+  Last month he sold five large vases and three small vases.
+  If he sold three times as much this month, how much is his sales for this month?
+
+  Answer: Let's think step by step.
+  He sold 5 large vases and 3 small vases last month.
+  He sold three times as many this month.
+  5 large vases x $30 = << 5*30= 150 >> 150
+  3 small vases x $10 = << 3*10= 30 >> 30
+  So he sold << 150+30= 180 >> 180 vases last month.
+  Therefore he sold << 180*3= 540 >> this month.
+  The answer is $540.
+
+- repeat:
+    document:
+    - "\nQuestion: "
+    - def: QUESTION
+      model: watsonx/ibm/granite-20b-code-instruct
+      parameters:
+        stop_sequences:
+        - Answer
+        include_stop_sequence: false
+    - "Answer: Let's think step by step.\n"
+    - repeat:
+        document:
+        - def: REASON_OR_CALC
+          model: watsonx/ibm/granite-20b-code-instruct
+          parameters:
+            stop:
+            - '<<'
+            include_stop_sequence: true
+        - if: '{{ REASON_OR_CALC.endswith("<<") }}'
+          then:
+            document:
+            - def: EXPR
+              model: watsonx/ibm/granite-20b-code-instruct
+              parameters:
+                stop:
+                - '='
+                - "\n"
+                include_stop_sequence: false
+            - '= '
+            - def: RESULT
+              lan: python
+              code: result = {{ EXPR }}
+            - ' >>'
+          else: ""
+      until: '{{ "The answer is" in REASON_OR_CALC }}'
+      as: document
+    - "\n\n"
+  num_iterations: 3
+  as: document
diff --git a/examples/arith/Arith.pdl b/examples/arith/Arith.pdl
@@ -32,7 +32,7 @@ document:
     - "\nQuestion: "
     - def: QUESTION
       model: ibm/granite-20b-code-instruct
-      parameters:
+      params:
         stop_sequences:
         - Answer
         include_stop_sequence: false
@@ -41,7 +41,7 @@ document:
         document:
         - def: REASON_OR_CALC
           model: ibm/granite-20b-code-instruct
-          parameters:
+          params:
             stop_sequences:
             - '<<'
             include_stop_sequence: true
@@ -50,7 +50,7 @@ document:
             document:
             - def: EXPR
               model: ibm/granite-20b-code-instruct
-              parameters:
+              params:
                 stop_sequences:
                 - '='
                 - "\n"

diff --git a/examples/arith/litellm.txt b/examples/arith/litellm.txt
@@ -0,0 +1,76 @@
+Question: Noah charges $60 for a large painting and $30 for a small painting.
+Last month he sold eight large paintings and four small paintings.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 8 large paintings and 4 small paintings last month.
+He sold twice as many this month.
+8 large paintings x $60 = << 8*60= 480 >> 480
+4 small paintings x $30 = << 4*30= 120 >> 120
+So he sold << 480+120= 600 >> 600 paintings last month.
+Therefore he sold << 600*2= 1200 >> this month.
+The answer is $1200.
+
+Question: Noah charges $30 for a large vases and $10 for a small vases.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as many this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10= 30 >> 30
+So he sold << 150+30= 180 >> 180 vases last month.
+Therefore he sold << 180*3= 540 >> this month.
+The answer is $540.
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold 50 small paintings last month.
+Therefore he sold 50 x 2 = << 50*2 = 100 >> 100 small paintings this month.
+The answer is 100
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+ Answer: Let's think step by step.
+ He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10 = 50 >>  50
+So he sold 50 small paintings last month.
+Therefore he sold 50 x 2 = << 50*2 = 100 >>  100 small paintings this month.
+The answer is 100
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+  Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10  = 50 >>    50
+So he sold 50 small paintings last month.
+Therefore he sold 50 x 2 = << 50*2   = 100 >>    100 small paintings this month.
+The answer is 100 
+
+Question:  
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+    Answer: Let's think step by step.
+    He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10     = 50 >>
diff --git a/examples/arith/watsonx.txt b/examples/arith/watsonx.txt
@@ -0,0 +1,71 @@
+Question: Noah charges $60 for a large painting and $30 for a small painting.
+Last month he sold eight large paintings and four small paintings.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 8 large paintings and 4 small paintings last month.
+He sold twice as many this month.
+8 large paintings x $60 = << 8*60= 480 >> 480
+4 small paintings x $30 = << 4*30= 120 >> 120
+So he sold << 480+120= 600 >> 600 paintings last month.
+Therefore he sold << 600*2= 1200 >> this month.
+The answer is $1200.
+
+Question: Noah charges $30 for a large vases and $10 for a small vases.
+Last month he sold five large vases and three small vases.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 large vases and 3 small vases last month.
+He sold three times as many this month.
+5 large vases x $30 = << 5*30= 150 >> 150
+3 small vases x $10 = << 3*10= 30 >> 30
+So he sold << 150+30= 180 >> 180 vases last month.
+Therefore he sold << 180*3= 540 >> this month.
+The answer is $540.
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold twice as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold twice as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold << 50*2= 100 >> this month.
+The answer is $100.
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold three times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold three times as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold << 50*3= 150 >> this month.
+The answer is $150.
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold five times as much this month, how much is his sales for this month?
+
+Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold five times as many this month.
+5 small paintings x $10 = << 5*10= 50 >>
+
+
+Question: 
+Noah charges $10 for a small painting.
+He sold five small paintings last month.
+If he sold ten times as much this month, how much is his sales for this month?Answer: Let's think step by step.
+He sold 5 small paintings last month.
+He sold ten times as many this month.
+5 small paintings x $10 = << 5*10= 50 >> 50
+So he sold << 50*10= 500 >> this month.
+The answer is $500.
diff --git a/examples/cldk/cldk-assistant.pdl b/examples/cldk/cldk-assistant.pdl
@@ -34,7 +34,7 @@ document:
   - "\n***Generating PDL code for your query:\n"
   - if: "{{ query != 'quit'}}"
     then: 
-    - model: ibm/granite-34b-code-instruct
+    - model: watsonx/ibm/granite-34b-code-instruct
       def: PDL
       show_result: true
       input: 
@@ -116,7 +116,7 @@ document:
               method = PDL_SESSION.cldk_state.get_method("org.ibm.App", "Foo(string)")
               result = method
           - "\n\nGenerate a summary of method Foo\n\n"
-          - model: meta-llama/llama-3-70b-instruct
+          - model: watsonx/meta-llama/llama-3-70b-instruct
           ```
 
           Question: Generate a different comment for method Foo(string) in class org.ibm.App?
@@ -129,7 +129,7 @@ document:
               method = PDL_SESSION.cldk_state.get_method("org.ibm.App", "Foo(string)")
               result = method
           - "\n\nGenerate a different comment for method Foo(string)\n\n"
-          - model: meta-llama/llama-3-70b-instruct
+          - model: watsonx/meta-llama/llama-3-70b-instruct
           ```
 
           If the query contains something about a field be sure to call a model.
@@ -138,7 +138,7 @@ document:
 
 
       parameters:
-        stop_sequences: ["Question"]
+        stop_sequence: ["Question"]
         include_stop_sequence: false
     - "\n\n***Executing the above PDL code:\n\n"
     - lan: python

diff --git a/examples/hello/hello.pdl b/examples/hello/hello.pdl
@@ -1,5 +1,9 @@
 description: Hello world through LiteLLM on watsonx
 document:
 - Hello,
-- model: watsonx/ibm/granite-3b-code-instruct
-
+- model: watsonx/ibm/granite-20b-code-instruct
+  parameters:
+    stop:
+    - '!'
+    include_stop_sequence: true
+- "\n"