feat: prompt generation strategy #6010

mingxuanzhangsfdx · 2025-01-16T00:19:45Z

What does this PR do?

Implement a bidder for prompt generation based on different strategies.

What issues does this PR fix or reference?

@W-17587267@

Functionality Before

The prompt generation is hard-coded with little flexibility.

Functionality After

Prompt generation will pick a strategy based on the effectiveness and pre-set preference.

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

peternhale · 2025-01-16T14:54:27Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

@@ -0,0 +1,128 @@
+/*


Please move this file out of the the commands folder. Maybe to src/oas, or something like that.

Also, I would like to see the two strategies exposed as distinct classes, with a proper interface defined for them.

public interface GenerationStratey { bid (..): GenerationStrategyBid; generate: (..): GenerationResult; getBid: (): GenerationStrategyBid; getPromptTokenCount(): number; }

public class FooStrategy implements GenerationStrategy { public static bid (..): GenerationStrategyBid { ... return this; } public generate: (..): GenerationResult { ... return fooResults; } public getBid: (): GenerationStrategyBid { return this.bid } getPromptTokenCount(): number { return return Math.floor(this.bid.prompt.length / 4); } }

const bidders = [ KitchenSinkStrategy.bid, VeryPickyStrategy.bid, ... FooStrategy.bid ]; ... const bids = await Promise.all(bidders.map(bidMethod => bidMethod()));

peternhale · 2025-01-16T14:59:14Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+  BAD_CLASS
+}
+
+export const PROMPT_TOKEN_MAX_LIMIT = 14 * 1024;


Let's use 95% of these two values as the limits we will impose. The 95% size is the limit being imposed by callLLM.
Don't forget the sum of prompt and response tokens must be < 16K * 95%

peternhale · 2025-01-16T14:59:59Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+const END_OF_PROMPT_TAG = '<|endofprompt|>';
+const USER_TAG = '<|user|>';
+const ASSISTANT_TAG = '<|assistant|>';
+const systemPrompt = `


Think about how we can tune these prompts without having to rebuild.

peternhale · 2025-01-16T15:00:55Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+    return input;
+  }
+
+  public getTokenCount(prompt: string): number {


Suggested change

public getTokenCount(prompt: string): number {

public getPromptTokenCount(prompt: string): number {

peternhale · 2025-01-16T15:12:11Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+      return [GenerationStrategy.KITCHEN_SINK, 1];
+    } else {
+      let count = 0;
+      for (const method of context.methods) {


I believe context.methods will include all methods in the class, this should be methods enumerated by eligibility check.

peternhale · 2025-01-16T15:13:16Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+    }
+  }
+
+  public generatePrompt(


Should be deligated to a strategy class

peternhale · 2025-01-16T15:14:49Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+    const promptKitchenSink = this.generatePrompt(GenerationStrategy.KITCHEN_SINK, metadata, context);
+    const tokenCountKitchenSink = this.getTokenCount(promptKitchenSink);
+    if (tokenCountKitchenSink <= PROMPT_TOKEN_MAX_LIMIT) {
+      return [GenerationStrategy.KITCHEN_SINK, 1];


Can we formalize the return type into its own type?

+1 would recommend creating interface GenerationStrategy and concrete classes responsible to the generation logic inside them. So we can keep the Orchestrator simple and add as many strategies as we need.

peternhale · 2025-01-16T15:21:31Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+Ensure no sensitive details are included. Decline requests unrelated to OpenAPI v3 specs or asking for sensitive information.`;
+
+export class promptGenerationOrchestrator {
+  public pickGenerationStrategy(


Let's imagine there are 50 strategies and how to handle finding the best bid.
The most direct approach would be to call each one, collect the bids and then determine which of those wins.

Are you thinking something like factory pattern or something a little more involved where bidding happens in parallel?

madhur310 · 2025-01-16T18:21:27Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+const END_OF_PROMPT_TAG = '<|endofprompt|>';
+const USER_TAG = '<|user|>';
+const ASSISTANT_TAG = '<|assistant|>';
+const systemPrompt = `


This should probably be coming from a config file of some sort?

madhur310 · 2025-01-16T18:35:31Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+    console.log('document text = ' + documentText);
+
+    const userPrompt = // to be adjusted for rest resource class only
+      'Generate an OpenAPI v3 specification for the following Apex class. The OpenAPI v3 specification should be a YAML file. The paths should be in the format of /{ClassName}/{MethodName} for all the @AuraEnabled methods specified. For every `type: object`, generate a `#/components/schemas` entry for that object. The method should have a $ref entry pointing to the generated `#/components/schemas` entry. Only include methods that have the @AuraEnabled annotation in the paths of the OpenAPI v3 specification. I do not want AUTHOR_PLACEHOLDER in the result.';


I suspect these might evolve quicker than the code itself, may be good to keep them in some config file or a constant.

madhur310 · 2025-01-16T18:36:48Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+} from '../openApiUtilities/schemas';
+
+enum GenerationStrategy {
+  KITCHEN_SINK,


nit: can we go ahead and call these: WHOLE_CLASS, METHOD_BY_METHOD and INVALID or something like that?

madhur310 · 2025-01-16T18:47:33Z

packages/salesforcedx-vscode-apex/src/commands/promptGenerationOrchestrator.ts

+Ensure no sensitive details are included. Decline requests unrelated to OpenAPI v3 specs or asking for sensitive information.`;
+
+export class promptGenerationOrchestrator {
+  public pickGenerationStrategy(


Are you thinking something like factory pattern or something a little more involved where bidding happens in parallel?

packages/salesforcedx-vscode-apex/src/oas/generationStrategy/methodByMethodStrategy.ts

peternhale · 2025-01-17T13:24:48Z

packages/salesforcedx-vscode-apex/src/oas/generationStrategy/generalPrompt.json

@@ -0,0 +1,7 @@
+{


Please combine this with whole class json into a single json file keyed by strategy name

What is the plan to expose these to the user?

This will help us maintain and tweak the prompt. We declared the prompt info in typescript in local scope only before.

I am also hesitating to introduce json files since it might introduce extra maintenance concern considering that the string format is a bit different from that in a typescript file, and json file is not that native as a javascript file during esbuild bundling. But I think it does not hurt for now before we clarify the next step.

What kinds of options are you considering to use to support various templates? Like environment variables, or VSCode configurations... Is the option of various templates only for our exploration to find the best prompt and will we keep the best prompt template as the default one if we are going to have a "template library" when it comes to TDX?

peternhale · 2025-01-17T13:29:45Z

packages/salesforcedx-vscode-apex/src/oas/generationStrategy/wholeClassStrategy.ts

+      `${generalPrompt.SYSTEM_TAG}\n${generalPrompt.systemPrompt}\n${generalPrompt.END_OF_PROMPT_TAG}\n${generalPrompt.USER_TAG}\n` +
+      wholeClassPrompt.USER_PROMPT +
+      '\nThis is the Apex class the OpenAPI v3 specification should be generated for:\n```\n' +
+      documentText +


This should account for the ability to apply additional context.

Agree and what do you think of supporting the flexibility in another story?

packages/salesforcedx-vscode-apex/src/oas/promptGenerationOrchestrator.ts

peternhale · 2025-01-17T13:36:12Z

packages/salesforcedx-vscode-apex/src/openApiUtilities/schemas.ts

@@ -97,3 +97,13 @@ export enum ApexOASResource {
 //   method: ApexMethodOASFilter;
 //   property: ApexPropertyOASFilter;
 // };
+


If we no longer need the comment code above, please remove it.

eslint.config.mjs

…ethodByMethodStrategy.ts Co-authored-by: peternhale <[email protected]>

…estrator.ts Co-authored-by: peternhale <[email protected]>

mingxuanzhangsfdx added 3 commits January 15, 2025 16:14

feat: prompt generation strategy

3be0211

chore: reduce dummy

5892f41

chore: rename class

d45d578

peternhale requested changes Jan 16, 2025

View reviewed changes

madhur310 reviewed Jan 16, 2025

View reviewed changes

mingxuanzhangsfdx added 2 commits January 16, 2025 17:57

feat: refactor generation strategy bidding

58a2fe4

chore: add test in include option in tsconfig

6acface

mingxuanzhangsfdx requested review from madhur310 and peternhale January 17, 2025 02:09

peternhale requested changes Jan 17, 2025

View reviewed changes

mingxuanzhangsfdx and others added 4 commits January 17, 2025 10:03

Update packages/salesforcedx-vscode-apex/src/oas/generationStrategy/m…

b3c8641

…ethodByMethodStrategy.ts Co-authored-by: peternhale <[email protected]>

Update packages/salesforcedx-vscode-apex/src/oas/promptGenerationOrch…

2c754ff

…estrator.ts Co-authored-by: peternhale <[email protected]>

chore: remove useless comments

cb86001

chore: apply reviewer suggestion

74f33c5

mingxuanzhangsfdx requested a review from peternhale January 17, 2025 18:31

mingxuanzhangsfdx marked this pull request as ready for review January 17, 2025 18:35

mingxuanzhangsfdx requested a review from a team as a code owner January 17, 2025 18:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: prompt generation strategy #6010

feat: prompt generation strategy #6010

mingxuanzhangsfdx commented Jan 16, 2025 •

edited

Loading

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025 •

edited

Loading

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025

peternhale Jan 16, 2025

madhur310 Jan 16, 2025

peternhale Jan 16, 2025

madhur310 Jan 16, 2025

madhur310 Jan 16, 2025

madhur310 Jan 16, 2025

madhur310 Jan 16, 2025

madhur310 Jan 16, 2025

peternhale Jan 17, 2025

peternhale Jan 17, 2025

mingxuanzhangsfdx Jan 17, 2025

mingxuanzhangsfdx Jan 17, 2025

peternhale Jan 17, 2025

mingxuanzhangsfdx Jan 17, 2025

peternhale Jan 17, 2025

	public getTokenCount(prompt: string): number {
	public getPromptTokenCount(prompt: string): number {

feat: prompt generation strategy #6010

Are you sure you want to change the base?

feat: prompt generation strategy #6010

Conversation

mingxuanzhangsfdx commented Jan 16, 2025 • edited Loading

What does this PR do?

What issues does this PR fix or reference?

Functionality Before

Functionality After

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peternhale Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mingxuanzhangsfdx commented Jan 16, 2025 •

edited

Loading

peternhale Jan 16, 2025 •

edited

Loading