-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* resolved merge conflicts * create webllm.ts chat model file * add prior developed code to webllm.ts with credit * feat: add webllm entry point * update @mlc-ai/web-llm dependency * change @mlc-ai/web-llm dependency to 0.2.28 * change @mlc-ai/web-llm dependency * Fix imports and spacing * Updated to use Engine instead of ChatModule (being deprecated) * Replace chatCompletionAsyncChunkGenerator with more general chat.completions.create * Add examples and update documentation * feat: add webllm entry point * feat: add webllm entry point * create webllm.ts chat model file * add prior developed code to webllm.ts with credit * update @mlc-ai/web-llm dependency * change @mlc-ai/web-llm dependency to 0.2.28 * Fix imports and spacing * Updated to use Engine instead of ChatModule (being deprecated) * Replace chatCompletionAsyncChunkGenerator with more general chat.completions.create * Add examples and update documentation * did integration tests for mlcllm and mlcaiwebllm * Make example file for webllm.ts * Add example code for webllm.ts integration * Update community package.json and gitignore with WebLLM integration, rework integration example * chore: format * remove incorrect comment in webllm.ts * changed the import statement in test file for mlcllm * fixed test file to use the right parameters * changed function to role * Lint and format * Comment out test for now * Fix streaming * Add docs page, fixes * Update lock * Update colors * Update docstring --------- Co-authored-by: edwincha <edwincha@DESKTOP-0I6H3O4> Co-authored-by: echao-2 <[email protected]> Co-authored-by: Joshua Charat-Collins <[email protected]> Co-authored-by: Jeffrey Zhang <[email protected]> Co-authored-by: Joshua Charat-Collins <[email protected]> Co-authored-by: jacoblee93 <[email protected]>
- Loading branch information
7 people
authored
May 13, 2024
1 parent
a57d58e
commit 5c36137
Showing
10 changed files
with
364 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| --- | ||
| sidebar_class_name: web-only | ||
| --- | ||
|
|
||
| # WebLLM | ||
|
|
||
| :::tip Compatibility | ||
| Only available in web environments. | ||
| ::: | ||
|
|
||
| You can run LLMs directly in your web browser using LangChain's [WebLLM](https://webllm.mlc.ai) integration. | ||
|
|
||
| ## Setup | ||
|
|
||
| You'll need to install the [WebLLM SDK](https://www.npmjs.com/package/@mlc-ai/web-llm) module to communicate with your local model. | ||
|
|
||
| import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; | ||
|
|
||
| <IntegrationInstallTooltip></IntegrationInstallTooltip> | ||
|
|
||
| ```bash npm2yarn | ||
| npm install -S @mlc-ai/web-llm @langchain/community | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| Note that the first time a model is called, WebLLM will download the full weights for that model. This can be multiple gigabytes, and may not be possible for all end-users of your application depending on their internet connection and computer specs. | ||
| While the browser will cache future invocations of that model, we recommend using the smallest possible model you can. | ||
|
|
||
| We also recommend using a [separate web worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) when invoking and loading your models to | ||
| not block execution. | ||
|
|
||
| import CodeBlock from "@theme/CodeBlock"; | ||
| import Example from "@examples/models/chat/integration_webllm.ts"; | ||
|
|
||
| <CodeBlock language="typescript">{Example}</CodeBlock> | ||
|
|
||
| Streaming is also supported. | ||
|
|
||
| ## Example | ||
|
|
||
| For a full end-to-end example, check out [this project](https://github.com/jacoblee93/fully-local-pdf-chatbot). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| // Must be run in a web environment, e.g. a web worker | ||
|
|
||
| import { ChatWebLLM } from "@langchain/community/chat_models/webllm"; | ||
| import { HumanMessage } from "@langchain/core/messages"; | ||
|
|
||
| // Initialize the ChatWebLLM model with the model record and chat options. | ||
| // Note that if the appConfig field is set, the list of model records | ||
| // must include the selected model record for the engine. | ||
|
|
||
| // You can import a list of models available by default here: | ||
| // https://github.com/mlc-ai/web-llm/blob/main/src/config.ts | ||
| // | ||
| // Or by importing it via: | ||
| // import { prebuiltAppConfig } from "@mlc-ai/web-llm"; | ||
| const model = new ChatWebLLM({ | ||
| model: "Phi2-q4f32_1", | ||
| chatOptions: { | ||
| temperature: 0.5, | ||
| }, | ||
| }); | ||
|
|
||
| // Call the model with a message and await the response. | ||
| const response = await model.invoke([ | ||
| new HumanMessage({ content: "What is 1 + 1?" }), | ||
| ]); | ||
|
|
||
| console.log(response); | ||
|
|
||
| /* | ||
| AIMessage { | ||
| content: ' 2\n', | ||
| } | ||
| */ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
69 changes: 69 additions & 0 deletions
69
libs/langchain-community/src/chat_models/tests/webllm.int.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| // TODO: Fix for Node environments | ||
|
|
||
| // import { ChatWebLLM, WebLLMInputs } from "../webllm.js"; | ||
| // import * as webllm from "@mlc-ai/web-llm"; | ||
|
|
||
| // jest.mock("@mlc-ai/web-llm", () => ({ | ||
| // Engine: jest.fn().mockImplementation(() => ({ | ||
| // reload: jest.fn().mockResolvedValue(undefined), | ||
| // setInitProgressCallback: jest.fn(), | ||
| // chat: { | ||
| // completions: { | ||
| // create: jest.fn().mockImplementation(() => { | ||
| // const messages = [ | ||
| // { | ||
| // choices: [ | ||
| // { | ||
| // delta: { content: "Hello" }, | ||
| // logprobs: null, | ||
| // finish_reason: "complete", | ||
| // }, | ||
| // ], | ||
| // }, | ||
| // { | ||
| // choices: [ | ||
| // { | ||
| // delta: { content: "How are you?" }, | ||
| // logprobs: null, | ||
| // finish_reason: "complete", | ||
| // }, | ||
| // ], | ||
| // }, | ||
| // ]; | ||
| // return (async function* () { | ||
| // for (let msg of messages) { | ||
| // yield msg; | ||
| // } | ||
| // })(); | ||
| // }), | ||
| // }, | ||
| // }, | ||
| // })), | ||
| // })); | ||
|
|
||
| describe("ChatWebLLM Integration Tests", () => { | ||
| // let chatWebLLM: ChatWebLLM; | ||
| // let modelRecord = { model_id: "test-model" }; | ||
|
|
||
| // beforeEach(() => { | ||
| // const inputs: WebLLMInputs = { | ||
| // modelRecord: modelRecord, | ||
| // appConfig: {}, | ||
| // chatOpts: {}, | ||
| // }; | ||
| // chatWebLLM = new ChatWebLLM(inputs); | ||
| // }); | ||
|
|
||
| test("ChatWebLLM initializes and processes messages correctly", async () => { | ||
| // const options = {}; // Adjust options as necessary | ||
| // const response = await chatWebLLM.invoke("Hello", options); | ||
| // expect(response).toBe("Hello"); | ||
| // expect(webllm.Engine).toHaveBeenCalled(); | ||
| // expect(webllm.Engine().chat.completions.create).toHaveBeenCalledWith({ | ||
| // stream: true, | ||
| // messages: [{ role: "user", content: "Hello" }], | ||
| // stop: null, | ||
| // logprobs: true, | ||
| // }); | ||
| }); | ||
| }); |
Oops, something went wrong.