Skip to content

Commit a49dafc

Browse files
committed
docs: add example for running gpt-oss
1 parent acaf41d commit a49dafc

File tree

1 file changed

+31
-1
lines changed

1 file changed

+31
-1
lines changed

README.md

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ llm.embeddings(prompt).foreach: embeddings =>
8686
llm.close()
8787
```
8888

89-
#### Self-contained [Scala CLI](https://scala-cli.virtuslab.org) example:
89+
#### Self-contained [Scala CLI](https://scala-cli.virtuslab.org) example (with basic [Llama 3](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) model):
9090

9191
`Run.scala`:
9292
```scala
@@ -113,3 +113,33 @@ object Main extends App:
113113
```sh
114114
scala-cli Run.scala
115115
```
116+
117+
#### Self-contained [Scala CLI](https://scala-cli.virtuslab.org) example (with configured [gpt-oss](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) model):
118+
119+
`Run.scala`:
120+
```scala
121+
//> using scala 3.3.0
122+
//> using jvm adoptium:17
123+
//> using java-opt --add-modules=jdk.incubator.foreign
124+
//> using java-opt --enable-native-access=ALL-UNNAMED
125+
//> using dep com.donderom::llm4s:0.13.0-b6109
126+
127+
import com.donderom.llm4s.{ContextParams, Llm, LlmParams}
128+
import java.nio.file.Paths
129+
import scala.util.Using
130+
131+
object Main extends App:
132+
System.load("./build/bin/libllama.dylib")
133+
val model = Paths.get("gpt-oss-20b-mxfp4.gguf")
134+
val prompt = "What is LLM?"
135+
// Use Flash attention and context size provided by the model
136+
val params = LlmParams(context = ContextParams(size = 0, flashAttention = true))
137+
Using(Llm(model)): llm => // llm : com.donderom.llm4s.Llm
138+
llm(prompt, params).foreach: stream => // stream : LazyList[String]
139+
stream.foreach: token => // token : String
140+
print(token)
141+
```
142+
143+
```sh
144+
scala-cli Run.scala
145+
```

0 commit comments

Comments
 (0)