File tree Expand file tree Collapse file tree 1 file changed +31
-1
lines changed Expand file tree Collapse file tree 1 file changed +31
-1
lines changed Original file line number Diff line number Diff line change @@ -86,7 +86,7 @@ llm.embeddings(prompt).foreach: embeddings =>
86
86
llm.close()
87
87
```
88
88
89
- #### Self-contained [ Scala CLI] ( https://scala-cli.virtuslab.org ) example:
89
+ #### Self-contained [ Scala CLI] ( https://scala-cli.virtuslab.org ) example (with basic [ Llama 3 ] ( https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF ) model) :
90
90
91
91
` Run.scala ` :
92
92
``` scala
@@ -113,3 +113,33 @@ object Main extends App:
113
113
``` sh
114
114
scala-cli Run.scala
115
115
```
116
+
117
+ #### Self-contained [ Scala CLI] ( https://scala-cli.virtuslab.org ) example (with configured [ gpt-oss] ( https://huggingface.co/ggml-org/gpt-oss-20b-GGUF ) model):
118
+
119
+ ` Run.scala ` :
120
+ ``` scala
121
+ //> using scala 3.3.0
122
+ //> using jvm adoptium:17
123
+ //> using java-opt --add-modules=jdk.incubator.foreign
124
+ //> using java-opt --enable-native-access=ALL-UNNAMED
125
+ //> using dep com.donderom::llm4s:0.13.0-b6109
126
+
127
+ import com .donderom .llm4s .{ContextParams , Llm , LlmParams }
128
+ import java .nio .file .Paths
129
+ import scala .util .Using
130
+
131
+ object Main extends App :
132
+ System .load(" ./build/bin/libllama.dylib" )
133
+ val model = Paths .get(" gpt-oss-20b-mxfp4.gguf" )
134
+ val prompt = " What is LLM?"
135
+ // Use Flash attention and context size provided by the model
136
+ val params = LlmParams (context = ContextParams (size = 0 , flashAttention = true ))
137
+ Using (Llm (model)): llm => // llm : com.donderom.llm4s.Llm
138
+ llm(prompt, params).foreach: stream => // stream : LazyList[String]
139
+ stream.foreach: token => // token : String
140
+ print(token)
141
+ ```
142
+
143
+ ``` sh
144
+ scala-cli Run.scala
145
+ ```
You can’t perform that action at this time.
0 commit comments