Skip to content

server : (refactor) no more json in server_task input #10691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Dec 7, 2024
Merged

server : (refactor) no more json in server_task input #10691

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
db97c8b
server : (refactor) no more json in server_task input
ngxson Dec 6, 2024
9bb1ae6
add test for slots endpoint
ngxson Dec 7, 2024
6bf6e30
Merge branch 'master' into xsn/refactor_server_struct_input
ngxson Dec 7, 2024
e721f4c
add tests for /props and /slots
ngxson Dec 7, 2024
090a113
remove task inf_type
ngxson Dec 7, 2024
65d2e6d
fix CI by adding safe_json_to_str
ngxson Dec 7, 2024
1949f68
add "model_path" to /props
ngxson Dec 7, 2024
89c2af9
update readme
ngxson Dec 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions examples/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -687,12 +687,14 @@ This endpoint is public (no API key check). By default, it is read-only. To make
}
},
"total_slots": 1,
"model_path": "../models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
"chat_template": "..."
}
```

- `default_generation_settings` - the default generation settings for the `/completion` endpoint, which has the same fields as the `generation_settings` response object from the `/completion` endpoint.
- `total_slots` - the total number of slots for process requests (defined by `--parallel` option)
- `model_path` - the path to model file (same with `-m` argument)
- `chat_template` - the model's original Jinja2 prompt template

### POST `/props`: Change server global properties.
Expand Down
Loading
Loading