[Refactor] OAI Server components

[JustinTong0323]

Motivation

• [OAI Server Refactor] Define Initial Serving Logic Structure #7104

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[JustinTong0323]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request undertakes a significant refactor of the OpenAI-compatible server components. The primary goal is to improve the code structure by introducing a base class for request handling, formalizing the API contract using Pydantic models, and separating endpoint-specific logic into dedicated handlers. This reorganization aims to enhance maintainability, testability, and provide a clearer foundation for implementing future features and ensuring better adherence to the OpenAI API specification.

Highlights

• Architectural Refactor: Introduced a new abstract base class OpenAIServingBase (python/sglang/srt/entrypoints/openai/serving_base.py) to provide common structure and logic for handling OpenAI-compatible requests across different endpoints.
• Formalized API Protocol: Defined comprehensive Pydantic models for OpenAI API requests and responses in a new protocol.py file (python/sglang/srt/entrypoints/openai/protocol.py), including SGLang-specific extensions, to formalize the API contract and enable Pydantic validation.
• Dedicated Endpoint Handlers: Created separate handler classes (OpenAIServingChat, OpenAIServingCompletion, OpenAIServingEmbedding) inheriting from the base class to encapsulate endpoint-specific logic for chat completions (python/sglang/srt/entrypoints/openai/serving_chat.py), text completions (python/sglang/srt/entrypoints/openai/serving_completions.py), and embeddings (python/sglang/srt/entrypoints/openai/serving_embedding.py).
• Centralized Utilities: Extracted common utility functions, such as template processing helpers, token usage calculation, and logprobs formatting, into a new utils.py module (python/sglang/srt/entrypoints/openai/utils.py).
• New Unit Tests: Added extensive unit tests for the new protocol models and the refactored serving handlers (test/srt/openai/test_protocol.py, test/srt/openai/test_serving_chat.py, test/srt/openai/test_serving_completions.py, test/srt/openai/test_serving_embedding.py) to ensure correctness and maintainability.

Changelog

• python/sglang/srt/entrypoints/openai/protocol.py
  • New file defining Pydantic models for OpenAI API entities: ModelCard, ModelList, ErrorResponse, LogProbs, TopLogprob, ChatCompletionTokenLogprob, ChoiceLogprobs, UsageInfo, StreamOptions, JsonSchemaResponseFormat, FileRequest, FileResponse, FileDeleteResponse, BatchRequest, BatchResponse, CompletionRequest, CompletionResponseChoice, CompletionResponse, CompletionResponseStreamChoice, CompletionStreamResponse, ChatCompletionMessageContentTextPart, ChatCompletionMessageContentImageURL, ChatCompletionMessageContentAudioURL, ChatCompletionMessageContentImagePart, ChatCompletionMessageContentAudioPart, ChatCompletionMessageContentPart (Union), FunctionResponse, ToolCall, ChatCompletionMessageGenericParam, ChatCompletionMessageUserParam, ChatCompletionMessageParam (Union), ResponseFormat, StructuresResponseFormat, StructuralTagResponseFormat, Function, Tool, ToolChoiceFuncName, ToolChoice, ChatCompletionRequest, ChatMessage, MultimodalEmbeddingInput, EmbeddingInput (Union), EmbeddingRequest, EmbeddingObject, EmbeddingResponse, ScoringRequest, ScoringResponse, OpenAIServingRequest (Union).
  • Includes SGLang-specific fields within standard models (e.g., top_k, min_p, regex, ebnf, repetition_penalty, stop_token_ids, no_stop_trim, ignore_eos, skip_special_tokens, lora_path, session_params, separate_reasoning, stream_reasoning, chat_template_kwargs, rid, bootstrap_* parameters).
  • Adds Pydantic field and model validators for max_tokens, messages, and tool_choice.
• python/sglang/srt/entrypoints/openai/serving_base.py
  • New file defining OpenAIServingBase, an abstract base class for OpenAI endpoint handlers.
  • Provides a common handle_request method for request validation, conversion to internal format, and dispatching to streaming/non-streaming handlers.
  • Includes abstract methods _request_id_prefix, _convert_to_internal_request, _handle_streaming_request, and _handle_non_streaming_request to be implemented by subclasses.
  • Implements base request ID generation (_generate_request_id_base).
  • Includes a base method for calculating streaming usage (_calculate_streaming_usage_base).
  • Provides helper methods for creating standard and streaming error responses (create_error_response, create_streaming_error_response).
• python/sglang/srt/entrypoints/openai/serving_chat.py
  • New file implementing OpenAIServingChat inheriting from OpenAIServingBase to handle /v1/chat/completions requests.
  • Implements _request_id_prefix for chat completion IDs.
  • Implements _validate_request with specific validation for chat messages.
  • Implements _convert_to_internal_request including logic for processing messages, applying Jinja or conversation templates, handling multimodal content, and building sampling parameters (including response formats, tools, and SGLang extensions).
  • Implements _handle_streaming_request for streaming chat responses, including handling delta messages, logprobs, reasoning content, and tool calls.
  • Implements _handle_non_streaming_request for non-streaming chat responses, building the final response structure including logprobs, reasoning content, and tool calls.
  • Includes helper methods for processing logprobs (_process_logprobs_tokens, _process_response_logprobs, _process_streaming_logprobs), tool calls (_process_tool_calls, _process_tool_call_stream), and reasoning content (_process_reasoning_stream).
• python/sglang/srt/entrypoints/openai/serving_completions.py
  • New file implementing OpenAIServingCompletion inheriting from OpenAIServingBase to handle /v1/completions requests.
  • Implements _request_id_prefix for completion IDs.
  • Implements _validate_request with specific validation for completion prompts (string, list of strings, list of token IDs, list of lists of token IDs).
  • Implements _convert_to_internal_request including logic for processing prompts (with optional suffix and completion template) and building sampling parameters.
  • Implements _handle_streaming_request for streaming completion responses, including handling echo and logprobs.
  • Implements _handle_non_streaming_request for non-streaming completion responses, building the final response structure including echo and logprobs.
  • Includes helper methods for handling echo text (_get_echo_text, _prepare_echo_prompts).
• python/sglang/srt/entrypoints/openai/serving_embedding.py
  • New file implementing OpenAIServingEmbedding inheriting from OpenAIServingBase to handle /v1/embeddings requests.
  • Implements _request_id_prefix for embedding IDs.
  • Implements _validate_request with specific validation for embedding input types.
  • Implements _convert_to_internal_request including logic for processing various input types (string, list of strings, list of token IDs, list of lists of token IDs, list of MultimodalEmbeddingInput) and noting that multimodal batch requests are not yet supported.
  • Implements _handle_non_streaming_request for embedding requests (streaming is not supported).
  • Implements _build_embedding_response to construct the final embedding response including embedding objects and usage information.
• python/sglang/srt/entrypoints/openai/utils.py
  • New file containing shared utility functions.
  • Includes logic for detecting Jinja template content format (detect_template_content_format) and processing message content based on format (process_content_for_template_format).
  • Provides functions for calculating and aggregating token usage (calculate_token_usage, aggregate_token_usage).
  • Includes a function to convert internal logprobs format to OpenAI style (to_openai_style_logprobs).
• test/pytest.ini
  • Added asyncio_mode = auto to the pytest configuration.
• test/srt/openai/test_protocol.py
  • New file with unit tests for the Pydantic models defined in protocol.py, covering creation, serialization, and validation.
• test/srt/openai/test_serving_chat.py
  • New file with unit tests for the OpenAIServingChat handler, covering request conversion, tool calls, multimodal content, template handling, reasoning content, sampling parameters, and compatibility with previous adapter logic.
• test/srt/openai/test_serving_completions.py
  • New file with unit tests for the OpenAIServingCompletion handler, covering prompt handling, echo functionality, request conversion, response building, and async method testing.
• test/srt/openai/test_serving_embedding.py
  • New file with unit tests for the OpenAIServingEmbedding handler, covering request validation and conversion, response building, async method testing, and edge cases like multimodal batching.

Activity

• The author @JustinTong0323 requested a review and summary from the bot.
• The bot provided an initial summary and a detailed review with High and Medium priority feedback.
• High priority feedback included concerns about global caching in serving_chat.py, unimplemented multimodal batching in serving_embedding.py, and incorrect top_logprobs processing in streaming chat.
• Medium priority feedback covered various aspects like docstrings, broad exception handling, the inclusion of SGLang-specific extensions in standard protocol models, details of validation logic, missing text_offset in logprobs, and the use of 'padding' for multimodal text.
• @JustinTong0323 responded to high-level design questions raised in the review, discussing the necessity of RequestContext, strategies for _build_sampling_params, and the placement of validation logic.
• @CatherineSue provided review comments, questioning the usage of certain functions, the need for RequestContext, the placement of validation logic (suggesting moving it to individual serving classes or noting existing validation in sampling_params.py), and suggesting moving error response helpers to the base class.
• @JustinTong0323 addressed some of @CatherineSue's comments, confirming some functions could be deleted and discussing validation placement.
• The bot provided a second review, identifying Critical issues related to passing the entire request object instead of request.n to _calculate_streaming_usage_base in both chat and completions handlers.
• The second bot review also highlighted a High priority syntax issue in the input_file_id type hint in protocol.py.
• Medium priority issues from the second bot review included the presence of 'abort' and 'function_call' in finish reasons, the tool_call_constraint TODO, the 'padding' approach for multimodal text, the text_offset TODO, the source of the enable_thinking parameter, chat message validation strictness, and the warning about echo + logprobs.

[CatherineSue]

This is in good shape. I think after resolving the comments in tests, we can run a local ut and make sure all tests pass. Merge this.

Great work!

[JustinTong0323]

All related tests passed:

[CatherineSue]

Approved. Looks like lint is failing. Let's fix the lint.

[jhinpan]

Wondering do we need to add test files here to CI suite as well like test_serving_chat.py and test_serving_completions.py?