# SGLang Go gRPC SDK A high-level Go SDK for interacting with SGLang gRPC API, designed with an OpenAI-style API for familiarity and ease of use. **Location**: `sgl-model-gateway/bindings/golang/` ## Table of Contents - [Features](#features) - [Installation](#installation) - [Quick Start](#quick-start) - [Basic Usage](#basic-usage) - [Streaming Usage](#streaming-usage) - [Examples](#examples) - [Configuration](#configuration) - [API Reference](#api-reference) - [Testing](#testing) - [Unit Tests](#unit-tests) - [Integration Tests](#integration-tests) - [Benchmarks](#benchmarks) - [Documentation](#documentation) - [Development](#development) - [Troubleshooting](#troubleshooting) - [License](#license) ## Features - **OpenAI-style API**: Familiar interface similar to OpenAI Go SDK - **Streaming Support**: Real-time streaming chat completions - **Non-streaming Support**: Simple request/response API - **Tool Calling**: Support for function calling and tool use - **Type-safe**: Full Go type definitions for requests and responses - **Comprehensive Testing**: 18+ unit and integration tests - **Thread-safe**: All public methods are safe for concurrent use - **Well-documented**: Full API documentation with examples ## Installation ```bash go get github.com/sglang/sglang-go-grpc-sdk ``` ### Sync Dependencies ```bash cd sgl-model-gateway/bindings/golang go mod tidy ``` ### Build Requirements - Go 1.21+, Rust toolchain, Python 3.x ## Quick Start ### Benchmark Run the OpenAI-compatible server and benchmark: ```bash # Set environment variables export SGL_TOKENIZER_PATH="/Users/yangyanbo/tokenizer" export SGL_GRPC_ENDPOINT="grpc://10.109.185.20:8001" # Run server cd examples/oai_server bash run.sh # Run E2E benchmark cd ../.. make e2e E2E_MODEL=/work/models/qwencoder-3b E2E_TOKENIZER=/Users/yangyanbo/tokenizer E2E_INPUT_LEN=1024 E2E_OUTPUT_LEN=512 ``` ## Examples The SDK includes several examples in the `examples/` directory: - **simple**: Basic non-streaming chat completion example - **streaming**: Real-time streaming with performance metrics ### Running Examples ```bash # Run simple example cd bindings/golang/examples/simple bash run.sh # Run streaming example cd bindings/golang/examples/streaming bash run.sh # Or use Makefile from bindings/golang directory cd bindings/golang make run-simple make run-streaming ``` ### Basic Usage (Non-streaming) ```go package main import ( "context" "fmt" "log" "github.com/sglang/sglang-go-grpc-sdk" ) func main() { // Create client client, err := sglang.NewClient(sglang.ClientConfig{ Endpoint: "grpc://localhost:20000", TokenizerPath: "/path/to/tokenizer", }) if err != nil { log.Fatal(err) } defer client.Close() // Create completion resp, err := client.CreateChatCompletion(context.Background(), sglang.ChatCompletionRequest{ Model: "default", Messages: []sglang.ChatMessage{ {Role: "user", Content: "Hello!"}, }, Stream: false, }) if err != nil { log.Fatal(err) } fmt.Println(resp.Choices[0].Message.Content) fmt.Printf("Usage: Prompt=%d, Completion=%d, Total=%d\n", resp.Usage.PromptTokens, resp.Usage.CompletionTokens, resp.Usage.TotalTokens) } ``` ### Streaming Usage ```go package main import ( "context" "fmt" "io" "log" "github.com/sglang/sglang-go-grpc-sdk" ) func main() { // Create client client, err := sglang.NewClient(sglang.ClientConfig{ Endpoint: "grpc://localhost:20000", TokenizerPath: "/path/to/tokenizer", }) if err != nil { log.Fatal(err) } defer client.Close() // Create streaming completion ctx := context.Background() stream, err := client.CreateChatCompletionStream(ctx, sglang.ChatCompletionRequest{ Model: "default", Messages: []sglang.ChatMessage{ {Role: "user", Content: "Tell me a story"}, }, Stream: true, MaxCompletionTokens: intPtr(500), }) if err != nil { log.Fatal(err) } defer stream.Close() // Read streaming response for { chunk, err := stream.Recv() if err == io.EOF { break } if err != nil { log.Fatal(err) } for _, choice := range chunk.Choices { if choice.Delta.Content != "" { fmt.Print(choice.Delta.Content) } } } fmt.Println() // newline } // Helper functions for optional pointer fields func intPtr(i int) *int { return &i } func float32Ptr(f float32) *float32 { return &f } ``` Examples automatically detect the server endpoint and tokenizer path via environment variables or defaults. ## Configuration ### Environment Variables - `SGL_GRPC_ENDPOINT`: gRPC server endpoint (default: `grpc://localhost:20000`) - `SGL_TOKENIZER_PATH`: Path to tokenizer directory (required) - `CARGO_BUILD_DIR`: Rust build output directory (auto-detected if not set) ### ClientConfig ```go type ClientConfig struct { // Endpoint is the gRPC endpoint URL (e.g., "grpc://localhost:20000") // Required field. Must include the scheme (grpc://) and port number. Endpoint string // TokenizerPath is the path to the tokenizer directory containing // tokenizer configuration files (e.g., tokenizer.json, vocab.json) // Required field. TokenizerPath string } ``` ## API Reference ### Client Methods ```go type Client struct { // Thread-safe client for SGLang gRPC API } // Creates a new client with the given configuration func NewClient(config ClientConfig) (*Client, error) // Closes the client and releases all resources func (c *Client) Close() error // Creates a non-streaming chat completion func (c *Client) CreateChatCompletion(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionResponse, error) // Creates a streaming chat completion func (c *Client) CreateChatCompletionStream(ctx context.Context, req ChatCompletionRequest) (*ChatCompletionStream, error) ``` ### Request Types - `ChatCompletionRequest`: Main request type for chat completions - Model, Messages, Stream, Temperature, TopP, MaxCompletionTokens, Tools, etc. - `ChatMessage`: Individual message in a conversation - Role, Content - `Tool`: Tool/function definition for function calling - Type, Function (name, description, parameters) ### Response Types - `ChatCompletionResponse`: Non-streaming response - ID, Model, Created, Choices, Usage - `ChatCompletionStreamResponse`: Streaming response chunk - Same structure as above but for incremental updates - `Message`: Complete message with content and tool calls - `ToolCall`: Tool call information with function and arguments - `Usage`: Token usage statistics - PromptTokens, CompletionTokens, TotalTokens ## Testing The SDK includes comprehensive testing infrastructure with both unit and integration tests. ### Unit Tests Unit tests are located in `client_test.go` and test individual components without requiring a server. #### Running Unit Tests ```bash # Run all unit tests go test ./... # Run with verbose output go test -v ./... # Run specific test go test -run TestClientConfig # Run tests with race detector (detects concurrency issues) go test -race ./... # Run with coverage analysis go test -cover ./... # Generate detailed coverage report go test -coverprofile=coverage.out ./... go tool cover -html=coverage.out -o coverage.html ``` #### Unit Test Coverage - Configuration validation, type structures, response handling, concurrent operations, and benchmarks - `client_test.go` - 10 unit tests covering core functionality ### Integration Tests Integration tests require a running SGLang server and test the full client-server interaction. #### Prerequisites 1. Start SGLang server: `python -m sglang.launch_server --model-path ` 2. Set environment variables: ```bash export SGL_GRPC_ENDPOINT=grpc://localhost:20000 export SGL_TOKENIZER_PATH=/path/to/tokenizer ``` #### Running Integration Tests ```bash # Run all integration tests go test -tags=integration ./... # Run specific integration test go test -tags=integration -run TestIntegrationNonStreamingCompletion # Run with verbose output go test -tags=integration -v ./... # Run with race detector go test -tags=integration -race ./... ``` #### Integration Test Coverage **Test File**: `integration_test.go` - 4 integration tests - `TestIntegrationNonStreamingCompletion` - Basic non-streaming request/response - `TestIntegrationStreamingCompletion` - Streaming response handling - `TestIntegrationConcurrentRequests` - Multiple simultaneous requests - `TestIntegrationContextCancellation` - Context timeout and cancellation ### Benchmarks ```bash go test -bench=. -benchmem ./... ``` ## Documentation All public types and functions include comprehensive documentation with usage examples. ### Key Documented Components - `Client` - Main client with thread-safety notes - `ClientConfig` - Configuration requirements and validation rules - `ChatCompletionRequest` - Request structure with field descriptions - `ChatCompletionResponse` - Response structure and usage - `ChatCompletionStreamResponse` - Streaming response format - `Usage` - Token usage information structure - `Tool`, `Function`, `ToolCall` - Tool call structures ### Viewing Documentation ```bash godoc -http=:6060 # Visit: http://localhost:6060/pkg/github.com/sglang/sglang-go-grpc-sdk/ ``` ## Development ```bash cd bindings/golang make build # Build Go bindings go vet ./... # Check code quality go fmt ./... # Format code go test -race ./... # Run tests ``` ### Project Structure ``` bindings/golang/ ├── client.go # Main client implementation ├── client_test.go # Unit tests ├── integration_test.go # Integration tests ├── README.md # This file ├── Makefile # Build automation ├── Cargo.toml # Rust FFI dependencies ├── examples/ # Example programs │ ├── simple/ # Non-streaming example │ └── streaming/ # Streaming example ├── src/ # Rust FFI source │ ├── client.rs # Client FFI │ ├── stream.rs # Stream handling │ ├── grpc_converter.rs # Response conversion │ └── ... └── internal/ # Internal packages └── ffi/ # FFI bindings ``` ## Troubleshooting ### Missing Dependencies Run `go mod tidy` to sync dependencies. ### Connection Errors Ensure SGLang server is running and check `SGL_GRPC_ENDPOINT`. ### Tokenizer Not Found Set `SGL_TOKENIZER_PATH` environment variable. 2. Verify path contains required files: `ls $SGL_TOKENIZER_PATH` 3. Files should include: `tokenizer.json`, `vocab.json`, `config.json` ### Build Failures **Error**: `library 'sgl_model_gateway_go' not found` **Solution**: 1. Rebuild Rust library: `cd sgl-model-gateway/bindings/golang && make build` 2. Or manually with cargo: `cd sgl-model-gateway/bindings/golang && cargo build --release` 3. Set `CARGO_BUILD_DIR` if using non-standard build location 4. Ensure Rust toolchain is installed: `rustup toolchain list` ### Tests Hanging **Error**: Tests seem to hang indefinitely **Solution**: 1. Use timeout for hanging tests: `timeout 30s go test ./...` 2. Run with verbose output to see which test hangs: `go test -v ./...` 3. Ensure server is responsive: `grpcurl -plaintext localhost:20000 list` ### Memory Issues **Error**: Out of memory during tests **Solution**: ```bash # Run with memory limit for long-running tests GODEBUG=madvdontneed=1 go test -timeout 5m ./... # Monitor memory during tests watch -n1 'ps aux | grep test' ``` ## Contributing When adding new features: 1. Add comprehensive documentation to public types/functions 2. Include usage examples for complex APIs 3. Add unit tests covering happy path and error cases 4. Add integration tests if server interaction required 5. Ensure code passes `go vet` and `go test -race` 6. Update this README if adding new features ## License See LICENSE file for details. --- **Need Help?** - Check examples in `examples/` directory - Run tests to see working code: `go test -v ./...` - Review function documentation: `godoc` or inline comments - Check troubleshooting section above