Local LLM inference, hand-tuned for one chip. The Ryzen AI Max+ 395.

OS
Install
GPU
Run 

          
        

      

    


    

  


  

    

      
      62.4tok/s
      

        

          
Model
Qwen3.6-35B-A3B-FP8

          
Machine
Ryzen AI Max+ 395

          
Shape
8K in / 1K out, one user

          
Spec
MTP depth 2, AL 2.371

          
4 users
128.4 tok/s total

          
AIME 2026
91.7% avg@4, ±4.2 — Qwen reports 92.7 avg@8

        

      

      
    


    

      
      
        
          
          
          
          
        
        
        
          
          
          
          
        
        
        
          1K
          8K
          32K
          128K
          220K
        
        
          67.4
          39.4
        
      
      
      
Output tok/s by input context, one user — five measured points, evenly spaced

    

  


  

    
Benchmarks

    

      
One client, one server per arm, the same OpenAI request at the same five context
        lengths. Prompts are real text (Project Gutenberg via PG-19) sized exactly by tokenizer,
        not synthetic filler — see benchmarks/perf_sweep.sh. The rungs are
        chosen lengths, drawn evenly spaced — the x axis has no scale.

      
A line is one (engine, model, drafter). Metrics sharing a unit share a chart; since
        decode tok/s = AL × step/s, the gap between an arm's two lines there is
        its acceptance.


      

      
Loading charts…


      

    

  


  

    
Run

    

      
snowllm pull qwen3.6-35b-a3b-fp8
snowllm qwen3.6-35b-a3b-fp8

      
A recipe is one model at one precision, and snowllm recipes
        lists them. The pull runs over several connections at once and resumes where it stopped,
        checking every file against the sha256 the repository publishes. A local safetensors
        directory works just as well.

      
A GGUF works too — snowllm pull qwen3.6-35b-a3b-q4-k-xl-dflash
        gets the UD-Q4_K_XL quant with its DFlash drafter beside it: a smaller download, and
        llama.cpp serves the same file.

      
Serves an OpenAI-compatible API. snowllm --help lists every flag.

      
If huggingface.co is slow from where you are, the pull honours the
        HF_ENDPOINT environment variable — point it at a mirror you trust.

    

  


  

    
Recipes

    

      
Read live off the catalogue snowllm recipes shows and
        snowllm pull resolves — never fresher than your last page reload.


      

      
    

  


  

    
Flags

    

      

        

          
            
FlagWhat it does

          
          
            

              --host --port
              Where the OpenAI-compatible API listens. 127.0.0.1:8000 by default — loopback, so reaching it from another machine takes --host 0.0.0.0.
            

            

              --max-num-seqs
              Concurrency ceiling, up to 256. A memory choice: each request pins linear-attention state for its whole life.
            

            

              --max-model-len
              Context length. A per-request ceiling, not a reservation — the KV pool is shared, and a request that outgrows it is preempted and re-prefilled.
            

            

              --gpu-memory-utilization
              Fraction of the GPU the server may occupy, 0.9 by default.
            

            

              --prefix-memory-ratio
              Share of the memory budget spent remembering prompt prefixes, 0.08 by default, 0 to disable. A repeat behind a 16K prefix costs 0.36 s instead of 5.1 s.
            

            

              --num-spec
              Speculative depth. The optimum depends on the workload; 2 is a robust default.
            

            

              --kv-cache-dtype
              bf16 or int8. int8 halves KV bytes: more decode throughput at long context, less prefill.
            

            

              --dflash PATH
              Speculate with a block-diffusion drafter instead of the MTP head, given the draft's directory. A recipe that ships one sets this for you.
            

            

              --device-map
              Which weight groups live in pinned host memory instead of the carve-out, trading a slower decode step for KV pool. auto is what the DeepSeek recipes use to fit their context.
            

          
        

      


      

        

          
            
EnvironmentWhat it does

          
          
            

              SNOWLLM_MODELS
              Where checkpoints land and are looked for, ~/models by default.
            

            

              HF_ENDPOINT
              Mirror to pull from instead of huggingface.co. The token below goes to whatever host this names.
            

            

              HF_TOKEN
              For gated repositories. SNOWLLM_HF_TOKEN overrides it where you want one token for this and another for everything else.
            

            

              SNOWLLM_RECIPES_URL
              Catalogue to resolve recipe ids against. A local path works, which is how you serve a recipe that is not published yet.
            

            

              SNOWLLM_HOME
              Where install.sh puts the environment, ~/.local/share/snowllm by default; SNOWLLM_BIN_DIR and SNOWLLM_PYTHON move the launcher and pick the interpreter.
            

          
        

      

    

  


  

    
Next

    

      
    
        
  • More models
    • 
        
  • Tensor parallelism at any degree
    • 
        
  • NPU/GPU co-working prefill
    • 
        
  • Native INT4 quantization
    •