download model from modelhub via http_proxy

[nayihz]

What would you like to be added:

Support to use http/https proxy to download model from modelhub.

Why is this needed:

In China, downloading models from huggingface often fails because of GFW.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[nayihz]

/assign

[kerthcet]

Could you share your ideas about how to implement this, right now, we can inject envs to the container, but I forgot whether the init-container would inject the envs as well, which is responsible for downloading models.

[nayihz]

Sorry I didn't explain that clearly.

we can inject envs to the container,

Yes, we need to check if user has set http_proxy and then inject it to init container's envs.

llmaz/pkg/controller_helper/modelsource/modelhub.go

Lines 79 to 84 in e3f2714

	initContainer.Env = append(
	initContainer.Env,
	corev1.EnvVar{Name: "MODEL_SOURCE_TYPE", Value: MODEL_SOURCE_MODELHUB},
	corev1.EnvVar{Name: "MODEL_ID", Value: p.modelID},
	corev1.EnvVar{Name: "MODEL_HUB_NAME", Value: p.modelHub},
	)

But before this, we need to provide users with a configurable entry point. So I think a new field should be added in OpenModel.Spec.Source, like HTTPProxy. Wdyt?

[kerthcet]

We have Envs entrypoint in Playground, let's make sure the envs will spread to the iniContainer as well, then this will be solved.

[nayihz]

Gotcha. I'll check it laster.

[nayihz]

make sure the envs will spread to the iniContainer as well

No, it doesn't.

  spec:
    containers:
    - args:
      - -m
      - /workspace/models/qwen2-0_5b-instruct-q5_k_m.gguf
      - --host
      - 0.0.0.0
      - --port
      - "8080"
      - -fa
      command:
      - ./llama-server
      env:
      - name: LWS_LEADER_ADDRESS
        value: qwen2-0--5b-0.qwen2-0--5b.default
      - name: LWS_GROUP_SIZE
        value: "1"
      - name: http_proxy            # Have injected to containers but not initContianers
        value: 192.168.100.103:7890
      - name: https_proxy
        value: 192.168.100.103:7890
      image: ghcr.io/ggerganov/llama.cpp:server
      initContainers:
    - env:
      - name: LWS_LEADER_ADDRESS
        value: qwen2-0--5b-0.qwen2-0--5b.default
      - name: LWS_GROUP_SIZE
        value: "1"
      - name: MODEL_SOURCE_TYPE
        value: modelhub
      - name: MODEL_ID
        value: Qwen/Qwen2-0.5B-Instruct-GGUF
      - name: MODEL_HUB_NAME
        value: Huggingface
      - name: MODEL_FILENAME
        value: qwen2-0_5b-instruct-q5_k_m.gguf
      - name: REVISION
        value: main
      - name: HUGGING_FACE_HUB_TOKEN
        valueFrom:
          secretKeyRef:
            key: HF_TOKEN
            name: modelhub-secret
            optional: true
      - name: HF_TOKEN
        valueFrom:
          secretKeyRef:
            key: HF_TOKEN
            name: modelhub-secret
            optional: true
      image: inftyai/model-loader:v0.0.10

Will send a PR to fix this later.

[kerthcet]

Yes, I mean right now we didn't spread the envs to the initContainer, we can do this with a fix.

[nayihz]

How ablout splitting playground.Spec.BackendRuntimeConfig.Envs to ContainerEnvs and InitContainerEnvs? Maybe there are some envs that the initContainer does not need.

[kerthcet]

Let's follow the simple way right now, YAGNI, we can revisit this part later if requested by users.

[nayihz]

Are we spreading all envs to initContainer or just http_proxy and https_proxy?