text-generation task fails with temperature: 0

[nico-martin]

System Info

transformers.js 3.7.4 in a vitejs/React application

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

when setting the temperature to 0, text-generation will generate some random tokens

Reproduction

const pipe = await pipeline(
  "text-generation",
  "HuggingFaceTB/SmolLM3-3B-ONNX",
  {
    device: "webgpu",
    dtype: "q4f16",
  }
);

await pipe(
  [
    {
      role: "system",
      content: "You are a helpful assistant",
    },
    {
      role: "user", 
      content: "Who are you?",
    },
  ],
  {
    max_new_tokens: 10,
    temperature: 0,
  }
);

This will generate something like this: !!!!!!

[xenova]

Thanks! The reason for this is because setting temperature to 0 should set do_sample: false, but currently we divide by 0, leading to Infinity, and we select the first token (! in this case).

[xenova]

In fact, I believe a similar warning was added to transformers recently -- something we can replicate:

/usr/local/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.