Weird generated image result

[congson1293]

Hi, I tried using your repo to reproduce the result of the Flux-schnell q8_0 at https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/flux.md.
The quantized model I got from: https://huggingface.co/leejet/FLUX.1-schnell-gguf
vae from https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors
clip_l from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/clip_l.safetensors
and t5xxl from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp16.safetensors
The command line I ran:

./bin/sd --diffusion-model ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf --vae ../models/FLUX.1-schnell-gguf/ae.safetensors --clip_l ../models/FLUX.1-schnell-gguf/clip_l.safetensors --t5xxl ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'flux.cpp'" --cfg-scale 1.0 --sampling-method euler -v --steps 4

Finally, I got the weird, low-quality image result

I'm using the latest version.

[Green-Sky]

Can you post the full log? And add -v for a more verbose output.

What backend are you using (CUDA, Vulkan, CPU...) ?

("low-quality" lmao)

[Green-Sky]

It is possible flux requires --clip-on-cpu.

@leejet I noticed that it now runs it on the acceleration device by default, did this change in the wan pr?

[Green-Sky]

$ result/bin/sd --diffusion-model models/flux1-schnell-q4_k.gguf --t5xxl models/flux-extra/t5xxl_fp16.safetensors --clip_l models/flux-extra/clip_l-f16.gguf -t 8 --vae models/flux-extra/ae-f16.gguf --sampling-method euler --scheduler smoothstep --steps 4 --cfg-scale 1 -p "a lovely cat holding a sign says 'flux.cpp'" -v --diffusion-fa --clip-on-cpu --vae-conv-direct -b 1 --offload-to-cpu

(contains an experimental conv2d patch)

• --diffusion-fa try this for lower vram usage and faster sampling (depending on backend, profile yourself)
• --scheduler smoothstep might look better, especially for low step counts, as required by schnell, and more aggressive quantizations. (compare yourself)
• --vae-conv-direct don't use this yet, except when you run out of vram, but for now prefer --vae-tiling instead.
• --offload-to-cpu loads the model to ram (instead of vram) until it is needed. I don't think you need this.

[congson1293]

First, I'm using CUDA device running on an Nvidia L40 card.

This is the full logs

Option:
n_threads: 56
mode: img_gen
model_path:
wtype: unspecified
clip_l_path: ../models/FLUX.1-schnell-gguf/clip_l.safetensors
clip_g_path:
clip_vision_path:
t5xxl_path: ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors
diffusion_model_path: ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf
high_noise_diffusion_model_path:
vae_path: ../models/FLUX.1-schnell-gguf/ae.safetensors
taesd_path:
esrgan_path:
control_net_path:
embedding_dir:
stacked_id_embed_dir:
input_id_images_path:
style ratio: 20.00
normalize input image: false
output_path: output.png
init_image_path:
end_image_path:
mask_image_path:
control_image_path:
ref_images_paths:
increase_ref_index: false
offload_params_to_cpu: false
clip_on_cpu: false
control_net_cpu: false
vae_on_cpu: false
diffusion flash attention: false
diffusion Conv2d direct: false
vae_conv_direct: false
control_strength: 0.90
prompt: a lovely cat holding a sign says 'flux.cpp'
negative_prompt:
clip_skip: -1
width: 512
height: 512
sample_params: (txt_cfg: 1.00, img_cfg: 1.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler, sample_steps: 4, eta: 0.00)
high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler_a, sample_steps: -1, eta: 0.00)
moe_boundary: 0.875
flow_shift: inf
strength(img2img): 0.75
rng: cuda
seed: 42
batch_count: 1
vae_tiling: false
upscale_repeats: 1
chroma_use_dit_mask: true
chroma_use_t5_mask: false
chroma_t5_mask_pad: 1
video_frames: 1
fps: 16
System Info:
SSE3 = 1
AVX = 1
AVX2 = 1
AVX512 = 1
AVX512_VBMI = 1
AVX512_VNNI = 1
FMA = 1
NEON = 0
ARM_FMA = 0
F16C = 1
FP16_VA = 0
WASM_SIMD = 0
VSX = 0
[DEBUG] stable-diffusion.cpp:143 - Using CUDA backend
[INFO ] ggml_extend.hpp:65 - ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
[INFO ] ggml_extend.hpp:65 - ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
[INFO ] ggml_extend.hpp:65 - ggml_cuda_init: found 1 CUDA devices:
[INFO ] ggml_extend.hpp:65 - Device 0: NVIDIA L40, compute capability 8.9, VMM: yes
[INFO ] stable-diffusion.cpp:208 - loading diffusion model from '../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf'
[INFO ] model.cpp:1035 - load ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf using gguf format
[DEBUG] model.cpp:1052 - init from '../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf'
[INFO ] stable-diffusion.cpp:224 - loading clip_l from '../models/FLUX.1-schnell-gguf/clip_l.safetensors'
[INFO ] model.cpp:1038 - load ../models/FLUX.1-schnell-gguf/clip_l.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from '../models/FLUX.1-schnell-gguf/clip_l.safetensors', prefix = 'text_encoders.clip_l.transformer.'
[INFO ] stable-diffusion.cpp:248 - loading t5xxl from '../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors'
[INFO ] model.cpp:1038 - load ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from '../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors', prefix = 'text_encoders.t5xxl.transformer.'
[INFO ] stable-diffusion.cpp:255 - loading vae from '../models/FLUX.1-schnell-gguf/ae.safetensors'
[INFO ] model.cpp:1038 - load ../models/FLUX.1-schnell-gguf/ae.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from '../models/FLUX.1-schnell-gguf/ae.safetensors', prefix = 'vae.'
[INFO ] stable-diffusion.cpp:267 - Version: Flux
[INFO ] stable-diffusion.cpp:301 - Weight type: q8_0
[INFO ] stable-diffusion.cpp:302 - Conditioner weight type: f16
[INFO ] stable-diffusion.cpp:303 - Diffusion model weight type: q8_0
[INFO ] stable-diffusion.cpp:304 - VAE weight type: f32
[DEBUG] stable-diffusion.cpp:306 - ggml tensor size = 400 bytes
[WARN ] stable-diffusion.cpp:335 - !!!It appears that you are using the T5 model. Some backends may encounter issues with it.If you notice that the generated images are completely black,try running the T5 model on the CPU using the --clip-on-cpu parameter.
[DEBUG] clip.hpp:171 - vocab size: 49408
[DEBUG] clip.hpp:182 - trigger word img already in vocab
[INFO ] flux.hpp:937 - Flux blocks: 19 double, 38 single
[INFO ] flux.hpp:941 - Flux guidance is disabled (Schnell mode)
[DEBUG] ggml_extend.hpp:1609 - clip params backend buffer size = 307.44 MB(VRAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1609 - t5 params backend buffer size = 9083.77 MB(VRAM) (219 tensors)
[DEBUG] ggml_extend.hpp:1609 - flux params backend buffer size = 12208.62 MB(VRAM) (776 tensors)
[DEBUG] ggml_extend.hpp:1609 - vae params backend buffer size = 94.57 MB(VRAM) (138 tensors)
[DEBUG] stable-diffusion.cpp:565 - loading weights
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf
|==================================================| 1435/1435 - 306.76it/s
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/clip_l.safetensors
|==================================================| 1435/1435 - 6674.42it/s
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors
|==================================================| 1435/1435 - 514.34it/s
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/ae.safetensors
|==================================================| 1435/1435 - 1267.67it/s
[INFO ] model.cpp:2216 - loading tensors completed, taking 8.93s (process: 0.09s, read: 5.54s, memcpy: 0.00s, convert: 0.32s, copy_to_backend: 1.87s)
[INFO ] stable-diffusion.cpp:646 - total params memory size = 21694.40MB (VRAM 21694.40MB, RAM 0.00MB): text_encoders 9391.21MB(VRAM), diffusion_model 12208.62MB(VRAM), vae 94.57MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:691 - running in Flux FLOW mode
[DEBUG] stable-diffusion.cpp:725 - finished loaded file
[DEBUG] stable-diffusion.cpp:2181 - generate_image 512x512
[INFO ] stable-diffusion.cpp:2314 - TXT2IMG
[INFO ] stable-diffusion.cpp:859 - attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:879 - apply_loras completed, taking 0.00s
[DEBUG] stable-diffusion.cpp:880 - prompt after extract and remove lora: "a lovely cat holding a sign says 'flux.cpp'"
[DEBUG] conditioner.hpp:1067 - parse 'a lovely cat holding a sign says 'flux.cpp'' to [['a lovely cat holding a sign says 'flux.cpp'', 1], ]
[DEBUG] clip.hpp:311 - token length: 77
[DEBUG] t5.hpp:402 - token length: 256
[DEBUG] clip.hpp:741 - identity projection
[DEBUG] ggml_extend.hpp:1437 - clip compute buffer size: 1.40 MB(VRAM)
[DEBUG] clip.hpp:741 - identity projection
[DEBUG] ggml_extend.hpp:1437 - t5 compute buffer size: 68.25 MB(VRAM)
[DEBUG] conditioner.hpp:1182 - computing condition graph completed, taking 157 ms
[INFO ] stable-diffusion.cpp:1970 - get_learned_condition completed, taking 162 ms
[INFO ] stable-diffusion.cpp:1993 - sampling using Euler method
[INFO ] stable-diffusion.cpp:2042 - generating image: 1/1 - seed 42
[DEBUG] ggml_extend.hpp:1437 - flux compute buffer size: 407.50 MB(VRAM)
|==================================================| 4/4 - 3.44it/s
[INFO ] stable-diffusion.cpp:2078 - sampling completed, taking 1.23s
[INFO ] stable-diffusion.cpp:2086 - generating 1 latent images completed, taking 1.24s
[INFO ] stable-diffusion.cpp:2089 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1437 - vae compute buffer size: 1664.25 MB(VRAM)
[DEBUG] stable-diffusion.cpp:1458 - computing vae decode graph completed, taking 0.11s
[INFO ] stable-diffusion.cpp:2099 - latent 1 decoded, taking 0.11s
[INFO ] stable-diffusion.cpp:2103 - decode_first_stage completed, taking 0.11s
[INFO ] stable-diffusion.cpp:2386 - generate_image completed in 1.52s
save result PNG image to 'output.png'
(base) sontc@aiserver:~/stable-diffusion-python/stable-diffusion.cpp$ ./bin/sd --diffusion-model ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf --vae ../models/FLUX.1-schnell-gguf/ae.safetensors --clip_l ../models/FLUX.1-schnell-gguf/clip_l.safetensors --t5xxl ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'flux.cpp'" --cfg-scale 1.0 --sampling-method euler -v --steps 4
Option:
n_threads: 56
mode: img_gen
model_path:
wtype: unspecified
clip_l_path: ../models/FLUX.1-schnell-gguf/clip_l.safetensors
clip_g_path:
clip_vision_path:
t5xxl_path: ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors
diffusion_model_path: ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf
high_noise_diffusion_model_path:
vae_path: ../models/FLUX.1-schnell-gguf/ae.safetensors
taesd_path:
esrgan_path:
control_net_path:
embedding_dir:
stacked_id_embed_dir:
input_id_images_path:
style ratio: 20.00
normalize input image: false
output_path: output.png
init_image_path:
end_image_path:
mask_image_path:
control_image_path:
ref_images_paths:
increase_ref_index: false
offload_params_to_cpu: false
clip_on_cpu: false
control_net_cpu: false
vae_on_cpu: false
diffusion flash attention: false
diffusion Conv2d direct: false
vae_conv_direct: false
control_strength: 0.90
prompt: a lovely cat holding a sign says 'flux.cpp'
negative_prompt:
clip_skip: -1
width: 512
height: 512
sample_params: (txt_cfg: 1.00, img_cfg: 1.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler, sample_steps: 4, eta: 0.00)
high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: default, sample_method: euler_a, sample_steps: -1, eta: 0.00)
moe_boundary: 0.875
flow_shift: inf
strength(img2img): 0.75
rng: cuda
seed: 42
batch_count: 1
vae_tiling: false
upscale_repeats: 1
chroma_use_dit_mask: true
chroma_use_t5_mask: false
chroma_t5_mask_pad: 1
video_frames: 1
fps: 16
System Info:
SSE3 = 1
AVX = 1
AVX2 = 1
AVX512 = 1
AVX512_VBMI = 1
AVX512_VNNI = 1
FMA = 1
NEON = 0
ARM_FMA = 0
F16C = 1
FP16_VA = 0
WASM_SIMD = 0
VSX = 0
[DEBUG] stable-diffusion.cpp:143 - Using CUDA backend
[INFO ] ggml_extend.hpp:65 - ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
[INFO ] ggml_extend.hpp:65 - ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
[INFO ] ggml_extend.hpp:65 - ggml_cuda_init: found 1 CUDA devices:
[INFO ] ggml_extend.hpp:65 - Device 0: NVIDIA L40, compute capability 8.9, VMM: yes
[INFO ] stable-diffusion.cpp:208 - loading diffusion model from '../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf'
[INFO ] model.cpp:1035 - load ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf using gguf format
[DEBUG] model.cpp:1052 - init from '../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf'
[INFO ] stable-diffusion.cpp:224 - loading clip_l from '../models/FLUX.1-schnell-gguf/clip_l.safetensors'
[INFO ] model.cpp:1038 - load ../models/FLUX.1-schnell-gguf/clip_l.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from '../models/FLUX.1-schnell-gguf/clip_l.safetensors', prefix = 'text_encoders.clip_l.transformer.'
[INFO ] stable-diffusion.cpp:248 - loading t5xxl from '../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors'
[INFO ] model.cpp:1038 - load ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from '../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors', prefix = 'text_encoders.t5xxl.transformer.'
[INFO ] stable-diffusion.cpp:255 - loading vae from '../models/FLUX.1-schnell-gguf/ae.safetensors'
[INFO ] model.cpp:1038 - load ../models/FLUX.1-schnell-gguf/ae.safetensors using safetensors format
[DEBUG] model.cpp:1145 - init from '../models/FLUX.1-schnell-gguf/ae.safetensors', prefix = 'vae.'
[INFO ] stable-diffusion.cpp:267 - Version: Flux
[INFO ] stable-diffusion.cpp:301 - Weight type: q8_0
[INFO ] stable-diffusion.cpp:302 - Conditioner weight type: f16
[INFO ] stable-diffusion.cpp:303 - Diffusion model weight type: q8_0
[INFO ] stable-diffusion.cpp:304 - VAE weight type: f32
[DEBUG] stable-diffusion.cpp:306 - ggml tensor size = 400 bytes
[WARN ] stable-diffusion.cpp:335 - !!!It appears that you are using the T5 model. Some backends may encounter issues with it.If you notice that the generated images are completely black,try running the T5 model on the CPU using the --clip-on-cpu parameter.
[DEBUG] clip.hpp:171 - vocab size: 49408
[DEBUG] clip.hpp:182 - trigger word img already in vocab
[INFO ] flux.hpp:937 - Flux blocks: 19 double, 38 single
[INFO ] flux.hpp:941 - Flux guidance is disabled (Schnell mode)
[DEBUG] ggml_extend.hpp:1609 - clip params backend buffer size = 307.44 MB(VRAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1609 - t5 params backend buffer size = 9083.77 MB(VRAM) (219 tensors)
[DEBUG] ggml_extend.hpp:1609 - flux params backend buffer size = 12208.62 MB(VRAM) (776 tensors)
[DEBUG] ggml_extend.hpp:1609 - vae params backend buffer size = 94.57 MB(VRAM) (138 tensors)
[DEBUG] stable-diffusion.cpp:565 - loading weights
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/flux1-schnell-q8_0.gguf
|==================================================| 1435/1435 - 379.23it/s
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/clip_l.safetensors
|==================================================| 1435/1435 - 6737.09it/s
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/t5xxl_fp16.safetensors
|==================================================| 1435/1435 - 531.68it/s
[DEBUG] model.cpp:1998 - loading tensors from ../models/FLUX.1-schnell-gguf/ae.safetensors
|==================================================| 1435/1435 - 8391.81it/s
[INFO ] model.cpp:2216 - loading tensors completed, taking 6.98s (process: 0.09s, read: 3.69s, memcpy: 0.00s, convert: 0.24s, copy_to_backend: 1.86s)
[INFO ] stable-diffusion.cpp:646 - total params memory size = 21694.40MB (VRAM 21694.40MB, RAM 0.00MB): text_encoders 9391.21MB(VRAM), diffusion_model 12208.62MB(VRAM), vae 94.57MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:691 - running in Flux FLOW mode
[DEBUG] stable-diffusion.cpp:725 - finished loaded file
[DEBUG] stable-diffusion.cpp:2181 - generate_image 512x512
[INFO ] stable-diffusion.cpp:2314 - TXT2IMG
[INFO ] stable-diffusion.cpp:859 - attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:879 - apply_loras completed, taking 0.00s
[DEBUG] stable-diffusion.cpp:880 - prompt after extract and remove lora: "a lovely cat holding a sign says 'flux.cpp'"
[DEBUG] conditioner.hpp:1067 - parse 'a lovely cat holding a sign says 'flux.cpp'' to [['a lovely cat holding a sign says 'flux.cpp'', 1], ]
[DEBUG] clip.hpp:311 - token length: 77
[DEBUG] t5.hpp:402 - token length: 256
[DEBUG] clip.hpp:741 - identity projection
[DEBUG] ggml_extend.hpp:1437 - clip compute buffer size: 1.40 MB(VRAM)
[DEBUG] clip.hpp:741 - identity projection
[DEBUG] ggml_extend.hpp:1437 - t5 compute buffer size: 68.25 MB(VRAM)
[DEBUG] conditioner.hpp:1182 - computing condition graph completed, taking 155 ms
[INFO ] stable-diffusion.cpp:1970 - get_learned_condition completed, taking 160 ms
[INFO ] stable-diffusion.cpp:1993 - sampling using Euler method
[INFO ] stable-diffusion.cpp:2042 - generating image: 1/1 - seed 42
[DEBUG] ggml_extend.hpp:1437 - flux compute buffer size: 407.50 MB(VRAM)
|==================================================| 4/4 - 3.44it/s
[INFO ] stable-diffusion.cpp:2078 - sampling completed, taking 1.23s
[INFO ] stable-diffusion.cpp:2086 - generating 1 latent images completed, taking 1.24s
[INFO ] stable-diffusion.cpp:2089 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1437 - vae compute buffer size: 1664.25 MB(VRAM)
[DEBUG] stable-diffusion.cpp:1458 - computing vae decode graph completed, taking 0.11s
[INFO ] stable-diffusion.cpp:2099 - latent 1 decoded, taking 0.11s
[INFO ] stable-diffusion.cpp:2103 - decode_first_stage completed, taking 0.11s
[INFO ] stable-diffusion.cpp:2386 - generate_image completed in 1.52s
save result PNG image to 'output.png'

[Green-Sky]

Log looks expected. It also includes the warning I spoke of:
[WARN ] stable-diffusion.cpp:335 - !!!It appears that you are using the T5 model. Some backends may encounter issues with it.If you notice that the generated images are completely black,try running the T5 model on the CPU using the --clip-on-cpu parameter.

[congson1293]

Log looks expected. It also includes the warning I spoke of: [WARN ] stable-diffusion.cpp:335 - !!!It appears that you are using the T5 model. Some backends may encounter issues with it.If you notice that the generated images are completely black,try running the T5 model on the CPU using the --clip-on-cpu parameter.

Ok, I will try with --clip-on-cpu on.

[congson1293]

@Green-Sky, it works when turned on --clip-on-cpu. I think you should update the documentation for Flux-schnell, especially for a newbie like me.

BTW, do you have any plans for an HTTP serving tool?

[Green-Sky]

#367

and

https://github.com/leejet/stable-diffusion.cpp?tab=readme-ov-file#uis

[leejet]

It is possible flux requires --clip-on-cpu.

@leejet I noticed that it now runs it on the acceleration device by default, did this change in the wan pr?

Yes, UMT5 runs very well on the GPU, so I removed the restriction that T5 cannot run on the GPU. I'll update the documentation later.