Fix LongCat LoRA load/unload and add regression test

[surajyadav-research]

What does this PR do?

This PR fixes LoRA integration for LongCatImagePipeline so that load_lora_weights() properly applies the adapter during inference and unload_lora_weights() cleanly restores the base (non-LoRA) behavior.

It also adds a slow regression test that:

1. runs the pipeline without LoRA (baseline),
2. loads a LoRA and verifies the output changes,
3. unloads the LoRA and verifies the output returns close to the baseline.

Why?

Addresses the reported LoRA load/unload issue for LongCat:

• load_lora_weights(), unload_lora_weights these doesn't work with LongCatImagePipeline. #12859

Tests

• RUN_SLOW=yes pytest -q tests/pipelines/longcat_image/test_longcat_lora.py

[surajyadav-research]

Hi @sayakpaul
CI is currently “awaiting approval from a maintainer”. Could you please approve the workflow runs?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[surajyadav-research]

Hi @sayakpaul,
I’ve fixed the issue in the test file and pushed the update. Would appreciate your review when convenient.

[sayakpaul]

This seems quite incorrect to me.

Flux has two LoRA loadable modules:

diffusers/src/diffusers/loaders/lora_pipeline.py

Line 1493 in f6b6a71

_lora_loadable_modules = ["transformer", "text_encoder"]

For LongCat, it uses a different text encoder (Flux uses two text encoder, let along) and rest of its components also seems to be different from Flux:

diffusers/src/diffusers/pipelines/longcat_image/pipeline_longcat_image.py

Lines 214 to 222 in f6b6a71

	def __init__(
	self,
	scheduler: FlowMatchEulerDiscreteScheduler,
	vae: AutoencoderKL,
	text_encoder: Qwen2_5_VLForConditionalGeneration,
	tokenizer: Qwen2Tokenizer,
	text_processor: Qwen2VLProcessor,
	transformer: LongCatImageTransformer2DModel,
	):

So, could you please explain how using the FluxLoraLoaderMixin is appropriate here?

Instead, I suggest we write a dedicated LoRA loader mixin class for LongCat -- LongCatLoraLoaderMixin. You can refer to

diffusers/src/diffusers/loaders/lora_pipeline.py

Line 4775 in f6b6a71

class QwenImageLoraLoaderMixin(LoraBaseMixin):

as an example.

[sayakpaul]

This is not needed. Please try to consult the existing testing structure for pipeline-level LoRA testing c.f. https://github.com/huggingface/diffusers/tree/main/tests/lora/

[surajyadav-research]

Thanks for the clarification @sayakpaul
I agree the current inheritance from FluxLoraLoaderMixin is misleading here.

While implementing this, my reasoning was that FluxLoraLoaderMixin mainly reuses the generic LoRA load/fuse path and simply assumes the loadable entry points are ["transformer", "text_encoder"]. But since LongCat’s text_encoder is Qwen2_5_VLForConditionalGeneration and the transformer is LongCatImageTransformer2DModel (i.e., not Flux’s text-encoder setup), this can easily lead to subtle issues like incorrect key routing, mismatched target modules, or silent partial loads.

So I’ll switch to a dedicated LongCatLoraLoaderMixin to make the intent explicit and handle any LongCat-specific routing cleanly.