Mask2Former _init_weights

[tommiekerssies]

System Info

• transformers version: 4.45.2
• Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39
• Python version: 3.11.9
• Huggingface_hub version: 0.24.6
• Safetensors version: 0.4.4
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.4.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: yes
• GPU type: NVIDIA RTX A6000

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run _init_weights

Expected behavior

The _init_weights method of Mask2Former has multiple problems. It initializes nn.Embeddings with an std of .02 (original Mask2Former code uses PyTorch's default init with std of 1.0). Similarly, the mask MLP is initialised wrongly with zero biases. Finally, another example of a problem is that the initialisation of the multi-scale deformable attention is overwritten by the branch for the Mask2FormerPixelDecoderEncoderOnly.

[Rocketknight1]

cc @qubvel for Mask2Former, but I think we would accept a PR to fix this! If you make that PR, can you include links to the relevant lines in the original Mask2Former code so we can compare against the reference implementation?

[qubvel]

Hi @tommiekerssies, thanks for opening the issue! _init_weights are not ideal for some vision models in transformers, we would appreciate PR to fix it!

[sambhavnoobcoder]

hi @qubvel , i was having a look at this issue and wrote a quick fix . pushed in PR #35904 .feel free to give this a look when you find the time . i will address any comments you might have on the same .

also thank you to @tommiekerssies for raising the issue .🤗

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sambhavnoobcoder]

not stale , active PR in Progress .

[sambhavnoobcoder]

saying this once more , can i am trying to work this through , dont mark it as stale .

[qubvel]

Added WIP to avoid marking as stale

[sambhavnoobcoder]

thank you for the tag .i'll build on this and get back to you on the same .