Allow video objects (np array etc.) in apply_chat_template (not just paths or urls)

[FredrikNoren]

Feature request

Currently it seems like it only supports { "type": "video", "path/url": ".."}.

The code seems to be made to allow numpy arrays, torch tensors and other things as values: https://github.com/huggingface/transformers/blob/main/src/transformers/image_utils.py#L851
But the video.startswith will always crash if anything other than a string is passed in

Motivation

I'd like to pass in videos from memory instead of writing them to disk first.

Your contribution

I can update the code to make it support video objects

[zucchini-nlp]

Hey @FredrikNoren !

Yes, that was supposed to be supported and prob I missed it when testing. Feel free to open a PR, and add some test in test_processing_common. Also, would be better to do same for images, enabling passing PIL/array images

[akibjawad]

Hi @FredrikNoren , @zucchini-nlp ,
Is the issue still open? I would love to take this if the issue is still there.

[zucchini-nlp]

Sure, feel free, I don't think there was a PR for that

[FredrikNoren]

@akibjawad Yeah I never had time to do a PR, feel free

[devesssi]

@zucchini-nlp is this still free if then, i would like to contribute

[akibjawad]

@devesssi I have developed the feature and currently testing it. My PR is in WIP it should be completed by early next week.

[devesssi]

Thanks for clarifying @akibjawad . Let me know if there's anything I could do