Faster Inference using vllm

[AR-javis]

I have modified the model_inference script for a faster inference using vllm. I want to contribute it here.

I have also modified the naming convention to make it more aligned with HF repo ids.

[XXX616519]

I have modified the model_inference script for a faster inference using vllm. I want to contribute it here.

I have also modified the naming convention to make it more aligned with HF repo ids.

Hi there, Can you send me that vllm version? Thanks!

[AR-javis]

Hi, I have that version in my forked repo from my personal account (this is a work account).

Linking the repo here for reference.

https://github.com/Anmol42/FollowBench/tree/master

I have not removed your code, just added another file as an option.

[XXX616519]

Hi, I have that version in my forked repo from my personal account (this is a work account).

Linking the repo here for reference.

https://github.com/Anmol42/FollowBench/tree/master

I have not removed your code, just added another file as an option.

Wooooooo, that helps a lot! Thank you very much!!!!!

[AR-javis]

I have raised a PR here.

#16

We can continue the discussion in that thread.

[XXX616519]

Ok great, we're now working on how to parallelize GPT evaluation using multithreading

[AR-javis]

Is this some independent university project or are you working with the owner of this repo?

[XXX616519]

Is this some independent university project or are you working with the owner of this repo?

Oh, sry. It's just an independent university project. I'm not the owner