Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[serve] vllm example to serve llm models #45430

Merged
merged 23 commits into from
May 23, 2024
Merged

[serve] vllm example to serve llm models #45430

merged 23 commits into from
May 23, 2024

Conversation

can-anyscale
Copy link
Collaborator

@can-anyscale can-anyscale commented May 18, 2024 •
edited

Adds a documentation example using vLLM to serve LLM models on Ray Serve.

This is a copy of #45325 + add a build environment for ray serve + vllm.

Test:

  • CI

@can-anyscale
[ci] add serve vllm build+test
08a9655 
Signed-off-by: can <can@anyscale.com>
@can-anyscale
[ci] add serve gpu build
3d63f6a 
Signed-off-by: can <can@anyscale.com>
@can-anyscale can-anyscale force-pushed the can-vllm02 branch 2 times, most recently from 4c067d7 to 3c8ca11 Compare May 18, 2024 04:51
@can-anyscale
[ci] add serve gpu guild
423bfd8 
Signed-off-by: can <can@anyscale.com>
@can-anyscale
run other serve doc tests
059cd15 
@MicroCheck linux://doc:source/serve/doc_code/distilbert linux://doc:source/serve/doc_code/object_detection linux://doc:source/serve/doc_code/stable_diffusion

Signed-off-by: can <can@anyscale.com>
@can-anyscale can-anyscale changed the title Can vllm02 [serve] add a documentation example using vllm May 18, 2024
@can-anyscale can-anyscale changed the title [serve] add a documentation example using vllm [serve] vllm example to serve llm models May 18, 2024
@can-anyscale
add todo
9656ba3 
@MicroCheck //doc:source/serve/doc_code/distilbert //doc:source/serve/doc_code/object_detection //doc:source/serve/doc_code/stable_diffusion

Signed-off-by: can <can@anyscale.com>
@can-anyscale can-anyscale marked this pull request as ready for review May 18, 2024 20:47
@can-anyscale
Copy link
Collaborator Author

@akshay-anyscale, @edoakes i managed to create an environment for the test to run but it fails for some other reasons https://buildkite.com/ray-project/microcheck/builds/237#018f8c35-e5a1-443d-8cf9-bbb481af6c1e/177-2429; if this makes sense feel free to change this pr, thankkks

aslonnie
aslonnie reviewed May 19, 2024
View reviewed changes
Copy link
Collaborator

@aslonnie aslonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this just a PoC, or is this intended to get merged?

@can-anyscale
Copy link
Collaborator Author

@aslonnie intended to get merged, but will need serve folks to pick up and finish the job ;)

@edoakes
maybe hopefully
019be90 
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
@edoakes
Copy link
Contributor

edoakes commented May 20, 2024

@akshay-anyscale, @edoakes i managed to create an environment for the test to run but it fails for some other reasons https://buildkite.com/ray-project/microcheck/builds/237#018f8c35-e5a1-443d-8cf9-bbb481af6c1e/177-2429; if this makes sense feel free to change this pr, thankkks

Pushed a commit to change the dtype, hopefully that fixes things.

@akshay-anyscale
Update vllm_openai_example.py
7c46f9a 
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
@akshay-anyscale
Update vllm_openai_example.py
03c0ae5 
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
@carsonwang
Copy link

Is ray-llm going to be deprecated and this example will be the recommended way to run vllm on Ray?

akshay-anyscale and others added 4 commits May 21, 2024 09:21
@akshay-anyscale
try diff model
f1d1968 
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
@can-anyscale
increase test timedout
4a13704 
Signed-off-by: can <can@anyscale.com>
@akshay-anyscale
Update vllm_openai_example.py
3419b5e 
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
@can-anyscale
will work
23fad7f 
Signed-off-by: can <can@anyscale.com>
@akshay-anyscale
Update vllm_openai_example.py
046eefa 
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
@can-anyscale
Copy link
Collaborator Author

@angelinalg do you mind help review the doc content pieces, thankks

@can-anyscale can-anyscale enabled auto-merge (squash) May 23, 2024 00:09
@github-actions github-actions bot added the go Trigger full test run on premerge label May 23, 2024
angelinalg
angelinalg approved these changes May 23, 2024
View reviewed changes
Copy link
Contributor

@angelinalg angelinalg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some style nits.

doc/source/serve/doc_code/vllm_openai_example.py Outdated Show resolved Hide resolved
doc/source/serve/doc_code/vllm_openai_example.py Outdated Show resolved Hide resolved
doc/source/serve/doc_code/vllm_openai_example.py Outdated Show resolved Hide resolved
doc/source/serve/doc_code/vllm_openai_example.py Outdated Show resolved Hide resolved
doc/source/serve/doc_code/vllm_openai_example.py Outdated Show resolved Hide resolved
doc/source/serve/tutorials/vllm-example.md Outdated Show resolved Hide resolved
doc/source/serve/tutorials/vllm-example.md Outdated Show resolved Hide resolved
doc/source/serve/tutorials/vllm-example.md Outdated Show resolved Hide resolved
doc/source/serve/tutorials/vllm-example.md Outdated Show resolved Hide resolved
doc/source/serve/examples.yml Show resolved Hide resolved
can-anyscale and others added 10 commits May 22, 2024 18:56
@can-anyscale @angelinalg
Update doc/source/serve/doc_code/vllm_openai_example.py
481b5e4 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/doc_code/vllm_openai_example.py
fee3ff0 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/doc_code/vllm_openai_example.py
c8f3eca 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/doc_code/vllm_openai_example.py
2b7a5b3 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/doc_code/vllm_openai_example.py
b658f81 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/tutorials/vllm-example.md
acac849 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/tutorials/vllm-example.md
3eda046 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/tutorials/vllm-example.md
1df049d 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/tutorials/vllm-example.md
ab616cb 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale @angelinalg
Update doc/source/serve/examples.yml
7607fae 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
@can-anyscale can-anyscale enabled auto-merge (squash) May 23, 2024 01:58
@github-actions github-actions bot disabled auto-merge May 23, 2024 02:01
@can-anyscale can-anyscale enabled auto-merge (squash) May 23, 2024 02:51
@can-anyscale can-anyscale merged commit 4fe83c3 into master May 23, 2024
7 of 8 checks passed
@can-anyscale can-anyscale deleted the can-vllm02 branch May 23, 2024 02:52
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 6, 2024
@can-anyscale @edoakes
@akshay-anyscale @angelinalg @ryanaoleary
[serve] vllm example to serve llm models (ray-project#45430)
2efdddb 
Adds a documentation example using vLLM to serve LLM models on Ray
Serve.

This is a copy of ray-project#45325 + add a
build environment for ray serve + vllm.

Test:
- CI

---------

Signed-off-by: can <can@anyscale.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 6, 2024
@can-anyscale @edoakes
@akshay-anyscale @angelinalg @ryanaoleary
[serve] vllm example to serve llm models (ray-project#45430)
2971769 
Adds a documentation example using vLLM to serve LLM models on Ray
Serve.

This is a copy of ray-project#45325 + add a
build environment for ray serve + vllm.

Test:
- CI

---------

Signed-off-by: can <can@anyscale.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 7, 2024
@can-anyscale @edoakes
@akshay-anyscale @angelinalg @ryanaoleary
[serve] vllm example to serve llm models (ray-project#45430)
faf9d99 
Adds a documentation example using vLLM to serve LLM models on Ray
Serve.

This is a copy of ray-project#45325 + add a
build environment for ray serve + vllm.

Test:
- CI

---------

Signed-off-by: can <can@anyscale.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Signed-off-by: Cuong Nguyen <128072568+can-anyscale@users.noreply.github.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Co-authored-by: akshay-anyscale <122416226+akshay-anyscale@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aslonnie aslonnie aslonnie left review comments

@edoakes edoakes edoakes approved these changes

@angelinalg angelinalg angelinalg approved these changes

@shrekris-anyscale shrekris-anyscale Awaiting requested review from shrekris-anyscale shrekris-anyscale is a code owner

@zcin zcin Awaiting requested review from zcin zcin is a code owner

@GeneDer GeneDer Awaiting requested review from GeneDer GeneDer is a code owner

@akshay-anyscale akshay-anyscale Awaiting requested review from akshay-anyscale akshay-anyscale is a code owner

Assignees

@angelinalg angelinalg

Labels
go Trigger full test run on premerge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@can-anyscale @edoakes @carsonwang @aslonnie @angelinalg @akshay-anyscale