PERF-#4445: Stop recomputing both indices for axis-wide applies. #4460
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #4460 +/- ##
==========================================
- Coverage 86.82% 86.19% -0.63%
==========================================
Files 222 222
Lines 18041 18100 +59
==========================================
- Hits 15664 15602 -62
- Misses 2377 2498 +121
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
|
|
|
YarShev
reviewed
May 18, 2022
| @@ -1864,6 +1864,7 @@ def apply_full_axis( | |||
| new_index=None, | |||
| new_columns=None, | |||
| dtypes=None, | |||
| func_may_change_complementary_index_size=True, | |||
There was a problem hiding this comment.
What if we would pass new row lengths and new column widths from the QC?
There was a problem hiding this comment.
The QC probably shouldn't know anything about this (partition lengths) though. However, what do you think on this?
There was a problem hiding this comment.
func_may_change_complementary_index_size looks a bit clumsy.
added 2 commits
May 25, 2022 13:32
…applies. Signed-off-by: mvashishtha <mahesh@ponder.io>
Signed-off-by: mvashishtha <mahesh@ponder.io>
mvashishtha
force-pushed
the
4445-recompute-single-index-for-apply
branch
from
68c4302
to
a1b861d
Compare
mvashishtha
changed the title
There was a problem hiding this comment.
@mvashishtha I like the idea of splitting the functionality like this, but maybe we need a new operator instead of handling it like this?
The difference between the new operator and the current |
|
If we do not want to pass lengths from the QC to the core dataframe, I would suggest renaming |
|
@mvashishtha For the algebra, typically there are two things that qualify an operator:
We typically want each operator to have certain guarantees without requiring too much information from upper layers. This is why I believe it should be a different operator, since both metadata and data can change at the same time, which isn't something the current operators explicitly support. |
|
@devin-petersohn The new param I introduced to modin/modin/core/dataframe/pandas/dataframe/dataframe.py Lines 2101 to 2102 in cca9468 |
|
@mvashishtha for now, let's keep this fix local to the axis-wide apply. we should revisit after. |
8 tasks
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: mvashishtha mahesh@ponder.io
What do these changes do?
Stop recomputing both indices for axis-wide applies. These changes make the slow computation in the snippet from #4445 (comment) asynchronous.
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date