FEAT-#7249: Add reload_modin feature
#7280
Conversation
Signed-off-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com>
YarShev
requested review from
devin-petersohn,
mvashishtha,
RehanSD,
vnlitvinov,
anmyachev,
dchigarev and
a team
as code owners
| if an execution engine has been shut down and | ||
| is going to be started up once again. | ||
| """ | ||
| modules = sys.modules.copy() |
There was a problem hiding this comment.
Why is there a copy here?
There was a problem hiding this comment.
Got an error without a copy.
RuntimeError: dictionary keys changed during iteration| modules = sys.modules.copy() | ||
| for name, module in modules.items(): | ||
| if name.startswith("modin"): | ||
| importlib.reload(module) |
There was a problem hiding this comment.
Isn't it enough to just do importlib.reload(modin)?
There was a problem hiding this comment.
It's not enough since we have to re-import all previously imported Modin modules.
There was a problem hiding this comment.
Considering different comments from https://stackoverflow.com/questions/28101895/reloading-packages-and-their-submodules-recursively-in-python, this approach is error prone if you restart the modules not in the correct order.
For example, this comment:
consider there are two submodules A and B. the objects in A depends on B. Suppose A is reloaded first, A would import the old B (since B has not reloaded). It results in a not fully updated A.
So the main question here is, can we rely on the order of modules in sys.modules?
More reliable would be the engine reinitialization function, which also resets the entire cache that relates to the engine.
What do these changes do?
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date