Make create_def a side effect instead of marking the entire query as always red #115613

oli-obk · 2023-09-06T16:25:48Z

Before this PR:

query A creates def id D
query A is marked as depending on the always-red node, meaning it will always get re-run
in the next run of rustc: query A is not loaded from the incremental cache, but rerun

After this PR:

query A creates def id D
query system registers this a side effect (just like we collect diagnostics to re-emit them without running a query)
in the next run of rustc: query A is loaded from the incremental cache and its side effect is run (thus re-creating def id D without running query A)

r? @cjgillot

TODO:

need to make feeding queries a side effect, too. At least ones that aren't written to disk.
need to re-feed the def_span query
many more tests

cjgillot · 2023-09-06T16:36:33Z

Another tricky case:

ensure() query A -> executes side-effect from the cache;
fetch query A's result -> calls the provider for A -> gets to create_def -> ?

We wouldn't want to create 2 definitions where we only ask for one.

oli-obk · 2023-09-06T16:41:29Z

Huh... why does that happen? Shouldn't we already be getting weird diagnostics in that case?

cjgillot · 2023-09-06T16:50:01Z

This happens if we don't have the result of A in the on-disk cache, but still need it later.
IIUC, diagnostic deduplication catches it, so there is no observable effect.

cjgillot · 2023-09-06T16:51:45Z

More precisely: the first call is ensure(), so we don't attempt to compute the result, but still mark the dep-node as green and re-execute side effects. The second call is get(), so we need the result, we don't find it in the on-disk cache, and compute it the only way we know, by calling the provider function.

cjgillot · 2023-09-06T16:54:43Z

The logic is the fallback case in try_load_from_disk_and_cache_in_memory.

oli-obk · 2023-09-06T18:04:09Z

Thanks. I really need to dig into ensure and all its behaviours.

oli-obk · 2023-09-07T09:36:55Z

More precisely: the first call is ensure(), so we don't attempt to compute the result, but still mark the dep-node as green and re-execute side effects. The second call is get(), so we need the result, we don't find it in the on-disk cache, and compute it the only way we know, by calling the provider function.

I did some testing and a code dive, and I don't think that's what's happening.

ensure does not execute side effects. It checks if something is in the cache, and if not, executes that query immediately. The cache lookup itself doesn't perform any actions but set up the dep graph dependency in case of a cache hit.

cjgillot · 2023-09-07T10:56:29Z

ensure() calls get_query_incr with QueryMode::Ensure, which calls ensure_must_run, which calls try_mark_green, which calls try_mark_previous_green, which calls emit_side_effects.

oli-obk · 2023-09-07T11:12:14Z

yay, with this hint I was able to produce an example that actually exhibits an issue

index out of bounds: the len is 9 but the index is 9

oh wait... I even have this issue without doing any other changes to rustc. So it's not even ensure related yet

bors · 2023-09-22T03:24:02Z

☔ The latest upstream changes (presumably #115920) made this pull request unmergeable. Please resolve the merge conflicts.

cjgillot · 2023-09-25T10:41:55Z

The difficulty is to know when to skip creating the DefId and reuse the one created by side-effect replay.

What about adding a new variant Replay to TaskDepsRef?
That variant would hold the list of definitions created by this query in the previous invocation. The nth call to create_def in the query would return the nth DefId in that list.

oli-obk · 2023-09-25T15:04:48Z

Thanks! I was thinking about doing

The nth call to create_def in the query would return the nth DefId in that list.

but didn't know how. I'll investigate the TaskDepsRef solution you hinted at.

bors · 2024-02-16T10:24:59Z

☔ The latest upstream changes (presumably #120486) made this pull request unmergeable. Please resolve the merge conflicts.

oli-obk · 2024-02-16T17:04:26Z

@cjgillot I implemented replaying, and that fixes the issues I was able to coax out of incremental tests, could you have a look? I'll keep working on it and adding more tests, but I think I could benefit from a review

@bors try @rust-timer queue

Make create_def a side effect instead of marking the entire query as always red Before this PR: * query A creates def id D * query A is marked as depending on the always-red node, meaning it will always get re-run * in the next run of rustc: query A is not loaded from the incremental cache, but rerun After this PR: * query A creates def id D * query system registers this a side effect (just like we collect diagnostics to re-emit them without running a query) * in the next run of rustc: query A is loaded from the incremental cache and its side effect is run (thus re-creating def id D without running query A) r? `@cjgillot` TODO: * [ ] need to make feeding queries a side effect, too. At least ones that aren't written to disk. * [ ] need to re-feed the `def_span` query * [ ] many more tests

bors · 2024-06-05T15:44:05Z

⌛ Trying commit c490a96 with merge d30f6ef...

bors · 2024-06-05T16:08:49Z

💔 Test failed - checks-actions

oli-obk · 2024-06-05T16:25:14Z

@bors try @rust-timer queue

rust-timer · 2024-06-05T16:25:18Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2024-06-05T16:26:24Z

⌛ Trying commit c490a96 with merge 9b0899e...

Make create_def a side effect instead of marking the entire query as always red Before this PR: * query A creates def id D * query A is marked as depending on the always-red node, meaning it will always get re-run * in the next run of rustc: query A is not loaded from the incremental cache, but rerun After this PR: * query A creates def id D * query system registers this a side effect (just like we collect diagnostics to re-emit them without running a query) * in the next run of rustc: query A is loaded from the incremental cache and its side effect is run (thus re-creating def id D without running query A) r? `@cjgillot` TODO: * [ ] need to make feeding queries a side effect, too. At least ones that aren't written to disk. * [ ] need to re-feed the `def_span` query * [ ] many more tests

bors · 2024-06-05T16:50:07Z

💔 Test failed - checks-actions

Some minor query system cleanups * Improves diagnostics on conflicting query flags * removes unnecessary impls * `track_caller` pulled out of rust-lang#115613

Rollup merge of rust-lang#126035 - oli-obk:query_macro_errors, r=fmease Some minor query system cleanups * Improves diagnostics on conflicting query flags * removes unnecessary impls * `track_caller` pulled out of rust-lang#115613

bors · 2024-06-07T00:47:01Z

☔ The latest upstream changes (presumably #126104) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-06-07T11:38:16Z

☔ The latest upstream changes (presumably #125918) made this pull request unmergeable. Please resolve the merge conflicts.

…o we can reliably recreate the same `DefId` over and over again

cjgillot · 2024-06-09T16:06:53Z

compiler/rustc_hir_analysis/src/lib.rs

+            let _ = tcx.eval_static_initializer(item_def_id);
+        }
+    });
+


Why is this block necessary? The previous calls tcx.ensure().eval_static_initializer.

Just test code to replicate the bug I had initially, it should not land

cjgillot · 2024-06-09T16:10:19Z

compiler/rustc_middle/src/ty/context.rs

        //
        // This call also writes to the value of `source_span` and `expn_that_defined` queries.
        // This is fine because:
        // - those queries are `eval_always` so we won't miss their result changing;
        // - this write will have happened before these queries are called.


Stale comment (pre-existing).

cjgillot · 2024-06-09T16:12:20Z

compiler/rustc_middle/src/ty/context.rs

+                TaskDepsRef::Replay { prev_side_effects, created_def_ids } => {
+                    trace!(?created_def_ids, "replay side effects");
+                    trace!("num_defs : {}", prev_side_effects.definitions.len());
+                    let index = created_def_ids.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                    let prev_info = &prev_side_effects.definitions[index];
+                    let def_id = self.untracked.definitions.read().local_def_path_hash_to_def_id(
+                        prev_info.hash,
+                        &"should have already recreated def id in try_mark_green",
+                    );
+                    assert_eq!(prev_info.data, data);
+                    assert_eq!(prev_info.parent, parent);
+                    def_id
+                }


This logic is subtle, we should definitely document it somewhere. Here, or on DefIdInfo struct?

bors · 2024-06-14T05:06:13Z

☔ The latest upstream changes (presumably #126439) made this pull request unmergeable. Please resolve the merge conflicts.

Some minor query system cleanups * Improves diagnostics on conflicting query flags * removes unnecessary impls * `track_caller` pulled out of rust-lang/rust#115613

rustbot assigned cjgillot Sep 6, 2023

cjgillot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 9, 2023

oli-obk mentioned this pull request Sep 16, 2023

Tracking Issue for infallible promotion #80619

Closed

2 tasks

oli-obk mentioned this pull request Oct 27, 2023

Create definitions for promoted constants. #111693

Closed

oli-obk mentioned this pull request Dec 5, 2023

feed def_span in resolver #118633

Closed

oli-obk force-pushed the create_def_forever_red branch from 6f8f71c to a7f29ee Compare February 14, 2024 12:42

oli-obk force-pushed the create_def_forever_red branch 2 times, most recently from 6d6b1eb to 6831868 Compare February 16, 2024 17:00

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 16, 2024

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 5, 2024

oli-obk mentioned this pull request Jun 5, 2024

Some minor query system cleanups #126035

Merged

This comment has been minimized.

Sign in to view

oli-obk force-pushed the create_def_forever_red branch from c490a96 to d428f0a Compare June 7, 2024 05:42

oli-obk added 5 commits June 7, 2024 14:32

Add regression test

e7390dc

Generalize query side effect handling to make future additions easier

b3121e0

Recreate DefIds when a cached query gets replayed

350476c

Reproduce issue

874f34b

Also store the DefPathHash as part of a DefId creation side effect, s…

972210d

…o we can reliably recreate the same `DefId` over and over again

oli-obk force-pushed the create_def_forever_red branch from d428f0a to 972210d Compare June 7, 2024 14:32

cjgillot reviewed Jun 9, 2024

View reviewed changes

cjgillot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jun 16, 2024

oli-obk mentioned this pull request Jul 25, 2024

Accelerate GVN a little #126991

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make create_def a side effect instead of marking the entire query as always red #115613

Make create_def a side effect instead of marking the entire query as always red #115613

oli-obk commented Sep 6, 2023 •

edited

Loading

cjgillot commented Sep 6, 2023

oli-obk commented Sep 6, 2023

cjgillot commented Sep 6, 2023

cjgillot commented Sep 6, 2023

cjgillot commented Sep 6, 2023

oli-obk commented Sep 6, 2023

oli-obk commented Sep 7, 2023

cjgillot commented Sep 7, 2023

oli-obk commented Sep 7, 2023 •

edited

Loading

bors commented Sep 22, 2023

cjgillot commented Sep 25, 2023

oli-obk commented Sep 25, 2023

bors commented Feb 16, 2024

oli-obk commented Feb 16, 2024

This comment has been minimized.

bors commented Jun 5, 2024

This comment has been minimized.

bors commented Jun 5, 2024

oli-obk commented Jun 5, 2024

rust-timer commented Jun 5, 2024

bors commented Jun 5, 2024

This comment has been minimized.

bors commented Jun 5, 2024

bors commented Jun 7, 2024

bors commented Jun 7, 2024

cjgillot Jun 9, 2024

oli-obk Jun 9, 2024

cjgillot Jun 9, 2024

cjgillot Jun 9, 2024

bors commented Jun 14, 2024

Make create_def a side effect instead of marking the entire query as always red #115613

Are you sure you want to change the base?

Make create_def a side effect instead of marking the entire query as always red #115613

Conversation

oli-obk commented Sep 6, 2023 • edited Loading

cjgillot commented Sep 6, 2023

oli-obk commented Sep 6, 2023

cjgillot commented Sep 6, 2023

cjgillot commented Sep 6, 2023

cjgillot commented Sep 6, 2023

oli-obk commented Sep 6, 2023

oli-obk commented Sep 7, 2023

cjgillot commented Sep 7, 2023

oli-obk commented Sep 7, 2023 • edited Loading

bors commented Sep 22, 2023

cjgillot commented Sep 25, 2023

oli-obk commented Sep 25, 2023

bors commented Feb 16, 2024

oli-obk commented Feb 16, 2024

This comment has been minimized.

bors commented Jun 5, 2024

This comment has been minimized.

bors commented Jun 5, 2024

oli-obk commented Jun 5, 2024

rust-timer commented Jun 5, 2024

bors commented Jun 5, 2024

This comment has been minimized.

bors commented Jun 5, 2024

bors commented Jun 7, 2024

bors commented Jun 7, 2024

cjgillot Jun 9, 2024

Choose a reason for hiding this comment

oli-obk Jun 9, 2024

Choose a reason for hiding this comment

cjgillot Jun 9, 2024

Choose a reason for hiding this comment

cjgillot Jun 9, 2024

Choose a reason for hiding this comment

bors commented Jun 14, 2024

oli-obk commented Sep 6, 2023 •

edited

Loading

oli-obk commented Sep 7, 2023 •

edited

Loading