Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Make Debruijn indices zero-based #49840

Closed
wants to merge 2 commits into from

Conversation

muscar
Copy link

@muscar muscar commented Apr 10, 2018

This changes the DebruijnIndex type to be zero-based (resolves #49813).

@rust-highfive
Copy link
Collaborator

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @pnkfelix (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 10, 2018
@muscar
Copy link
Author

muscar commented Apr 10, 2018

This fails on my local machine when I run the tests with ./x.py test:

error: internal compiler error: librustc/infer/region_constraints/mod.rs:695: cannot relate bound region: ReLateBound(DebruijnIndex { depth: 0 }, BrAnon(0)) <= ReStatic
  --> libcore/panicking.rs:43:1
   |
43 | / pub fn panic(expr_file_line_col: &(&'static str, &'static str, u32, u32)) -> ! {
44 | |     // Use Arguments::new_v1 instead of format_args!("{}", expr) to potentially
45 | |     // reduce size overhead. The format_args! macro uses str's Display trait to
46 | |     // write expr, which calls Formatter::pad, which must accommodate string
...  |
51 | |     panic_fmt(fmt::Arguments::new_v1(&[expr], &[]), &(file, line, col))
52 | | }
   | |_^

thread 'main' panicked at 'Box<Any>', librustc_errors/lib.rs:488:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: rustc::util::common::panic_hook
   5: std::panicking::rust_panic_with_hook
   6: std::panicking::begin_panic
   7: rustc_errors::Handler::span_bug
   8: rustc::session::opt_span_bug_fmt::{{closure}}
   9: rustc::ty::context::tls::with_opt::{{closure}}
  10: rustc::ty::context::tls::with_context_opt
  11: rustc::ty::context::tls::with_opt
  12: rustc::session::opt_span_bug_fmt
  13: rustc::session::span_bug_fmt
  14: rustc::infer::region_constraints::RegionConstraintCollector::make_subregion
  15: rustc::infer::InferCtxt::sub_regions
  16: rustc::infer::outlives::obligations::TypeOutlives::components_must_outlive
  17: rustc::infer::outlives::obligations::TypeOutlives::type_must_outlive
  18: rustc::infer::outlives::obligations::<impl rustc::infer::InferCtxt<'cx, 'gcx, 'tcx>>::process_registered_region_obligations
  19: rustc_typeck::check::regionck::<impl rustc_typeck::check::FnCtxt<'a, 'gcx, 'tcx>>::regionck_item
  20: rustc::ty::context::tls::with_related_context
  21: rustc::infer::InferCtxtBuilder::enter
  22: rustc_typeck::check::wfcheck::check_item_well_formed
  23: rustc::dep_graph::graph::DepGraph::with_task_impl
  24: rustc::ty::context::tls::with_related_context
  25: rustc::ty::maps::<impl rustc::ty::maps::queries::check_item_well_formed<'tcx>>::force_with_lock
  26: rustc::ty::maps::<impl rustc::ty::maps::queries::check_item_well_formed<'tcx>>::try_get
  27: rustc::ty::maps::TyCtxtAt::check_item_well_formed
  28: rustc::ty::maps::<impl rustc::ty::maps::queries::check_item_well_formed<'tcx>>::ensure
  29: rustc::hir::Crate::visit_all_item_likes
  30: rustc::util::common::time
  31: rustc_typeck::check_crate
  32: rustc::ty::context::tls::enter_context
  33: <std::thread::local::LocalKey<T>>::with
  34: rustc::ty::context::TyCtxt::create_and_enter
  35: rustc_driver::driver::compile_input
  36: rustc_driver::run_compiler_impl
  37: syntax::with_globals
  38: rustc_driver::run
  39: rustc_driver::main
  40: std::rt::lang_start::{{closure}}
  41: std::panicking::try::do_call
  42: __rust_maybe_catch_panic
  43: std::rt::lang_start_internal
  44: main
  45: __libc_start_main
  46: _start

Any suggestions on how to tackle this would be appreciated :).

@muscar muscar changed the title Make Debruijn indices zero-based [WIP] Make Debruijn indices zero-based Apr 10, 2018
@TimNN
Copy link
Contributor

TimNN commented Apr 10, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
Resolving deltas: 100% (614042/614042), completed with 4868 local objects.
---
[00:00:46] configure: rust.quiet-tests     := True
---
[00:21:43]   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name core libcore/lib.rs --color always --error-format json --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=64878136ec7adadb -C extra-filename=-64878136ec7adadb --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/release/deps` (exit code: 101)
[00:21:43] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:21:43] expected success, got: exit code: 101
[00:21:43] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1085:9
[00:21:43] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:21:43] travis_fold:end:stage1-std
[00:21:43] travis_time:end:stage1-std:start=1523366143948361927,finish=1523366161171037840,duration=17222675913
[00:21:43] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:21:43] Build completed unsuccessfully in 0:16:45
[00:21:43] make: *** [all] Error 1
[00:21:43] Makefile:28: recipe for target 'all' failed
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:13dff2cc:start=1523366161637153135,finish=1523366161643385547,duration=6232412
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:30ff1b28
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:30ff1b28:start=1523366161648856525,finish=1523366161654690857,duration=5834332
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:18a3e3c4
$ dmesg | grep -i kill
[   10.409578] init: failsafe main process (1092) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@@ -1156,7 +1156,6 @@ impl<'a, 'tcx, 'gcx> PolyExistentialProjection<'tcx> {

impl DebruijnIndex {
pub fn new(depth: u32) -> DebruijnIndex {
assert!(depth > 0);
Copy link
Member

@lnicola lnicola Apr 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does DebruijnIndex::from_depth need to be changed to remove the shift by 1?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I'll give it a go locally.

@TimNN
Copy link
Contributor

TimNN commented Apr 10, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
Resolving deltas: 100% (613567/613567), completed with 4873 local objects.
---
[00:00:48] configure: rust.quiet-tests     := True
---
[00:22:25] thread 'main' panicked at 'attempt to subtract with overflow', librustc/ty/fold.rs:481:42
[00:22:25] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:22:26] error: Could not compile `core`.
[00:22:26]
[00:22:26] Caused by:
[00:22:26]   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name core libcore/lib.rs --color always --error-format json --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=64878136ec7adadb -C extra-filename=-64878136ec7adadb --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/release/deps` (exit code: 101)
[00:22:26] warning: build failed, waiting for other jobs to finish...
[00:22:29] error: build failed
[00:22:29] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:22:29] expected success, got: exit code: 101
[00:22:29] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1085:9
[00:22:29] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:22:29] travis_fold:end:stage1-std
[00:22:29] travis_time:end:stage1-std:start=1523374612033433137,finish=1523374628614435963,duration=16581002826
[00:22:29] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:22:29] Build completed unsuccessfully in 0:17:06
[00:22:29] Makefile:28: recipe for target 'all' failed
[00:22:29] make: *** [all] Error 1
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:2daa37f0:start=1523374629252340580,finish=1523374629261588778,duration=9248198
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:1cc367af
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:1cc367af:start=1523374629270437201,finish=1523374629280286749,duration=9849548
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0c010401
$ dmesg | grep -i kill
[   11.744187] init: failsafe main process (1093) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@muscar muscar force-pushed the zero-based-debruijn-indices branch from df3c32c to 8cdfe8c Compare April 10, 2018 15:47
@TimNN
Copy link
Contributor

TimNN commented Apr 10, 2018

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
Resolving deltas: 100% (613567/613567), completed with 4873 local objects.
---
[00:00:49] configure: rust.quiet-tests     := True
---
[00:21:53] error: internal compiler error: librustc/infer/region_constraints/mod.rs:695: cannot relate bound region: ReLateBound(DebruijnIndex { depth: 0 }, BrAnon(0)) <= ReStatic
[00:21:53]   --> libcore/panicking.rs:43:1
[00:21:53]    |
[00:21:53] 43 | / pub fn panic(expr_file_line_col: &(&'static str, &'static str, u32, u32)) -> ! {
[00:21:53] 44 | |     // Use Arguments::new_v1 instead of format_args!("{}", expr) to potentially
[00:21:53] 45 | |     // reduce size overhead. The format_args! macro uses str's Display trait to
[00:21:53] 46 | |     // write expr, which calls Formatter::pad, which must accommodate string
[00:21:53] ...  |
[00:21:53] 51 | |     panic_fmt(fmt::Arguments::new_v1(&[expr], &[]), &(file, line, col))
---
[00:21:53]   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name core libcore/lib.rs --color always --error-format json --crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=64878136ec7adadb -C extra-filename=-64878136ec7adadb --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/release/deps` (exit code: 101)
[00:21:53] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:21:53] expected success, got: exit code: 101
[00:21:53] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1085:9
[00:21:53] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:21:53] travis_fold:end:stage1-std
[00:21:53] travis_time:end:stage1-std:start=1523376692769816631,finish=1523376709307185145,duration=16537368514
[00:21:53] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:21:53] Build completed unsuccessfully in 0:16:29
[00:21:53] Makefile:28: recipe for target 'all' failed
[00:21:53] make: *** [all] Error 1
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:01c35728:start=1523376709848438196,finish=1523376709854123791,duration=5685595
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:03fab481
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:03fab481:start=1523376709859608763,finish=1523376709865107600,duration=5498837
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:2624761f
$ dmesg | grep -i kill
[   10.383662] init: failsafe main process (1097) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@nikomatsakis
Copy link
Contributor

r? @nikomatsakis

Copy link
Contributor

@nikomatsakis nikomatsakis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what's causing the failures, looks like some binder not catching. If you want me to help debug, I can, but I don't know what to do beyond dumping RUST_LOGs and trying to see where things are going astray.

@@ -137,14 +137,14 @@ impl Region {
match self {
Region::LateBound(debruijn, id, origin) => Region::LateBound(
ty::DebruijnIndex {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pre-existing, but from_depth sure needs a comment =) let me try to remember what it's purpose is...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it seems like it is basically a variant of shifted, but it goes in the opposite direction. I think it should probably be renamed; chalk calls this shift_down, I think. In any case, a comment would be good:


Removes depth level of binders from this region. region must not be bound within those binders or else this will panic.


To that end, up/down are not especially intuitive names. Maybe something like shift_out or pull_out (from binders)? We would then rename shifted to shift_in or push_down (through binders)? Maybe remove_binders and in_binders?

Copy link
Author

@muscar muscar Apr 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have a look and rename these :D Thanks for the hints.

@muscar
Copy link
Author

muscar commented Apr 11, 2018

Thanks for looking into this @nikomatsakis :). I'll keep working on this, and I'll let you know if I need help.

@nikomatsakis
Copy link
Contributor

@muscar ok -- I didn't do any digging yet, but let me know.

@nikomatsakis nikomatsakis added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 12, 2018
@muscar
Copy link
Author

muscar commented Apr 13, 2018

@nikomatsakis I'll dig further during the weekend. Keep you posted.

@muscar
Copy link
Author

muscar commented Apr 18, 2018

@nikomatsakis Hi, I'm a bit stuck on this one. It panics on this line: https://github.com/muscar/rust/blob/master/src/librustc/infer/region_constraints/mod.rs#L694 with

cannot relate bound region: ReLateBound(DebruijnIndex { depth: 0 }, BrAnon(0)) <= ReStatic

Since the left hand side is a late bound region (with a debruijn index of 0), I assume it wasn't resolved to another type of region in one of the revious passes. Since this happens after the index change, I guess some other piece of code assumes debruijn indexes are 1-based so it doesn't catch the late bound region. I don't know enough about the code to say if this is the case or not, it's just a guess. I'd be happy to work on this if you can point me in the right direction (I'm afraid I don't have a lot of time to familiarise myself with the code right now :().

@fabric-and-ink
Copy link
Contributor

fabric-and-ink commented Apr 21, 2018

Printing some debug information shows this:

 INFO 2018-04-21T18:13:20Z: rustc::traits::fulfill: selecting trait `Binder(TraitPredicate(<I as iter::traits::FusedIterator>))` at depth 0 yielded Err
 INFO 2018-04-21T18:13:20Z: rustc::traits::fulfill: selecting trait `Binder(TraitPredicate(<I as iter::traits::FusedIterator>))` at depth 0 yielded Err
 INFO 2018-04-21T18:13:20Z: rustc::traits::fulfill: selecting trait `Binder(TraitPredicate(<A as iter_private::TrustedRandomAccess>))` at depth 0 yielded Err
 INFO 2018-04-21T18:13:20Z: rustc::traits::fulfill: selecting trait `Binder(TraitPredicate(<B as iter_private::TrustedRandomAccess>))` at depth 0 yielded Err
 INFO 2018-04-21T18:13:20Z: rustc::traits::fulfill: selecting trait `Binder(TraitPredicate(<A as cmp::Ord>))` at depth 0 yielded Err
 INFO 2018-04-21T18:13:20Z: rustc::traits::fulfill: selecting trait `Binder(TraitPredicate(<T as marker::Copy>))` at depth 0 yielded Err

(Don't know what that means. Just want to help :) )

Edit: The information is generated here.

@muscar
Copy link
Author

muscar commented Apr 24, 2018

Thanks @fabric-and-ink. I ran the compiler with extra logs, and got the same errors. I'll try to add extra logging, and see if I can make sense of this.

@nikomatsakis
Copy link
Contributor

@muscar indeed, I'm not sure which code is wrong, but clearly something is. I'll try and do a local build and see if I can spot it.

@nikomatsakis
Copy link
Contributor

OK -- I started digging in and I think i've found the problem. A number of bits of the code prefer to use the term "depth", which basically means "number of enclosing binders (inclusive)". This is handy because 0 means "none". So e.g. they will start with a depth of 0 then, when they see a binder, increase it to 1. Then they expect to see debruijn indices at this depth (1) or lower -- that means the value is bound within the binders we've traversed.

Some of the code then does things like debruijn_index.depth > depth to check for things that "escape" the depth. But this is no longer quite right.

I think actually we should probably convert the code that currently tracks "present depth" so that it is tracking a DebruijnIndex. It would start at 0 -- the idea would be that there is kind of an implicit "outermost binder" that binds everything, and we are tracking its index -- this starts off as zero, but when we pass through a binder, it increments to 1. So if we see an index X that is strictly less than this index B, it must be one of the binders we've passed through; if X >= B, on the other hand, it refers to some binders we've not seen. This is basically equivalent to what we today but with a subtle "shift" in perspective, I suppose.

Anyway i'm trying to decide how much of this to hack and how much to leave to you @muscar =) this kind of change is sort of hard to do piecemeal, you lose track of what you have changed and what the other has done.

@nikomatsakis
Copy link
Contributor

@muscar so -- I'm concerned that if I push much more on this, I will wind up kind of "taking over" this PR. I was wondering if you'd be up for scheduling a slot to "pair program" on this remotely for a bit?

@nikomatsakis
Copy link
Contributor

I pushed my WIP commit here:

https://github.com/nikomatsakis/rust/tree/zero-based-debruijn-indices

As you'll see if you browse that commit, I also started doing some renamings and refactorings (e.g., renaming shift to shift_in). This was partly as a way to ensure that I am "visiting" each part of the code as I go, so that it's hard to overlook things.

I sort of remember that I had decided to take a slightly different approach than that commit, though, in particular eliminating the idea of "depth".

@shepmaster
Copy link
Member

Ping from triage @muscar ! Will you have time to work on this soon or to schedule time with @nikomatsakis ?

@nikomatsakis
Copy link
Contributor

Closing in favor of #50475, which is taking a more gradual approach. @muscar -- do ping me if you get more time!

@muscar
Copy link
Author

muscar commented May 11, 2018

@nikomatsakis First of all sorry for the lack of response. I got tied up at work, and looking for a new flat. Thanks for all the help with this--even if it didn't go anywhere. I can see why people say the Rust community is so welcoming. I hope this gets fixed. I'll ping you if I get more time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor DebruijnIndex to be 0-based
8 participants