Add linear type facility

[aidancully]

Rendered

[Diggsey]

The "Linear" trait is the wrong way around: implementing a trait for a type should mean that type supports a greater number of operations, in this case it's the opposite.

It should be the same as the "Sized" trait: the trait should be called "Affine", or "NonLinear", so that implementing the trait allows one to implicitly drop the type. Otherwise, generic methods will need to have an implicit "!Linear" bound on all of their generic types, which would be bizarre.

Additionally, this removes alot of the redundant complexity from the RFC. All of the stuff about a new special kind of pointer to get around the problem of having linear types in non-linear types can be removed, just give the "NonLinear" trait a method which can make "self" non-linear type just before drop, then types can explicitly implement "NonLinear" even if they contain linear fields, while the compiler will automatically implement "NonLinear" for all types which don't contain linear fields.

[ftxqxd]

I’ve always liked the idea (which I’ve seen in a few places by different people) of using ?Drop as a bound instead of Linear. Basically, Drop would become an implicit bound (like Sized) on type parameters by default (unless : ?Drop was specified), and would have a slightly different meaning from today: it would mean that the type can be dropped at all (rather than meaning that the type implements Drop explicitly, which is rather useless as a bound). Linear types would of course not satisfy Drop.

[aidancully]

@Diggsey I think the new pointer type is necessary whether the type-trait signifies "Linear" (removing a capability from an Affine var) or "Affine" (adding a capability to a Linear var). In this design, the linear property of a variable is viral: any Linear field in a compound variable suffices to make the borrow-checker treat the compound itself as linear. Moving the linear fields out allows the compound variable to be treated as droppable, again. This means partial moves will be necessary to transform a linear variable into a non-linear variable. No current or proposed pointer types allow partial moves, so if I want drop to allow cleaning up a linear instance, by partially-moving its fields out such that it the variable can be treated as affine again, then a new pointer type is necessary, right?

For what it's worth, the proposed change to drop is the most time-critical part of this RFC, in that it is non-backwards-compatible, so we'd want it in before 1.0. (In that light, I've tried to pick the most forward-compatible, least intrusive syntax for these drop pointers.) The rest of the proposal is backwards-compatible with current rust, so there is more time to get the details right.

[Ericson2314]

@Diggsey and @P1start are correct. To do this elegantly/properly, linearity needs to be the default. This is why it is very important that this be done before 1.0. Treating the new Drop like Copy (warning missing impl) or Size (default bounds) are demonstrated methods to minimize the ergonomic impact.

@aidancully I think the solution to get the affine fields out is:

let (mut first_affine_field, _) = linear_val;
do_something_with(&mut first_affine_field);

The rules are simple: you can can deconstruct a linear type in this way iff all variants and fields are public. [If https://github.com//pull/736 which adapts the some rule, all the better for consistency and mutual-intuitiveness.]

---

I've mentioned it before, but I like Copy : Clone :: Drop : Destroy. Drop and Copy are the magic empty traits that really speak to the semantics of the type. Clone and Destroy have methods that do the obvious. Allowing implicit Destroy::destroy can be bikeshedded just like implicit Clone::clone. I'd be fine allowing the former but not the latter to minimize churn, but I think there is little enough consensus to warrant (eventually) making this configurable, perhaps even configurable per type.

[aidancully]

@Ericson2314 The problem I was having wasn't getting the affine fields out of a linear variable - that part is fine, and works just like you say EDIT: no it doesn't, I misread initially[1]. The problem I have is getting the linear fields out of an affine variable during the drop hook, in order to support making an affine container of a linear variable. The current drop function cannot support that, given the type of self that it receives:

[1]: In this design, it's always inappropriate to silently drop the linear portion of a linear value. The major intent behind linear values is to allow a library author to define the ways in which it is safe to dispose of one. In your example, it appears that the linear portion of a compound value is implicitly dropped by some mechanism that the library author cannot control. This would violate the intention behind the proposal...

// try to define an Affine variable that contains a Linear field by having
// the drop hook use a specific strategy for cleaning up the linear field:
struct AffineOfLinear(MakeLinear);
impl Drop for AffineOfLinear {
  fn drop(&mut self) {
    // the following cannot compile, because partial moves are disallowed from &mut pointers.
    self.0.consume();
    // but moves are also the only "safe" way of consuming Linear instances. (we explicitly
    // do not want a MakeLinear instance to be automatically consumed during the
    // drop hook.) So how do we consume the MakeLinear field? In the proposal, I
    // added a new DropPtr<T> type to allow partial moves from `self` during drop.
  }
}

// intention is to support:
{
  let x = AffineOfLinear(MakeLinear);
  // implicit drops are allowed on AffineOfLinear types.
  // x's drop hook will be called when the scope closes.
}
// we want to force that x's drop hook explicitly cleans up after its linear field.

I'm amenable to inverting the meaning of the type-trait so that Affine is the function (if not the name) of the trait that allows implicit drops, and !Affine means "Linear", especially if that increases the chances that the proposal gets accepted. But I don't think that that changes the backwards-compatibility story in the proposal, at least not on its own. (Changing the meaning of the Drop trait certainly would be a breaking change. But if we kept Drop as meaning "the drop hook will be called when the variable goes out of scope" and had a new "Affine" trait, meaning "implicit drops are possible", then that could be introduced in a backwards-compatible way). The one part of this proposal that can't be added in a backwards compatible way (as far as I can tell) is the change to the signature of the Drop::drop function.

[aidancully]

I experimented with removing the Linear trait and adding a Droppable trait, and liked the results (y'all were right :-)), so that's what the proposal now shows. My position remains flexible on this point, though.

[Ericson2314]

@aidancully
I don't see why getting self by &move, or even by value, wouldn't work. (abi means pointer will be used on big struct by value anyways.) The infinite recursion (or better yet err message with a #[no_implicit_destructor] attribute) forces the user to deconstruct self to avoid all dropping/destructors on self.

---

Perhaps the marker should be called NoDrop analogous to http://doc.rust-lang.org/std/marker/struct.NoCopy.html? [end bikshead]

[kvark]

I'd just like to point out that we really need this for gfx-rs/gfx#288 and thank @aidancully for consistently pushing and developing linear types feature. As for the ways to implement it, I'd prefer having minimal complexity over backwards compatibility, because we are still in alpha.

[kvark]

I do feel that this PR introduces way too much complexity though, at least for our simple needs. The way we need linear types to work is basically just have their automatic destruction forbidden. Thus, simply disallow Drop implementation, which automatically requires the explicit destruction code to decompose it.

Our case in gfx-rs just needs to enforce (at compile time) a specific spot for resource destruction, that's it.

I believe this could be achieved with less intrusion to the language. For (an oversimplified) example, we could just have a Droppable kind (similar to Sized) that is the default and assumes a default drop() implementation for the type, but only as long as all the contained types are Droppable as well (again, I assume Sized works the same way).

Please note that I'm new to compiler design and realize that my understanding may be to flawed and simplified.

[Ericson2314]

What's the point of keeping a drop that takes &mut self? A destructor isn't suppose to to borrow what it destroys. &move self is strictly more powerful, and self is almost strictly more powerful.

[Ericson2314]

I don't see what unwinding semantics have to do with the signature of drop. Drop also has little to do with releasing memory on the language level. Moving out of a stack var does not (nessisarily) mean the stack pointer is manipulated differently, and the frees in smart pointers are just an arbitray function call as far as the language is concerned.

[aidancully]

I'll have a bit more to say, here, but for what it's worth:

An earlier design had &move self for the argument to Drop::drop (based on a comment by @nikomatsakis that "the natural pointer type for drop() would be &my" (which @glaebhoerl later called &move). When I mentioned this on IRC, it seemed that this approach, and the approach where the drop argument is self, were both abandoned, since they caused an infinite loop when the self value went out of scope at the end of the drop call (thus causing another invocation of drop).

It's probably possible to change the drop behavior inside the drop hook itself... but it'd be weird if the same type (&move T) is treated differently in different routines (the drop hook vs. everywhere else). If the variable is treated differently by the compiler, it should be differently typed - then that new type gives a new reader a handle to understand how the language works. It'd also be possible to say "your fault for not cleaning up after self" if we get such an infinite loop. I'm sympathetic to that approach, but it would have the effect of making the language harder to deal with (no longer possible to write impl Drop for MyStruct { fn drop(self) { println!("Drop called!"); } }, since it would lead to that same infinite loop when self isn't cleaned up. And you couldn't even call std::mem::drop(self) within the drop routine to fix the issue! Thus, new pointer type.

[Ericson2314]

First of all, as I think we agree, the sneaky infinite loop is not a soundness problem, just an ergonomic problem: If panic!() inhabits all types, infinite recursion with moving should consume all types.

[bvssvni]

@aidancully Are you on IRC (#rust)? I'd like to ask a few questions to get a better picture...

[Ericson2314]

My solution starts with disabling custom drops everywhere. Period. Drop is a trait with no methods that just signifies whether you want magic silent compiler drop (affinity), or not (linearity). Next add a normal trait Destroy with destroy that takes self by value (or &move) and returns nothing. This Destroy is for custom dropping. Lastly, destructuring, whether explicitly or semi-explicity in a functional update, requires certain level of privacy, e.g. "all fields / variants be public".

At this point, let's review the situation:

• +1 Most types with #[deriving(Copy, Clone, Drop, Destroy)]. If you don't mind the former two, I'd hope you don't mind the latter. We could use NoCopy and NoDrop markers exclusively, but Rust has moved away from that with Copy.
• +1 The (core) language is simpler: there is no drop glue.
• +1 Working with types with silent drop is exactly like today.
• +0 It is silly to implement an effectful Destroy and Drop (same as effectful Clone and Copy).
• -1 Working with types that only implement destroy (most linear types, and everything with custom drop glue today, requires explicit .destroy(). Maybe good for MutexGuard, bad for Vec.
• +1 But, the above means little risk of accidentally infinite-recurring in a destroy method.
• -1 When Working with generic code that consumes stuff, one will want to add a Destroy bound. Non-consuming code doesn't need this, so I do not propose Destroy becomes a default bound*.
• +1 statics might no longer need any restrictions wrt Drop or Destroy (EDIT)

Room for improvement, but not the worst ergonomic fallout? Right?!

[Ericson2314]

How do we improve this ergonomic story? Finer grained implicits and warnings for all four traits Copy, Clone, Drop, Destroy:

I see potentially four categories:

1. Copy is sound, but probably not what you wanted (Random number generators)
2. Implicit Clone is nice to have, Drop is not implemented (Rc?)
3. Drop is sound, but probably not what you wanted (zeroing passwords or other sensitive data?)
4. Implicit Destroy is nice to have, Drop is not implemented (Vec)

The larger point is that currently both Copy and Drop are used both to provide an ergonomic story, like an operator trait, and say something deeply important about the semantics of the type. I think this is a mistake, and either additional traits or annotations can be used to signify membership of a type in (1, 2, or neither) and (3, 4, or neither). [Being in categories 1 and 2, or 3 and 4 does not make sense.]

[Ericson2314]

If annotating a type to show its membership in those four categories globally is too course, it different scopes can override the default per scope. The biggest use cases I can think of are:

• Ensuring no accidental recursion in the impl Destroy for a type that intends to allow the use of its custom destructor implicitly (e.g. Vec). The idea is by default dropping Vec will implicitly call destroy, but in the definition of Vec's destroy this is overridden so that this doesn't happen inadvertently in the definition of Destroy.
• Placating users with mutually incompatible desires/tolerances for implicit behavior.

These finer-grained granularity can be added backwards comparably, and I don't think users will implement types with custom destructors as ubiquitous and innocuous as Rc or Vec, so were my plan to be expected I would postpone this bit until post 1.0.

[Ericson2314]

The one exception to the coppying-destroying symmetry is that unwinding would want to call destroy if Destroy was implemented. So unfortunately while Destory is "denotationally" a normal trait, it would need special backend (unwinding) and frontend (implicit->explicit desugaring) support.

[aidancully]

I'd like to move the drop discussion to another RFC, so that the smaller piece can be considered independently, and possibly accepted for 1.0 without requiring the whole rest of this proposal to be accepted. I've started a thread on internals for discussion of this narrow point, maybe the design can be resolved there?

[aidancully]

Regarding the "complexity" concerns, I may not be am not the best person to judge them, but is it possible that this seems complex because of the, for lack of a better term, completeness of the design? The core proposal is bracketed by a characterization of its prerequisites (drop-pointers, the Finalize trait) and some additional proposals that seemed to fall out from adding linear types to the language (the "explicit_bounds" lint, refactoring the standard library with linear types in mind). This was deliberate, in that I thought the prerequisite-proposals might be better motivated by having an overall vision into which they fit, while the consequence-proposals would show that linear types could be made to fit cleanly into the language. This does mean there's significantly more text in the proposal to work through, and there are multiple design ideas at play, probably making it feel complex. But hopefully it's made up of simple parts, and hopefully those parts aren't narrowly targeted towards linear types, so that maybe this is essential complexity, rather than accidental?

[bvssvni]

@aidancully The summary makes sense to me now that I got a better picture of it. It doesn't seems to add more complexity than necessary, considering the alternative might be to make all types linear by default to be solved properly. I understand the drop problems must be dealt with one way or another, but I don't think this will hurt ergonomics.

+1 Assuming this works, and I don't know whether some minor things could be improved. I think that supporting this way or through similar functionality would solve the problems we have.

[pnkfelix]

postponing for some point after 1.0.

[nikomatsakis]

(See issue #814)

[glaebhoerl]

@pnkfelix Given the extensive discussion of backwards (in)compatibility issues in the RFC and comments, I think the postponement might have deserved some justification and those aspects some mention in it.

[pnkfelix]

@glaebhoerl the team discussed briefly, but its possible we overlooked backwards compatibility issues that would make postponement untenable. I will review the RFC and comment thread, and try to extract out a high-level summary of what we would need to do now to make doing this feasible later.

[pnkfelix]

@Ericson2314 I think your text above is massive because you did not include a newline between your paragraph and the ---- beneath it, so the ---- is not treated as a horizontal rule, and instead it emboldens the text above it (like an underline, I assume).

(I am going to take the liberty of editing your comment to fix the formatting.)

[pnkfelix]

@glaebhoerl okay, so obviously the fn drop signature is the big question mark with respect to backwards compatibility.

I had been musing about ways we might try to keep the current API while adding other destructor-enabling traits that would provide alternative signatures. But even if that were a palatable option (and it may well not be), one would still want all of the various destructor signatures to be composable, and that sounds like a harder nut to crack.

So, probably better to either put in forward-compatible changes now, or decide up front that we are not going to be supporting this kind of feature without deeper changes to the language and/or APIs of Rust 1.0.

[pnkfelix]

(I do appreciate @aidancully 's posting of http://internals.rust-lang.org/t/moves-from-self-during-the-drop-hook/1536, which seems to factor out the main backwards-compatibility concern. So I plan to focus my attention on that, i.e. evaluating the options given in the thread there and trying to figure out whether we can adopt any of them.)

[glaebhoerl]

I don't have a very clear understanding of the issues myself, just wanted to bring some attention to them. Thanks!

[Ericson2314]

@glaebhoerl Thanks for bringing up the backwards computability issues. @pnkfelix thanks for double checking the back-compat situation (and fixing my markdown error).

[1fish2]

Questions on what, why, how researchy, and criteria for adding features to Rust.

(1) Can you please explain the idea to ordinary software engineers who aren't math majors?

Web searches turn up more info that I don't understand and/or don't seem to fit, like "Use-Once" Variables on c2 and Substructural type system on Wikipedia. "Used exactly once" is not about drop().

Does "like sprinkling free() calls" mean the idea is to allow explicitly freeing resources in a non-LIFO order with the compiler checking that we didn't forget?

(2) Why is this compelling?

In the first example, "Force explicit mutex drop", the everyday way to move code outside the mutex is to move it outside that RAII block. That may require additional temporary state but that's life with locks.

Notably, avoiding deadlocks requires a globally consistent lock order and releasing locks in LIFO order. So enabling a different unlock order is ungood. (If we don't all switch from locking shared state to sending messages between threads, a lock-checker like Spin would be nice.)

The other example says "Now we must manually drop(data); and drop(r) here, othewise check will segfault. because data will already be dropped." How so? We must manually drop(data) that's already dropped? How will check() segfault now that it's finished? I missed the point of this example.

(3) Is this a research feature or a proven production feature?
Where has it been field-tested? Does it need iterating?

To quote the Java Language Spec

It is intended to be a production language, not a research language, and so, as C. A. R. Hoare suggested in his classic paper on language design, the design has avoided including new and untested features.

Contrast Java with Scala -- a research language to rapidly try ideas. Of course some ideas don't pan out (like implicits).

Rust has one big research experiment (ownership/borrowing). If it aims squarely to be a production language, it ought to hinge on that one and do other experiments in forks.

(4) Can this be simplified and made approachable, akin to the scope idea?

(5) Does it really have to be built into the type system? Is it better as a property of variables than a property of types?

(6) Can this be implemented as a compiler plug-in or a separate code analysis tool like the Spin model checker and the SPARK 2014 validation tools?

(7) What are the criteria for adding features to Rust? I suggest factors like high bang/buck, demonstrated broad utility and usability for Rust's target uses, worth the total costs & opportunity costs to the Rust ecosystem, not implementable in a library, and minimalist.

[bvssvni]

@1fish2 The idea is to allow more strict control of how objects are used in a library interface. An example from nature: Some quantities in a physics are preserved and should neither be duplicated or destroyed without being counter balanced with something else. Linear types gives you a tool to build APIs with this property. For example, when you are dealing with references to external resources, such as textures in GPU memory, which will cause memory leaks if not destroyed through a device object. It can also be used for transforming messages that keeps information that should never be lost. Because you can enforce this at compile time, you can make safer zero cost abstractions. It puts a constraint on the number of valid programs, and therefore must be part of the type system.

[pnkfelix]

@1fish2 wrote:

The other example says "Now we must manually drop(data); and drop(r) here, othewise check will segfault. because data will already be dropped." How so? We must manually drop(data) that's already dropped? How will check() segfault now that it's finished? I missed the point of this example.

I think the example is trying to describe a problem with eager-drop semantics. If we had eager-drop semantics, then the explicit drop call would keep the data alive to the end of the block (without it, the data would be dropped by the compiler much sooner, at least according to the rules of that other RFC; see the "source" link). And this RFC is saying that linear types would allow the code to be more self-documenting. But I'm inclined to say that given that we are almost certainly not going to switch to an eager-drop semantics, this particular example does not really drive me much towards the feature.

[aidancully]

@1fish2 @pnkfelix Yes, I believed this RFC addressed the major shortcoming in eager drop, in that eager drop didn't interact well with FFI on its own, but with linear types, it could be made to do so. I had hoped to see eager drop adopted in the language before 1.0, but since that isn't happening, this part of the RFC can be removed.

But that's certainly not the only reason I wanted the feature. I don't go into this in the RFC because it's a bigger lift, but I want to support intrusive structures, in which ownership of a containing structure can be moved via ownership of a field. If you do this kind of thing, then dropping the field-pointer is a bug: you need to get back to the container before you have something that can be dropped. So I wanted a linear Box, by which ownership could be transferred, but which could not be dropped: you'd need to convert back to the containing structure to get something droppable.

I also wanted to support more advanced release'ing facilities, so that a client is forced to make a decision about how a variable should be released. For example, I've got a message-passing architecture in which a request-message is sent to a receiver, the receiver handles the request, then returns the request back to its originator, but with the message modified to indicate success or failure status of the operation. With linear types, I could force the release operation to take a success: bool parameter, and know that the message will be well-formed as it returns to its source. It's possible for a drop to enforce that the message can be returned to its originator, but if we're forced to rely on drop alone, it would not be possible to make sure it's well-formed as it returns.

I currently have both of these ideas implemented in C and C++ in a several-hundred-KLOC embedded application at my day job. I have to simulate linearity because I'm using C and C++ (by using panics when linearity constraints might be violated), but the ideas have (IMO) worked very well for us. I went to the effort of producing the RFC for two main reasons:

1. (Primarily) I hope to be able to use Rust at my day job, some day, and this feature would make the language a much easier sell. (We still occasionally run into a panic when one of our linearity panics trips, and having this happen at compile-time rather than run-time would help our productivity. On the other hand, the language's tendency to encourage implicit drops would, I think, make it more likely for our developers to get used to the rust-style, so that implicit drops do not jump out during a review in the same way that a missing "free" would, so that inappropriate-drop bugs might actually become more likely in our environment. But that's just a guess, and there are probably ways to mitigate the problem.)
2. I hoped that demonstrating a solution to the major problem with eager drop would make it more likely that it be adopted (which is why I wanted to submit this before 1.0).

Of course, I'm not the only one interested in the facility, but those were my reasons.

[Ericson2314]

@1fish2 @pnkfelix @aidancully Related to intrusive data structures is that certain idioms of fine-grained locking are currently not possible to express/enforce in Rust.

struct Node<T> { payload: T, children: [Option<Box<SomeLock<Node<T>>>>; 2] }

Consider a tree with lock per node. To traverse the tree one locks a node, locks its child, then unlocks the parent (hand-over-hand locking I think this is sometimes called?). To remove a node one must have it locked (to ensure that nobody else has access to it) and have its parents locked (to ensure nobody else is trying to lock a lock that is about to be destroyed).

struct Node<T> { payload: T, children: [SomeLock<Option<Box<Node<T>>>>; 2] }

Well, first of all, the locking of the parents---and the problems it is supposed to solve--are a big jank red flag. The pointer to the node represents a capability to the node, and seeing that this is going to be tree and not the dag, might as well lock the pointer and not the node. This also can help avoid "dummy head nodes". Then just lock the owning pointer to remove the node -- simple as that. The rest of this comment is going to assume the pointers are locked instead of nodes because this seems less jank and less hopelessly incompatible with Rust, but I make no claims that doing this isn't somehow less performant or otherwise doesn't introduce other complications.

Assuming that works and has no performance problems or other downside (which I am not at all sure about), there is still a still a problem with just traversing the tree. Rust insists that one borrow a lock to lock it (which, don't get me wrong, is a normally a good idea). This means the to access a node, one must borrow its lock, which is owned by its parents so borrow that lock... all the way back to the root. This means that the tree is globally locked after all! A big reason for this is that locking a node gives one power over not just it, but subtree of it and nodes reachable by it, allowing one to all or large parts of that subtree en masse without further locking. The borrows ensure nobody else gets the locks, and therefore nobody can drop the subtree.

[Perhaps Arc could be used to get around the need for such borrowing by cloning the arcs but a) this is suboptimal b) when I tried to do this (granted using locked nodes and not locked pointers) I had nasty overlapping borrows due to the hand-over-handing which meant I was stuck with a (non-tail-)recursive function that probably held on to the locks--and thus was no better. Also RwLock provide another way to achieve fine grained locking, but RwLock + this is totally possible algorithmically and could be be strictly better. More generally I am trying to present the difficulties with a specific algorithm for the purpose of illustrating limits to the expressiveness of the language, so please forgive me for not considering every other way of doing things.]

I don't think there is any hope of getting Rust to understand this idiom "natively" (like it understands borrowing), but it would be nice to be able to encapsulate this idiom once so data structures and algorithms can be ensured to respect it with the type system. I don't have an exact plan for how this is done, but a key ingredient is to fake the "shallow" ownership the hand over hand locking uses in order to ensure the children of the node cannot be dropped with non-aliasable reference to the node (Box<Node<T>, &mut Node<T>, etc). Enforcing linearity is the way to ensure a nodes children cannot be dropped. With linearity I think just simple tricks like encapsulated fields and encapsulated casting can do the rest.

Hopefully that makes some sense.