Executable Justfiles

[lylemoffitt]

Motivation

I use Make to make simple script tools. I find it's much easier to use (and maintain) than my previous solution -- bash switch blocks. Recently I've been porting these Makefiles to Just. But, Just has one weakness, I can't make a single file executable like I can with shell scripts. This means that when I want to distribute my tools to others I have to explain how to them how to use them, how it's like a Makefile, and how it's not. Being similar to a Makefile, creates friction, as I find that many devs don't like seeing Makefiles for things other than building code. To me the structure of a Makefile is ideal for writing composable scripts. For more complex tooling I even have multiple Makefiles in a directory tree acting like modules, separating out different common utilities into each file. If you think that this sounds like madness, take a look at BuildRoot. It's a production codebase that's almost entirely written in Makefiles!

I also spend some amount of time using and maintaining a large recursive-make build script. This may cause some to shudder, but that's just how it is. One day, I'd like to port this to Just, as I think it has the potential to smooth out many of the developer pain points. The similarity in syntax makes this especially appealing over other alternatives that would require a larger rewrite. However, I think Just lacks some really killer features that would make this work more justifiable (pun intended).

Features

Just has many advantages over Make as pointed out by its docs, but I suggest here some features that could make it even more useful for writing declarative tooling scripts and handling complex builds. Though they may seem unrelated, they are really two sides of the same coin.

Top Level Shebang

This is very simple, and at most would require some low-level changes to how Just handles file inputs. What I want to support is having a line like the following at the top of the file.

#!/usr/bin/env just <args> --justfile

This should look and act just like a regular shebang you may see in a single-file python/js/shell script. The <args> is a placeholder for whatever args Just might need to support this feature. For example, the way Just is written right now it would likely need --working-directory to be supplied; though it could easily default to the CWD.

Currently this kind of usage is explicitly blocked with an error:

error: `#!` is reserved syntax outside of recipes
  |
1 | #!/usr/bin/env just --working-directory ./ --justfile
  | ^

I understand this is likely done to avoid confusion with the shebangs allowed in recipes, but it shouldn't be too hard to disambiguate, especially if you only allow shebangs on the first line. I think dropping this error is probably about all you need to make this work.

Alternate Naming Convention

If a justfile is to be executable, it makes sense to allow for it to be named something other than Justfile. Here I'd suggest the *.just extension. Supporting this may be more about editor support more than anything, but within Just I'd at least hope that it would support opening *.just files by default if no Justfiles or justfile is present in in the CWD.

Ideally, I'd hope that multiple *.just files could be selected from based on target, logically acting like one large Justfile. Collisions between targets in separate files could be disambiguated with a fully qualified (Make compatible) target like path/to/filename@target. See below for more.

Inter-file Targets

The dream-usage I'm aiming for is that one could execute a top-level *.just file that could call out to other "child" *.just files by specifying target alone. This would be similar to the recursive make idiom, except with a few huge advantages.

1. Composition: Targets of another Justfile can be specified as dependent targets of a recipe alone. This allows them to be interleaved with other targets in the same file, leading to concise and introspectable composition.

get-build-set THING: 
    getters/thing.just@get {{THING}} \
    build {{THING}} \
    setters/thing.just@set {{THING}} \

2. Target Inlining: Using something like$(MAKE) to execute dependent targets makes those actions opaque and increases process overhead. In a large build, the cost of creating all those extra processes can be rather burdensome. Hoisting the target up into the recipe makes it much easier to introspect the dependency graph and perform optimizations.

3. Introspection: While you'd probably only want to supply targets from the CWDs file/s by default when --list or --summary is used, this behavior could be changed with a --recursive flag. The upshot of this being that the entire graph can be inspected and debugged by the programmer, without having to execute any recipes. Anyone who's spent time working with a recursive-make build knows how much of a lifesaver this would be.

[casey]

Heyo! Thanks for your thoughtful comments!

I think allowing a shebang in the first level Justfile is a good idea. You're right that it's illegal to avoid confusion with the #! in recipes, but I didn't consider how this would disallow executable justfiles. Would you like to submit a patch?

#355 should land soon, which would make it unnecessary to supply --working-directory, unless you want to use a different directory from the one the justfile is in, like ./.

What do you see executable justfiles as being good for? Just curious, since I don't see any downside in allowing them.

...within Just I'd at least hope that it would support opening *.just files by default if no Justfiles or justfile is present in in the CWD.

I think that this might be problematic. For example, if there's a file called Justfile one directory up, but a foo.just file in the current directory, which one should be executed? What about if there are multiple .just files in the current directory, which one of them should be executed?

I'm definitely open to the idea of allowing multiple files. Recipes can't be slash separated though (like path/to/justfile@recipe) since that's used in the CLI. (You can invoke bar in sub/directory/Justfile with just sub/directory/bar, and recipe foo in ../Justfile with just ../foo). I was thinking that recipes could be dot or :: delimited, since those are illegal characters in recipe names. Unless necessary, I'd prefer to require @ before the recipe name, since it should be enough to know it's in another justfile and resolve it if a . is present.

I'm not sure how I feel about loading justfiles without some kind of mod or import statement, just since I usually like things to be explicit, but I could be convinced otherwise if there's good reason.

I think it would be good to figure out a minimal design that makes multiple files useful, to avoid trying to do too much at once.

Unfortunately, I'm very busy with other responsibilities for the time being, so I wouldn't be able to do much in the way of implementation, and reviews and merges would probably be slow.

[lylemoffitt]

Hey! Thanks for the thoughtful reply, and sorry for the long delay. Work/holidays and whatnot.

I'm glad you like the shebang. It did seem like a design oversight, but I wasn't sure if there had been a strong reason to keep it out. I would love to submit a PR, but I don't know rust. I've always fancied learning it, but never could quite find the time. Maybe I'll find sometime to learn what i need in the coming month or two.

---

What do you see executable justfiles as being good for? Just curious, since I don't see any downside in allowing them.

Essentially, I see it as a script with self-documenting CLI. The tasks/targets are composable functions. A justfile itself can be either a single executable or a module/library depending on usage. A build process is really just another kind of data-processing program, after all. Just, like Make before it, is a declarative programming language. This is a design that is well suited to writing build/tool scripts. Unfortunately, it (like it's predecessor) lacks many of the conveniences of more conventional programming languages, namely modularity and flexible composition. That's what I'm aiming to address with this set of improvement suggestions.

---

...within Just I'd at least hope that it would support opening *.just files by default if no Justfiles or justfile is present in in the CWD.

For this context is everything, but syntax should be uniform. The resolution order I'd imagine is current file -> current directory with Justfiles superseding *.just files. A target outside the current file/directory would not be resolved unless it was fully specified with a relative path.

... if there's a file called Justfile one directory up, but a foo.just file in the current directory, which one should be executed?

The one in the current directory. The Justfile shouldn't even be considered as part of resolution, unless the target was specified with a relative path indicating to start resolution in that directory.

What about if there are multiple *.just files in the current directory, which one of them should be executed?

When considering multiple *.just files in the current directory, they should be considered in a uniform order, probably alphanumerically. All files should be checked for the target (lazy resolution could be an opt-in flag). If more than one match is found, resolution should abort with an error that the target is ambiguous (perhaps list the conflicting files), and that a fully specified target is needed.

An exception to this process might by that if a Justfile exists in the current path it (and only it) will be considered for resolution. This solves the path/target specification problem you bring up, is backwards compatible, and helps naturally encourage a user to organize their*.just files into subdirectories, while leaving a top-level Justfile as the intuitive point of entry.

---

Recipes can't be slash separated though (like path/to/justfile@recipe) since that's used in the CLI.

I don't see why not. If that's how you would refer to in on the CLI, why should it be any different in the task recipes? Uniformity is a boon and should only be discarded with good reason. Where possible, I'd like the syntax for referring to a target in another file from a recipe to work as if I'd invoked just with the same target from the same directory. This would make debugging syntax issues easier, among other things. This wouldn't be true the other way around, though. Allowing a recipe target to resolve to any file by default would hurt performance. Just should only look in other files when a recipe target fully specified.

I was thinking that recipes could be dot or :: delimited ...

With respect to a fully specified target, it only matters that it be possible to refer to the specific file in which to look for the target. All other constraints are free. From there, of the three ways to specify target foo in file bar.just, I think all are equally valid and have the following tradeoffs:

1. bar::foo
   • Pro: It mirrors the task declaration syntax task-name ARG:.
   • Pro: It mirrors how one might refer to a method in a class which is not too dissimilar to referring to a task in a file.
   • Pro: Obviously distinguishes the file/path from the task name
2. bar.foo
   • Pro: It mirrors existing dot-notation syntax for methods of a class.
   • Con: May be ambiguous to the reader (though not to the program) as to whether bar.foo names a file or a file/target pair.
   • Con: Possible ambiguity/confusion with make-style special targets.
3. bar@foo
   • Pro: Obviously distinguishes the file/path from the task name
   • Con: Unlike any other hierarchical syntax in common usage.

---

I'm not sure how I feel about loading justfiles without some kind of mod or import statement ...

I think that's a good design constraint to have. Though, I don't think it's necessary since requiring inter-file tasks to be fully qualified makes them already stand out. Having a special statement calling it out is redundant, though it could aid in debugging task resolution problems. There may also be performance/implementation advantages to resolving the includes ahead of task parsing. Overall, I'm for it. Syntax TBD.

---

I think it would be good to figure out a minimal design that makes multiple files useful, to avoid trying to do too much at once.

Given what we've discussed so far, and aside from the shebang, it sounds like the MVP would have the following features:

1. Just module imports. These would add fully specified tasks to the task database (or whatever internal store). Paths are relative to the CWD as of just invocation. Import paths can be required to resolve to a *.just file in a child directory.
2. Inter-file task support. This prepends the CWD of the current justfile (or *.just file) context to the fully specified task path and invokes it via task database lookup.
3. Fully specified task syntax. This makes path/to/file::task invoke task in path/to/file.just. As before, path/to/task will invoke task in path/to/justfile. This would work the same from CLI or task recipe.

Things left out of the MVP:

1. Dynamix lookup. We would not resolve tasks against multiple *.just files in the CWD. Only fully specified tasks would be supported. If this feature is needed it can be implemented with an index.just file that imports all the files that you would like to include in resolution, and aliases them with tasks local to that file.
2. Implicit *just file support. Automatically looking for a *.just file in the current directory would not be supported. The *.just file convention would only work for explicit file argument (via -f), explicit import, or with a shebang invocation (via the CLI flag).

---

Unfortunately, I'm very busy with other responsibilities for the time being...

That's alright. :) I am too. Though, bits of "free" time occasionally pop up here and there. If I can get to it, I'll try to. I really just wanted to get your feedback on my idea/s and try to get it in the pipeline. In the interim, I have my own hacky workarounds.

[casey]

I finally added this in #393. Thanks for opening the issue, I think it's a good feature, and I'm curious how it winds up getting used. Also, since #392 landed, --working-directory is optional, so you can choose which directory you'd like as the cwd.