fish code in the background

[JanKanis]

Although I find fish much better than bash for everyday use, there's one major feature bash supports but fish does not: running fish code while doing anything else. Specifically, backgrounding a fish function doesn't work, and running fish code as part of a pipeline blocks the whole pipeline.

Try this in bash:

$ function foo() {
> echo bar
> sleep 10
> }
$ foo | cat
bar
# notice that output arrives immediately, though the whole job only finishes after 10 seconds
$ foo &
[1] 1126
bar
$ # you can continue your interactive session. After 10 seconds: 
[1]+ Done        foo

If you try the same in fish, you'll notice that in a pipeline the output only arrives after the foo function has exited, i.e. after 10 seconds, and running a fish function in the background basically has no effect. I run into this sometimes when building pipelines to filter stuff or when using aliases, for example this one:

function gitg
    # to see a nice graphical history of the git repository without gtk error messages
    command gitg $argv ^/dev/null
end

If I then run gitg &, the shell is blocked until I quit gitg.

Pressing ctrl-Z while a function or loop or other compound statement is being executed backgrounds only the current external command, and continues the fish code in the foreground:

> begin
      sleep 20
      echo done
  end
# press ^Z
Job 1, “sleep 20” has stopped
done
> bg
Send job 1 “sleep 20” to background
# some seconds later...
Job 1, “sleep 20” has ended
> 

Ideally this should background the entire block. (But this part does not work in bash either) In fact, backgrounding a compound statement that consists only of functions and builtins doesn't work at all:

> while test 1; echo -n o; end
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo....
# lots of o's. ^Z doesn't do anything (though ^C works)

Fixing all of this would require some way of having fish execute multiple pieces of fish shellcode in parallel, so that is probably not a simple fix. However I think this issue should be discussed so we can decide on how to move forward with respect to fixing this at some point in the future. Fish's parser could really use an overhaul (see also bug #557), and if/when that happens this issue should also be taken into consideration.

[JanKanis]

There are two closely related issues that would require solving: stopping and resuming a block of fish code with ctrl-Z, and running fish code in parallel, e.g. in pipelines or using the & character.

There are (as far as I have thought out) three options:

1. fork
2. multithreading
3. make the evaluator stoppable and resumable

Forking (option 1) is what bash and similar shells do, but they only use it to solve the second problem, not the first one. Using this option in fish could I think be done. When a ctrl-Z arrives fish forks, the parent handles user input and the child maintains the fish script state. This option is rather hairy, first of all it needs some way to synchronize internal state such as variables between the two processes, and second it needs some way to have the new child processes handle all external resources such as file descriptors and other child processes. This does not seem like a very practical approach to me, but I still wanted to name it.

In option 2, fish would always execute shell code on a secondary thread. On a ctrl-Z this thread could be suspended and a new thread started to process user input. This would require making the evaluator and most of fish's internals thread safe.

In option 3, the evaluator (parser and related things) would need to be refactored so that multiple instances of it could exist at the same time, with all state stored in an object. If a ctrl-Z arrived, the evaluator should then unwind without destroying its state, so that later on it could be resumed again. To have fish code run in parallel within this option, there are again several possibilities: resume the stopped evaluator in a forked child (which has the same problems as option 1), resume it in a different thread (which also requires making fish's internals thread safe), or having some way to interleave multiple evaluators: fish would run one evaluator for some time, then the other, then the first one again, etc.

I think either option 2 or 3 (or some kind of combination) would need to be chosen, but I'm not sure which of those would be best.

[ridiculousfish]

The parser is already instanced, and in fact we use that when loading completions (see completer_t::complete_from_args). I did at one point have grand ambitions of loading functions, etc. off of the main thread; those were never realized but I think they're still interesting, for responsiveness and performance.

A stackless-type parser, that doesn't depend on the C stack for storing its state, seems achievable to me. Such a parser could be driven externally ("do some work"). The big win here would be sane signal handling: cancellation would be handled by just stopping doing work, without needing to unwind the C stack.

The biggest obstacle I see to a fully multithreaded shell is that there really is per-process state out of our control, such as the current working directory or the process that owns the term (tcgetpgrp).

[JanKanis]

Great, so the parser is in fact already multithread safe!

I must say I don't immediately see the advantage of the 'sane signal handling' you mention. On a ctrl-C the execution of code is cancelled, then why is it a problem to unwind the stack? That has to happen some way or another, now it happens by explicitly unwinding the C stack, but in a stackless parser stack frames would be deleted by doing something like parser->reset() or so.

The two examples of kernel controlled state you give seem solvable (using openat and friends, and there is only one terminal so that needs to be under a lock anyway, though I'm not an expert in process group arcana). Are there any other such items that can't be fixed?

[ridiculousfish]

Handling signals by unwinding the C stack might be OK. I think it's more the current haphazard design that I dislike - for example, the parser_t::skip_all_blocks, which is hard to reason about its interaction with other uses of the skip flag (like if/else handling), or what happens if that's invoked during parser_t::push_block, or any of the expansions we do (brackets, pid, tilde, globbing, command substitutions...).

I guess the real win would be centralizing SIGINT handling so it's easier to reason about.

Regarding the kernel controlled state, consider a command like this:

cd /tmp/(random) ; command pwd

The pwd command should reflect the directory that we cd'd into in the previous command. That works today because the pwd process inherits the working directory from the parent shell. But if we were to run this code on multiple threads, each one attempting to cd someplace else, how would we ensure that each pwd sees a different working directory?

[JanKanis]

Have a look at the openat manpage. The openat and other *at bunch of functions were specifically designed to allow for an (emulated) per-thread working directory.

The fish process obviously cannot have more than one working directory, but it can open other directories that other threads use as working directory using the normal directory open calls. Then if the second thread needs to do things relative to its working directory, fish can use openat/fstatat/etc to open files relative to the directory file descriptor. When the second thread needs to spawn a program like pwd in your example, it can do fork, cd to its per-thread working dir using fchdir(fd), and exec. The external command executes in the right directory just fine.

[ridiculousfish]

openat doesn't appear to be implemented on OS X. I see it in Linux.

[JanKanis]

Hmm, bummer. The manpage says it's part of POSIX, so I figured it would be portable. But apparently not. OSX does have fchdir, so it is possible to emulate openat doing: lock mutex, fchdir(thread_pwd_fd), open(...), fchdir(main_pwd_fd), unlock mutex. But it is less than ideal.

[ridiculousfish]

Revisiting this, being able to safely run shell code on other threads would be a total coup. Fixing up stuff like the working directory after fork is plausible. It's certainly worth exploring.

[ekg]

I would absolutely love to have this feature in fish. It's one of the most important features of the other shells I've used!

[ridiculousfish]

In the case of foo | cat there ought to be no reason why fish cannot pass the output of the function to the pipe, instead of filling up a buffer.

[ridiculousfish]

Another thought I had - background processes may be implemented with a sort of queue / barrier mechanism:

1. Add a thread barrier. No new threads start until the barrier is released.
2. You get a callback when all extant threads have completed. At that point, fork for each waiting background process.
3. Release the barrier, allowing new threads to start up again

This is very bash-like, and would be an alternative to a fully threaded model.

[JanKanis]

I've also thought about that, and there are several things that came to
mind.

• It's probably possible to get a very basic forking implementation without
  too much work. There's already an iothread_drain_all function that could
  easily be changed to suspend new thread creation. Doing so waits for all
  outstanding threads to be completed rather than having them hit a barrier,
  so there may be a longer delay if i/o is slow. On the other hand, while a
  background thread is blocking on some long running io it is not hitting the
  barrier either so there could still be a significant delay. (Unless you
  want to wrap all potentially blocking io in some kind of barrier, but that
  seems like a lot of work.
• However, doing the above does not meet what fish intends to be. You would
  need to synchronize e.g. global variable updates done in such a forked and
  backgrounded function. Bash doesn't do that, but it is clearly the fish
  way. All other global state should also be synchronized, so once you start
  thinking about that it probably becomes much more attractive to make the
  fish interpreter thread safe so background functions can run on different
  threads rather than forked processes.
• Forking with threads at a barrier rather than with all threads drained as
  I suggested in the first point would afaik still invoke undefined behavior.
  I don't know if that would be a problem but I can imagine that the libc
  thread management datastructures could end up in an inconsistent state if
  threads just vanish in the forked child, even if they are waiting at a
  barrier. That could make it impossible to safely start new background
  threads in the child.

So IMO the 'right' way forward is making the interpreter multithreaded.
Unfortunately I don't have time to implement that myself in the near
future.

[ridiculousfish]

At first, I thought it would be weird to allow setting a variable from a background task...it would just sort of change at some point in the future. But now I think it makes sense at least for global variables - you might want to spawn off a slow background task that reports data back when it's done, and one mechanism for that could be setting a variable.

We will have to take care that things like redefining functions while they are executing does not cause a crash.

[faho]

The backgrounding half of this is #238, the pipe-blocking is related to #936.

[floam]

openat doesn't appear to be implemented on OS X. I see it in Linux.

FWIW not the case anymore.

[floam]

from the manpage:

HISTORY
     An open() function call appeared in Version 6 AT&T UNIX.  The openat()
     function was introduced in OS X 10.10

[floam]

full list of added documented syscalls in OS X 10.10:

• fchmodat
• fchownat
• getattrlistat
• mkdirat
• openat
• readlinkat
• fstatat
• symlinkat
• unlinkat

it would seem these are the first new public syscalls added since 10.5, which added sendfile and the now-deprecated *stat64 and *statfs64 syscalls.

[faho]

I'm closing this as a duplicate of #238.