Improve performance

[kkmuffme]

While wp i18n is a great tool, the performance especially for themes is abysmal.

E.g. I have a (bigger) theme, where wp i18n takes 3 minutes to extract all strings/create the pot (wp cli 2.3.0, PHP7.3; issue was there with earlier versions too though), while it takes literally 5 seconds with poedit.

Could this be improved? Atm I am building multiple themes daily (and since I need to wait for the output to check if any errors found and then having to rebuild it multiple times if there are errors found), I wait roughly an hour daily for it to finish, so this would be greatly appreciated.

[schlessera]

We have been focussing on correctness so far to precisely match the previous output that was needed for WP projects.

I'm sure that we can improve the performance if we do an audit about the bottlenecks.

Did you by chance do some profile runs against the command already?

[kkmuffme]

I only run some plain time to measure how long these take while identifying what is slowing down my deploy scripts.

I don't really have experience debugging cli (not even sure if new relic/tideways works for that), so would love to get some additional feedback if you want me to further trace this.

[swissspidy]

Did you try running the command with the --debug flag? It might be that the script is scanning some unnecessary folders/files that should be excluded. If you could perhaps share your code so that we can have a look that would be nice.

[kkmuffme]

Yes I did, it goes until like:

Excluding the following files: node_modules,.git,.svn,.CVS,.hg,vendor,Gruntfile.js,webpack.config.js,*.min.js (0.272s)

Then there's no output, then it finishes.

In another example I ran there were a couple "Could not parse file ..." in the meantime, but it basically was just like a progress bar (accidentally) until it finishes after 7m3s

I set it to only 1 textdomain via the --textdomain flag.

The issue seems to be that the parsing is slow as.

when I parse the same folder with grep like:
grep -rnw '/my/path/' -e "'my-textdomain'"
it takes 2 seconds. Obviously its not as exact, but its a first rough selection, which then could be parsed by PHP for exact string extraction.

If you can't use exec, perhaps the parsing of each file could be parallelized using:
WP_CLI::runcommand()
(maybe up to a limit of X children at once)
but I honestly have no experience with taht.

[kkmuffme]

What is also weird is that sometimes (for unknown reasons, as there is no output in --debug beyond: Excluding the following files: node_modules,.git,.svn,.CVS,.hg,vendor,Gruntfile.js,webpack.config.js,*.min.js (0.272s)) it gets completely stuck.
e.g. what normally takes 3-4 minutes will take hours and not be done (until I kill it)

No errors are logged either.

So as a first quick fix I suggest to improve the debug output, to perhaps output the file name it is currently processing? (optionally behing a --very-verbose flag or something if its too much for --debug even though it's helpful in any case of debug)
This would allow to at least see where it gets stuck or give some insight into wtf is going on.

[schlessera]

@kkmuffme That can point to an infinite loop of some sorts. I've hit that when working on a patch as well, in the JS extraction. There's definitely something very brittle happening there that needs further inspection.

[kkmuffme]

@schlessera I just saw that this was implemented right now #183 - thanks to this I found the issue now.

Anyway, I was suffering from #185 as the include somehow overwrote the exclude and it was going through 100000s of included node_modules,min.js,... which made it seem stuck, but in reality there were just loads of files that it shouldn't have checked.