Very slow on large files (and hangs, can't cancel)

[dardoor]

I'm trying the latest version and it's still pretty much unusable on large files (and by "large files" I mean like 10MB).
It can make N++ hang maybe for hours and it's not even possible to cancel the compare process. I don't know if it's the size alone or the number of lines that causes this.

If it's not possible to fix this, maybe you can add a warning when a user tries to compare large files.

[pnedev]

What is actually "the latest" version you are using ? ComparePlus 1.1.0 ?

Are the files confidential ? If they are not, could you upload them here or share them so I can analyze what is happening ?

BR

[dardoor]

Yes, version 1.1.0.
Here are two files I can share: https://www.sendspace.com/file/gdaiau
I've had some worse examples too, where N++ was stuck for over an hour. But I can't find them right now and it is possible that it was with an older version of the plugin.

[pnedev]

Hello @dardoor ,

Thank you for the example files.
These are really big binary files with lots of long lines and a lot of differences which causes such long compare time.
I believe other compare tools/programs will also have troubles comparing them and will also take considerable time doing so.
Unfortunately this cannot be avoided, ComparePlus plugin is aimed at comparing mainly shorter text files or bigger files that do not differ a lot.
Your idea about the warning before comparing such file types is really good and I will also fix the Cancel functionality to be more responsive in such cases.

BR

[dardoor]

Thanks.

I believe other compare tools/programs will also have troubles comparing them and will also take considerable time doing so.

Some programs can compare them instantly. For example Total Commander (select both files and then Files->Compare By Content...). It can show them as binary or as text lines like in N++. It just doesn't look as nice as in N++ with your plugin.

And WinMerge is free, looks nice and is very fast too. https://winmerge.org/

I think some of the slowness here is also the fault of N++ itself, which takes a while to open big files even when not comparing.

[pnedev]

Hello @dardoor ,

Some programs can compare them instantly.

Interesting.
Anyway, I don't think I can do much about making ComparePlus faster when comparing such files, sorry.
I'll improve the Cancel responsiveness and add a warning before starting the compare process for large files.
Thank once again.

BR

[pnedev]

Hello @dardoor ,

I have made Cancel much more responsive and added a warning message before comparing very large files.
The changes will be included in the next release but meanwhile you can get the dev build from here: win32 , win64 or arm64. The dev builds are for Notepad++ versions >= 8.4.7.

P.S. About slow responsiveness of Notepad++ when dealing with large files - if you haven't done that already disabling Word Wrap function could speed things up significantly.

BR

[Yaron10]

Hello Pavel,

Thank you very much for the recent improvements. 👍

Comparing the files @dardoor provided in #339 (comment), and pressing Cancel after ~2 seconds - NPP still hangs on my machine. Isn't that the case on yours?

BR

[pnedev]

Hello Yaron,

Thank you for testing this.
I don't see a considerable delay after pressing Cancel and the actual compare cancel. No more than 2 seconds (even less).
Most of the time it is almost instant. I tried in different moments after starting compare on the example files provided by @dardoor.
What exactly do you observe? By "NPP hangs" do you mean that you see the progress message "Cancelling compare, please wait..." but you wait very long time before operation is cancelled or it is not cancelled at all?

My change might depend on the computer speed so is your computer processor older generation?

Thanks and BR

[Yaron10]

Hello Pavel,

On Cancel:

Before the recent commits:
I get a black screen with the title "Not responding".
The progress message remains on that black screen until I press the Close button.
After pressing the Close button, I get the Windows message "Notepad++ is not responding...".

With the latest commit:
I get a black screen with the title "Not responding".
The progress message disappears almost instantly. 👍
After pressing the Close button, I get the Windows message "Notepad++ is not responding...".

So there's certainly some improvement, but NPP still hangs/crashes.

I've tested it on a relatively powerful PC.

Thank you again.

BR

[pnedev]

Hello Yaron,

That is totally not what I was expecting :))

Well, then my fix works OK - the progress disappears almost instantly meaning that compare has been cancelled but the black screen is something new to me and might be either another bug or most probably NPP related behavior itself.

Do you see that black screen when you open the files initially?
Have you tried waiting some time before closing NPP?
Could you please try the same but make sure that 'Word Wrap' is disabled (if it has been enabled in your previous tests)?
Could you also try removing all other plugins just to make sure they are not interfering?
I did my tests on NPP 8.4.7 and 8.4.8, I suppose you are using one of those versions too (unaltered in any way).

Thank you very much! 👍

BR

[Yaron10]

Hello Pavel,

Well, then my fix works OK - the progress disappears almost instantly meaning that compare has been cancelled but the black screen is something new to me and might be either another bug or most probably NPP related behavior itself.

👍

Could you please try the same but make sure that 'Word Wrap' is disabled (if it has been enabled in your previous tests)?

👍
I occasionally get a black screen (without the "Not responding" in the title) for a split second, but then the Compare is canceled and everything is OK.

I did my tests on NPP 8.4.7 and 8.4.8, I suppose you are using one of those versions too (unaltered in any way).

8.4.8. Unaltered in any way. :)

Could you also try removing all other plugins just to make sure they are not interfering?

I've removed all other plugins. Same results.

Do you see that black screen when you open the files initially?

Interesting.
When I first open the files (by d&d or from the Recent Files), I do not get the black screen. Word wrap ON or OFF.
But if start Compare with Word wrap unchecked, Cancel, close the files, check Word wrap and then open the files from the Recent Files - I do get the black screen.

Have you tried waiting some time before closing NPP?

With Word wrap ON -
I have waited for some 10 seconds. No change.
Don't you get the black screen on your machine? Wine?

Thank you maestro. I appreciate your work. 👍

BR

[pnedev]

Hello Yaron,

Thank you for the quick re-test and the reply. 👍

On Wine I do not get a black screen.
I just tried on native Win11 and without 'Word wrap' the compare starts and can be cancelled normally (no freeze or 'Not responding'). With 'Word wrap' enabled after cancelling compare NPP hangs and displays 'Not responding').
I'll do some more tests to see if I can figure out what is going on.

BR

[Yaron10]

Hello Pavel,

So we get the same results on native Windows.

Thank you for further investing it, and best of luck finding the cause. 👍
I do appreciate it.

And as you wrote: "or most probably NPP related behavior itself".
NPP is certainly a great editor, but - as we all know - it has some performance known-issues.

BR

[Yaron10]

Hello again Pavel,

Regarding the new large-files warning:
Comparing these 800-lines files (Ignore Spaces: OFF), I get a black screen after a couple of seconds and the Compare process is very slow.
I'm wondering if the 50,000 lines is the appropriate criterion, and if the warning should be helpful at all.
Perhaps users should accept it that Notepad++ is not built/optimized for some long complex operations.

What do you think?

Thank you.
BR

[pnedev]

Hello Yaron,

I don't get the black screen on my Win11 but the compare is very slow, yes.
About the 50000 lines warning criterion I think it is better than nothing. The criterion cannot be very precise because the speed depends on the lines count, the lines size and on how much the files differ so it is a preliminary assumption.
The good thing is that now compare can be cancelled almost instantly by clicking on Cancel so the missing warning message is not that crucial.

It is also true that NPP and ComparePlus definitely cannot handle all file types and file contets equally well so there will always be files that are not easily and quickly handled.

Thank you for providing these test files I will try to do some speed optimizations but even if I don't manage to improve things a lot it is a expected limitation in both NPP and CP.

BR

[Yaron10]

Hello Pavel,

About the 50000 lines warning criterion I think it is better than nothing.

👍

I will try to do some speed optimizations...

Great.
Thanks again for your hard work. Much appreciated.

Have a nice weekend.

[informatorius]

Would compare be faster on large files with multicore enabled? Could that be an option in Compare Plus settings?

I see that option in code:
https://github.com/pnedev/comparePlus/blob/master/src/Engine/Engine.cpp:40
#define MULTITHREAD 1

The cancel check interval maybe is too long. Could it be one second?
diff.h
static constexpr int _cCancelCheckItrInterval {3000};

[pnedev]

Hi,
I already made the Cancel compare functionality more responsive (in my development version, it is not released yet).
As to the multithread implementation - it is not working properly currently and is disabled ATM.
I will fix that some time in the future. I suspect it won't happen soon, Compare is currently not in my priority list.
BR

[Yaron10]

Hello my friend Pavel,

Just wanted to express my sincere hope that you're doing well.

Best wishes.

[informatorius]

Maybe that diff algo gives a speedup for large files?

https://github.com/gliese1337/fast-myers-diff

[pnedev]

Hello and thank you @informatorius ,
I'll check that algo if it can be used as well but it won't be soon.

Hello Yaron,

Thank you friend, I am OK just a lot of work to do and it's difficult to switch to "Compare" mode.

BR

[Yaron10]

Hello Pavel,

I'm glad you're OK.

and it's difficult to switch to "Compare" mode.

I can relate to that. :)

BR

[FunkyFr3sh]

There's clearly a bug, once the files exceed a certain size it will hang. Other compare tools like visual studio and beyond compare do it almost instantly, so I don't think the files are too big or too complex. Pretty sure this can be fixed. I hope someone will have a look into it, I really like the look of the compare plugins in N++

[BastienGauthier]

Another idea could be to switch to a very simple diff comparison (line-to-line) above a certain threshold (I have seen the 50k lines mentionned in the discussion for exemple).

[pnedev]

Maybe that diff algo gives a speedup for large files?

https://github.com/gliese1337/fast-myers-diff

Hello @informatorius ,

A long time has passed but I finally was able to try that algorithm and unfortunately it is around 3 times slower than the currently used algorithm.
Thank you for the suggestion anyway 👍

BR