Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve performance for large files #22

Merged
merged 1 commit into from
Mar 9, 2017
Merged

improve performance for large files #22

merged 1 commit into from
Mar 9, 2017

Conversation

targos
Copy link
Contributor

@targos targos commented Mar 8, 2017

Using buf = Buffer.concat([buf, data]); on the data event copies the
data in a new larger buffer everytime. Instead, store the intermediate
small buffers in an array and concat only at the end.

Test with a 117MB file:

  • before: 83.868 seconds
  • after: 2.230 seconds

@coveralls
Copy link

coveralls commented Mar 8, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling 156bba8 on targos:improve-perf into ac309ee on richardgirges:master.

@targos
improve performance for large files
e23f337 
Using `buf = Buffer.concat([buf, data]);` on the data event copies the
data in a new larger buffer everytime. Instead, store the intermediate
small buffers in an array and concat only at the end.
@coveralls
Copy link

coveralls commented Mar 8, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

4 similar comments
@coveralls
Copy link

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@coveralls
Copy link

coveralls commented Mar 8, 2017 •
edited
Loading

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@richardgirges
Copy link
Owner

Beautiful. Amazing increase in performance! Thank you very much :)

@richardgirges richardgirges merged commit 90267a3 into richardgirges:master Mar 9, 2017
@targos targos deleted the improve-perf branch March 9, 2017 19:16
@richardgirges richardgirges mentioned this pull request Mar 9, 2017
Closed
@mikeatlas
Copy link

Wow. This is a huge difference. Thanks @targos !

This was referenced Nov 18, 2019
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Closed
Open
Open
Open
Open
Open
Open
Open
This was referenced Apr 15, 2020
Open
Open
Open
Open
Open
Open
Open
Open
Open
This was referenced Apr 22, 2020
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
@snyk-bot snyk-bot mentioned this pull request Apr 30, 2020
Open
This was referenced Nov 30, 2023
Open
Open
Open
@snyk-matt snyk-matt mentioned this pull request May 25, 2024
Open
@johanants johanants mentioned this pull request Jun 8, 2024
Open
@florinMiro florinMiro mentioned this pull request Jun 11, 2024
Closed
@zolamk zolamk mentioned this pull request Jun 12, 2024
Open
@snyk-matt snyk-matt mentioned this pull request Jul 17, 2024
Open
@frogthehun frogthehun mentioned this pull request Aug 25, 2024
Open
@cdobru cdobru mentioned this pull request Aug 30, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@targos @coveralls @richardgirges @mikeatlas