Incremental builds (faster publish)

[hamoid]

Due to the size of my blog, running build takes between 30 and 120 seconds.

There are 279 posts, but I suspect what makes it slow is the 220 Mb of images the posts contain.

Possible solutions:

1. Use symlinks to media, instead of copying media files. This would also avoid having all files duplicated (in my case over 200 Mb "wasted")
2. When copying media, copy it preserving attributes. Before copying media, check if items already exist and have the same size and timestamp. This would already avoid copying existing files, and might make it many times faster (if copying files is the issue).
3. Keep an internal list of all files and detect changed ones. Then build only those that have changed. One way to do this is to write the code that explores all folders recursively and gets data for each file. To avoid making hundreds of calls to functions that get file info one could use the 'ls' command. Execute "ls -lR contents/articles/ >file_details.txt" and then compare the content of that text file with past versions, finding what has changed. Mmm I think I just forgot that there are Windows users out there... :)

Thanks!

[jnordberg]

The problem is that not all content is derived from a file. I'm thinking of adding something like a isModifiedSince method to content plugins that you can implement to enable incremental builds. This would also allow for caching in the preview server.

[hamoid]

Sounds like a good solution.

[mjpizz]

Do you think incremental builds could be done in a way that enables caching to disk? For example, to survive a restart of the preview server.

[jnordberg]

@mjpizz yeah, that would be possible. When this is implemented i plan to put a memory cache in front of the preview server, that cache could just as easily be a disk cache. But why would you want to have persistent caching for the preview server?

[mjpizz]

The main issue is performance during development and previewing. Once we're over 1000 articles, it would be nice if wintersmith didn't have a 30s-1min bootup time, and if individual page changes were updated more-or-less instantly.

For example: consider a "related articles" plugin. This might require scanning all 1000 articles to find the ones that share a specific tag before rendering. Cold booting the preview server in this case might become a pretty intensive process of basically rebuilding an in-memory index every time somebody sits down to hack on the design.

Optional cache persistence via a simple database (or even JSON flatfile) would probably help cold-boot-for-large-websites quite a bit. Do you think it makes sense?

[jnordberg]

@mjpizz hmm, the cache im talking about here wouldn't help in that case. Wintersmith needs to build up the content tree on startup, that would have to be a separate cache and each content plugin instance have to be able to be (de)serialized . When harmony proxies lands in node we could do something like only create the instances on the content tree when they are accessed, that along with the output cache would solve most performance problems for large projects.

[tusharmath]

Sorry guys but this might be a little tangent - How and where do you deploy your content? I personally have created a Gruntfile to publish to Github which I don't feel is very elegant.

[hamoid]

@tusharmath Very tangent :) I rsync to my own server. It takes seconds.