-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge swt.binaries repo in the swt one #2
Merge swt.binaries repo in the swt one #2
Comments
Is the SWT binaries repo really necessary? What's the harm of building the natives on demand? Github provides build machines for the three main platforms, right? I was hoping to radically simplify SWT build process after the Github migration. Also, if we combine SWT repos, we could attach platform specific source to the platform specific fragments and leave the main bundle empty. Physically, the source might stay in o.e.swt, but logically every binary project like o.e.swt.win32.win32_x86_64 would contain both the binaries and the source. No more .classpath switcheroo and it would be possible to work on all platforms from the same workspace simultaneously. |
If you bisect SWT bugs, you won't (and probably can't) build on every commit. Building natives locally is a non trivial task (requires extra dependencies to be installed with root rights etc). |
You can't bisect across two repos anyway, can you? TBH I never tried to bisect SWT bugs. We can make the natives build almost a trivial task, but dependencies are of course unavoidable. JDK is a dependency too, why single out native code? |
It is not the question of "can" but it is a "must have" because if you bisect SWT issues your native library version is essential to be able to start anything. And yes, I've bisected two repos and that is NOT FUN, because you have to look which commit corresponds to which native commit. Having both the code & binaries in LFS and in one repo would simplify that A LOT.
Because a "simple" java compile with maven doesn't need to install extra gtk/win native packages etc, and in order to do that you must be able to run tasks as a root / have access to the required packages at all. Once you have setup that, it's fine, but it is not trivial to setup. Most people don't even need that at all if they fix something in simple Java code. |
Adding build artifacts to a Git repo also sounds wrong to me. We can publish the artifacts somewhere else to help with searching for errors. WDYT @akurtakov |
There have been already a discussion with @sravanlakkimsetti about that with potentially just using something like downloads.eclipse.org/eclipse/swt-natives/4567a/win32/ and having maven/ant download them. |
Corporates rebuild older versions of eclipse including SWT to release eclipse based products. Building on demand is not exactly possible as many of the dependencies would not be available. One solution is to store the built libraries in a separate repo(that is the current solution). Regarding git LFS, The individual libraries are not that big to start with less than 1 MB. I am not sure we are qualified for the LFS here.
What we are doing is not wrong. These are intermediate files used for the build purposes. I feel most of the thought process is going towards building master. for master we will have machines and other dependencies available for the build purposes. But that cannot be said when we want to build older release. For example IBM builds 4.23, 4.19, 4,15 regularly. By removing these artifacts you are forcing the end users to maintain native build infrastructure for different versions. A lot of duplicate effort and it turn into cost to the end users. We will definitely need to have the binaries stored. Where is going to be a question. Also we should make building aggregator simpler. Before triggering a build we should not fetch artifacts by some other means. Same problem exists for team, equinox.framework, filesystem components as well. |
So probably instead of check them into the git, simply deploy them as an additional attached artifact along with the maven deployment? |
This is what git LFS seamlessly does for you without any 3rd party tool chain.
The number of files in history does it. Cloning SWT binaries will move over wire ~1 GB just because every binary file is in the history. The sources itself are minimal, and with git LFS only last native libraries version will be needed to copy from server, not all of them. |
@sravanlakkimsetti |
I believe we are all in agreement that for everyone but seasoned SWT developer having the native bits downloadable is beneficial. But this comes at a price - almost every release we have one broken build by bump of version of the host without doing so for the fragments. Not to mention other missed opportunities to simplify things. So the question is how do we organize a change in order to get more benefits than complications?
One step at a time. Let's keep these discussions for other places. If we fix it in one place we will have better idea how to fix it in the other one. Btw, the team one is gone now after Linux moved to JNA 2 releases ago and Windows did so in master (https://bugs.eclipse.org/bugs/show_bug.cgi?id=578341). So we are getting there although the solution is totally different. |
@iloveeclipse A big plus from migrating to github is that cloning is much faster. For me SWT binaries clones in just under 3 minutes... |
Coping from(eclipse-platform/eclipse.platform#7 (comment)):Merging of SWT repos will break all previous SWT build setups and back-porting of patches which will become challenging if we merge repos:
Currently both of SWT-Sources & SWT-Binaries repo itself is combination of code from below 3 different platforms:
In general and specifically existing Eclipse/SWT wiki articles for setup/configuration are written with SWT(Source & Binaries) repo wise and it may also need to be fixed. Sorry it's a NoGo for SWT(Sources & Binaries) repo merging. Not only new comers but also all the existing corporate customers who just depend on Eclipse/SWT sources to build their products(on non-supported versions or non-supported platforms) are prone to break and may get out of picture for same set of reasons. |
One of the reasons could be a bug fix required in older releases. Atleast in IBM a product needs to be supported for 5 years. Any bugs reported needs to be fixed. so they end up rebuilding historical releases. Making it difficult to build will make customers to backdown. This would a very bad idea. I am all for optimizing. At the same time we should also make it simpler for customers to build their own products and make contributing easy. to me both are equally important one gets us funding and another gets us good fixes. |
@merks : that depends on your provider / country. @niraj-modi : I see your point. Is moving SWT binaries to git LFS alone worth it then? If we will go for LFS, we would need to:
Pros:
Cons:
|
@niraj-modi is SWT doing a lot of backporting? Looks to me that 4.23, 4.22, 4.21 and 4.20 maintenance branch still points to commit tagged for the release. |
So let's try to summarize discussion so far to see whether we can come to an agreement.
Does it sound more applicable now? |
Nope, exact that is the proposal. Have everything in one repo will simplify bisecting / changes a lot, we would not need to synchronize anything across two different repos / different locations, because everything will be in same git repository.
Git LFS stores the text url to the file in a server in the commit, not the binary itself in the repository (it goes to the extra server storage). This means, we can push 10 MB binaries but the commit record itself will be few bytes. |
Thanks for explaining. What does it mean for clone time? When is the binary for given commit downloaded ? At clone time? If it practically means no slowdown for clone while having the binary when needed sounds exactly what we need. |
I think on clone you get only the last version of the binaries (with the checkout).
|
@iloveeclipse Your above comments suggests that the old SWT Binaries repo would be read-only. |
@niraj-modi : the whole point of git LFS is to avoid huge repo size. So binaries are all on server and only fetched on checkout of a commit, in the "placeholder" there is just plain text link to the server url. I expect the SWT repo size after merge with SWT binaries be ~5 - 10 MB higher only (but I haven't tested, just looking how much space binaries taking on the actual binaries repo and how much the rest). And for the read-only repo I meant the old one, not the new one (main SWT), which will be of course writable. |
We will need writable repo not only for main master but also for maintenance branches as well(for SWT build input to work) Note: At-least in IBM a product needs to be supported for 5 years and please check below log: Hope you understand my concerns w.r.t. to SWT binaries repo and reason/importance of it to be writable on master/old as well :) |
@iloveeclipse There is no requirement for old repo to become readonly, right? We can e.g. clean master stating that it's no longer used and is there only for support or old releases. |
Regarding read only: sure. I assumed no one need write access to the main repo and all rebuilds happen on local clones. Regarding playing with git lfs: I surely can try if that all worka at all as planned. |
@niraj-modi Do you have more principal concerns?
|
Note that this is 1GB bandwidth. Here are my calculations, correct me if I'm wrong:
My conclusion: Free GitHub bandwidth is not sufficient for SWT binaries repo. We could probably use a separate LFS server, but this approach sounds quite weird in the grand picture of trying to move things to GitHub. |
Here's my attempt to summarize (and discuss) mentioned benefits:
I'm not sure if merging is worth the trouble. |
Configure this Git-repository to store the SWT native binaries, i.e. all *.dll, *.so and *jnilib files, using Git's Large File Storage (LFS) and move the platform-specific SWT fragments from the eclipse.platform.swt.binaries repository into the binaries folder in this repository. The platform specific projects are (minimally adjusted) copies of the corresponding projects from https://github.com/eclipse-platform/eclipse.platform.swt.binaries/tree/9cdcee51a79a3f72e10e074b87326f80d3d6ada2 The native binary files are not copied because the build triggered by this commit will re-build and add them anyways. Additionally the 'forceQualifier.txt' are also left out since the main bundle only drives the qualifier. The commit procedure when the binaries are re-build is changed to commit the binary and pom changes in one commit. Fixes eclipse-platform/eclipse.platform.swt.binaries#2
Configure this Git-repository to store the SWT native binaries, i.e. all *.so, *.dll and *jnilib files, using Git's Large File Storage (LFS) and move the platform-specific SWT fragments from the eclipse.platform.swt.binaries repository into the 'binaries' folder in this repository. The platform specific projects are copies (with only the adjustments necessary due to the move) of the corresponding projects from https://github.com/eclipse-platform/eclipse.platform.swt.binaries/tree/e58be4cbe6e638e5a190d019643bb11db49c3ade The native binary files are not copied because the build triggered by this commit will re-build and add them anyways. Additionally the 'forceQualifier.txt' from the native fragments are also left out since the main bundle only drives the qualifier. The commit procedure when the binaries are re-build is changed to commit the binary and pom changes in one commit. Fixes eclipse-platform/eclipse.platform.swt.binaries#2
With eclipse-platform/eclipse.platform.swt#956 the swt native binaries are stored in the eclipse.platform.swt repository using git-lfs. The eclipse.platform.swt.binaries repository is therefore obsolete. Part of eclipse-platform/eclipse.platform.swt.binaries#2
Configure this Git-repository to store the SWT native binaries, i.e. all *.so, *.dll and *jnilib files, using Git's Large File Storage (LFS) and move the platform-specific SWT fragments from the eclipse.platform.swt.binaries repository into the 'binaries' folder in this repository. The platform specific projects are copies (with only the adjustments necessary due to the move) of the corresponding projects from https://github.com/eclipse-platform/eclipse.platform.swt.binaries/tree/e58be4cbe6e638e5a190d019643bb11db49c3ade The native binary files are not copied because the build triggered by this commit will re-build and add them anyways. Additionally the 'forceQualifier.txt' from the native fragments are also left out since the main bundle only drives the qualifier. The commit procedure when the binaries are re-build is changed to commit the binary and pom changes in one commit. Fixes eclipse-platform/eclipse.platform.swt.binaries#2
Configure this Git-repository to store the SWT native binaries, i.e. all *.so, *.dll and *jnilib files, using Git's Large File Storage (LFS) and move the platform-specific SWT fragments from the eclipse.platform.swt.binaries repository into the 'binaries' folder in this repository. The platform specific projects are copies (with only the adjustments necessary due to the move) of the corresponding projects from https://github.com/eclipse-platform/eclipse.platform.swt.binaries/tree/e58be4cbe6e638e5a190d019643bb11db49c3ade The native binary files are not copied because the build triggered by this commit will re-build and add them anyways. Additionally the 'forceQualifier.txt' from the native fragments are also left out since the main bundle only drives the qualifier. The commit procedure when the binaries are re-build is changed to commit the binary and pom changes in one commit. Fixes eclipse-platform/eclipse.platform.swt.binaries#2
With eclipse-platform/eclipse.platform.swt#956 the swt native binaries are stored in the eclipse.platform.swt repository using git-lfs. The eclipse.platform.swt.binaries repository is therefore obsolete. Part of eclipse-platform/eclipse.platform.swt.binaries#2
and add a README to inform about the move. Last part of eclipse-platform#2
and add a README to inform about the move. Last part of eclipse-platform#2
and add a README to inform about the move. Last part of eclipse-platform#2
@HannesWell : many thanks, it was really huge amount of hard work. |
and add a README to inform about the move. Last part of #2
Thanks and you are welcome. It was indeed a lot of work but also a lot to learn. Since there were no reports about any problems over the day I have completed this and merged #70 to clear and retire this repository. |
Fix paths that are still relevant and remove obsolete occurrences. Especially this fixes the provision of the SWT.zip files at the Eclipse download page. Part of eclipse-platform/eclipse.platform.swt.binaries#2
Fix paths that are still relevant and remove obsolete occurrences. Especially this fixes the provision of the SWT.zip files at the Eclipse download page. Part of eclipse-platform/eclipse.platform.swt.binaries#2
Fix paths that are still relevant and remove obsolete occurrences. Especially this fixes the provision of the SWT.zip files at the Eclipse download page. Part of eclipse-platform/eclipse.platform.swt.binaries#2
There was a good question asked on eclipse-platform/eclipse.platform#7 - why don't we use git LFS for SWT binaries? As of today, SWT is about 250 MB and it seems that the github provides 1 GB free bandwidth for git LFS repos, so it should be enough.
The benefit is surely that we don't need to synchronize two repos and have platform code & binaries in one, plus much faster clone.
The text was updated successfully, but these errors were encountered: