Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release CmdStan 2.33 #1181

Closed
27 tasks done
WardBrian opened this issue Aug 17, 2023 · 50 comments
Closed
27 tasks done

Release CmdStan 2.33 #1181

WardBrian opened this issue Aug 17, 2023 · 50 comments
Assignees
@rok-cesnovar @WardBrian @serban-nicusor-toptal
Labels
release/packaging

Comments

@WardBrian
Copy link
Member

WardBrian commented Aug 17, 2023 •
edited by serban-nicusor-toptal
Loading

Feature freeze (August 21, 2023):

  • Ensure all expiring language deprecations have been removed or given new removal dates.
  • Create and merge version updating pull requests in Math/Stan/Cmdstan. These should be the last PRs accepted before the freeze.
  • Create Math/Stan RC releases.
  • Create Stanc3 RC binary.
  • Create a release candidate tarball for x86. Make sure RC tarballs include stanc3 binaries.
  • Check external links in docs (i.e. TBB docs link) - use https://github.com/tcort/markdown-link-check
  • Create a release candidate feature/bugfix list (major features/bugfixes that need testing, link to new docs in Github)
  • Run CmdStanR tests with the RC tarball.
  • Run CmdStanPy tests with the RC tarball.
  • Make a Discourse RC post.
  • Post a tweet with a link to the Discourse RC post.

Release (planned for September 5, 2023):

  • Create the Math Release notes.
  • Create the Stan Release notes.
  • Create the Cmdstan Release notes.
  • Create the Stanc3 Release notes, include new deprecations and removals.
  • Rebuild and publish docs for the new version.
  • Check that docs for the previous release links correctly to the newest docs.
  • Create the Math release.
  • Create the Stan release.
  • Create the Stanc3 release.
  • Create x86 CmdStan tarballs (check version, check that the extracted folder is in the cmdstan-version format).
  • Create non-x86 CmdStan tarballs.
  • Run CmdStanR tests with the release tarball.
  • Merge release branch into master for math, stan, cmdstan
  • Make a Stan blog release announcement post (thank the sponsors and all contributors, mention new devs).
  • Link to the blog post in a Discourse thread.
  • Make a Twitter announcement.
@WardBrian
Copy link
Member Author

I believe #1184 is the only thing we're waiting on before we can freeze (stan-dev/stanc3#1347 would be nice, if it is ready)

@WardBrian WardBrian mentioned this issue Aug 21, 2023
Merged
3 tasks
@WardBrian
Copy link
Member Author

I finally made the release-notes script a bit nicer and automatic

Here's the raw output, we can work on editing them:
https://gist.github.com/WardBrian/2898738905ab3592e33b0897d7129020

@WardBrian
Copy link
Member Author

The issues mentioned above have been resolved, so @serban-nicusor-toptal you can run the RCs whenever you are available tomorrow

@wds15
Copy link
Contributor

wds15 commented Aug 22, 2023

This is quite a BIG release in terms of new stuff. Awesome!

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Aug 22, 2023 •
edited
Loading

Good morning,

Release Candidates are out, see:

I've created a release/vx.x.x branch to use for RCs and the release.
Created a merge to develop for the version increase, we can either merge now for the version update or wait for the release, see:

CmdStanR running at https://github.com/stan-dev/cmdstanr/actions/runs/5939209477

@WardBrian
Copy link
Member Author

Thanks @serban-nicusor-toptal!

Re: merging those PRs, when do we update the change logs in the repos historically? Does it make sense to wait until those are finalized and do them with the version numbers? stanc3 would need a PR too in that case

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Aug 22, 2023 •
edited
Loading

Ah, you're right about stanc3, I'll open it now. ( can't do it now since there are no changes on the branch, will do it before release )

In general right before the release, I add the release notes ( they're ready usually after to catch possible fixes during RC ), merge them into the release branch, and then that branch into develop and master.

The only downside is that develop is still tied to an older version in the meanwhile, to fix this we can merge once now and once again during the release for the release notes.

wdyt is best to follow here ?

@WardBrian
Copy link
Member Author

I’m happy to wait personally

@serban-nicusor-toptal
Copy link
Contributor

Sounds good! Then I'll add release notes to the release branch and merge them during the release.
Please let me know if there's anything I can help with now! Are the cmdstanpy tests part of GH Actions here https://github.com/stan-dev/cmdstanpy/actions/workflows/main.yml ?

@WardBrian
Copy link
Member Author

Yep, running here: https://github.com/stan-dev/cmdstanpy/actions/runs/5939297312/job/16105419727

I know from local development we have one test which will fail because we check too specific of an error message, but otherwise I expect it to pass

@WardBrian
Copy link
Member Author

The following docs PRs should get merged during this period:

stan-dev/docs#665
stan-dev/docs#651
stan-dev/docs#625

@mitzimorris
Copy link
Member

reviewed and approved docs#625

@rok-cesnovar
Copy link
Member

RC announcements are out on Discours, Slack & Twitter. Please share to other channels where you think we could find some eager testers.

@WardBrian
Copy link
Member Author

WardBrian commented Aug 25, 2023 •
edited
Loading

Bug fixes so far:

@WardBrian
Copy link
Member Author

WardBrian commented Aug 28, 2023 •
edited
Loading

I found a pretty nasty bug in #1191, which I think should be fixed in this release.

And in the compiler, a simple one: stan-dev/stanc3#1351

@WardBrian
Copy link
Member Author

https://gist.github.com/WardBrian/2898738905ab3592e33b0897d7129020

I believe these release notes are up to date. If you want to update the version bump PRs @serban-nicusor-toptal i think everything else is ready for the release tomorrow

@serban-nicusor-toptal
Copy link
Contributor

Hey @WardBrian everything should be ready.
On a side note before the release, what do you think about increasing the quality gate threshold to 3 from 1 here https://github.com/stan-dev/cmdstan/blob/develop/Jenkinsfile#L271 ?
And that's because I think it's too strict, any small warning will make a pipeline fail/unstable, eg. https://jenkins.flatironinstitute.org/job/Stan/job/Math/job/PR-2932/1/

@WardBrian
Copy link
Member Author

I would be in favor but arguably that’s just hiding the issue. See also discussions stan-dev/math#2919

which reminds me, did we make a final call on adding -w to CXXFLAGS as part of this release?

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Sep 5, 2023 •
edited
Loading

which reminds me, did we make a final call on adding -w to CXXFLAGS as part of this release?

I'm not entirely sure, I don't see any PR merged wrt to it.
Edit: Math pipeline passed now https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FMath/detail/PR-2932/2/pipeline
If everything is ready to go, I'll start with the release shortly.
Edit: Release in progress, waiting for stanc3 builds ...

@mitzimorris
Copy link
Member

people keep running into the sprintf warnings issue - someone brought it up at last week's online meeting.
cons:

  • adding -w to CXXFLAGS could hide warnings that might be useful, but those don't come up very often.
    pros:
  • avoid user confusion.
    at this point, we can wait and see, and perhaps document this behavoir somehow - "spurious warnings"?

@serban-nicusor-toptal
Copy link
Contributor

From my side I can create a PR in cmdstan to just exclude that warning, the problem with this approach is that any new warning will fail the pipeline and need an exclude.

@WardBrian
Copy link
Member Author

@serban-nicusor-toptal the proposal is to add that flag only in the release tarball, never in anything checked in to git

@serban-nicusor-toptal
Copy link
Contributor

Oh I see, I misunderstood. I'll add it now then.

@serban-nicusor-toptal
Copy link
Contributor

Do we add this flag in the Math tarball? To which variable in here should we add it?
Do we do anything with cmdstan flags ?

@WardBrian
Copy link
Member Author

CXXFLAGS_WARNINGS seems like a reasonable place to add it. I'm not sure how it interacts with other flags, so we'd want to either completely override that variable or set it as the last argument.

This could also be something we test out locally before next release, rather than setting last minute

@serban-nicusor-toptal
Copy link
Contributor

I can wait if you want to give it a local test. If everything is working as expected, we can just replace the tarball later or wait until the next release.

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Sep 5, 2023 •
edited
Loading

Release wise we're almost ready, the -w flag was not added. As Brian said it's better to discuss it and test it properly instead of rushing it.
PRs for docs are ready if anyone wants to take a look before merging so new docs don't 404:

I'm waiting on the collab tarball for cmdstan (done) and will need to merge develop -> master but other than this everything should be good to go.

CmdStanR tests running at https://github.com/stan-dev/cmdstanr/actions/runs/6085703885

@WardBrian
Copy link
Member Author

Docs look good. Thanks @serban-nicusor-toptal!

@rok-cesnovar are you able to make the blog/forum/social media posts?

@rok-cesnovar
Copy link
Member

Yes, once the docs are live I can edit the blog draft with the new links and post it.

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Sep 5, 2023 •
edited
Loading

Created also PRs for develop -> master:

I'll now merge the docs and after fixing the conflicts we can merge the above PRs.

@serban-nicusor-toptal
Copy link
Contributor

Update: docs are now up!

@rok-cesnovar
Copy link
Member

Blog post as well as Discourse, Slack and Twitter/X announcements have been posted.

Thanks to everyone who made this release possible, it's a big one! Thanks to @WardBrian and @serban-nicusor-toptal for preparing the release!

@WardBrian
Copy link
Member Author

I believe at this point we're only waiting on CI on stan-dev/math#2941 before all the items on the checklist are done

@WardBrian
Copy link
Member Author

Thank you to everyone who helped with this release!

At this point the feature freeze for this release is over. We can monitor for bug reports but the RC period was pretty quiet all things considered

@WardBrian
Copy link
Member Author

@serban-nicusor-toptal what day this week works well for you to run a release for 2.33.1?

Currently, the only changes are still in stanc3 based off of stan-dev/stanc3#1356, but the other repos do still get version bumps I believe

@serban-nicusor-toptal
Copy link
Contributor

Hey Brian, any day works for me, Wednesday I'll have a busier morning but the afternoon is fine. Do we need to do it also for Math and Stan ? Cmdstan will change because of the stanc3 binaries but if nothing changed in Stan/Math the submodule references can stay the same I think.

@WardBrian
Copy link
Member Author

I'd prefer tomorrow or Wednesday to avoid releasing close to a weekend.

Good question. It looks like for 2.32.2 (which only updated the Stan repo) we did also have a minor version bump for Math. It's probably not necessary, however.

@rok-cesnovar - thoughts?

@serban-nicusor-toptal
Copy link
Contributor

That's great, If we want to wait a day more just in case anything pops up, Wednesday is fine by me!

@serban-nicusor-toptal
Copy link
Contributor

Good morning, if everything is ready I'll do the patch release a bit later today. @WardBrian @rok-cesnovar do I skip Math, Stan?

@rok-cesnovar
Copy link
Member

rok-cesnovar commented Sep 13, 2023 •
edited
Loading

Yes, only stanc3 and cmdstan tarballs.

@WardBrian
Copy link
Member Author

Thanks @serban-nicusor-toptal!

@serban-nicusor-toptal
Copy link
Contributor

Hey @WardBrian as release notes for stanc3 should I use just?:

  • Fix two issues with tuple functions (#1356)

For cmdstan I will use:

  • Updated stanc3 binaries to v2.33.1

@WardBrian
Copy link
Member Author

I think "Fixed several issues with the code generation of user-defined functions which accepted tuples (#1356)" (it ended up being more than two)

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Sep 13, 2023 •
edited
Loading

Thanks! I'll start shortly. ( Jenkins is scheduled for a reboot, most probably maintenance stuff. Might take a bit longer as builds will pass through only after the reboot )

@serban-nicusor-toptal
Copy link
Contributor

serban-nicusor-toptal commented Sep 13, 2023 •
edited
Loading

Everything should be ready now!

I've created a PR for stanc3 and 2 for cmdstan(I think we forgot to merge the release branch into develop too):

@WardBrian
Copy link
Member Author

Those all look good @serban-nicusor-toptal.

Thanks!

@serban-nicusor-toptal
Copy link
Contributor

The cmdstan PRs are failing, weird as the one for develop is successful.
https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FCmdStan/detail/PR-1201/3/pipeline
Does anyone know why would it fail here? I'm not sure why that test failed.

@WardBrian
Copy link
Member Author

#1197 was the fix for that, but it won’t be in this branch I don’t think

@serban-nicusor-toptal
Copy link
Contributor

Should I try to merge it in ? I hope that won't require a release.

@WardBrian
Copy link
Member Author

Just merge it I think. The only changes were in test code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rok-cesnovar rok-cesnovar

@WardBrian WardBrian

@serban-nicusor-toptal serban-nicusor-toptal

Labels
release/packaging
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mitzimorris @wds15 @rok-cesnovar @WardBrian @serban-nicusor-toptal