3937 rewrite installer in python again

[landreev]

What this PR does / why we need it:
New Installer Implementation, in Python
Which issue(s) this PR closes:

Closes #3937

Special notes for your reviewer:

Suggestions on how to test this:

Does this PR introduce a user interface change?:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

Coverage decreased (-0.009%) to 19.504% when pulling a35e3f6 on 3937-rewrite-installer-in-python-again into b70cd18 on develop.

[pdurbin]

Overall, I think this looks good so I'm sending it to QA. That said, her are some (minor) suggestions, questions, and comments. @landreev should feel free to pull this out of QA if he wants to work on any of this.

Suggestions:

• in the Perl install script, mention the new Python script
• in requirements.txt, put version of psycopg2
• consider recommending Python 3's venv (virtual environment)

Questions:

• 10.5072 is configured but when they contact DataCite do they get a new authority for testing?
• Any thoughts on when we'll stop supporting the Perl installer?

Comments:

• @donsizemore is planning on testing the new Python installer as part of support new Python 3 installer (formerly in Perl) dataverse-ansible#139
• I'm not sure if anyone will miss the "podName" stuff but I guess it doesn't hurt to leave it in there. Recently @poikilotherm mentioned OpenShift (which podName was added for) at Test for OpenShift 4.x+ compatibility gdcc/dataverse-kubernetes#129
• It would be nice to support Windows dev environments some day. Out of scope for now.

[djbrooke]

@donsizemore, @landreev mentioned that you were discussing this quite a bit. Your review here would be very appreciated if you have the time to do so. Thanks in advance!

[donsizemore]

@landreev what's the relationship between default.config and interactive.config? I popped the latter into Ansible as a template but the install script seems to want default.config's structure instead.

I can swap out the templates, but default.config has 30 lines of content while interactive.config has 47. It's looking like I'll need both?

[landreev]

@landreev what's the relationship between default.config and interactive.config? I popped the latter into Ansible as a template but the install script seems to want default.config's structure instead.

I can swap out the templates, but default.config has 30 lines of content while interactive.config has 47. It's looking like I'll need both?

(@pdurbin mentioned he had already answered this, but just to clarify and document this here:)

Yes, the script needs both.
The default.config is for supplying the configuration values. The installer also has a new option --config_file= that can be used to use a custom file, instead of the default.config.

intreractive.config stores the prompts and info/help messages for the interactive setup dialog. It's the stuff that's hard-coded in %CONFIG_PROMPTS and %CONFIG_COMMENTS in the old installer. Storing it in a config file outside the script should make it cleaner/easier to maintain (or even localize the installer in a different language, if anyone ever feels a need for that...).

Thinking about it, there's no need to load and parse this file at all, when the installer is running in the non-interactive mode. I'll add an if statement for that.

[donsizemore]

@landreev one more question: previously the dataverse installer captured the output of glassfish-setup into glassfish-setup.out — do we want to keep that, or offer an output file flag? helpful for debugging...

[poikilotherm]

Nobody asked for it, but my 2 cents: as long as @donsizemore is not relying on the OpenShift workarounds, feel free to let anything related to it RIP in good ol' Perl version.

@pdurbin and I where going to cut out the Openshift stuff from the guide anyway and move all cloud/container regarding OpenShift to dataverse-k8s. See also gdcc/dataverse-kubernetes#129

[pdurbin]

Oh, I guess I'll throw in a few cents too. Back when the (Perl) installer required root I wrote a little shell script called "devinstall" at f181543 that seemed to work for me as an alternative. These days you can also find a slight variation of this script at https://github.com/IQSS/dataverse/blob/v4.18.1/scripts/deploy/phoenix.dataverse.org/install

These days I just use the standard installer for dev environments rather than my "devinstall" script. But on servers, I use dataverse-ansible, which doesn't use the old Perl script. I guess if dataverse-ansible uses the new Python script I'll start using it on servers.

I'm not exactly sure where I'm going with this. I guess I'm happy that the new Python script will be exercised regularly by dataverse-ansible. I think somewhere in docker-aio there's an option to use the old Perl script but I'm not in the habit of using docker-aio. In short, I'm glad the new code will be tested regularly, at least the non-interactive mode.

[donsizemore]

dataverse-ansible now deploys the 4.18.1 release normally using the python3 installer instead of a bunch of calls I had broken out into Ansible-ese. I did keep the JDBC jar placement in Ansible's realm as that's a configurable switch for testing of newer jars. If anyone would like to give the work-in-progress a whirl, it's https://github.com/IQSS/dataverse-ansible/tree/139_python3_installer

[kcondon]

Issues found so far:

1. Add to readme: yum install postgresql96-devel needed so .h files available for pip install psycopg2
2. default.config from original non-interactive installer has same name as default.config used for python install and may conflict, resulting in failed install if downloaded from doc.
3. *.config files for python missing from dvinstall.zip
4. Replace default system email, nobody@mailinator.com, with blank
5. Maybe tighten up README to indicate supported python versions for psycopg: http://initd.org/psycopg/docs/install.html v2.7 or v3.4-3.8
6. Need python-devel to be installed for Python.h to build psycopg.

[landreev]

Yay! I'll give it a try.

@donsizemore

dataverse-ansible now deploys the 4.18.1 release normally using the python3 installer instead of a bunch of calls I had broken out into Ansible-ese.

[pdurbin]

@donsizemore wow! Looking at https://github.com/IQSS/dataverse-ansible/compare/139_python3_installer I see you've been busy!! Thanks!! 🎉

[kcondon]

All set testing, passing back to dev.

[landreev]

@kcondon, I addressed the feedback below. With the exception of the old-style default.config - I figured it wasn't necessary, just because of how unlikely it is to happen to anyone else. (and easy to fix if it does :)

Issues found so far:

1. Add to readme: yum install postgresql96-devel needed so .h files available for pip install psycopg2

2. default.config from original non-interactive installer has same name as default.config used for python install and may conflict, resulting in failed install if downloaded from doc.

3. *.config files for python missing from dvinstall.zip

4. Replace default system email, [nobody@mailinator.com](mailto:nobody@mailinator.com), with blank

5. Maybe tighten up README to indicate supported python versions for psycopg: http://initd.org/psycopg/docs/install.html v2.7 or v3.4-3.8

6. Need python-devel to be installed for Python.h to build psycopg.

[landreev]

@pdurbin

* Any thoughts on when we'll stop supporting the Perl installer?

My thought was to release it as beta/experimental in the next release; invite people to give it a try; incorporate any feedback we receive and then switch to it as the default installer for the release after that.
But idk - I'm feeling a little discouraged by the fact that a user needs to do more work to run the new installer than the old one. For ex., having to install psycopg2 - at least some people will definitely run into some issues with that.
By which I mean, none of it is going to be a problem for another developer. But not everybody trying to install Dataverse from a release bundle is a developer, or super technical necessarily.

This is something we may want to discuss as a team at some point during the next release cycle. I personally would be ok to say that you have to be somewhat of a sysadmin/be comfortable with python, etc. to install Dataverse.