Merge branch 'develop' into 8531-refactor-validators

conf/docker-aio/1prep.sh

@@ -12,10 +12,10 @@ cd ../../
 cp -r scripts conf/docker-aio/testdata/
 cp doc/sphinx-guides/source/_static/util/createsequence.sql conf/docker-aio/testdata/doc/sphinx-guides/source/_static/util/
 
-wget -q https://downloads.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz
-tar xfz apache-maven-3.8.4-bin.tar.gz
+wget -q https://downloads.apache.org/maven/maven-3/3.8.5/binaries/apache-maven-3.8.5-bin.tar.gz
+tar xfz apache-maven-3.8.5-bin.tar.gz
 mkdir maven
-mv apache-maven-3.8.4/* maven/
+mv apache-maven-3.8.5/* maven/
 echo "export JAVA_HOME=/usr/lib/jvm/jre-openjdk" > maven/maven.sh
 echo "export M2_HOME=../maven" >> maven/maven.sh
 echo "export MAVEN_HOME=../maven" >> maven/maven.sh

doc/release-notes/5.10.1-release-notes.md

@@ -0,0 +1,96 @@
+# Dataverse Software 5.10.1
+
+This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Bug Fix for Request Access
+
+Dataverse Software 5.10 contains a bug where the "Request Access" button doesn't work from the file listing on the dataset page if the dataset contains custom terms. This has been fixed in PR #8555.
+
+### Bug Fix for Searching and Selecting Controlled Vocabulary Values
+
+Dataverse Software 5.10 contains a bug where the search option is no longer present when selecting from more than ten controlled vocabulary values. This has been fixed in PR #8521.
+
+## Major Use Cases and Infrastructure Enhancements
+
+Changes and fixes in this release include:
+
+- Users can use the "Request Access" button when the dataset has custom terms. (Issue #8553, PR #8555)
+- Users can search when selecting from more than ten controlled vocabulary values. (Issue #8519, PR #8521)
+- The default file categories ("Documentation", "Data", and "Code") can be redefined through the `:FileCategories` database setting. (Issue #8461, PR #8478)
+- Documentation on troubleshooting Excel ingest errors was improved. (PR #8541)
+- Internationalized controlled vocabulary values can now be searched. (Issue #8286, PR #8435)
+- Curation labels can be internationalized. (Issue #8381, PR #8466)
+- "NONE" is no longer accepted as a license using the SWORD API (since 5.10). See "Backward Incompatibilities" below for details. (Issue #8551, PR #8558).
+
+## Notes for Dataverse Installation Administrators
+
+### PostgreSQL Version 10+ Required Soon
+
+Because 5.10.1 is a bug fix release, an upgrade to PostgreSQL is not required. However, this upgrade is still coming in the next non-bug fix release. For details, please see the release notes for 5.10: https://github.com/IQSS/dataverse/releases/tag/v5.10
+
+### Payara Upgrade
+
+You may notice that the Payara version used in the install scripts has been updated from 5.2021.5 to 5.2021.6. This was to address a bug where it was not possible to easily update the logging level. For existing installations, this release does not require upgrading Payara and a Payara upgrade is not part of the Upgrade Instructions below. For more information, see PR #8508.
+
+## New JVM Options and DB Settings
+
+The following DB settings have been added:
+
+- `:FileCategories` - The default list of the pre-defined file categories ("Documentation", "Data" and "Code") can now be redefined with a comma-separated list (e.g. `'Docs,Data,Code,Workflow'`).
+
+See the [Database Settings](https://guides.dataverse.org/en/5.10.1/installation/config.html#database-settings) section of the Guides for more information.
+
+## Notes for Developers and Integrators
+
+In the "Backward Incompatibilities" section below, note changes in the API regarding licenses and the SWORD API.
+
+## Backward Incompatibilities
+
+As of Dataverse 5.10, "NONE" is no longer supported as a valid license when creating a dataset using the SWORD API. The API Guide has been updated to reflect this. Additionally, if you specify an invalid license, a list of available licenses will be returned in the response.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the [5.10.1 Milestone](https://github.com/IQSS/dataverse/milestone/102?closed=1) in Github.
+
+For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org.
+
+## Installation
+
+If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.10.1/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.10.1/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already.
+
+## Upgrade Instructions
+
+0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.10.1.
+
+If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user.  
+
+In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed.
+
+`export PAYARA=/usr/local/payara5`
+
+(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell)
+
+1\. Undeploy the previous version.
+
+- `$PAYARA/bin/asadmin list-applications`
+- `$PAYARA/bin/asadmin undeploy dataverse<-version>`
+
+2\. Stop Payara and remove the generated directory
+
+- `service payara stop`
+- `rm -rf $PAYARA/glassfish/domains/domain1/generated`
+
+3\. Start Payara
+
+- `service payara start`
+
+4\. Deploy this version.
+
+- `$PAYARA/bin/asadmin deploy dataverse-5.10.1.war`
+
+5\. Restart payara
+
+- `service payara stop`
+- `service payara start`

doc/sphinx-guides/source/admin/metadatacustomization.rst

@@ -574,8 +574,6 @@ The scripts required can be hosted locally or retrieved dynamically from https:/
 Tips from the Dataverse Community
 ---------------------------------
 
-When creating new metadatablocks, please review the :doc:`/style/text` section of the Style Guide, which includes guidance about naming metadata fields and writing text for metadata tooltips.
-
 If there are tips that you feel are omitted from this document, please open an issue at https://github.com/IQSS/dataverse/issues and consider making a pull request to make improvements. You can find this document at https://github.com/IQSS/dataverse/blob/develop/doc/sphinx-guides/source/admin/metadatacustomization.rst
 
 Alternatively, you are welcome to request "edit" access to this "Tips for Dataverse Software metadata blocks from the community" Google doc: https://docs.google.com/document/d/1XpblRw0v0SvV-Bq6njlN96WyHJ7tqG0WWejqBdl7hE0/edit?usp=sharing

doc/sphinx-guides/source/api/sword-atom-entry.xml

@@ -27,8 +27,8 @@
    <dcterms:coverage>United States</dcterms:coverage>
    <dcterms:coverage>Canada</dcterms:coverage>
    <!-- license and restrictions -->
-   <dcterms:license>NONE</dcterms:license>
-   <dcterms:rights>Downloader will not use the Materials in any way prohibited by applicable laws.</dcterms:rights>
+   <dcterms:license>CC0 1.0</dcterms:license>
+   <!--<dcterms:rights>Downloader will not use the Materials in any way prohibited by applicable laws.</dcterms:rights>-->
    <!-- related publications -->
    <dcterms:isReferencedBy holdingsURI="http://dx.doi.org/10.1038/dvn333" agency="DOI" IDNo="10.1038/dvn333">Peets, J., &amp; Stumptown, J. (2013). Roasting at Home. New England Journal of Coffee, 3(1), 22-34.</dcterms:isReferencedBy>
 </entry>

doc/sphinx-guides/source/api/sword.rst

@@ -65,6 +65,8 @@ Differences in Dataverse Software 4 from DVN 3.x lead to a few minor backward in
 
 - The Service Document will show a single API Terms of Use rather than root level and Dataverse collection level Deposit Terms of Use.
 
+- As of Dataverse Software 5.10, ``NONE`` is no longer supported as a license.
+
 New features as of v1.1
 -----------------------
 
@@ -80,7 +82,7 @@ New features as of v1.1
 
 - "Contributor" can now be populated and the "Type" (Editor, Funder, Researcher, etc.) can be specified with an XML attribute. For example: ``<dcterms:contributor type="Funder">CaffeineForAll</dcterms:contributor>``
 
-- "License" can now be set with ``dcterms:license`` and the possible values are "CC0" and "NONE". "License" interacts with "Terms of Use" (``dcterms:rights``) in that if you include ``dcterms:rights`` in the XML, the license will be set to "NONE". If you don't include ``dcterms:rights``, the license will default to "CC0". It is invalid to specify "CC0" as a license and also include ``dcterms:rights``; an error will be returned. For backwards compatibility, ``dcterms:rights`` is allowed to be blank (i.e. ``<dcterms:rights></dcterms:rights>``) but blank values will not be persisted to the database and the license will be set to "NONE".
+- "License" can now be set with ``dcterms:license`` and the possible values determined by the installation ("CC0 1.0" and "CC BY 4.0" by default). "License" interacts with "Terms of Use" (``dcterms:rights``) in that if you include ``dcterms:rights`` in the XML and don't include ``dcterms:license``, the license will be "Custom Dataset Terms" and "Terms of Use" will be populated. If you don't include ``dcterms:rights``, the default license will be used. It is invalid to specify a license and also include ``dcterms:rights``; an error will be returned. For backwards compatibility, ``dcterms:rights`` is allowed to be blank (i.e. ``<dcterms:rights></dcterms:rights>``) but blank values will not be persisted to the database and the license will be set to "Custom Dataset Terms".
 
 - "Contact E-mail" is automatically populated from dataset owner's email.
 
@@ -143,9 +145,9 @@ Dublin Core Terms (DC Terms) Qualified Mapping - Dataverse Project DB Element Cr
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
 |dcterms:coverage             |              otherGeographicCoverage         |              |  General information on the geographic coverage of the Dataset.                                                                                             |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|dcterms:license              |                   license                    |              |  Set the license to CC0 (default in a Dataverse installation for new Datasets), otherwise enter "NONE" and fill in the dcterms:rights field.                |                
+|dcterms:license              |                   license                    |              |  Set the license. Alternatively, use the dcterms:rights field instead.                                                                                      |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|dcterms:rights               |                 termsofuse                   |              |  If not using CC0, enter any terms of use or restrictions for the Dataset.                                                                                  |
+|dcterms:rights               |                 termsofuse                   |              |  If not using dcterms:license, enter any terms of use or restrictions for the Dataset.                                                                      |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
 |dcterms:isReferencedBy       |             publicationCitation              |              |  The publication (journal article, book, other work) that uses this dataset (include citation, permanent identifier (DOI), and permanent URL).              |
 +-----------------------------+----------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+

doc/sphinx-guides/source/conf.py

@@ -65,9 +65,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '5.10'
+version = '5.10.1'
 # The full version, including alpha/beta/rc tags.
-release = '5.10'
+release = '5.10.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/sphinx-guides/source/style/index.rst

@@ -14,4 +14,3 @@ This style guide is meant to help developers implement clear and appropriate UI
 
    foundations
    patterns
-   text

doc/sphinx-guides/source/user/tabulardataingest/excel.rst

@@ -1,10 +1,32 @@
 Excel 
 +++++++
 
+Microsoft Excel files (XLSX format).
+
 .. contents:: |toctitle|
 	:local:
 
-Excel files (**New** in Dataverse Software 4.0!)
+Supported Formats
+------------------
+
+Only the newer XLSX Excel files are supported. We are not planning to add support for the old-style, binary XLS files.
+
+Limitations
+-----------
+
+If an Excel file has multiple sheets, only the first sheet of the file will be ingested. The other sheets will be available when a user downloads the original Excel file. To have all sheets of an Excel file ingested and searchable at the variable level, upload each sheet as an individual file in your dataset.
+
+Ingest Errors
+-------------
+
+You may encounter ingest errors after uploading an Excel file if the file is formatted in a way that can't be ingested by the Dataverse software. Ingest errors can be caused by a variety of formatting inconsistencies, including:
+
+* line breaks in a cell
+* blank cells
+* single cells that span multiple rows
+* missing headers
 
-Note: only the "new", XLSX Excel files are supported. We are not planning to add support for the old-style, binary XLS files.
+Example Data
+------------
 
+`An example of an Excel file that successfully ingests <https://github.com/IQSS/dataverse-sample-data/blob/master/data/dataverses/dataverseno/datasets/tabular-sample-data/files/Tabular_Sample_Data.xlsx>`_ is available in the Dataverse Sample Data GitHub repository.

doc/sphinx-guides/source/user/tabulardataingest/rdata.rst

@@ -6,20 +6,19 @@ Support for R (.RData) files has been introduced in DVN 3.5.
 .. contents:: |toctitle|
     :local:
 
-Overview.
+Overview
 ===========
 
-
 R has been increasingly popular in the research/academic community,
 owing to the fact that it is free and open-source (unlike SPSS and
 STATA). Consequently, there is an increasing amount of data available
 exclusively in R format.  
 
-Data Formatting Requirements.
+Data Formatting Requirements
 ==============================
 
-The data must be formatted as an R dataframe (data.frame()). If an
-.RData file contains multiple dataframes, only the 1st one will be
+The data must be formatted as an R dataframe (``data.frame()``). If an
+.RData file contains multiple dataframes, only the first one will be
 ingested (this may change in the future).
 
 Data Types, compared to other supported formats (Stat, SPSS)
@@ -82,7 +81,7 @@ Boolean values
 R Boolean (logical) values are supported. 
 
 
-Limitations of R, as compared to SPSS and STATA. 
+Limitations of R, as compared to SPSS and STATA
 ------------------------------------------------
 
 Most noticeably, R lacks a standard mechanism for defining descriptive

doc/sphinx-guides/source/versions.rst

@@ -6,8 +6,9 @@ Dataverse Software Documentation Versions
 
 This list provides a way to refer to the documentation for previous versions of the Dataverse Software. In order to learn more about the updates delivered from one version to another, visit the `Releases <https://github.com/IQSS/dataverse/releases>`__ page in our GitHub repo.
 
-- 5.10
+- 5.10.1
 
+- `5.10 </en/5.10/>`__
 - `5.9 </en/5.9/>`__
 - `5.8 </en/5.8/>`__
 - `5.7 </en/5.7/>`__

modules/dataverse-parent/pom.xml

@@ -129,7 +129,7 @@
 
     <properties>
         <!-- This is a special Maven property name, do not change! -->
-        <revision>5.10</revision>
+        <revision>5.10.1</revision>
 
         <target.java.version>11</target.java.version>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>