Merge branch 'develop' into 7307-get-user-by-token

doc/release-notes/5.1.1-release-notes.md

@@ -24,9 +24,12 @@ As mentioned above, we encourage 5.1.1 instead of 5.1 for production use.
 
 Larger installations may want to increase the number of open S3 connections allowed (default is 256). For example, to set the value to 4096:
 
-``./asadmin create-jvm-options "-Ddataverse.files.<id>.connection-pool-size=4096"`
+`./asadmin create-jvm-options "-Ddataverse.files.<id>.connection-pool-size=4096"`
+(where `<id>` is the identifier of your S3 file store (likely `"s3"`). The JVM Options section of the [Configuration Guide](http://guides.dataverse.org/en/5.1.1/installation/config/) has more information.
 
-The JVM Options section of the [Configuration Guide](http://guides.dataverse.org/en/5.1.1/installation/config/) has more information.
+### New S3 Bucket CORS setting for Direct Upload/Download
+
+When using S3 storage with direct upload and/or download enabled, one must now expose the ETag header as documented in the [updated cors.json example](https://guides.dataverse.org/en/5.1.1/developers/big-data-support.html?highlight=etag#s3-direct-upload-and-download).
 
 ## Complete List of Changes
 
@@ -44,16 +47,17 @@ If this is a new installation, please see our [Installation Guide](http://guides
 
 1. Undeploy the previous version.
 
-<payara install path>/payara/bin/asadmin list-applications
-<payara install path>/payara/bin/asadmin undeploy dataverse
+`<payara install path>/bin/asadmin list-applications`
+`<payara install path>/bin/asadmin undeploy dataverse<-version>`
 
 2. Stop payara and remove the generated directory, start.
 
-- service payara stop
-- remove the generated directory: rm -rf <payara install path>payara/payara/domains/domain1/generated
-- service payara start
+- `service payara stop`
+- remove the generated directory:
+`rm -rf <payara install path>/glassfish/domains/domain1/generated`
+- `service payara start`
 
 3. Deploy this version.
-<payara install path>/payara/bin/asadmin deploy <path>dataverse-5.1.1.war
+`<payara install path>/bin/asadmin deploy dataverse-5.1.1.war`
 
 4. Restart payara

doc/release-notes/6919-preview-tools.md

@@ -0,0 +1,38 @@
+## Release Highlights
+
+### File Preview When Guestbooks or Terms Exist
+
+Previously, file preview was only available when files were publicly downloadable. Now if a guestbook or terms (or both) are configured for the dataset, they will be shown in the Preview tab and once they are agreed to, the file preview will appear (#6919).
+
+### Preview Only External Tools
+
+A new external tool type called "preview" has been added that prevents the tool from being displayed under "Explore Options" under the "Access File" button on the file landing page (#6919).
+
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Users can now preview files that have a guestbook or terms. (Issue #6919)
+- External tools authors can indicate that their tool is "preview only". (Issue #6919)
+
+
+## Notes for Dataverse Installation Administrators
+
+### Converting Explore External Tools to Preview Only
+
+When the war file is deployed, a SQL migration script will convert [dataverse-previewers][] to have both "explore" and "preview" types so that they will continue to be displayed in the Preview tab.
+
+If you would prefer that these tools be preview only, you can delete the tools, adjust the JSON manifests (changing "explore" to "preview"), and re-add them.
+
+[dataverse-previewers]: https://github.com/GlobalDataverseCommunityConsortium/dataverse-previewers
+
+## Notes for Tool Developers and Integrators
+
+### Preview Only External Tools
+
+A new external tool type called "preview" has been added that prevents the tool from being displayed under "Explore Options" under the "Access File" button on the file landing page (#6919). This "preview" type replaces "hasPreviewMode", which has been removed.
+
+### Multiple Types for External Tools
+
+External tools now support multiple types. In practice, the types "explore" and "preview" are the only combination that makes a difference in the UI as opposed to only having only one or the other type (see "preview only" above). Multiple types are specified in the JSON manifest with an array in "types". The older, single "type" is still supported but should be considered deprecated.

doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv

@@ -1,3 +1,4 @@
+Tool	Type	Scope	Description
 TwoRavens	explore	file	A system of interlocking statistical tools for data exploration, analysis, and meta-analysis: http://2ra.vn. See the :doc:`/user/data-exploration/tworavens` section of the User Guide for more information on TwoRavens from the user perspective and the :doc:`/installation/r-rapache-tworavens` section of the Installation Guide. 
 Data Explorer	explore	file	A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Explorer for the instructions on adding Data Explorer to your Dataverse; and the :doc:`/installation/prerequisites` section of the Installation Guide for the instructions on how to set up **basic R configuration required** (specifically, Dataverse uses R to generate .prep metadata files that are needed to run Data Explorer). 
 Whole Tale	explore	dataset	A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide <https://wholetale.readthedocs.io/en/stable/users_guide/integration.html>`_.

doc/sphinx-guides/source/_static/installation/files/root/external-tools/dynamicDatasetTool.json

@@ -2,7 +2,9 @@
   "displayName": "Dynamic Dataset Tool",
   "description": "Dazzles! Dizzying!",
   "scope": "dataset",
-  "type": "explore",
+  "types": [
+    "explore"
+  ],
   "toolUrl": "https://dynamicdatasettool.com/v2",
   "toolParameters": {
     "queryParameters": [

doc/sphinx-guides/source/_static/installation/files/root/external-tools/fabulousFileTool.json

@@ -3,7 +3,10 @@
   "description": "Fabulous Fun for Files!",
   "toolName": "fabulous",
   "scope": "file",
-  "type": "explore",
+  "types": [
+    "explore",
+    "preview"
+  ],
   "toolUrl": "https://fabulousfiletool.com",
   "contentType": "text/tab-separated-values",
   "toolParameters": {

doc/sphinx-guides/source/admin/external-tools.rst

@@ -1,7 +1,7 @@
 External Tools
 ==============
 
-External tools can provide additional features that are not part of Dataverse itself, such as data exploration.
+External tools can provide additional features that are not part of Dataverse itself, such as data file previews, visualization, and curation.
 
 .. contents:: |toctitle|
   :local:
@@ -12,7 +12,7 @@ Inventory of External Tools
 ---------------------------
 
 .. csv-table:: 
-   :header: "Tool", "Type", "Scope", "Description"
+   :header-rows: 1
    :widths: 20, 10, 5, 65
    :delim: tab
    :file: ../_static/admin/dataverse-external-tools.tsv
@@ -31,14 +31,12 @@ To add an external tool to your installation of Dataverse you must first downloa
 
 Go to :ref:`inventory-of-external-tools` and download a JSON manifest for one of the tools by following links in the description to installation instructions.
 
-In the curl command below, replace the placeholder "fabulousFileTool.json" placeholder for the actual name of the JSON file you downloaded.
+Configure the tool with the curl command below, making sure to replace the ``fabulousFileTool.json`` placeholder for name of the JSON manifest file you downloaded.
 
 .. code-block:: bash
 
   curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file fabulousFileTool.json 
 
-Note that some tools will provide a preview mode, which provides an embedded, simplified view of the tool on the file pages of your installation. This is controlled by the `hasPreviewMode` parameter. 
-
 Listing All External Tools in Dataverse
 +++++++++++++++++++++++++++++++++++++++
 
@@ -75,17 +73,26 @@ Testing External Tools
 
 Once you have added an external tool to your installation of Dataverse, you will probably want to test it to make sure it is functioning properly.
 
+File Level vs. Dataset Level
+++++++++++++++++++++++++++++
+
+File level tools are specific to the file type (content type or MIME type). For example, a tool may work with PDFs, which have a content type of "application/pdf". 
+
+In contrast, dataset level tools are always available no matter what file types are within the dataset.
+
 File Level Explore Tools
 ++++++++++++++++++++++++
 
-File level explore tools are specific to the file type (content type or MIME type) of the file. For example, Data Explorer is tool for exploring tabular data files.
+File level explore tools provide a variety of features from data visualization to statistical analysis.
 
-An "Explore" button will appear (on both the dataset page and the file landing page) for files that match the type that the tool has been built for. When there are multiple explore tools for a filetype, the button becomes a dropdown.
+For each supported file type, file level explore tools appear in the file listing of the dataset page as well as under the "Access" button on each file page.
 
 File Level Preview Tools
 ++++++++++++++++++++++++
 
-File level explore tools can be set up to display in preview mode, which is a simplified view of an explore tool designed specifically for embedding in the file page. 
+File level preview tools allow the user to see a preview of the file contents without having to download it.
+
+When a file has a preview available, a preview icon will appear next to that file in the file listing on the dataset page. On the file page itself, the preview will appear in a Preview tab either immediately or once a guestbook has been filled in or terms, if any, have been agreed to.
 
 File Level Configure Tools
 ++++++++++++++++++++++++++
@@ -95,12 +102,12 @@ File level configure tools are only available when you log in and have write acc
 Dataset Level Explore Tools
 +++++++++++++++++++++++++++
 
-When a dataset level explore tool is added, an "Explore" button on the dataset page will appear. This button becomes a drop down when there are multiple tools.
+Dataset level explore tools allow the user to explore all the files in a dataset.
 
 Dataset Level Configure Tools
 +++++++++++++++++++++++++++++
 
-Configure tools at the dataset level are not currently supported. No button appears in the GUI if you add this type of tool.
+Configure tools at the dataset level are not currently supported.
 
 Writing Your Own External Tool
 ------------------------------

doc/sphinx-guides/source/admin/geoconnect-worldmap.rst

@@ -35,15 +35,15 @@ SQL commands to point a Dataverse installation at different Geoconnect servers:
 Removing Dead Explore Links
 ---------------------------
 
-After a map has been created in WorldMap (assuming all the setup has been done), an "Explore" button will appear next to the name of the file in Dataverse. The "Explore" button should open the map in WorldMap. In rare occasions, the map has been deleted on the WorldMap side such that the "Explore" button goes nowhere, resulting in a dead link, a 404.
+After a map has been created in WorldMap (assuming all the setup has been done), in Dataverse the file will display WorldMap as an explore option. In rare occasions, the map has been deleted on the WorldMap resulting in a dead link, a 404 page not found error.
 
 Functionality has been added on the Dataverse side to iterate through all the maps Dataverse knows about (stored in the ``maplayermetadata`` database table) and to check for the existence of each map in WorldMap. The status code returned from WorldMap (200, 404, etc.) is recorded in Dataverse along with a timestamp of when the check was performed. To perform this check, you can execute the following ``curl`` command:
 
 ``curl -X POST http://localhost:8080/api/admin/geoconnect/mapLayerMetadatas/check``
 
 The output above will contain the ``layerLink`` being checked as well as the HTTP response status code (200, 404, etc.) in the ``lastVerifiedStatus`` field. 200 means OK and 404 means not found. 500 might indicate that the map is only temporarily unavailable. The ``lastVerifiedStatus`` and ``lastVerifiedTime`` will be persisted to the ``maplayermetadata`` database table.
 
-Armed with this information about WorldMap returning a 404 for a map, you may want to delete any record of the map on the Dataverse side so that the "Explore" button goes away (and so that thumbnail files are cleaned up). To accomplish this, use the following ``curl`` command, substituting the id of the file:
+Armed with this information about WorldMap returning a 404 for a map, you may want to delete any record of the map on the Dataverse side removing WorldMap as an explore option (and so that thumbnail files are cleaned up). To accomplish this, use the following ``curl`` command, substituting the id of the file:
 
 ``curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://localhost:8080/api/files/{file_id}/map``
 

doc/sphinx-guides/source/admin/mail-groups.rst

@@ -55,6 +55,32 @@ To load it into your Dataverse installation, either use a ``POST`` or ``PUT`` re
 
 ``curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/groups/domain --upload-file domainGroup1.json``
 
+Matching with Domains or Regular Expressions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Adding simple domain names requires exact matches of user email domains and the configured domains of a group. Although you could add multiple domains to a group, those still require exact matches. 
+
+You can also use one or multiple regular expressions instead of simple domains for a group. Those should not be mixed, although it would work. Regular expressions still require exact matches, but are much more flexible and are designed to support installation-specific use cases for group management.
+
+Some hints:
+
+- Due to their excessive CPU usage, regular expressions should be used rarely.
+- Remember to properly escape "\" in your regular expression. Both Java and JSON are a bit picky about this. E.g. a character class "\d" would have to be escaped as "\\d". Plenty of tutorials on the web explain this in more detail.
+- There is no way Dataverse can detect a wrong regular expression for you. Be sure to do extensive testing, as a misconfigured group could result in privilege escalation or an unexpected influx of support contacts.
+- Remember to enable the regular expression support for a group by adding ``"regex": true``!
+
+A short example for a group using regular expressions:
+
+.. code-block:: json
+
+  {
+    "name": "Users from @example.org",
+    "alias": "exampleorg-regex",
+    "description": "Any verified user from x@example.org or x@sub.example.org will be included.",
+    "regex": true,
+    "domains": ["example\\.org", "[a-z]+\\.example\\.org"]
+  }
+
 Updating a Mail Domain Group
 ----------------------------
 

doc/sphinx-guides/source/api/apps.rst

@@ -65,10 +65,10 @@ OSF allows you to view, download, and upload files to and from a Dataverse datas
 
 https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
 
-GeoConnect
+Geoconnect
 ~~~~~~~~~~
 
-GeoConnect allows Dataverse files to be visualized on http://worldmap.harvard.edu with the "Explore" button. Read more about it in the :doc:`/user/data-exploration/worldmap` section of the User Guide.
+Geoconnect allows Dataverse files to be visualized and explored on `WorldMap <http://worldmap.harvard.edu>`_. Read more about it in the :doc:`/user/data-exploration/worldmap` section of the User Guide.
 
 https://github.com/IQSS/geoconnect