Merge branch 'master' of https://github.com/panosc-eu/panosc

USE CASES/Use Case 28 - Online visualisation, exploration and analysis of HDF5 files with h5nuvola.md

@@ -0,0 +1,69 @@
+Use Case 28 - Online visualisation, exploration and analysis of HDF5 files with h5nuvola.md
+=========================================================
+[HDF5](http://www.hdfgroup.org/) is a de-facto standard for storage of heterogeneous, n-dimensional  scientific data. It is portable, without vendor lock-in, self-describing thanks to included metadata, with libraries in different languages. It offers high I/O performance, optimised for both time and storage.
+
+One of the disadvantages of this format is the need for specialised applications to work with data from hdf5 files. HDFGroup itself offers standalone software like HDFView that can be downloaded, while other actors shared tools such as H5Web with the scientific community, to visualise such data.
+
+With this use case we designed and developed h5nuvola, a web tool written in Python, based on the popular Django framework. h5nuvola can be linked to one or more data sources, being local folders, NFSs, etc., and uses metadata and hard logic to select from a set of visualisation plugins, being them general purposed (like the ones for 1 and 2 dimensional datasets) or customised for specific applications (such as one designed to view, explore and analyse X-Ray Fluorescence datasets).
+
+Custom applications are easy to implement, relying on Dash-Plotly applications, written in  pure Python, and making easy the integration of interactive controls, callbacks, and the use of scientists’ analytic pipelines.
+
+
+Main Contributors
+------
+* **Andrea Lorenzon** – Software engineering, main developer
+* **Georgios Kourousias** – Concept, supervisor, initial development
+* **Marco De Simone** – System integration, DevOps
+* **Emiliano Coghetto** – Integration with Elettra Virtual Unified Office
+* **Alessandro Olivo** – Testing
+
+Use Case Action Flow
+------
+* **Browing** – Visit h5nuvola home page, and browse file included in your attached data sources
+* **Select a file** – Quick search available: typing few letters of the filename will help in finding your data
+* **Select a dataset** – Quick search available
+* **Automatic redirection to an appropriate app** – h5nuvola will redirect users, dispatching to an appropriate app to explore your data
+* **Perform analytics** – Use app’s analytical tools to extract information from data
+
+Description of needs
+------
+The service will be integrated into the CERIC-ERIC EOSC service portfolio. h5nuvola will progressively fit more in the PaNOSC universe, and as soon as metadata will be included in h5 files and users will create and share more apps.
+
+Furthermore, the Search API and Data Catalog will integrate current data browsing. Eventually, OAUTH2 or other authentication authorities (that are already supported) will be included.
+
+Impacts from the implementation
+------
+Remote authorised users will be able to visualise and analyse shared hdf5 files datasets without the need to install software. Immediate visual feedback, with interactive plots and online analysis tools, will make their workflow more transparent, reproducible and machine-independent. 
+
+A galaxy of h5nuvola apps can be shared by users, improving collaboration and progress of research.
+
+Generalisation of the use case
+Any hdf5 user can benefit from h5nuvola. Any kind of data, even non-PaN, can have a dedicated app.
+
+It is also possible to include backend-powered computation for online processing of datasets, making the data pipeline process potentially transparent to scientists, letting them concentrate on results.
+
+While h5nuvola is not indicated for non-standard analysis pipelines, that could find a better tool in a Jupyter Notebook instance, it makes fast exploration of hdf5 a reality.
+
+Other partners interested in this use case
+------
+Elettra, and any other partner using hdf5 as a standard data format, could be interested.
+
+Resources
+------
+* **Web framework**: django
+* **HDF5 access**: h5py
+* **web app management**: Dash
+* **Dash application wrapper for Django**: django-plotly-dash
+* **Session management**: redis
+* **app database**: postgres
+* **Data visualisation**: plotly, ev. matplotlib
+* **Data processing**: numpy, scipy, opencv2…
+* **[Software and documentation repository](https://gitlab.elettra.eu/panosc/h5nuvola)**
+
+PaNOSC related work packages
+------
+[WP4 – Data analysis services](https://www.panosc.eu/work-packages/work-package-4-data-analysis-services/)
+
+Contact person
+------
+[Andrea Lorenzon](mailto:andrea.lorenzon@ceric-eric.eu) (CERIC-ERIC)

Work Packages/WP1 Management/Meetings/Project Management Committee/2022-01-12-PMC/2022-01-12-PMC_summary.md

@@ -0,0 +1,35 @@
+Project Management Committee Meeting 12/01/2022
+===============================================
+
+Participants: Andy Goetz (ESRF), Jordi Bodera Sempere (ESRF), Jean-Francois Perrin (ESRF), Jayesh Wagh (ESRF), Fabio Dal Antonia (XFEL),  Ornella De Giacomo (CERIC ERIC), Dario Rocella (CERIC-ERIC), Nicoletta Carboni (CERIC-ERIC), Thomas Holm Rod (ESS), Tobias Richter (ESS) Erwan Le Gall (ILL), Jamie Hall (ILL), Stuart Caunt (ILL), Giuseppe La Rocca (EGI), Teodor Ivanoaica (ELI)
+
+--
+
+**1.	Deliverables & Milestones**
+
+o	D6.2 - Teodor is awaiting some brief inputs from ILL and ESS under WP3. Once these inputs are in, document is ready to be submitted to the reviewer.
+
+o	D6.3 - Jean-Francois has received feedback from ILL and minor comments should be addressed today itself. 
+
+o	MS6.1 and M6.3 - Jean-Francois will be able to submit the text today or latest by tomorrow. Data portals from some of the facilities ex. ESS will be registered on the EOSC portal during the course of the project. Jordi suggested to provide reference to D6.3 in the milestone text. 
+
+o	D7.2 - Ornella informed that some of the comments from Rudolf are yet to be resolved. Ornella expects to submit the deliverable by Friday 14th January.
+
+o	MS8.5- Thomas informed MS8.5 should be achieved by next week. 
+
+**2.	Perioidic Report Update**
+
+o	Jordi presented PANOSC google drive folder and other relevant documents. Andy discussed to which extent activities in the last month should be elaborated.
+
+o	Jordi discussed recommendations from the previous review, deviations, data management plan, risks sections of the periodic report.
+
+o	Jordi reminded the partners that the deadline 28 January 2022 for the submission of periodic report. 
+
+**3.	Project Brief EOSC**
+
+o	Jordi informed the project brief submission. It is a public document and can be disseminated further. 
+
+**4. Any other business**
+
+o	Jean-Francois informed participants about the invitation from CS3MESH project to participate in a workshop on 26th January. Jean-Francois and Nicoletta would attend the meeting from PaNOSC. Jean-Francois will present CVMFS activities. 
+

Work Packages/WP1 Management/Meetings/Project Management Committee/2022-01-19-PMC/2022-01-19-PMC_summary.md

@@ -0,0 +1,36 @@
+Project Management Committee Meeting 19/01/2022
+===============================================
+
+Participants: Jordi Bodera Sempere (ESRF), Rudolf Dimper (ESRF), Jayesh Wagh (ESRF), Fabio Dal Antonia (XFEL), Dario Rocella (CERIC-ERIC), Nicoletta Carboni (CERIC-ERIC), Carsten Fortmann Grote(XFEL), Thomas Holm Rod (ESS), Tobias Richter (ESS), Sophie Servan (ExPaNDs),Jamie Hall (ILL), Stuart Caunt (ILL), Giuseppe La Rocca (EGI), Teodor Ivanoaica (ELI)
+
+--
+
+**1. Update on LATE deliverables/milestones and WPs**
+D6.2 Compute cloud report - Submitted.
+D6.3 AAI report - Submitted
+MS6.1 Implementation of AAI integration - Achieved
+M6.3 Second release of PaNOSC services, data and resources - Achieved. 
+D7.2 - Photon and Neutron EOSC metrics - To be submitted by the end of the week. 
+MS8.5 e-learning virtual facilities - Milestone document should be ready by the end of the week. 
+
+**2. Project Review news**
+Jordi informed participants about the project review communication. None of the project member notified any conflict of interest with the proposed external reviewer from University of Manchester.
+Jordi is not certain if it is a review with or without a meeting with the consortium. It is not specified in the letter.
+Sophie - The ext reviewer was reviewer for EXPANDS and from EXPANDS experience the reviewer tends to request more precise planning on the technical activities.
+Jordi will share the previous review report with project partners. 
+
+**3. Periodic Report update**
+Jordi went through the report chapters and comments on the technical report.
+
+1. Objectives section - Thomas suggested to include collaboration with EXPANDS. Jordi assigned project members to specific objectives and asked project members to ensure quality of the content and avoid repetitions.
+Rudolf will update EOSC Future activities in the objectives section. Rudolf and Tobias will separately discuss point about harvesting ESRF metadata in the EOSC Future search API. 
+2. Explanation of the work carried out per wp - Jordi asked project partners to add atleast a page or two page long description per wp. Thomas suggested to write a brief overview on all the workpackages.
+Jordi suggested WP leads to include tables and figures in the WP text. 
+3. Major recommendations : Jordi will share (with Sophie's consent) EXPANDS KPIs because some of them are same as PANOSC ones.
+4. Jordi will work with WP leaders to upload the deliverables on Zenodo to get a DOI for each of them. 
+5. Jordi urged all the project partners to complete the assigned sections on time to avoid delay in report submission.
+
+
+
+
+

Work Packages/WP1 Management/Meetings/Project Management Committee/2022-01-26-PMC/2022-01-26-PMC_summary.md

@@ -0,0 +1,33 @@
+Project Management Committee Meeting 12/01/2022
+===============================================
+
+Participants: Andy Goetz (ESRF), Jordi Bodera Sempere (ESRF), Jean-Francois Perrin (ESRF), Fabio Dal Antonia (XFEL),  Ornella De Giacomo (CERIC ERIC), Dario Rocella (CERIC-ERIC), Nicoletta Carboni (CERIC-ERIC), Thomas Holm Rod (ESS), Tobias Richter (ESS), Jamie Hall (ILL), Stuart Caunt (ILL), Giuseppe La Rocca (EGI), Teodor Ivanoaica (ELI), Carsten (XFEL) and Rudolf (ESRF)
+
+--
+
+**1. Update on LATE deliverables/milestones and WPs**
+
+Jordi reported that D7.2 has been submitted and that the WP8 milestone is still pending. Thomas explained that there is a little issue with integrating a solution from WP5 into the e-learning platform. The milestone is nearly achieved, but not yet.
+
+Jordi pointed out that if the milestone is not achieved then the periodic report needs to be changed to explain why it is the case.
+
+**2.	Perioidic Report Update**
+
+The forms:
+* Risks : Jordi thanked Tobias and Carsten for the clarifications regarding risks 7, 11 and 12. The PMC agreed that other identified risks are not required to be updated in the EC Funding and Tenders Portal, and they will be followed up via our Google spreadsheet.
+* Publications: Jordi reminded to everyone that any PaNOSC sponsored-publication has to be listed in that section.
+* Dissemination & Communication: Nicoletta confirmed that the information was updated in the Google Drive. Jordi said that he will updated in the EC Portal
+* Open Data: Andy explained that given that PaNOSC is all about open FAIR data there will be hundreds of open datasets and it is not feasible to update this area.
+* Gender: Jordi reminded all partners to update this section. Thomas enquired about what is considered a "researcher" and the recommendation from Jordi is to use common sense, given that we have PhDs doing engineering work and some bits of work overlap with research.
+* Financial statements: Jordi reminded all partners to update this section. Teodor said that ELI should contact Jordi about incorrect data provided and Jordi replied that this has already taken place and only affects the Internal Financial Report and not the Periodic Report. Jordi is now awaiting the corrected information from ELI.
+
+The Technical Report Part B
+
+Jordi scrolled through the document commenting the different sections and comments made. **It was agreed that all partners and contributors to the Periodic Report should do all their contributions by Thursday 27th of January at 12:00h**. At this point, Jordi will download the Google Docs file and start working with a local copy.
+
+Jordi highlighted that the Deviations section is still missing content from the different partners and asked WP leaders to solve any comments and suggested content in their areas.
+
+The deadline to submit the report is Saturday 29th of January and Jordi ask partners to provide timely contributions in order to avoid having to work on Saturday or having to submit the report late.	
+
+--ENDS--
+

Work Packages/WP1 Management/Meetings/Project Management Committee/2022-02-09-PMC/2022-02-09-PMC_Meeting.md

@@ -0,0 +1,52 @@
+PaNOSC Project Management Committee Meeting - 09/02/2022 
+=========================================================
+
+Agenda
+------	
+
+1. Update on late WP8 milestone *e-learning virtual facilities*
+
+2. Periodic Report & Review update / news
+* we will wait until after the review to start the Grant Amendment
+
+3. News from ExPaNDS
+* face to face meeting in the summer?
+
+4. News from the EOSC
+
+5. WP leaders updates
+
+| item |    Title    | due date |
+| ---- | ----------- | -------- |
+| D3.3 | Catalog Service | 31/03/2022 |
+| D1.9 | Mid-year summary | 31/05/2022 |
+| D3.5 | NeXus Metadata Schema | 31/05/2022 |
+| D4.3 | Remote desktop and Jupyter service deployed at EOSC | 31/05/2022 |
+| MS13 | Data analysis services accessible through EOSC | 31/05/2022 |
+| D5.3 | Documented simulation tasks executable | 31/05/2022 |
+| MS16 | VINYL software release | 31/05/2022 |
+| D7.3 | PaN EOSC Business model reference doc | 31/05/2022 |
+| D8.2 | Report on lessons learned for adopting e-learning platform at PaNOSC facilities | 31/05/2022 |
+| D8.3 | Teaching material for users | 31/05/2022 |
+
+6. Upcoming events
+- 7-9 February, online - [ESRF User Meeting 2022](https://www.esrf.fr/fr/home/events/conferences/2022/user-meeting-2022.html), [Registration](https://events.esrf.fr/misapps/workshops/login.do) 
+- 28 March - 1 April 2022, online [14th International Conference on Synchrotron Radiation Instrumentation - SRI2021](https://www.sri2021.eu/)
+- 18-20 May 2022, Utrecht, the Netherlands - [Instruct Biennial Structural Biology Conference](https://instruct-eric.org/ibsbc2022) | [Abstract submission](https://instruct-eric.org/submit-call/instruct-eric-biennial-2022---abstract-submission) within 1 March 2022 | [Registration](https://www.eventbrite.co.uk/e/instruct-biennial-structural-biology-conference-2022-tickets-161765698559) within 1 April 2022
+- 5-7 September 2022, Berlin - Germany, [SNI2022 - German Conference for Research with Synchrotron Radiation, Neutrons and Ion Beams at Large Facilities](https://www.helmholtz-berlin.de/events/sni-2022/scientific-programme_en.html) - [Important dates](https://www.helmholtz-berlin.de/events/sni-2022/important-dates_en.html) - Abstract submission deadline: 24 April 2022, Registration deadline: 21 August 2022.
+- 5-7 October 2022, Lund - Sweden, [Joint ESS-ILL User Meeting 2022](http://www.neutrons4europe.com/)
+- 14-17 November 2022, Prague - Czech Republic & online, [EOSC Symposium 2022](https://eosc-portal.eu/events/eosc-symposium-2022)
+
+7. Use Cases Dashboard reminder (~1 min)
+
+| Partner | Use Cases Submitted | Comments |
+| ------- | ------------------- | -------- |
+| ESRF  |  10  | 1 planned   |
+| ILL   |  1  | 3 planned  | of which 1 w CERIC and EGI)
+| ESS   |  2  |   |
+| XFEL  |  7  |   |
+| CERIC |  8  | of which 1 with ILL and EGI, 2 planned |
+| ELI   |  1  | 1 planned  |
+| EGI   |  1  | of which 1 with CERIC and ILL | 
+
+8. AOB

Work Packages/WP1 Management/Meetings/Project Management Committee/2022-02-09-PMC/2022-02-09-PMC_summary.md

@@ -0,0 +1,53 @@
+Project Management Committee Meeting 09/02/2022
+===============================================
+
+Participants: Jordi Bodera Sempere (ESRF), Rudolf Dimper (ESRF), Jean-Francois Perrin (ESRF), Jayesh Wagh (ESRF), Fabio Dal Antonia (XFEL),  Ornella de Giacomo (CERIC-ERIC), Nicoletta Carboni (CERIC-ERIC), Tobias Richter (ESS), Sophie Servan (ExPaNDs),
+Erwan le Gall (ILL), Teodor Ivanoaica (ELI)
+
+
+**1. Update on late WP8 milestone e-learning virtual facilities**
+
+Tobias is expecting to hear from Thomas. Jordi informed that this milestone has been reported as achieved because the e-learning platform was demonstrated as working for all facilities (several RIs of the same type may share a "virtual facility"). 
+Teodor mentioned that there will be additional activities in the upcoming months.
+
+**2. Periodic Report & Review update / news**
+Jordi informed the report has been submitted and he does not have further information on the review procedure. 
+We should await feedback from Project officer on the periodic report and review to move ahead with grant amendment concerning ELI ERIC and EGI. 
+
+**3. EXPANDS news - Sophie Servan**
+Patrick and Andy are planning to organize a joint PANOSC-EXPANDS all hands f2f meeting to be scheduled in June 2022 or September 2022 in Grenoble. 
+Event could be organized in Prague with ELI as local organizer.
+EXPANDS has setup a dashboard for objectives on https://github.com/ExPaNDS-eu/ExPaNDS/wiki/ExPaNDS-objectives-dashboard-per-partner
+All the EXPANDS services with the relevant metadata will be added in the FAIR tools catalogue. EXPANDS will share the feedback with PANOSC on this activity. 
+
+**3. EOSC news - Rudolf Dimper**
+EOSC association is gathering data on in kind contribution to EOSC. EOSC association members from all the member states will provide this data through a survey. Rudolf suggested to not reply to the survey as it is not clear what counts as in kind contribution. 
+EOSC Future had another review and a consolidated rewiew report is yet to be delivered. Next review in June 2022. The project partners are gathering use cases. 
+ESRF science projects will require IT resources and it could be an interesting use case to share with the EC.
+GEANT is leading procurement activities in EOSC Future similar to OCRE. GEANT is now requesting use cases from the project partners. A direct access to compute resources is essential for users and it has been highlighted in one of the deliverable.
+ESRF data catalogue is registered on EOSC thanks to support from EGI colleagues. Discussions are ongoing with OpenAIRE to making ESRF data FINDABLE.
+
+
+
+**4. WP leaders updates**
+
+WP1 - Jordi will send out an internal financial report by 11 February 2022. Jordi is preparing for upcoming grant amendment. 
+
+WP6 - Jean-Francois shared outcomes of the UmbrellaID AAI meeting. All RIs will share UmbrellaID contact person/persons (both management and technical activities) with Jean-Francois. All the representatives will vote on activities planned for the next year. CVMFS activities are ongoing, VISA is being deployed by the PANOSC RIs. ESRF used VISA for training session at the user meeting and we had a positive feedback from users. Very likely VISA will be deployed on multiple ESRF beamlines. 
+
+WP4- Fabio informed that VISA deployment activities are on track. ESS is linking the federated search to VISA and future activities will include HPC resources connection. EXPANDS partners are also putting in efforts for VISA deployment.
+
+WP3- Tobias informed that preparations for D3.3 ,D3.4 and D3.5 are ongoing. A writing session is schedule next week for the deliverables? Tobias will add Teodor to WP3 mailing list.
+
+WP9 - Nicoletta shared WP9 updates. 3 video interviews were released in Jan 2022. WP leads were invited to share and communicate achievements with WP9 to further disseminate to wider community. Jordi discussed the low spending under WP9, this could be due to reporting activities under the technical wps and not under WP9. PANOSC activities could be shared in the future ESRF user meetings. ESS and ELI could disseminate PANOSC activities in their user meetings. INSTRUCT will organize a structural biology conference and some use cases from EXPANDS could be used. Jean-Francois discussed if UmbrellaID communication through WP9 would be eligible for declaring costs on PANOSC. Jordi will check and get back to Jean-Francois.Teodor will participate in ELI data workshop and could present PANOSC activities from technical WPs. 
+
+ WP8- Jordi informed that next meeting will take place on 23rd Feb and it will focus on WP8.
+
+
+
+
+
+
+
+
+