diff --git a/README.md b/README.md index d7ecf306..889ba9d9 100644 --- a/README.md +++ b/README.md @@ -149,13 +149,9 @@ iop4conf = iop4lib.Config(config_db=True, gonogui=False, jupytermode=True) ``` ### Tips -You can get an IPython interactive terminal after running iop4 using the `-i` option. You can override any config option using the `-o` option, e.g. +You can get an IPython interactive terminal after running iop4 using the `-i` option. You can override any config option using the `-o` option, or by setting environment variables, e.g. ```bash - $ iop4 -i -o nthreads=20 -o log_file=test.log --epoch-list T090/230313 T090/230317 -``` -or by setting environment variables, e.g. -```bash - $ IOP4_NTHREADS=20 IOP4_LOG_FILE=test.log iop4 -i --epoch-list T090/230313 T090/230317 + $ IOP4_NTHREADS=20 iop4 -i -o log_file=test.log --epoch-list T090/230313 T090/230317 ``` Check `iop4 --help` for more info. @@ -166,6 +162,8 @@ To build and show the documentation, run $ make docs-show ``` +The documentation for the lastest release is hosted in this repository's [GitHub Pages](https://juanep97.github.io/iop4/). + ## Contribute You are welcome to contribute to IOP4. Fork and create a PR! diff --git a/docs/data_reduction_details.rst b/docs/data_reduction_details.rst index 2d463423..a0d577a0 100644 --- a/docs/data_reduction_details.rst +++ b/docs/data_reduction_details.rst @@ -8,7 +8,7 @@ Data reduction details Introduction ------------ -The following steps are done by IOP4 to reduce data from one epoch (when invoked as `iop4 --epoch-list `): +The following steps are done by IOP4 to reduce data from one epoch (when invoked as :code:`iop4 --epoch-list `): #. Registering the epoch in the database, listing, registering and downloading the raw files. diff --git a/docs/iop4_concepts.rst b/docs/iop4_concepts.rst index 3f4cbe10..5b28d8b4 100644 --- a/docs/iop4_concepts.rst +++ b/docs/iop4_concepts.rst @@ -40,7 +40,7 @@ models: * **AperPhotResult**: a result of aperture photometry, defined by its associated `reducedfit` (ReducedFit), the `astrosource` (AstroSource) for which it was computed, - and the aperture `aperpix`` (float) used. + and the aperture `aperpix` (float) used. * **PhotoPolResult**: a result of photo-polarimetry, the end product of IOP4, as pipeline for optical photo-polarimetry. @@ -87,8 +87,8 @@ This should open a tab in your browser with the IOP4 web interface. used in production, or entirely replaced by a new Django project and used only as a guide. See :doc:`serving iop4 in production ` for more information. -After login in with the credentials that you supplied during the `set up` - you will have access to the following tabs: +After login in with the credentials that you supplied during the `set up +`_ you will have access to the following tabs: * Explore > Plot: to plot and inspect the photometry and polarimetry results, flag data and download plots. * Explore > Data: to inspect, filter and download data (e.g. in CSV format). @@ -103,13 +103,130 @@ FITS header keywords, different polarimeters, different pixel scales, etc. IOP4 details from the main code. Telescope-specific code is relegated to the :code:`iop4lib.telescopes` submodule, while instrument-specific code is relegated to the :code:`iop4lib.instruments` submodule. Adding a new telescope or instrument to IOP4 is as simple as adding a new class to these submodules, inheriting the -:code:`iop4lib.telescopes.Telescope` or code:`iop4lib.instrument.Instrument` base classes, and implementing the required methods +:code:`iop4lib.telescopes.Telescope` or :code:`iop4lib.instrument.Instrument` base classes, and implementing the required methods (like methods to list the available data in the remote observatory archives, reading of non-standard FITS header keywords, or specific reduction steps). Information and details about the different telescopes and instruments can be found at :ref:`data_reduction_details`. +IOP4 data directory structure +----------------------------- + +IOP4 data directory structure follows the following hierarchical schema. +In this schema, all raw data is stored and isolated under a single folder +(``raw/``), with the intent of establishing a local archive of the original data +without any modifications for long-term conservation. +Under the raw directory, data is organized first by telescope and then by night +of observation. Other files such as built master calibration frames and reduced +images are stored separately. Also auxiliary images such as automatically built +previews, finding charts, summary plots, etc, which are too heavy to be stored +in a database, are stored under different folders. + +.. + Tree generated with the follwoing ascii input (https://tree.nathanfriend.io/): + datadir (e.g. ~/.iop4data/) + - raw + - telescope 1 (e.g. OSN-T090) + - night 1 (e.g. 2024-04-08) + - file 1 (e.g. sciencefile_1.fits) + - file 2 (e.g. bias_1.fits) + - ... + - ... + - ... + - masterbias + - telescope 1 (e.g. OSN-T090) + - night 1 (e.g. 2024-04-08) + - file 1 (e.g. masterbias_1.fits) + - ... + - ... + - ... + - masterdark + - ... + - masterflat + - ... + - calibration + - telescope 1 (e.g. OSN-T090) + - night 1 (e.g. 2024-04-08) + - file 1 folder (e.g. sciencefile_1.fits.d) + - RawFit + - auxiliary file 1 (e.g. preview.png) + - ... + - ReducedFit + - reduced fit file (e.g. sciencefile_1.fits) + - auxiliary file 1 (e.g. astrometry_summary.png) + - ... + - file 2 folder (e.g. masterbias_1.fits.d) + - MasterBias + - auxiliary file 1 (e.g. preview.png) + - ... + - ... + - ... + - ... + - logs + - log file 1 + - ... + - astrosource + - source 1 + auxiliary file 1 (e.g. finding_chart.png) + - ... + - database file (e.g. iop4.db) + +.. code-block:: text + + datadir (e.g. ~/.iop4data/) + ├── raw + │ ├── telescope 1 (e.g. OSN-T090) + │ │ ├── night 1 (e.g. 2024-04-08) + │ │ │ ├── file 1 (e.g. sciencefile_1.fits) + │ │ │ ├── file 2 (e.g. bias_1.fits) + │ │ │ └── ... + │ │ └── ... + │ └── ... + ├── masterbias + │ ├── telescope 1 (e.g. OSN-T090) + │ │ ├── night 1 (e.g. 2024-04-08) + │ │ │ ├── file 1 (e.g. masterbias_1.fits) + │ │ │ └── ... + │ │ └── ... + │ └── ... + ├── masterdark + │ └── ... + ├── masterflat + │ └── ... + ├── calibration + │ ├── telescope 1 (e.g. OSN-T090) + │ │ ├── night 1 (e.g. 2024-04-08) + │ │ │ ├── file 1 folder (e.g. sciencefile_1.fits.d) + │ │ │ │ ├── RawFit + │ │ │ │ │ ├── auxiliary file 1 (e.g. preview.png) + │ │ │ │ │ └── ... + │ │ │ │ └── ReducedFit + │ │ │ │ ├── reduced fit file (e.g. sciencefile_1.fits) + │ │ │ │ ├── auxiliary file 1 (e.g. astrometry_summary.png) + │ │ │ │ └── ... + │ │ │ ├── file 2 folder (e.g. masterbias_1.fits.d) + │ │ │ │ └── MasterBias + │ │ │ │ ├── auxiliary file 1 (e.g. preview.png) + │ │ │ │ └── ... + │ │ │ └── ... + │ │ └── ... + │ └── ... + ├── logs + │ ├── log file 1 + │ └── ... + ├── astrosource + │ ├── source 1 + │ │ └── auxiliary file 1 (e.g. finding_chart.png) + │ └── ... + └── database file (e.g. iop4.db) + + +By default, IOP4 will use ``~/.iop4data/`` as the data directory root. You can +indicate a different path with the ``datadir`` configuration option. You can +also specify a different (and independent) database file location with the +``db_path`` option. + .. rubric:: Footnotes .. [#otherORMs] There exists many other ORM engines, such as SQLAlchemy, diff --git a/docs/serving_iop4_in_production.rst b/docs/serving_iop4_in_production.rst index 69b02d1a..c2ba7e71 100644 --- a/docs/serving_iop4_in_production.rst +++ b/docs/serving_iop4_in_production.rst @@ -1,6 +1,11 @@ Serving IOP4 in production ========================== +.. _production_web_server: + +Setting up a production web server +---------------------------------- + .. warning:: This section requires some familiarity with system @@ -55,4 +60,94 @@ your Django project are: ALLOWED_HOSTS = ["domain", "domain2"] # Configure static files to the path served by nginx - STATIC_ROOT = '/path/to/static/' \ No newline at end of file + STATIC_ROOT = '/path/to/static/' + + +.. _production_cron_job: + +Creating a cron job for reducing new observations +------------------------------------------------- + +You might be interested in creating a cron job that routinely reduces new +observations every morning. Here we provide an example. This assumes you are +working in a linux system. Other OS provide different methods to create jobs. + +Use ``crontab -e`` to edit your crontab file and add a line like the following + +.. code-block:: text + + 00 08 * * * /home/vhega/run_iop4_daily.sh > /home/vhega/run_iop4_daily.log 2>&1 + +Then, create a file ``run_iop4_daily.sh``, give it execution permissions (``chmod +x run_iop4_daily.sh``) and add the following content: + +.. code-block:: bash + + #!/usr/bin/bash + + # save the current datetime + printf -v date '%(%Y-%m-%d_%H%M)T' -1 + + echo "#########################################" + echo "Run daily job for IOP4: $date" + echo "#########################################" + + . /home/vhega/miniconda3/bin/activate iop4 + + # make sure all files created by iop4 are editable by the current user only + umask 0022 + + # Run iop4 for new observations (i.e. last night) + iop4 --discover-missing -o log_file=/home/vhega/iop4data/logs/daily_$date.log + + # Create and send a summary of the results for last night + iop4-night-summary --fromaddr '{{YOUR SENDER ADDRESS}}' \ + --mailto '{{ADDRESS 1}},{{ADDRESS 2}},{{ADDRESS 3}}' \ + --contact-name '{{CONTACT NAME}}' \ + --contact-email '{{CONTACT EMAIL}}' \ + --site-url '{{DEPLOYMENT SITE URL}}' \ + --saveto "/home/vhega/iop4data/logs/daily_$date.html" + +The above script will run iop4 every morning, disovering and proccessing new +observations. + +The last command is optional, and will send an email with a summary with the +results from last night to the specified email addresses. +The cron job does not need to be run on a daily basis, and you can run it +whenever you expect new observations to become available in the telescope +remote archives (e.g. every few hours). Alternatively, you can pass an argument +to ``iop4-night-summary`` specifying the night that you want to generate the +summary for. The email will be in HTML format (viewable in any modern browser or +email client), and can optionally be saved to any path. You can indicate the url +of your deployed site so the links in the email (e.g. for files with error) +point directly to the corresponding page in the iop4 web interface or admin +site. + + +.. _production_share_datadir: + +Sharing the data directory with other system users +-------------------------------------------------- + +If you are running IOP4 in a server with multiple users, and have created an +user to run the IOP4 pipeline as a service, you might be interested in making +the local archive available to other users, so they can process the data +independently. By default, IOP4 will remove write permissions on raw files, +protecting them from accidental modification. For example, you can link your raw +directory to the service account raw directory, + +.. code-block:: bash + + ln -s /home/iop4user/.iop4data/raw ~/home/myuser/.iop4data/raw + +The other directories can also be linked, but keep in mind that other users +might not be able to modify and reprocess reduced files (they will still be able +to inspect them). You will still need to create your database following the +installation instructions. + +Multiple users can also share the same data directory, but this is not +recommended. + +You should not confuse system users (which can run the iop4 pipeline) with IOP4 +portal users, that can access and inspect data from the web interface. These +should be created only after following See :ref:`Setting up a production web server ` (the debug web +server is not recommended for multiple users). \ No newline at end of file