Improve docs (#113)

README.md

@@ -149,13 +149,9 @@ iop4conf = iop4lib.Config(config_db=True, gonogui=False, jupytermode=True)
 ```
 
 ### Tips
-You can get an IPython interactive terminal after running iop4 using the `-i` option. You can override any config option using the `-o` option, e.g.
+You can get an IPython interactive terminal after running iop4 using the `-i` option. You can override any config option using the `-o` option, or by setting environment variables, e.g.
 ```bash
-    $ iop4 -i -o nthreads=20 -o log_file=test.log --epoch-list T090/230313 T090/230317
-```
-or by setting environment variables, e.g.
-```bash
-    $ IOP4_NTHREADS=20 IOP4_LOG_FILE=test.log iop4 -i --epoch-list T090/230313 T090/230317
+    $ IOP4_NTHREADS=20 iop4 -i -o log_file=test.log --epoch-list T090/230313 T090/230317
 ```
 Check `iop4 --help` for more info.
 
@@ -166,6 +162,8 @@ To build and show the documentation, run
     $ make docs-show
 ```
 
+The documentation for the lastest release is hosted in this repository's [GitHub Pages](https://juanep97.github.io/iop4/).
+
 ## Contribute
 
 You are welcome to contribute to IOP4. Fork and create a PR!

docs/data_reduction_details.rst

@@ -8,7 +8,7 @@ Data reduction details
 Introduction
 ------------
 
-The following steps are done by IOP4 to reduce data from one epoch (when invoked as `iop4 --epoch-list <epochname>`):
+The following steps are done by IOP4 to reduce data from one epoch (when invoked as :code:`iop4 --epoch-list <epochname>`):
 
 #. Registering the epoch in the database, listing, registering and downloading the raw files.
 

docs/iop4_concepts.rst

@@ -40,7 +40,7 @@ models:
 
 * **AperPhotResult**: a result of aperture photometry, defined by its associated 
   `reducedfit` (ReducedFit), the `astrosource` (AstroSource) for which it was computed, 
-  and the aperture `aperpix`` (float) used.
+  and the aperture `aperpix` (float) used.
 
 * **PhotoPolResult**: a result of photo-polarimetry, the end product of IOP4, as pipeline for
   optical photo-polarimetry.
@@ -87,8 +87,8 @@ This should open a tab in your browser with the IOP4 web interface.
    used in production, or entirely replaced by a new Django project and used only as a guide.
    See :doc:`serving iop4 in production <serving_iop4_in_production>` for more information.
 
-After login in with the credentials that you supplied during the `set up` 
-</iop4/docs/#usage> you will have access to the following tabs:
+After login in with the credentials that you supplied during the `set up 
+</iop4/docs/#usage>`_ you will have access to the following tabs:
 
 * Explore > Plot: to plot and inspect the photometry and polarimetry results, flag data and download plots.
 * Explore > Data: to inspect, filter and download data (e.g. in CSV format).
@@ -103,13 +103,130 @@ FITS header keywords, different polarimeters, different pixel scales, etc. IOP4
 details from the main code. Telescope-specific code is relegated to the :code:`iop4lib.telescopes` submodule, while 
 instrument-specific code is relegated to the :code:`iop4lib.instruments` submodule.
 Adding a new telescope or instrument to IOP4 is as simple as adding a new class to these submodules, inheriting the 
-:code:`iop4lib.telescopes.Telescope` or code:`iop4lib.instrument.Instrument` base classes, and implementing the required methods 
+:code:`iop4lib.telescopes.Telescope` or :code:`iop4lib.instrument.Instrument` base classes, and implementing the required methods 
 (like methods to list the available data in the remote observatory archives, reading of non-standard FITS header keywords, or 
 specific reduction steps).
 
 Information and details about the different telescopes and instruments can be found at :ref:`data_reduction_details`.
 
 
+IOP4 data directory structure
+-----------------------------
+
+IOP4 data directory structure follows the following hierarchical schema. 
+In this schema, all raw data is stored and isolated under a single folder 
+(``raw/``), with the intent of establishing a local archive of the original data 
+without any modifications for long-term conservation. 
+Under the raw directory, data is organized first by telescope and then by night 
+of observation. Other files such as built master calibration frames and reduced 
+images are stored separately. Also auxiliary images such as automatically built 
+previews, finding charts, summary plots, etc, which are too heavy to be stored 
+in a database, are stored under different folders.
+
+.. 
+  Tree generated with the follwoing ascii input (https://tree.nathanfriend.io/):
+  datadir (e.g. ~/.iop4data/)
+  - raw
+    - telescope 1 (e.g. OSN-T090)
+      - night 1 (e.g. 2024-04-08)
+        - file 1 (e.g. sciencefile_1.fits)
+        - file 2 (e.g. bias_1.fits)
+        - ...
+      - ...
+    - ...
+  - masterbias
+    - telescope 1 (e.g. OSN-T090)
+      - night 1 (e.g. 2024-04-08)
+        - file 1 (e.g. masterbias_1.fits)
+        - ...
+      - ...
+    - ...
+  - masterdark
+    - ...
+  - masterflat
+    - ...
+  - calibration
+    - telescope 1 (e.g. OSN-T090)
+      - night 1 (e.g. 2024-04-08)
+        - file 1 folder (e.g. sciencefile_1.fits.d)
+          - RawFit
+            - auxiliary file 1 (e.g. preview.png)
+            - ...
+          - ReducedFit
+            - reduced fit file (e.g. sciencefile_1.fits)
+            - auxiliary file 1 (e.g. astrometry_summary.png)
+            - ...
+        - file 2 folder (e.g. masterbias_1.fits.d)
+          - MasterBias
+            - auxiliary file 1 (e.g. preview.png)
+            - ...
+        - ...
+      - ...
+    - ...
+  - logs
+    - log file 1
+    - ...
+  - astrosource
+    - source 1
+      auxiliary file 1 (e.g. finding_chart.png)
+    - ...
+  - database file (e.g. iop4.db)
+
+..  code-block:: text
+
+    datadir (e.g. ~/.iop4data/)
+    ├── raw
+    │   ├── telescope 1 (e.g. OSN-T090)
+    │   │   ├── night 1 (e.g. 2024-04-08)
+    │   │   │   ├── file 1 (e.g. sciencefile_1.fits)
+    │   │   │   ├── file 2 (e.g. bias_1.fits)
+    │   │   │   └── ...
+    │   │   └── ...
+    │   └── ...
+    ├── masterbias
+    │   ├── telescope 1 (e.g. OSN-T090)
+    │   │   ├── night 1 (e.g. 2024-04-08)
+    │   │   │   ├── file 1 (e.g. masterbias_1.fits)
+    │   │   │   └── ...
+    │   │   └── ...
+    │   └── ...
+    ├── masterdark
+    │   └── ...
+    ├── masterflat
+    │   └── ...
+    ├── calibration
+    │   ├── telescope 1 (e.g. OSN-T090)
+    │   │   ├── night 1 (e.g. 2024-04-08)
+    │   │   │   ├── file 1 folder (e.g. sciencefile_1.fits.d)
+    │   │   │   │   ├── RawFit
+    │   │   │   │   │   ├── auxiliary file 1 (e.g. preview.png)
+    │   │   │   │   │   └── ...
+    │   │   │   │   └── ReducedFit
+    │   │   │   │       ├── reduced fit file (e.g. sciencefile_1.fits)
+    │   │   │   │       ├── auxiliary file 1 (e.g. astrometry_summary.png)
+    │   │   │   │       └── ...
+    │   │   │   ├── file 2 folder (e.g. masterbias_1.fits.d)
+    │   │   │   │   └── MasterBias
+    │   │   │   │       ├── auxiliary file 1 (e.g. preview.png)
+    │   │   │   │       └── ...
+    │   │   │   └── ...
+    │   │   └── ...
+    │   └── ...
+    ├── logs
+    │   ├── log file 1
+    │   └── ...
+    ├── astrosource
+    │   ├── source 1
+    │   │   └── auxiliary file 1 (e.g. finding_chart.png)
+    │   └── ...
+    └── database file (e.g. iop4.db)
+
+
+By default, IOP4 will use ``~/.iop4data/`` as the data directory root. You can 
+indicate a different path with the ``datadir`` configuration option. You can 
+also specify a different (and independent) database file location with the 
+``db_path`` option.
+
 .. rubric:: Footnotes
 
 .. [#otherORMs] There exists many other ORM engines, such as SQLAlchemy, 

docs/serving_iop4_in_production.rst

@@ -1,6 +1,11 @@
 Serving IOP4 in production
 ==========================
 
+.. _production_web_server:
+
+Setting up a production web server
+----------------------------------
+
 .. warning::
 
     This section requires some familiarity with system 
@@ -55,4 +60,94 @@ your Django project are:
     ALLOWED_HOSTS = ["domain", "domain2"]
 
     # Configure static files to the path served by nginx
-    STATIC_ROOT = '/path/to/static/'
+    STATIC_ROOT = '/path/to/static/'
+
+
+.. _production_cron_job:
+
+Creating a cron job for reducing new observations
+-------------------------------------------------
+
+You might be interested in creating a cron job that routinely reduces new 
+observations every morning. Here we provide an example. This assumes you are 
+working in a linux system. Other OS provide different methods to create jobs.
+
+Use ``crontab -e`` to edit your crontab file and add a line like the following
+
+..  code-block:: text
+  
+    00 08 * * * /home/vhega/run_iop4_daily.sh > /home/vhega/run_iop4_daily.log 2>&1
+
+Then, create a file ``run_iop4_daily.sh``, give it execution permissions (``chmod +x run_iop4_daily.sh``) and add the following content:
+
+..  code-block:: bash
+
+    #!/usr/bin/bash
+
+    # save the current datetime
+    printf -v date '%(%Y-%m-%d_%H%M)T' -1
+
+    echo "#########################################"
+    echo "Run daily job for IOP4: $date"
+    echo "#########################################"
+
+    . /home/vhega/miniconda3/bin/activate iop4
+
+    # make sure all files created by iop4 are editable by the current user only
+    umask 0022
+
+    # Run iop4 for new observations (i.e. last night)
+    iop4 --discover-missing -o log_file=/home/vhega/iop4data/logs/daily_$date.log
+
+    # Create and send a summary of the results for last night
+    iop4-night-summary  --fromaddr '{{YOUR SENDER ADDRESS}}' \
+                        --mailto '{{ADDRESS 1}},{{ADDRESS 2}},{{ADDRESS 3}}' \
+                        --contact-name '{{CONTACT NAME}}' \
+                        --contact-email '{{CONTACT EMAIL}}' \
+                        --site-url '{{DEPLOYMENT SITE URL}}' \
+                        --saveto "/home/vhega/iop4data/logs/daily_$date.html"
+
+The above script will run iop4 every morning, disovering and proccessing new 
+observations. 
+
+The last command is optional, and will send an email with a summary with the 
+results from last night to the specified email addresses. 
+The cron job does not need to be run on a daily basis, and you can run it 
+whenever you expect new observations to become available in the telescope 
+remote archives (e.g. every few hours). Alternatively, you can pass an argument 
+to ``iop4-night-summary`` specifying the night that you want to generate the 
+summary for. The email will be in HTML format (viewable in any modern browser or
+email client), and can optionally be saved to any path. You can indicate the url 
+of your deployed site so the links in the email (e.g. for files with error) 
+point directly to the corresponding page in the iop4 web interface or admin 
+site.
+
+
+.. _production_share_datadir:
+
+Sharing the data directory with other system users
+--------------------------------------------------
+
+If you are running IOP4 in a server with multiple users, and have created an 
+user to run the IOP4 pipeline as a service, you might be interested in making 
+the local archive available to other users, so they can process the data 
+independently. By default, IOP4 will remove write permissions on raw files, 
+protecting them from accidental modification. For example, you can link your raw
+directory to the service account raw directory,
+
+.. code-block:: bash
+
+      ln -s /home/iop4user/.iop4data/raw ~/home/myuser/.iop4data/raw
+
+The other directories can also be linked, but keep in mind that other users
+might not be able to modify and reprocess reduced files (they will still be able
+to inspect them). You will still need to create your database following the 
+installation instructions.
+
+Multiple users can also share the same data directory, but this is not 
+recommended.
+
+You should not confuse system users (which can run the iop4 pipeline) with IOP4 
+portal users, that can access and inspect data from the web interface. These 
+should be created only after following See :ref:`Setting up a production web server <production_web_server>` (the debug web 
+server is not recommended for multiple users).