-
Notifications
You must be signed in to change notification settings - Fork 10
IRIS_Packaging_Solution.md
This document provides comprehensive information about the packaging solution adopted by Infrastructure and Release Information System (IRIS) team, including the following:
- Rules and conventions
- Technical choices
- Implementation specifications
Note: The current solution is verified in openSUSE 12.1.
The purpose of this document is to reveal the technical background, decision-making process, and implementation details of the IRIS packaging solution, hoping other similar projects can use these experiences for reference.
This document is valid for pure Python applications.
This section describes the essentials of this technical solution, including the following:
- Rules and conventions
- Technical choices
Deploying diverse Python application requires solid and consistent deployment standards. Our main consideration is to adopt simple solution rather than easy ones, because easy ones can cause messes in the long run.
In light of this, rules and conventions for the purpose of keeping the packaging process simple and effective are followed, as shown below:
-
Use virtual environment.
Always use virtual environment to install dependency packages to avoid potential contradictions of dependency packages required for different projects.
A virtual environment is an isolated working copy of Python that allows engineers to work on a specific project without worrying about affecting other projects. For example, users can work on a project that requires Django 1.3 while also maintaining a project that requires Django 1.0. For more information, refer to Virtual Environments.
-
Never use Python packages from Linux distributions.
The only good Linux distributions may be doing is automatically updating packages with security vulnerabilities that users may have missed. But the deployers are the ones who have responsibilities to monitor the deployed softwares. Relying on the Linux distributions give deployers a false sense of security. If a customers' data gets hacked, they don't care that Ubuntu was too slow to issue a security update.
-
Configuration is not part of the application.
The production configuration doesn't belong to the same source repository. There are configuration management tools like Salt, Puppet or Chef that these jobs for engineers.
Based on the rules and conventions in ealier section, main technical choices made are as follows:
-
Use virtual environment and include it in RPM packages.
In addition to the superiorities mentioned in earlier section, our solution also allows users to deploy IRIS by running a single command without taking dependency issue into account and get everything ready once and for all by including virtual environment in PRM packages, making the handling of IRIS RPM package a breeze.
-
List all Python dependency packages in requirements.txt.
Requirements.txt serves as a dependency package list for IRIS.
During the make process, virtual environment uses requirements.txt to create env.bundle, in which all Python dependency packages are included.
-
Extract configuration file from source code.
IRIS configuration file is stored independently in /etc/iris directory instead of written in source code, thus avoiding the hard-coding issue and making further maintenance easier.
#3 Technical Specifications
This section describes the technical specifications of the following:
- makefile
- setup.py
- spec file
Makefiles specifies how to derive target files, including the following:
-
<Module_Name>-<Version>.tar.gz - env.pybundle
- prefetch.tar
Key points of makefile for IRIS are as follows:
-
Generate
<Module_Name>-<Version>.tar.gz.git archive --format=tar --prefix=$(NVR_DIR)/ $(TAG) | tar xpf - git show $(TAG) --oneline | head -1 > $(COMMIT_ID_FILE) tar cfz $(PKG_FILE) -C $(DIST_DIR) $(NVR) -
Generate env.pybundle based on the dependency package list specified in requirements.txt.
Make brand new virtual environment, add "--no-site-packages" option to ignore the dependency support provided by the system, add "--distribute" option to use distribute instead of setuptools. virtualenv --no-site-packages --distribute $(ENV_DIR) . $(ENV_DIR)/bin/activate && \ pip bundle $(BUNDLE_FILE) -r $(REQ_FILE) -
Generate prefetch.tar.
-
Confirm the pip version and distribute version required by virtualenv and define corresponding variables.
PIP_URL=https://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz PIP_FILE=$(DIST_DIR)/$(shell basename $(PIP_URL)) DIS_URL=https://pypi.python.org/packages/source/d/distribute/distribute-0.6.19.tar.gz DIS_FILE=$(DIST_DIR)/$(shell basename $(DIS_URL)) PREFETCH_FILE=packaging/prefetch.tar -
Download pip and distribute.
$(PIP_FILE): wget -O $(PIP_FILE) $(PIP_URL) touch $(PIP_FILE) $(DIS_FILE): wget -O $(DIS_FILE) $(DIS_URL) touch $(DIS_FILE) -
Combine distribute-
<Version>.tar.gz and pip-<Version>.tar.gz into prefetch.tar. Note that the version information is masked so that version issue needn't to be taken into account when creating the spec file.$(PREFETCH_FILE): $(PIP_FILE) $(DIS_FILE) tar cvf $(PRFEETCH_FILE) -C $(DIST_DIR) $(shell basename $(PIP_FILE)) $(shell basename $(DIS_FILE))
-
Setup.py facilitates the installation process on local host.
Key points of setup.py for IRIS are as follows:
#Comment below 2 lines to avoid obs build error, un-comment them to install manually.
#from distribute_setup import use_setuptools
#use_setuptools()
setup(name = 'iris-<Module_Name>',
version = version,
description = 'IRIS api <Module_Name> service',
author = '<Author>',
author_email = '<E-mail_Address>',
url = 'http://www.tizen.org/',
scripts = ['tools/iris-<Module_Name>-server.py'],
packages = find_packages(),
)
The spec file contains information required by OBS to build packages, as well as instructions that tells OBS how to build it. The spec file also dictates exactly what files are a part of the package, and where they should be installed.
Key points of customizing the spec file for IRIS are as follows:
- Find out the python site-packages path in virtualenv, which will be used in the post scriptlets.
Refer to the hellopack project for the position to insert it.
%{!?pylib_site: %define pylib_site %(%{__python} -c "from distutils.sysconfig import get_python_lib;import sys;print get_python_lib().lstrip(sys.prefix)")}
- Define the preamble.
The preamble contains information that will be displayed when users request information about the package.
Name: iris-<Module_Name>
Version: <Version>
Release: 1
License: GPLv2
Summary: Http API gateway for IRIS services
Url: http://www.tizen.org
Group: Web/Services
Source: %{name}-%{version}.tar.gz
Source1: prefetch.tar
Source2: env.pybundle
Source3: iris-<Module_Name>.conf
Source4: iris-<Module_Name>.json
BuildRequires: python-virtualenv
BuildRequires: python-devel
BuildRequires: openldap2-devel
BuildRequires: gawk
Requires: python
Requires: apache2
Requires: apache2-mod_wsgi BuildRoot: %{_tmppath}/%{name}-%{version}-build AutoReq: no
-
Define the build scriptlets.
-
Copy all the sources to the OBS server and untar prefetch.tar.
cp %{SOURCE1} . cp %{SOURCE2} . cp %{SOURCE3} . cp %{SOURCE4} . tar xf %{SOURCE1} -
Create brand new virtual environment, activate it, and then install Python dependency packages from env.pybundle.
By adding "--never-download" option, virtualenv will use distribute-
<Version>.tar.gz and pip-<Version>.tar.gz in the project directory instead of downloading the former mentioned tarballs again.By adding "--distribute" option, virtualenv will use distribute instead of setuptools.
virtualenv --never-download --distribute env source env/bin/activate pip install env.pybundle -
Remove env/build to prevent warnings during OBS build process.
rm -rf env/build -
Fix shebang issue, that is, replace Python relative paths provided by virtualenv with default Python path, more specifically, replace "{buildroot}/env/bin/python" with "/usr/bin/python".
grep -nIE '^#!.python' env/bin/ | awk -F':' '$2=="1" {print $1}' | xargs sed -i '1c#!/usr/bin/python'; grep -rnIE '^#!.*python' env | awk -F':' '$2=="1" {print $1}' | grep easy_install.py | xargs sed -i '1c#!/usr/bin/python'
-
-
Define the install scriptlets.
-
Use setup.py to perform installation in virtualenv directory, more specifically, to install in "/srv/iris/<Module_Name>/env/lib/python2.7/site-packages/<Module_Name>".
python setup.py install --root=env --prefix='' -
Copy the entire virtual environment on OBS server to local host, with all the installed dependencies, for example, Flask.
cp -a env/ %{buildroot}/srv/iris/<Module_Name> -
Install mod_wsgi file to work with apache to launch IRIS web service.
install -m644 iris-<Module_Name>.wsgi %{buildroot}/srv/iris/<Module_Name>/bin -
Copy distribute-
<Version>.tar.gz and pip-<Version>.tar.gz to local host to fulfil requrments for "post" section virtualenv installation.cp *.tar.gz %{buildroot}/srv/iris/<Module_Name> -
Install index.html.
install -m644 apps/index.html %{buildroot}/srv/iris/<Module_Name>/apps -
Install configuration file for specific module to system-wide IRIS configuration directory.
install -m644 iris-<Module_Name>.json %{buildroot}/etc/iris -
Install apache configuration file for specific module.
install -m644 iris-<Module_Name>.conf %{buildroot}/etc/apache2/vhosts.d -
Prevent possible error regarding symlink during RPM installation.
The following error message will be displayed if potential error regarding symlink exists.
"OSError: [Errno 17] File exists: 'env/lib64'"
The command below eleminates this possibility. A brand new /lib64 symlink will be automatically created in post phase.
rm -rf %{buildroot}/srv/iris/<Module_Name>/env/lib64
-
-
Define the post scriptlets.
-
Create virtualenv named env by using pip--tar.gz and distribute-.tar.gz to fix Python file shebangs and correct symlinks. Virtualenv can also be automatically adjusted for local enviroment and paths.
cd /srv/iris/<Module_Name>;virtualenv --never-download --distribute env -
Make new symlink to link /static to /apps in order to store static files, including html files and javascript files.
ln -s /srv/iris/<Module_Name>/apps /srv/iris/<Module_Name>/env/%{pylib_site}/<Module-Name>/static
-
-
Define the postun scriptlets.
Remove all the project files during uninstallation.
rm -rf /srv/iris/<Module_Name> *Note:* RPM can remove all the files that owned by specific package, without taking the files generated during runtime into account, that is, RPM can handle the removal partially instead of completely, that's where "rm -rf /srv/iris/<Module_Name> " comes in. Simply put, this *rm* command guarantees all the unnessary files are removed once and for all. -
Define the files directive.
%files %defattr(-,root,root) /srv/iris/ # Use RPM to track and manage the conf file for specific module, noreplace, tells RPM not to overwrite, or replace a configuration file. %dir /etc/iris %config(noreplace) /etc/iris/iris-<Module_Name>.json # Use RPM to track and manage the apache conf file for specific module, noreplace, tells RPM not to overwrite, or replace a configuration file. %dir /etc/apache2 %dir /etc/apache2/vhosts.d %config(noreplace) /etc/apache2/vhosts.d/iris-<Module_Name>.conf