Skip to content

Commit

Permalink
Fixed #22/#73: made everything except ML install optional, dont updat…
Browse files Browse the repository at this point in the history
…e os by default
  • Loading branch information
grtjn committed Sep 6, 2015
1 parent d7550de commit 966e8e2
Show file tree
Hide file tree
Showing 13 changed files with 285 additions and 104 deletions.
55 changes: 52 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Scripts for bootstrapping a local MarkLogic cluster for development purposes using [Vagrant](https://www.vagrantup.com/) and [VirtualBox](https://www.virtualbox.org/).

By default these scripts create 3 'grtjn/centos-6.6' Vagrant VMs, running in VirtualBox. The names and ips will be recorded in /etc/hosts of host and VMs with use of vagrant-hostmanager. MarkLogic (including dependencies) will be installed on all three vms, and bootstrapped to form a cluster. The OS will be fully updated initially, and "Development Tools" installed as well. Zip/Unzip, Java, MLCP, Nodejs, Bower, Gulp, Forever, Ruby, Git, and Tomcat will be installed, and configured. A bare git repository will be prepared in /space/projects. All automatically with just a few commands.
By default these scripts create 3 'grtjn/centos-6.7' Vagrant VMs, running in VirtualBox. The names and ips will be recorded in /etc/hosts of host and VMs with use of vagrant-hostmanager. MarkLogic (including dependencies) will be installed on all three vms, and bootstrapped to form a cluster. The OS will be fully updated initially, and "Development Tools" installed as well. Zip/Unzip, Java, MLCP, Nodejs, Bower, Gulp, Forever, Ruby, Git, and Tomcat will be installed, and configured. A bare git repository will be prepared in /space/projects. All automatically with just a few commands.

Each VM takes roughly 2.5Gb. The VM template, together with 3 VMs will take about 10Gb of disk space. In addition, each VM that is launched will claim 2Gb of RAM, and 2 CPU cores. Make sure you have sufficient resources!

Expand Down Expand Up @@ -34,6 +34,8 @@ You first need to download and install prerequisites and mlvagrant itself:
- `sudo chmod 775 /opt/vagrant`
- Copy mlvagrant/opt/vagrant to /opt/vagrant

**IMPORTANT:**

You will also need to get hold of a valid license key. Put the license key info in the appropriate ml license properties file in /opt/vagrant. You will need an Enterprise (Developer) license for setting up clusters.

Above steps need to taken only once. For every project you wish to create VMs, you simply take these steps:
Expand Down Expand Up @@ -91,7 +93,9 @@ VM naming pattern - defaults to {project_name}-ml{i}, also allowed: {ml_version}
**IMPORTANT: DON'T CHANGE ONCE YOU HAVE CREATED THE VM'S!!**

### vm_version
CentOS base VM version - defaults to 6.6, allowed: 6.5/6.6/7.0/7.1
CentOS base VM version - defaults to 6.7, allowed: 5.11/6.5/6.6/6.7/7.0/7.1

Note: CentOS 5(.11) does not support MarkLogic 8

### ml_version
Major MarkLogic release to install - defaults to 8, allowed: 5,6,7,8 (installers need to be present)
Expand Down Expand Up @@ -123,7 +127,52 @@ Override hard-coded MarkLogic installers (file is searched in /space/software, o
### mlcp_installer
Override hard-coded MLCP installers (file is searched in /space/software, or c:\space\software\ on Windows)

## Using public_network (fixing IP issues)

### update_os
Run full OS updates - defaults to false

Note: doing this with CentOS 6.5 or 7.0 will take it up to the very latest minor release (6.7+ resp 7.1+)

### install_dev_tools
Install group "Development tools" - defaults to false

### install_zip
Install zip/unzip - defaults to true

Note: Zip/unzip **not** required for MLCP (provided through Java)

### install_java
Install Java - defaults to true

Note: necessary for MLCP
Note: installs JDK 8 currently

### install_mlcp
Install MarkLogic Content Pump - defaults to true

Note: installs an MLCP version that matches ml_version, unless an explicit mlcp_installer was specified

### install_nodejs
Install Node.js, npm, bower, gulp, forever (globally) - defaults to true

## install_ruby
Install Ruby - default to true

Note: Ruby is mostly already installed on CentOS, this is just to be certain

### install_git
Install Git command-line tools - defaults to true

### install_git_project
Initializes a bare Git repository under /space/projects, along with a user named {project_name} to use it

### install_tomcat
Install Tomcat, and enable the service - defaults to true

Note: Tomcat could be pre-installed, but usually isn't enabled by default. This will make sure it is installed, and enabled.
Note: on CentOS 5 you get Tomcat 5 (tomcat5), on CentOS 6 you get Tomcat 6 (tomcat6), on CentOS 7 you get Tomcat 7 (tomcat)

## Fixing IP issues with public_network

The earlier version of mlvagrant was using public_network, and that will likely reappear as option soon. Handing out of IPs in that case depends on the external DHCP of the network you happen to be connected with. If you are running on a laptop, and take it elsewhere, your laptop, and public_network VMs will get new IPs. At that moment the hosts tables become outdated. You can fix that with a simple command though:

Expand Down
2 changes: 1 addition & 1 deletion opt/vagrant/bootstrap-centos-extra.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ fi

# Run the installers
sudo /opt/vagrant/restore-yum-cache.sh
sudo /opt/vagrant/update-os.sh
sudo /opt/vagrant/update-os.sh $4
sudo /opt/vagrant/install-ml-centos.sh $4
sudo /opt/vagrant/setup-ml-extra.sh $@
sudo /opt/vagrant/backup-yum-cache.sh
6 changes: 3 additions & 3 deletions opt/vagrant/bootstrap-centos-master.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ fi

# Run the installers.
sudo /opt/vagrant/restore-yum-cache.sh
sudo /opt/vagrant/update-os.sh
sudo /opt/vagrant/update-os.sh $4
sudo /opt/vagrant/install-ml-centos.sh $4
sudo /opt/vagrant/setup-ml-master.sh $1 $2 $3
sudo /opt/vagrant/install-node.sh
sudo /opt/vagrant/install-node.sh $4
sudo /opt/vagrant/install-mlcp.sh $4
sudo /opt/vagrant/install-user.sh $4
sudo /opt/vagrant/setup-git.sh $4
sudo /opt/vagrant/setup-tomcat.sh
sudo /opt/vagrant/setup-tomcat.sh $4
sudo /opt/vagrant/backup-yum-cache.sh
55 changes: 34 additions & 21 deletions opt/vagrant/install-mlcp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,43 @@ echo "running $0 $@"

# Defaults
ml_version=8
install_zip=true
install_java=true
install_mlcp=true

# Load the normalized project properties.
source /tmp/$1.project.properties

yum -y install zip unzip
yum -y install java-1.8.0-openjdk-devel

# Determine installer to use.
if [ -n "${mlcp_installer}" ]; then
installer=${mlcp_installer}
elif [ "${ml_version}" = "8" ]; then
installer=mlcp-1.3-3-bin.zip
elif [ "${ml_version}" = "7" ]; then
installer=mlcp-Hadoop2-1.2-4-bin.zip
else
installer=mlcp-Hadoop2-1.0-5-bin.zip
# Zip/unzip not required for MLCP (provided through Java)
if [ install_zip == true ]; then
yum -y install zip unzip
fi

echo "Installing MLCP using $installer ..."
install_dir=$(echo $installer | sed -e "s/-bin.zip//g")
if [ ! -d /opt/$install_dir ]; then
cd /opt && unzip "/space/software/$installer"
fi
if [ ! -h /usr/local/mlcp ]; then
echo "setting sym-link: /opt/$install_dir for mlcp"
cd /usr/local && ln -s "/opt/$install_dir" mlcp
fi
if [ install_mlcp == true ]; then
# Java required for MLCP
yum -y install java-1.8.0-openjdk-devel

# Determine installer to use.
if [ -n "${mlcp_installer}" ]; then
installer=${mlcp_installer}
elif [ "${ml_version}" = "8" ]; then
installer=mlcp-1.3-3-bin.zip
elif [ "${ml_version}" = "7" ]; then
installer=mlcp-Hadoop2-1.2-4-bin.zip
else
installer=mlcp-Hadoop2-1.0-5-bin.zip
fi

echo "Installing MLCP using $installer ..."
install_dir=$(echo $installer | sed -e "s/-bin.zip//g")
if [ ! -d /opt/$install_dir ]; then
cd /opt && unzip "/space/software/$installer"
fi
if [ ! -h /usr/local/mlcp ]; then
echo "setting sym-link: /opt/$install_dir for mlcp"
cd /usr/local && ln -s "/opt/$install_dir" mlcp
fi

elif [ install_java == true ]; then
yum -y install java-1.8.0-openjdk-devel
fi
57 changes: 35 additions & 22 deletions opt/vagrant/install-node.sh
Original file line number Diff line number Diff line change
@@ -1,30 +1,43 @@
#! /bin/sh
echo "running $0 $@"

yum -y install curl epel-release # epel-release necessary to install nodejs/npm on CentOS 5, and no harm otherwise
curl --silent --location https://rpm.nodesource.com/setup | bash -
yum -y install gcc-c++ make nodejs

if [ hash bower 2> /dev/null ]; then
echo 'Bower is already installed'
else
npm -q install -g bower
fi
# Defaults
install_nodejs=true
install_ruby=true

if [ hash gulp 2> /dev/null ]; then
echo 'Gulp is already installed'
else
npm -q install -g gulp
fi
# Load the normalized project properties.
source /tmp/$1.project.properties

if [ install_nodejs == true ]; then

yum -y install curl epel-release # epel-release necessary to install nodejs/npm on CentOS 5, and no harm otherwise
curl --silent --location https://rpm.nodesource.com/setup | bash -
yum -y install gcc-c++ make nodejs

if [ hash bower 2> /dev/null ]; then
echo 'Bower is already installed'
else
npm -q install -g bower
fi

if [ hash gulp 2> /dev/null ]; then
echo 'Gulp is already installed'
else
npm -q install -g gulp
fi

if [ hash forever 2> /dev/null ]; then
echo 'Forever is already installed'
else
npm -q install -g forever
fi

if [ hash forever 2> /dev/null ]; then
echo 'Forever is already installed'
else
npm -q install -g forever
fi

if [ hash ruby 2> /dev/null ]; then
echo 'Ruby is already installed'
else
yum -y install ruby
if [ install_ruby == true ]; then
if [ hash ruby 2> /dev/null ]; then
echo 'Ruby is already installed'
else
yum -y install ruby
fi
fi
16 changes: 12 additions & 4 deletions opt/vagrant/install-user.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,16 @@
echo "running $0 $@"

if [ -d /vagrant ]; then
# creation of goups and users is limited on demo servers
/usr/sbin/groupadd -r sshuser
pass=$(perl -e 'print crypt($ARGV[0], "password")' $1)
/usr/sbin/useradd -m -G sshuser -p $pass $1
# Defaults
install_git_project=true

# Load the normalized project properties.
source /tmp/$1.project.properties

if [ install_git_project == true ]; then
# creation of goups and users is limited on demo servers
/usr/sbin/groupadd -r sshuser
pass=$(perl -e 'print crypt($ARGV[0], "password")' $1)
/usr/sbin/useradd -m -G sshuser -p $pass $1
fi
fi
65 changes: 38 additions & 27 deletions opt/vagrant/setup-git.sh
Original file line number Diff line number Diff line change
@@ -1,36 +1,47 @@
#! /bin/sh
echo "running $0 $@"

# Install git
yum -y install git
# Defaults
install_git=true
install_git_project=true

# create a local git repo
mkdir -p /space/projects/$1.git
# Load the normalized project properties.
source /tmp/$1.project.properties

# initialize the git repo
cd /space/projects/$1.git
if [ ! -f HEAD ]; then
git --bare init
fi
if [ ! -f hooks/post-receive ]; then
printf "#!/bin/sh\nin=\$(cat)\nbranch=\${in##*/}\nGIT_WORK_TREE=/space/projects/$1.live git checkout -f \$branch\nchown -R :sshuser /space/projects/$1.live\nchmod -R g+rw /space/projects/$1.live\n" > hooks/post-receive
chmod 755 hooks/post-receive
if [ install_git == true ]; then
# Install git
yum -y install git
fi

# create a project deploy dir
mkdir -p /space/projects/$1.live
if [ install_git_project == true ]; then
# create a local git repo
mkdir -p /space/projects/$1.git

# initialize the git repo
cd /space/projects/$1.git
if [ ! -f HEAD ]; then
git --bare init
fi
if [ ! -f hooks/post-receive ]; then
printf "#!/bin/sh\nin=\$(cat)\nbranch=\${in##*/}\nGIT_WORK_TREE=/space/projects/$1.live git checkout -f \$branch\nchown -R :sshuser /space/projects/$1.live\nchmod -R g+rw /space/projects/$1.live\n" > hooks/post-receive
chmod 755 hooks/post-receive
fi

# create a project deploy dir
mkdir -p /space/projects/$1.live

# change permissions to allow remote deployment
cd /space/projects
if [ -d /vagrant ]; then
chown -R $1:sshuser $1.git
chown -R $1:sshuser $1.live
else
# creation of users is limited on demo servers
chown -R $USER:sshuser $1.git
chown -R $USER:sshuser $1.live
# change permissions to allow remote deployment
cd /space/projects
if [ -d /vagrant ]; then
chown -R $1:sshuser $1.git
chown -R $1:sshuser $1.live
else
# creation of users is limited on demo servers
chown -R $USER:sshuser $1.git
chown -R $USER:sshuser $1.live
fi
chmod -R g+rw $1.git
chmod -R g+rw $1.live
chmod g+s $1.git
chmod g+s $1.live
fi
chmod -R g+rw $1.git
chmod -R g+rw $1.live
chmod g+s $1.git
chmod g+s $1.live
2 changes: 2 additions & 0 deletions opt/vagrant/setup-ml-extra.sh
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ echo "LICENSEE is ${LICENSEE}"

# Make sure curl is installed
yum -y install curl
# Make sure perl URI::Escape is installed
yum -y install perl-URI

# Suppress progress meter, but still show errors
CURL="curl -s -S"
Expand Down
2 changes: 2 additions & 0 deletions opt/vagrant/setup-ml-master.sh
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ echo "LICENSEE is ${LICENSEE}"

# Make sure curl is installed
yum -y install curl
# Make sure perl URI::Escape is installed
yum -y install perl-URI

# Suppress progress meter, but still show errors
CURL="curl -s -S"
Expand Down
39 changes: 24 additions & 15 deletions opt/vagrant/setup-tomcat.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,28 @@ echo "running $0 $@"

os=`cat /etc/redhat-release`

# tomcat likely pre-installed on demo servers, but run this anyhow
if [[ $os == *"7."* ]]; then
yum -y install tomcat
/sbin/chkconfig --levels 2345 tomcat on
/sbin/service tomcat start
elif [[ $os == *"5."* ]]; then
yum -y install tomcat5
/sbin/chkconfig --levels 2345 tomcat5 on
/sbin/service tomcat5 start
else
yum -y install tomcat6
/sbin/chkconfig --levels 2345 tomcat6 on
/sbin/service tomcat6 start
fi
# Defaults
install_tomcat=true

# Load the normalized project properties.
source /tmp/$1.project.properties

if [ install_tomcat == true ]; then

echo "Note: consider running Tomcat behind a firewall if you intend to keep it open!"
# tomcat likely pre-installed on demo servers, but run this anyhow
if [[ $os == *"7."* ]]; then
yum -y install tomcat
/sbin/chkconfig --levels 2345 tomcat on
/sbin/service tomcat start
elif [[ $os == *"5."* ]]; then
yum -y install tomcat5
/sbin/chkconfig --levels 2345 tomcat5 on
/sbin/service tomcat5 start
else
yum -y install tomcat6
/sbin/chkconfig --levels 2345 tomcat6 on
/sbin/service tomcat6 start
fi

echo "Note: consider running Tomcat behind a firewall if you intend to keep it open!"
fi
Loading

0 comments on commit 966e8e2

Please sign in to comment.