dictyBase Developers

Solving one problem at a time

Setup Remote Machine for Deployment

Local machine setup

Clone the git repository with deploy tasks

1

git://github.com/dictyBase/deploy-task.git

Setup perl environment

Documented here

Install Rex

The perl module for writing and running all deployment tasks.

First create a project specific local lib environment and install all required modules

1
2
3

perlbrew lib create deploy-tasks
perlbrew use lib deploy-perl@deploy-tasks
cpanm -n Rex

Before we start

The remote box should be accessible by ssh.

It also have to be configured for public key based SSH authentication with any passphrase. It’s recommend not to use the default one,

1

ssh-keygen -t rsa -f [filename]

Then add the content of ~/.ssh/[filename]_rsa.pub file from local machine to the ~/.ssh/authorized_keys of remote machine. Refer to the ssh-key rex task below for this transfer.

Running rex tasks

All the rex task are going to run remotely, so it is not at all needed to be installed on remote machine. Create a default Rexfile and add the correct user name and ssh key credentials.

1

cp Rexfile.sample Rexfile
‘Sample rexfile public key’ (Rexfile) download
1
2
3
4
5
6
7
8

use lib 'lib';

user 'user_name';
private_key 'path_to_private_key';
public_key 'path_to_public_key';
key_auth;

require Tasks;
‘Sample Rexfile with password’ (Rexfile_pass) download
1
2
3
4
5
6
7

use lib 'lib';

user 'user_name';
password 'pass';
pass_auth;

require Tasks;

Check the list of tasks that are available to run. Make sure which are mentioned to run as sudo, pass the sudo password (-S)in the argument

1
2

rex -T
rex -H <host> -s -S <sudo pass> <task_name>

To run sudo globally, add it to the Rexfile

‘Rexfile with sudo on’ (Rexfile_sudo) download
1
2
3
4
5
6
7
8
9
10
11
12
13

use lib 'lib';

user 'user_name';

# global sudo
sudo_password '48343jfen';
sudo -on;

private_key 'path_to_private_key';
public_key 'path_to_public_key';
key_auth;

require Tasks;

To run a non-default Rexfile

1

rex -H <host> -f [other_file] task_name

Last but not the least

  • The shared folder is assumed to be hosted on a separated partition. It has to be created with an ext4 filesystem before any of the task is being run.
  • However in case of shared device which is already mounted the above step is not needed. Refer to the Setup shared folder section below for detail.
  • Various configuration files needed for sys admin tasks are either present in conf.d folder or mentioned in the detail guide. Make sure to consult them if something is unclear.

Sys admin tasks

The tasks are assumed to be run on a remote CentOS box and so yum/rpm is selected for package installation. At this point no heuristics is performed to identify the type of remote box.

Quick steps

Create a yaml file in $HOME/.rex/config.yml and run the given Rex tasks

‘sample rex config’ (config.yml) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

# sets up various command aliases and two groups sys and developer
# with varying level of access
sudo:
  file: conf.d/sudoers.multigenome

# create the groups and then add the user to that group
# could add extra groups separated by colon
group:
  name: 'developer:sys:deploy'
user:
  user: newuser
  pass: newpass
  groups: 'developer:deploy'

# the device should have an ext4 partition and shared folder will be the only folder
# occupying the partition
shared:
  group: deploy
  folder: '/shared'
  device: '/dev/sdb1'

# uncomment the following section for shared folder if it has to coexist with other folder in
# the device. Make sure you comment the previous section. 
#'shared-remount':
#  group: deploy
#  folder: '/home/shared'
#  base: '/home'

# Download rpms from here
# http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html
oracle:
  rpm: '/home/user/downloads/rpms'
  file: 'conf.d/tnsnames.ora'
  host: localhost
  sid:  orcl
  service: served
mojo:
  mode: staging

# look in the step by step section for sample files
apache:
  envvars:
    file: 'env.sh'
  vhost:
    file: 'conf.d/apache_site.conf'
    name: genome
  perl:
     file: 'conf.d/perl.conf'

# this should match the shared folder 
perlbrew:
  root: '/shared'
  perl: '5.10.1'

Rex tasks

1
2
3
4
5
6

rex -H <host> -s setup:dicty:box
rex -H <host> setup:dicty:perl

# check for status of perl installation before running the toolchain task
rex -H <host> perl:install-status
rex -H <host> setup:dicty:perl-toolchain

Alternate step by step

Skip this entire section if you have already gone through Quick steps

Add extra repositories

1

rex -H <host> -s -S <sudo_pass> add:repos

Add extra sudoers configuration file

1

rex -H <host> -s -S <sudo_pass> add:sudoers --file=conf.d/sudoers.multigenome

Add user to developers and deploy groups

1
2

rex -H <host> -s -S <sudo_pass> add:groups --name=developer:deploy
rex -H <host> -s -S <sudo_pass> add:user --user=<user> --pass=<pass> --groups=developer:deploy

Install packages

1

rex -H <host> -s -S <sudo_pass> install:dicty-pack

Setup shared folder

1
2
3

Remember the device is expected to have a ext4 partition
rex -H <host> -s -S <sudo_pass> setup:shared-folder --group=[deploy] \ 
               --folder=[/dictybase] --device=[]

The above task is suitable when the entire device is dedicated for the shared folder. However, if the shared folder has to coexist in a device(shared folder) with other folders, the run the following task …

1
2

rex -H <host> -s -S <sudo_pass> setup:shared-folder-remount --group=[deploy] \ 
               --folder=[/dictybase] --base-folder[$HOME]

It will remount the base folder with acl and sets the proper share permission for the folder. However, it is not persistent over reboot, so add the acl entry to fstab manually to make it stick.

Set daemontools

1

rex -H <host> -s -S <sudo_pass> setup:daemontools

Mojolicious web application setup

1

rex -H <host> -s -S <sudo_pass> setup:global-mojo --mode=[staging|production]

Apache setup

1

rex -H <host> -s -S <sudo_pass> setup:apache:envvars --file=[file]

The following envvars have to be defined

1
2
3
4
5
6

export ASSETS_DIR=....
export WEBAPPS_DIR=...
export WEBHOST_NAME=....
export DICTYHOST_NAME=....
export MULTIGENOME_WEBROOT=....
export GBROWSE_HOST=....

1

rex -H <host> -s -S <sudo_pass> setup:apache:vhost --file=[file] --name=multigenome.conf

A minimal configuration file

‘multigenome virtualhost configration’ (multigenome.conf) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

<VirtualHost ${WEBHOST_NAME}:80>
    ServerAdmin admin@admin.edu
    ServerName multigenome


    #
    # LogLevel: Control the number of messages logged to the error_log.
    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    #

    LogLevel warn

    # RewriteLogLevel 3 

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_common
    ErrorLog /var/log/httpd/${WEBHOST_NAME}_error.log
    CustomLog /var/log/httpd/${WEBHOST_NAME}_access.log vhost_common

    # HTTP compression logging

    #DeflateFilterNote Input input_info
    #DeflateFilterNote Output output_info
    #DeflateFilterNote Ratio ratio_info
    #LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
    #CustomLog /var/log/apache2/deflate_log deflate

    DocumentRoot ${MULTIGENOME_WEBROOT}
    DirectoryIndex index.html index.htm default.htm multigenome.html

    RewriteEngine On

    #Redirect for fasta download page
    RedirectMatch ^/downloads/fasta   http://${DICTYHOST_NAME}/downloads/fasta
    RewriteRule ^/discoideum(\S*)  http://${DICTYHOST_NAME}$1 [NC,L,R]


## -- serving static files through apache
#
    Alias /assets ${ASSETS_DIR}
    <Directory "${ASSETS_DIR}">
        Allow from all
          <IfModule mod_expires.c>
            ExpiresActive On

            ExpiresByType image/jpg "access plus 6 months"
              ExpiresByType image/gif "access plus 6 months"
              ExpiresByType image/png "access plus 6 months"
              ExpiresByType application/javascript "access plus 3 months"
              ExpiresByType application/x-javascript "access plus 3 months"
              ExpiresByType text/ecmascript "access plus 3 months"
              ExpiresByType text/javascript "access plus 3 months"

              </IfModule>
    </Directory>

    <Location /assets>
        SetHandler default-handler
    </Location>

    HeaderName /assets/include/default/page-header.html
    ReadmeName /assets/include/default/page-footer.html

    # gbrowse2 proxy
    <Proxy *>
        Allow from all
    </Proxy>

    ProxyPass         /browser  ${GBROWSE_HOST}/browser
    ProxyPassReverse  /browser  ${GBROWSE_HOST}/browser
    ProxyPass         /gbrowse2 ${GBROWSE_HOST}/gbrowse2
    ProxyPassReverse  /gbrowse2 ${GBROWSE_HOST}/gbrowse2


</VirtualHost>

1

rex -H <host> -s -S <sudo_pass> setup:apache:perl-code --file=[file]
‘mod_perl configuration’ (perl.conf) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

    ## -- Including all conf file for each web application
    ## -- Assuming here that mod_perl is included in apache2
    <Perl>
           my $folder = $ENV{WEBAPPS_DIR};
           if (-e $folder) {
              opendir my $web,  $folder or die "cannot open dir $!";
              my @dirs = map {"$folder/$_"} grep {/^[A-Z, a-z,\-]+$/} readdir $web;
              closedir $web;
              for my $name(@dirs) {
                 my $config = "$name/deploy/apache.conf";
                 if (-e $config) {
                     push @Include,$config;
                 }
              }
            }
    </Perl>

1

rex -H <host> -s -S <sudo_pass> setup:apache:startup

Oracle client setup

It installs the instant client rpms and sets up oracle environment globally. Oracle instantclient could be downloaded from here. For here we need three rpms

  • oracle-instantclient-basic-10.2.0.4-1.x86_64.rpm
  • oracle-instantclient-sqlplus-10.2.0.4-1.x86_64.rpm
  • oracle-instantclient-devel-10.2.0.4-1.x86_64.rpm
1
2
3

rex -H <host> -S <sudo_pass> setup:oracle-client --rpm=rpms
rex -H <host> -S <sudo_pass> setup:oracle:tnsnames --file=[file] \ 
    --host=[host] --sid=[sid] --service=[service_name]

Setup perl environment

perl environment for deployment will be setup using perlbrew in the shared folder(/dictybase). The first task requires sudo access if system argument is given.

1
2
3
4
5

rex -H <host> -s perlbrew:install --install-root=/dictybase/perl5 --system=1
rex -H <host> perlbrew:install-cpanm
rex -H <host> perl:install-notest
rex -H <host> perlbrew:switch --version=perl-5.10.1
rex -H <host> perl:install-toolchain

Posted by Siddhartha Basu