RSync Info/Tips for Metawerx Customers

RSync Info/Tips for Metawerx Customers - v1.1

This document describes the use of RSync at metawerx. It is recommended
that the SSH tutorial is read before using this document, as RSync
runs over an SSH connection at metawerx.

This information is intended for use by hosting customers of
Metawerx Developer Hosting, however, it is useful as a general
guide for using RSync.

Contents

1. Introduction to RSync

2. Using RSync

3. Example: Developing and deploying a Tomcat application from home

4. Example: Advanced use of RSync, and automating the PLINK connection

1. Introduction to SSH - RSync and PuTTY

RSync allows you to update your site using compressed differential file
transfer. It has options to only update files that are changed, and is
much faster that updating using FTP.

Another major advantage is that you can sync entire folders up or down,
enabling you to instantly deploy your updated application, without having
to search for which files or images have changed, or go through each folder
separately.

RSync runs over our SSH encrypted connection, so if you are not already
using our SSH services, please read the SSH Notes File first
to establish your connection.

If you do not yet have RSync installed on your site, and you would like
it to be installed, please contact support.

2. Using RSync

RSync allows you to transfer folders to and from the metawerx servers, over
secure connection created by PLINK. It is faster than FTP because it can use
compression and can also be set to only transfers files that have changed.

Download CWRSYNC (http://www.itefix.no/cwrsync/)

In RSYNC, file paths are specified using the 'cygwin directory format'
c:\inetpub\myfiles is specified as /cygdrive/c/inetpub/myfiles/
(don't forget the trailing slash)

In the batch file below, you would set c:\myfolder\myfiles as /cygdrive/c/myfolder/myfiles/
Your RSYNC folder is specified as your username.

PLINK or PUTTY must be running to make RSYNC work, because we have it running
over an SSH tunnel, so start a PLINK connection to metawerx.net first. You
will then connect to the rsync drive on 127.0.0.1, which will connect through
PLINK to the metawerx server.

It is necessary to set an environment variable named RSYNC_PASSWORD before
running RSYNC to enter the password automatically. If this is not done,
you will be prompted for the password each time RSYNC is run.

Use RSYNC as follows:

rem set username, password and root folders (note: always use a leading and trailing slash on paths)
set RSYNC_USERNAME=your_username
set RSYNC_PASSWORD=your_password
set LOCALROOT=/cygdrive/c/myfolder/myfiles/
set REMOTEROOT=/

Note: in the examples below, quotes are provided around the path parameters in case
you are using a space in your folder names.

To transfer your local files up to the server:

rsync -vvrzut "%LOCALROOT%" "rsync://%RSYNC_USERNAME%@127.0.0.1/%RSYNC_USERNAME%%REMOTEROOT%"

To simulate what would be copied without actually copying anything, add "n" to the options, as follows:

rsync -nvvrzut "%LOCALROOT%" "rsync://%RSYNC_USERNAME%@127.0.0.1/%RSYNC_USERNAME%%REMOTEROOT%"

To transfer your files on the server, onto your local system:

rsync -vvrzut "rsync://%RSYNC_USERNAME%@127.0.0.1/%RSYNC_USERNAME%%REMOTEROOT%" "%LOCALROOT%"

To transfer your tomcat ROOT application from /private-cgi-bin/tomcat/ROOT on the server, to
your local system path c:\myfolder\myApplication

set LOCALROOT=/cygdrive/c/myfolder/myApplication/
set REMOTEROOT=/private-cgi-bin/tomcat/ROOT/
rsync -vvrzut "rsync://%RSYNC_USERNAME%@127.0.0.1/%RSYNC_USERNAME%%REMOTEROOT%" "%LOCALROOT%"

To transfer your local application from c:\myfolder\myApplication to the tomcat ROOT folder
/private-cgi-bin/tomcat/ROOT on the server:

set LOCALROOT=/cygdrive/c/myfolder/myApplication/
set REMOTEROOT=/private-cgi-bin/tomcat/ROOT/
rsync -vvrzut "%LOCALROOT%" "rsync://%RSYNC_USERNAME%@127.0.0.1/%RSYNC_USERNAME%%REMOTEROOT%"

Example without environment variables (note you will be prompted for the password)

rsync -vvrzut "rsync://username@127.0.0.1/username/private-cgi-bin/tomcat/ROOT/" "/cygdrive/c/myfolder/myApplication/"

Example with spaces in your folder name (this works because the paths are surrounded by quotes):

rsync -vvrzut "rsync://username@127.0.0.1/username/private-cgi-bin/tomcat/ROOT/" "/cygdrive/c/myfolder/my App/"

For a list of options, run RSYNC by itself.
Options used in these examples are as follows:

vv - very verbose
v - verbose
r - recursive
z - enable compression
u - update mode (don't replace files that don't need it)
t - enable timestamp syncing (required to make updates work properly)
n - dry run only, do not transfer any files, just simulate

For help, contact support@metawerx.net

3. Example: Developing and deploying a Tomcat application from home

This tutorial demonstrates how to develop a tomcat application from home,
or your office.

First, install the same version of Tomcat as the metawerx VM that you have
been provided with (for example, Tomcat 5.0.28 or Tomcat 4.1.30).

The Tomcat server uses a folder called /webapps which contains the web
applications.

/jakarta/webapps/ROOT
/jakarta/webapps/jsp-examples
/jakarta/webapps/servlet-examples

You will find a similar structure in your metawerx folder as follows:

/private-cgi-bin/tomcat/ROOT
/private-cgi-bin/tomcat/jsp-examples
/private-cgi-bin/tomcat/servlet-examples

Your /private-cgi-bin/tomcat folder is the same as your local /webapps
folder.

This example will handle the following problems:

a. creating a batch file to deploy the application easily

b. using a different web.xml file which contains different JDBC
information for your "home" version and your "server" version.

c. avoiding uploads of .BAK files and .java files to the server

a. creating a batch file to deploy the application easily
------------------------------------------------------------------

First, create a batch file or a quick way to get a PLINK connection
started (connect.bat)

For example:

CONNECT.BAT
-----------
c:\installers\putty\plink -C -v -l ssh_MyUserName -pw MyPassword -L 873:localhost:873 metawerx.net
rem -- uses compression (-C)
rem -- use verbose output (-v)
rem -- create a tunnel for port 873 only (RSync)

Next, create a batch file to upload the files to the server (sync.bat).

SYNC.BAT
--------
set RSYNC_PASSWORD=MyPassword
set LOCALROOT=/cygdrive/c/tomcat5/webapps/ROOT/
set SERVERROOT=rsync://MyUserName@127.0.0.1/MyUserName/private-cgi-bin/tomcat/ROOT/
c:\installers\rsync\rsync -vvrzut --delete "%LOCALROOT%" "%SERVERROOT%"
rem -- verbose output, recursive, zipped, update-mode, timestamp-syncing, delete files that don't exist locally

Now whenever you have a new version ready and tested locally, you can deploy it
using the following steps:

1. Start CONNECT.BAT
2. Start SYNC.BAT
3. Wait for upload to complete
4. Close CONNECT.BAT window
5. Connect to manager application (http://your_tomcat_site/manager/html)
6. Restart the application using the manager
7. Test on server

Only the files that have been modified will be uploaded, so performing
a change to the live server is very fast using this method, and avoids
going through all the folders using FTP to make sure I upload the right
files.

b. using a different web.xml file for the server
------------------------------------------------------------------

This works for most applications, but in some cases you will need to do something
special. For example, in the application described above, the web.xml file needs
to be modified for the server version because I want to use a different set of JDBC
settings on the live server.

The JDBC settings are stored in the web.xml file and locally, I want to connect
to the host "dbserver", with the username "sa" and no password. On the server
however, I want to use "127.0.0.1" for the host, "MyUsername" for the username
and "MyPassword" for the password.

In this case, I have made two web.xml files. One called "web.xml" and one called
"web.xml.live".

Using RSync, I sync the entire application, then copy the "web.xml.live" file over
the top of the "web.xml" file on the server.

This saves time, because I don't have to remember that the settings are different,
or upload the application and avoid uploading "web.xml" because it might override
the server settings. I just have two versions of the file and leave it to the
batch file to put the server version in the correct place.

The following commands are added to SYNC.BAT to handle this:

c:\installers\rsync\rsync -vvzut "%LOCALROOT%WEB-INF/web.xml.live" "%SERVERROOT%WEB-INF/web.xml"
rem -- copy up a single file "web-inf/web.xml.live" to "web-inf/web.xml"

c. avoiding uploads of .BAK files and .java files to the server
------------------------------------------------------------------

It is great having RSync upload everything, but I don't really want the .java
files or the .BAK files created by my text editor to be uploaded. This is just
a waste of time, and I also don't really want my source files on the server.

For this I use the "--exclude-from" command for RSync.

Create a file named c:\exclude.lst and put the following the lines in it:
*.bak
*.BAK
*.java

In the SYNC.BAT file you created above, there is a command that looks like this:

c:\installers\rsync\rsync -vvrzut --delete "%LOCALROOT%" "%SERVERROOT%"

Modify that to add the list of excluded files as follows:

c:\installers\rsync\rsync -vvrzut --delete --exclude-from=/cygdrive/c/exclude.lst "%LOCALROOT%" "%SERVERROOT%"

Now when you run the SYNC.BAT batch file, the *.bak, *.BAK and *.java files won't
be uploaded. As we are using "very-verbose" mode, you will see a list of the
excluded files as RSync runs through it's steps.

4. Example: Advanced use of RSync, and automating the PLINK connection

The above tutorial is a good example of how to use PLINK and RSync as part
of your deployment.

The steps in this section are more advanced, but allow you to use even
faster deployment.

If you have more than 1 site, perform daily-builds, or frequent website
updates, it is nice to have a completely automated deployment system.

The following steps are not as simple to follow as the above example
however, so I have separated them into this section.

Background
----------

If you use PLINK a lot, you will find that occasionally the connection drops
out, which can be irritating but hopefully something that will be fixed in
a later version. I also wanted a way to automatically stop and start PLINK
during SYNC sessions, so I am currently using the batch file layout below.

D:\SYNC\EXCLUDE.LST - list of files to exclude
D:\SYNC\CONNECT.BAT - actual commands for PLINK
D:\SYNC\SYNC2.BAT - start PLINK if necessary, one RSync call, error handling
D:\SYNC\SYNC.BAT - main batch, sync all applications

The SYNC.BAT batch file is part of my windows Start Menu (above Programs in
the menu, so it's visible as soon as I click the Start button). Naturally
everyone has their own preferences here, but the main thing is that it's
easy to get to.

For any web applications, I use a web.xml.live file as described in the above
section. This means there's no messing around with changing any settings
before an upload.

To use this method, you will need to have the program KILL.EXE which comes
with the Windows Resource Kit, or PSKILL.EXE from sysinternals.com. This
is a useful command-line utility for killing processes.

You will also need a program such as WAIT.EXE which is capable of providing
a pause. A 10 second pause is used to ensure PLINK has started in a second
window, and a 1 second pause is inserted between calls to RSYNC to help it
run smoothly.

Note that contrary to the examples above, I actually include the trailing
slash for folder names in the RSYNC commands instead of in environment
variables. This is just a personal preference.
(I prefer %VAR%/path instead of %VAR%path in the commands for readability).

Files
----------

The batch files are as follows:

EXCLUDE.LST
-----------

*.bak
*.BAK
*.java

CONNECT.BAT
-----------

c:\installers\putty\plink -C -v -l ssh_MyUserName -pw MyPassword -L 873:localhost:873 metawerx.net
rem -- uses compression (-C)
rem -- use verbose output (-v)
rem -- create a tunnel for port 873 only (RSync)

rem -- for a general purpose connection which allows RSync, SQL/Server, MySQL and FireBird, use
rem -- the command below instead
rem -- this will map the following:
rem -- Local 1434 -> SSH -> Metawerx 1433
rem -- Local 873 -> SSH -> Metawerx 873
rem -- Local 3307 -> SSH -> Metawerx 3306
rem -- Local 3051 -> SSH -> Metawerx 3050
rem c:\installers\putty\plink -C -v -l ssh_MyUserName -pw MyPassword -L 1434:localhost:1433 -L 873:localhost:873 -L 3307:localhost:3306 -L 3051:localhost:3050 metawerx.net

SYNC2.BAT
-----------

@echo off
@REM -- This file starts PLINK, performs a single RSYNC and performs error handling
@REM -- If any errors occur, PLINK is killed and then started again
@REM -- The first time through the batch file, PLINK is not running, but is
@REM -- assumed to be running. This causes the first RSYNC to crash with an error
@REM -- code, allowing the batch file to start PLINK, and continue on.
@REM -- PLINK is left running, so the next call to this file doesn't have to start
@REM -- it again.
@REM -- A 10 second pause is used after starting PLINK to ensure it has time to
@REM -- connect and get ready for connections. If this value is too low, this batch
@REM -- file will try to run RSYNC, crash, and stop and start PLINK over and over
@REM -- again. If it is too high, it introduces more wait-time to finish a SYNC.
@REM -- Fine tune this value for your own system, and add a few seconds to avoid
@REM -- problems.
@REM -- ARGS 1-6 are args for PLINK

SET plink_started=1

:start
if "%plink_started%"=="1" goto transfer

:start_plink
%STARTPLINK%
SET plink_started=1
wait 10

:transfer
echo %2 %3 %4 %5 %6
%RSYNC_EXE% %1 %2 %3 %4 %5 %6
if "%ERRORLEVEL%"=="0" goto end
echo ERRORLEVEL WAS: %ERRORLEVEL%, RETRYING...
SET plink_started=0
kill plink.exe
goto start

:end
wait 1

SYNC.BAT
----------

set RSYNC_PASSWORD=MyPassword
set RSYNC_EXE=c:\progra~1\cwrsync\rsync

set SERVERROOT=rsync://MyUserName@127.0.0.1
set LOCALSYNC=/cygdrive/d/sync
set LOCALECOMAPP=/cygdrive/d/mwtomcat_vm5/jakarta/webapps/ROOT

set EXCLUDE=/cygdrive/d/sync/exclude.lst

set SYNC2=D:\SYNC\SYNC2.BAT
set PLINK=D:\SYNC\CONNECT.BAT
set STARTPLINK=START /MIN "PLINK" %PLINK%

@echo.
@echo ***************************************************************
@echo ECOMAPP APP
@echo ***************************************************************

rem -- Note that "ecomapp" is the username on the server in this case.
rem -- This name has been modified for the purpose of this example, but
rem -- demonstrates uploading different apps to different sites
rem -- In the case that you only have one site, this will be your username
rem -- for that site, or might be placed in the SERVERROOT environment variable
rem -- as in SERVERROOT=rsync://MyUserName@127.0.0.1/MyUserName

CALL %SYNC2% -vvrzut --delete --exclude-from=%EXCLUDE% %LOCALECOMAPP%/ %SERVERROOT%/ecomapp/private-cgi-bin/tomcat/ROOT/
CALL %SYNC2% -vvzut --delete %LOCALECOMAPP%/WEB-INF/web.xml.live %SERVERROOT%/ecomapp/private-cgi-bin/tomcat/ROOT/WEB-INF/web.xml

@echo.
@echo ***************************************************************
@echo SYSLOGGER SETTINGS
@echo ***************************************************************

rem -- Copy setting files into d:\sync\mw_data and sync the d:\sync\mw_data folder up
rem -- this is easier than excluding all other possible files and uploading from
rem -- d:\syslog\vm in this case (I may have a lot of other files in there).

copy d:\syslog\vm\systemlog.ini d:\sync\mw_syslog
copy d:\syslog\vm\systemlog.lst d:\sync\mw_syslog
copy d:\syslog\vm\systemlog.dat d:\sync\mw_syslog
CALL %SYNC2% -rzut %LOCALSYNC%/syslog/ %SERVERROOT%/syslog/

@echo.
@echo ***************************************************************
@echo BI-DIRECTIONAL SYNC OF METAWERX SERVLETS (SITEWINDER, SQL ADMIN, FORMHANDLER)
@echo ***************************************************************

rem -- These are servlets for ServletExec, so the development team
rem -- copy these to the d:\sync\mw_servlets folder using a batch file
rem -- command from their text editors (UltraEdit - great editor) after
rem -- testing.
rem -- Therefore, the latest tested versions always exist in %LOCALSYNC%/mw_servlets
rem -- ready to be deployed.

CALL %SYNC2% -vrzut %LOCALSYNC%/mw_servlets/ %SERVERROOT%/metawerx/private-cgi-bin/servlet/

rem -- other deployment sections removed for example

@echo.
@echo ***************************************************************
@echo FINISHED SYNC
@echo ***************************************************************

rem -- kill the plink window, and pause to show everything went OK

kill plink.exe
pause
:end

Conclusion
----------

The above batch file is run whenever we need to upload changes to the metawerx
servlets, metawerx website, the fictional "ecomapp" or the "syslog" applications.
We start it off and a minute or so later, all the changes have been uploaded
succesfully and securely (just enough time to make a coffee).

If you have any questions on the batch files above, find an error in this file,
or are trying to implement a similar method for use in your own office, please
don't hesitate to contact us with your questions or comments.

** This file was deployed using the batch files above.

If you found this information helpful, additional guides can be found at
http://www.metawerx.net/admin under the 'Resources' link.

END OF DOCUMENT