Wigwam

In a traditional software project, each person working on the project gets their own copy of all the source code (a playpen) so they can change and test all components of the software without interference from anyone else. The changes they make and the changes from others are merged and tracked using a revision control system.

For some packages, such as a GUI utility run by a user or a library, this is the only type of change management necessary.

But with a server application such as a website, you also need to manage the operation of the software. This involves more than just a few software libraries linked into one executable. It means integrating and configuring external programs as well as the source code for the project itself.

The configuration files and other software integration data, such as the details of starting and stopping of services, need to be managed by revision control along with the source code. For example, in projects that maintain a webserver, the source-code repository for the project needs to store all information required to run the site at a given moment. For example, it will probably list the webserver's hostname and which port the services are configured to bind to. Likewise revision control retains all the configuration files, for example, they surely contain the location of the database server at a given time.

Ideally, the revision control system should only contain source files: it should contain no generated files. However, that is difficult to attain, given that most websites depend on many libraries, and furthermore they probably depend on configuration files.

Here is a concrete example of the problem. Suppose someone committed a configuration file to the revision control system for my application server that included a connection to some.machine.com:3301 for some database. Unfortunately, some other developer might be testing against a local server, say theirmachine.machine.com:3394. Worse yet, accidentally committing such a change could break the live running server application.

Perhaps you're now thinking that configuration should be left out of revision control. Actually, that can cause similar problems. Introducing new software and new configuration for that software into a project that does not get propagated to other developers or the live application server, can cause those other developers or the live application server to break as a result of running new code that depends on the new services being configured properly.

For example, developer A sets up a new database service as part of his website. He then configures (or hardcodes) the application logic to connect to that database. Then he checks into revision control all of his changes. Developer B or even the live website then gets a copy of the changes. There is no way for developer B or the live site to know that the site now depends on the new database and their versions of the site will break.

A hybrid approach is needed where a the small amount of information that must vary between playpens (such as user IDs, network ports, and network names) is allowed outside of revision control.

Wigwam accomplishes this by giving users standard places to store this type of information and standard methods for getting at it and using it for configuration. More specifically, Wigwam loads a bunch of environment variables from per-role and per-cluster configuration files Startup scripts for services can use those variables to generate runtime config files or scripts from templates that are checked into the project or provided by a package. (See the Section called Reusing the Effort of Others for an introduction to the Wigwam package system.) This way, almost everything is tracked by the revision control system, and the difference between development, testing, and live installations of the project is minimized.

How Wigwam Is Organized

The files in Wigwam are laid out to mimic the standard Unix/GNU filesystem hierarchy, with one major change. In Unix, /usr is used to represent "files only needed once the system is booted." We don't care about that, hence we don't have /usr. However, files which are installed as Wigwam packages should place their files in a similar directory, $PLAYPEN_ROOT/ext. You should never hand-edit files in $PLAYPEN_ROOT/ext, nor should ext be checked into your CVS repository.

A project using Wigwam contains a few major directories. They are generally checked into CVS, except that all the files installed by packages are in ext/, so those should not be cvs commit'd.

Usually the other directories are similar to the normal Unix hierarchy:

lib: Shared libraries and other language libraries: they contain helpful functions used by programs.
bin: Programs run by a users.
etc: Project configuration files.
www: Web documents: html and template files.
src: Source code that must be compiled (such as C or Java).
ext: External packages: programs, libraries and data provided by code outside your project. The directory is not checked into CVS.

In order to configure services, developers must put configuration information into the etc/roles/ROLE/config and etc/clusters/CLUSTER/config files. These environment variables may be interpreted in any way the services please, but often implementors use the sub-conf script to substitute patterns of the form __ENVAR__ with the value of the environment variable ENVAR. See the Section called Configuring Wigwam Services in Chapter 3 for more information.

Frequently Used Commands in Wigwam

Here are the three most commonly used commands provided by Wigwam:

packagectl: The package management program. See the Section called Using packagectl in Chapter 3.
servicectl: The service management program. See the Section called Using servicectl in Chapter 3.
pubtool: The publication program. See the Section called Using pubtool in Chapter 4.

Terminology

Here are some terms used throughout the documentation. They are perhaps more precisely defined than usual:

package

A collection of source code and auxiliary files that compiles or otherwise installs programs, libraries and/or data files.

There are many examples: apache, mysql, iota, etc.

depend

As with other common packaging systems (Debian, RPM), a package X can depend on another package Y, meaning that X cannot be installed until Y has been installed and usually Y will be installed automatically if X is requested.

playpen

A complete mirror of the site, used for development and testing. Each developer typically has a playpen on his or her box that they use for developing/testing new features, or debugging problems.

Sometimes called a sandbox.

cluster

A group of computers which act together to provide something to a user, like a website.

Package Archive

A place where all the packages are held. It contains all the files written by the packaging authors ( often the same as the project authors ) plus the raw source code from the web, and a package-list so you can easily find the latest version of a given package.

A package archive is just another wigwam project. As such the files found there are checked into CVS. For examle the public wigwam package archive is checked in as the module wigwam-package-archive.

There are files ending with .urls which contain web addresses for the official download sites for the remaining files. However, Because all the files are downloaded to the archive, and copies stored there, playpens do not depend on the continued availability of the packages on the Internet, only the availability of the archive. ( so if you back yours up, you can always rebuild any version of your project. )

Playpens may use more than one package archive simultaneously; the first package archive listed takes precendence if two identically numbered packages are available. Also if the package archive listed is the word 'LOCAL' it will be interpreted to be an in project package archive located in $PLAYPEN_ROOT/package-archive ( as of wigwam v3.0.48 )

If you wish to use a package archive that is protected with HTTP Basic Authentication, do one of the following:

Use the standard format for embedding usernames and passwords in a URL:
http://USERNAME:PASSWORD@HOST/PATH/...
Use auth-disk-http: instead of http: and auth-disk-https: instead of https:. The first time you use the playpen you will be prompted for user-names and passwords. After that the username and password will be stored on your local disk.
Use auth-env-http: instead of http: and auth-env-https: instead of https:. When you setup-env will be prompted for user-names and passwords, then the data will be stored in your environment.

building

The process of compiling the package. Generally that means taking the original authors' source code, applying any patches we might have, and then converting that to machine readable format. This includes a "source code configuration phase" to determine what flags are needed during actual compilation.

Building doesn't really have any direct effect on the project, but before you install a package, it must be built, and building takes a lot longer than installing. To save time, two conflicting packages can be built, and either one can be installed in short notice. This is often used to switch back and forth quickly between two versions of a piece of software.

For most packages, building consists of running these commands:
./configure make
but with several options, and some packages (e.g., "raw" and "service" type packages) do not even conform to this.

TODO: examples would be helpful

This process downloads the unbuilt package sources from the project's archive to ext/packages/, then it unpacks and compiles under ext/build/$package-$version. It should not modify any file outside of these two directories.

installing

The process of copying the public binaries, libraries and data files into publicly accessible areas. Building should only affect files under ext/build. Installed packages must not use ext/packages or ext/build.

For most packages, installing consists of running the command:
make install
but with several options.

TODO: examples would be helpful

Packages should never install anything outside of ext/.

publishing

The process of

updating packages and source code,
rebuilding and reinstalling the upgraded packages,
restarting services on the actual live servers.

service

A package or part of a project which has some functionality that can be stopped, started or restarted. Services are controlled using servicectl.

Note that services are not the same as server programs. For example, a commonly used service is static_apache, which runs the Apache web server in a configuration suitable for serving HTML, images, and other static content. This service is provided by the package service_static_apache. The actual apache program is provided by the apache package, which is different from the service_static_apache package. This is because there are other services (e.g., service_mod_perl_apache and service_jserv_apache) which run apache in different configurations for different purposes.

Most services are provided by packages whose names start with "service_". These packages will depend on the packages which provide the program they run. For example, the service_mysqld package depends on the mysqld package.

Services can also be provided by the project itself, so that custom project-specific programs can be controlled in the same way as the other services. When the service is provided by the project, the source code for should be in $PLAYPEN_ROOT/src, and the service configuration should be in $PLAYPEN_ROOT/services/$SERVICE_NAME.

md5sum

A checksum used in Wigwam to determine whether a file is correct. The program md5sum will tell you the md5 checksum (md5sum) of any stream of data or a file. It is a 32-digit hexidecimal number (with the a-f characters lowercase). If the md5sums of two files are identical, it is likely that the files are also identical. If the md5sum is different, the files are definitely different. This property is used to verify the integrity of downloaded packages.

Chapter 2. Starting a New Project with Wigwam

Starting a New Project

Starting a new project is now quite easy, since wigwam-bootstrap will know to download packages from wigwam's package archive.

Get the latest wigwam-bootstrap from the wigwam-base download area, and run it to make a skeletal Wigwam project as follows:
wigwam-bootstrap --project=project_name\ --package-archive=archive-URL
The wigwam-bootstrap program is in the CVS project wigwam-base, in the directory bootstrap/.
You can use the --initialize option to install packages you know in advance you are going to need.

Switch into the newly created project directory:
cd project_name

Use the cvs import command to check your newly created project into CVS.

wigwam-bootstrap produces all needed .cvsignore files, so that
cvs import project_name wigwam init
should work to correctly import the project.

Of course you'll have to have the CVS repository configured properly, either with CVSROOT pointed to it, or with -d. If you are connecting to a remote CVS server with ssh, make sure to set CVS_RSH to ssh.

Delete the version of the project which you just imported. It is useless, because it has no CVS control directories, so you cannot use cvs commit in it.

Checkout a fresh version of your project:
cvs checkout project_name

cd into the newly-created project directory and run ./autogen.sh in the checked-out copy of your project -- this will cause your ./ext directory to be rebuilt and the necessary packages to be downloaded and installed.
Next type
. ./setup-env
to set up your path.
Install any services that you'll want running. Use packagectl to install the relevant service. All the service packages begin with service_, for example, service_static_apache, service_mysqld, etc..

Checking out a New Project

If someone else has begun a project, and you want to check out a playpen version of it, all you should need to do is check the project out of the CVS repository and then run ./autogen.sh in the top directory of the project.

If you don't have access to the same package archive the original author was using, try making a etc/package-archives.local file that contains URLs for package archives you wish to use. You can list several URLs in a etc/package-archives; If identically-numbered versions of a package are are available at more than one listed package archive, the package from the first archive will be used.

Chapter 3. How to use wigwam-base

Every Wigwam installation has a version of wigwam-base. It contains scripts to install any other Wigwam package. This section describes how to use the programs in wigwam-base. Once you have sourced setup-env, all these programs should be on your path.

Using packagectl

Table of Contents
packagectl -- manage packages in your wigwam project

All Wigwam package management should be done through the packagectl interface; like the other *ctl programs it does one of several different things, depending on a command argument.

packagectl

Name

packagectl -- manage packages in your wigwam project

Synopsis

packagectl {[ install package [version] | update | build [--force] package [version] | update-packages | info package [version] ]}

packagectl install [--fix | --no-update | --update] {package} [version]

packagectl update

packagectl build [--force] {package} [version]

packagectl update-packages

packagectl update-local

packagectl upgrade [--force] [packages...]

packagectl project [build | install]

packagectl info {package} [version]

packagectl install

Add the appropriate version of the package into the project's package list and update your local packages.

You may install more than one package at once. If you want to install more than one package and want to use the latest version, you may use - instead of a version number.

If --fix is specified, all the dependencies will be added to etc/project-packages, otherwise we always get the latest version of all dependent packages. You must --fix the project file to store the versions of the dependencies. This is mostly important for rollback purposes, but it can be helpful to make sure two playpens are using the same versions even if they use different package-archives.

If --update is specified, only those packages which changed in the project-packages; will be installed; etc/project-packages will not modified.

If --no-update is specified, no packages will be built or installed at all; only etc/project-packages.

The default is --fix.

packagectl update-packages

Downloads the latest ext/etc/package-archives/* files. This updates the playpen so that you can install or upgrade to the latest versions available in the package-archive.

packagectl update

Download the lists of available packages from all the package archives.

packagectl update-local

Replace setup-env and bin/wigwam-bootstrap with the latest versions distributed with wigwam. (Normally these files are unaffected by upgrading your wigwam-base.)

This is separated from the upgrade of wigwam-base so that if you must make changes to wigwam-bootstrap or setup-env they will be preserved.

packagectl update-packages

Build and install any needed packages to make the installed packages match the etc/project-packages file.

packagectl upgrade

Upgrade packages to the latest versions, according to your local project_packages. This will install previously uninstalled packages too.

If no packages are specified, all packages in the project will be upgraded.

With --force, the package will be downloaded again from the package archive, rebuilt, and reinstalled, even if all these steps appear unnecessary.

packagectl project

Build or install the source code in src, in the standard way. See the Section called Building and Installing Binaries in a Project in Chapter 3. This can be convenient if you don't have a simple Makefile, but it's mostly for testing, since this is how the project's source code will be compiled on the live and staging servers.

packagectl info

Print information about the requested package. If the version is omitted, the installed version of the package is used, or if none is installed, the most recent available version.

Using System-Installed Versions of Software

Occasionally you wish to avoid the package manager altogether, and just inform Wigwam about a package's availability.

Wigwam has support for using versions of a package that are installed on the system, not in a playpen. There are several ways this can happen:

The package is autodetected by wigwam. Currently just Java 1.2 is autodetected.
You write a script in either
etc/system-packages-test
etc/roles/$PLAYPEN_ROLE/system-packages-test
etc/clusters/$PLAYPEN_CLUSTER/system-packages-test
which will be sourced by the packaging system with the variable $package and $version set to the package we wish to look for on the system. If your script detects or assumes the presence of that package, then exit 0 from that script.
For example, if you know libnutz is installed on the foodtypes cluster, you should make a etc/clusters/foodtypes/system-packages script with code like:
# You may also wish to check $version, # or test the system's version's availability. if test "$package" = libnutz ; then exit 0 ; fi
This should inhibit the installation of libnutz on foodtypes.
You make simple lists of packages in similar locations:
etc/system-packages
etc/roles/$PLAYPEN_ROLE/system-packages
etc/clusters/$PLAYPEN_CLUSTER/system-packages
which must contain one package per line, if a version number is omitted then the package satisfies any dependency on the package.

Using servicectl

Table of Contents
servicectl -- manage and query wigwam services

servicectl

Name

servicectl -- manage and query wigwam services

Synopsis

servicectl [start | stop | status | test | restart] [service...]

servicectl start [service...]

servicectl stop [service...]

servicectl restart [service...]

servicectl status [service...]

servicectl test [service...]

servicectl new-service [service]

servicectl check-config [role | cluster] [--verbose | --script] [cluster_or_role_name]

Description

Like cvs, servicectl really encapsulates several different functions. If a list of services is specified, only those services are affected; otherwise the command operates on all the services defined for your current role, which is specified in $PLAYPEN_ROOT/etc/role. This may be overridden by PLAYPEN_ROLE in the environment.

The role's services are specified as a shell variable, playpen_services, in $PLAYPEN_ROOT/etc/roles/$ROLE/config.

servicectl start

Start the services.

servicectl stop

Stop the services.

servicectl status

Check the status of the services according to their pid files. Normally these are in LOCAL_RUNVAR, which is usually LOCAL_VAR/run named SERVICE.pid.

servicectl test

Check the status of the services, using their provided self-test scripts (or you can place such a script in your project, in the services/$SERVICE_NAME directory).

servicectl restart

Restart the specified services, using SIGHUP if they take it, or their own restart script, or by stopping and starting the service.

After restarting:

the newest binaries will be in use,
the log files will be reopened,
the configuration files will be reread and the service's caches will be flushed.

servicectl check-config

Find out what environment variables are still left to be configured in the project for each service.

You can use the --verbose flag to print out descriptions of missing environment variables.

You can use the --script flag to make a script that will set the missing environments to their default values.

servicectl new-service

Mark a service as available in the project. Used internally by the packaging system.

Configuring Wigwam Services

Configuring the services comes in several parts:

Determine the Role for each Playpen.

Each playpen you run Wigwam on has a role. The role of a playpen is found in a file etc/role. This file should not be checked into CVS.

Role files on clusters are made by pubtool.

Configuring the Roles

The roles are arranged in directories, etc/roles/$role_name/. Currently there is only one file: a sh-script named config that is sourced before starting or stopping the services.

From a sh-script you may load these environment variables by sourcing the ext/bin/load-config script.

Service-Specific Configuration

[TODO: re write some of this info and put it in the how wigwam is organized section]

The configuration files used by the service are generally substituted using the sub-conf program. These means that they can be configured by environment variables, and it means that the package provides a configuration file in ext/etc which can be overridden a copy in etc. This allows you to make a project-specific version of that file.

If the environment variable applies across an entire cluster, you should put that variable in etc/clusters/$CLUSTER/config. If it applies to a role generally, you should put that variable in in etc/roles/$ROLE/config.

All such services' actual configuration files should put in ext. There are a few paths which are in standard locations relative to the installed config file. We will give examples, assuming the config file is in ext/foo/dummy.

Sometimes, even with substitution, the distributed configuration file is not flexible enough. In that case you may provide an equivalent configuration file as just foo/dummy (that is, the same path, but with ext cut off the front). Such a modified file should be checked into CVS and is global to the project.

Sometimes, for testing, you may wish to change the conf file without affecting the version in the CVS repository. For this purpose, if foo/dummy.local is available, it will be substituted instead.

The Playpen `config` File

Each playpen has a role specified by etc/role. Each role has a config file in etc/roles/$PLAYPEN_ROLE, which may set certain shell variables:

playpen_services: List of services which are run in this role by default.

Wigwam Cluster Configuration

A cluster is a group of machines that you can publish to. They have identical content, but they often run different services. We classify the machines into roles, which are nothing more than configuration variables and a list of services to run.

Clusters are configured in directories etc/clusters/$CLUSTER. Each directory contains a file hosts. Each line in that file starts with a comma-separated list of hosts, then a few key-value pairs. For example:
host1,host2,host3 role=webserver playpen_root=/tmp username=daveb
Here are what the keys mean:

role

The role the machine is supposed to play. There should be a directory checked into CVS, etc/roles/$PLAYPEN_ROLE/, that contains configuration for the role, including what services should be run.

playpen_root

The PLAYPEN_ROOT of this project on the host. Please note that it is extremely important that $PLAYPEN_ROOT point to different paths on different hosts. This won't work if playpen_root points to a shared drive.

username

The username to use when connecting to this host; if not specified, no username will be given to ssh.

ssh_identity

ssh identity file to use when performing commands in parallel on the cluster. If not specified, no identity file will be used.

ssh_flags

Extra options for ssh when performing commands in parallel on the cluster.

rsync_flags

Extra options for rsync when copying the project to the cluster.

A common option is -t, which will cause rsync to preserve the timestamps on the files it transfers. This can speed up publication significantly, especially if your project has a lengthy compilation phase. However, you must make sure the clocks on all the machines in your cluster are synchronized (e.g., using ntpd) for it to be safe. If you specify rsync_flags=-t and the machines' clocks are not synchronized, make and its ilk are likely to get confused and build the project incorrectly.

disconnected

This host or hosts is not attached to the same network as one of the package archives.

Wigwam will automatically run an http-proxy, tinyproxy, and use ssh-port-forwarding (the -R option) to download all the packages, so that only the machine from which you are publishing needs to be attached to the net.

You can specify default values for any of the keys by including a line with the special hostname _default. The default values only apply to hosts listed after the _default line. For example:
_default playpen_root=/web/myproject
Any hosts below this line will have the playpen root /web/myproject.

Clusters, like roles, may also have a config file which will be sourced by ext/bin/load-config. It should set variables for the cluster that override variables in project.conf.

There can also be a file default-tag-prefix in etc/clusters/$CLUSTER to control the default prefix to use when making CVS tags for publishing to the cluster. If this file is not present, the default tag prefix will be live.

Finally, on some clusters, services which are normally in Wigwam are available outside of Wigwam. For example, on some clusters, all projects may use a common database. To indicate that, make a file, etc/clusters/$CLUSTER/foreign-services which contains a list of services which are available globally on this cluster.

Using clusterctl

Table of Contents
clusterctl -- run something on all the hosts in a cluster

clusterctl

Name

clusterctl -- run something on all the hosts in a cluster

Synopsis

clusterctl [CLUSTER] [COMMAND] [COMMAND-ARGUMENTS]

clusterctl list-hosts [CLUSTER]

clusterctl exec {COMMAND} [ARGUMENTS...]

Description

Often you want to do the same thing to every machine on the cluster. The program which does this type of management is clusterctl. Of course, publishing affects all the machines on a cluster, but use pubtool for publishing (TODO: explain briefly why). clusterctl is for commands that are not part of publication.

The most common need is to start, stop and restart all the services on a cluster -- this is done very often during development. Most servicectl commands are recognized exactly as servicectl recognizes them, in particular:

start	test	new-service
stop	restart	list-services
status	query-restart	check-setup

The list-hosts command prints a sequence of USER@HOST pairs of the hosts in the cluster.

The exec command will run an arbitrary command on all the hosts.

Environment Variables

These environment variables are set by sourcing setup-env.

LD_LIBRARY_PATH

This is where the system's shared-library loader looks for shared libraries before looking in the system's default shared-library directories (usually /lib, /usr/lib, /usr/local/lib, and /usr/X11R6/lib).

It should contain the two Wigwam library directories, and they should be in this order:
LD_LIBRARY_PATH=$PLAYPEN_ROOT/lib:$PLAYPEN_ROOT/ext/lib
This allows projects to override the shared libraries installed by packages, since packages will install their libraries in $PLAYPEN_ROOT/ext/lib, and project developers will work in $PLAYPEN_ROOT/lib.)

PATH

This is the list of directories where the Unix operating system searches for executables if they don't have a "/" in the name.

It should always start with:
PATH=$PLAYPEN_ROOT/bin:$PLAYPEN_ROOT/ext/bin:...
This allows projects to override binaries installed by packages.

In general, you should not need to set "global" environment variables, but if you need them for development, you may add them to anywhere that load-config will source. See the Section called Scripts to Load Certain Configurations into sh-variables in Appendix B for the details.

You must also set interactive_variables to the names of the variables you want to export in to the user's shell. [1]

Generally, you should put environment variables in one of the following places instead. These variables are only set by sourcing the load-config shell script, but that is done automatically by Wigwam for running, building and installing. If you need to run a single command with all the cluster and role environment variables set, you may use wigdo.

etc/roles/ROLE/config: If the variable only applies to a particular role.
etc/roles/CLUSTER/config: If the variable only applies to a particular cluster.
etc/project.conf: If the variable applies cluster-wide.

Packages, however, may not directly set environment variables, yet; instead use service_required_envars and service_optional_envars from the service startup scripts and later the configuration tool will use the descriptions to make an interface to set these.

Note that in Wigwam, we recommend using all-capital names for environment variables and all-lowercase names for sh-variables which are not exported. Not exporting lowercase environment variables makes it easier for scripts to work recursively, and generally without side-effects. Be very clear in your documentation about variables you require to be exported.

Manage Project Source Code

Wigwam can help manage the compile and install phases for C or java code within your project as well.

You may use packagectl project as a publishing interface to building or installing the project's code; and projectctl is a slightly shorter interface that doesn't redirect the output, which will make debugging easier.

Where the Package Data is Stored

While technically there's no reason to know this, it is a little helpful to see a bit of the implementation. So here is how the information about your playpen and project packages are stored. For a lot more information, see Chapter 6.

There are two lists of packages maintained in the Wigwam tree.

ext/etc/installed-packages: the packages (and their versions) that are installed in this Wigwam playpen.
etc/project-packages: the packages (and their versions) that are supposed to be installed to use this project.

For each package archive, there is a list of packages available from that archive and information about the layout of the archive. ARCHIVE-NAME is the url with / converted to the underscore (_) character.

ext/package-archives/ARCHIVE-NAME/package_list: A list of all packages and their version numbers from the package archive.
ext/package-archives/ARCHIVE-NAME/style: The arrangement of the files in the package archive. This is mostly for compatibility; usually it is just set to deep.

Packages and their associated configuration and build files will be downloaded to ext/packages. Therefore, you should use packages to contain local overrides for the configuration.

You can use the update-packages command to synchronize the playpen's packages with the project's package list. It will only recompile packages that have changed.

Writing Project Services

Project services are services that are checked into the CVS module for the project, rather than being maintained as Wigwam packages.

They don't have explicit version numbers, because they are in the project's CVS module. Hence tagging the project uniquely identifies a "version" of the service.

They are useful for providing services that aren't needed by other projects, so should not be in the package archive.

Project services are just like package services, except:

They are not in ext/, they are simply in $PLAYPEN_ROOT/services/SERVICE_NAME.
They never have a version number, and wherever $service_name-$service_version is used, it is replaced with just $service_name.
They are managed by the project's CVS module -- not the packaging system.

Here is what you must do to define a new project service:

Edit or create etc/project-services. It must contain a line which is just the name of the service.
Make sure binaries exist for the service. If you are going to use packaged binaries, just packagectl install them. If you are going to implement your own daemons, the code should be in src; it should be possible to build and install that code as described in the Section called Building and Installing Binaries in a Project.
Create a directory services/$service_name. Make a config there that describes how to run the service; the variables are exactly the same as for packaged services.
At this point, you should be able to use servicectl to start and stop the new project service.

Building and Installing Binaries in a Project

This section applies to writing code in a project in order to run a project service.

This implies that you are writing code that you don't intend to reuse in another project. Please consider making your code general enough that it can be placed into a package to be used for many other projects.

Like packages, the project source code is installed in two phases.

If you use automake, all the building and installing is completely standardized, and Wigwam will handle it without the project-build and project-install scripts.

If you use automake and autoconf, be sure that your configure script, Makefile.ins, and Makefiles will be regenerated automatically when their corresponding input files (configure.in, Makefile.am, Makefile.in) change. Specifically, you probably do not want to use the AM_MAINTAINER_MODE macro, because it disables those rules by default.

Alternately, you may write a simple Makefile where
make PREFIX=$prefix
builds the project, and
make PREFIX=$prefix install
installs the project.

If you do not want to use automake, or if your project is very unusual, you may prefer to write custom build and install scripts.

build: If a file src/project-build exists, then that will be run to build the project. This process should check availability of the libraries and check that the code compiles, but should not touch any files outside of src.
install: If a file src/project-install exists, then that will be run to install the project. That means copying its libraries into lib, its executables into bin, and so on.
configure: Automake and other source code configuration tools have a separate phase for finding out what features are available on the local platform. If you are not using automake, a src/project-configure script will be tried. If that does not exist, we assume that this project has no source code configuration phase.
clean: Clean up compiled object files, libraries and executables. If you are not using automake or make, a src/project-clean script will be tried. If that does not exist, you cannot clean this project---you should fix that.

Overriding Packages

package overrides are can be used to override control files in a wigwam package. This is most usefull for overriding the .options file, to build with different options than the original package, without having to make a whole new package.

Simply create the directory $PLAYPEN_ROOT/package-overrides and drop replacement files in there. Note, this currently only works for files of the form PACKAGE_NAME-PACKAGE_VERSION.EXT where EXT is the common package extensions.

[TODO: this needs a better explaination]

Internals of Wigwam Services

Wigwam provides a way of packaging services, which are daemons that must be started, stopped, and restarted.

From the user's perspective we install services like normal packages and use the servicectl script to affect and query the services.

From there, servicectl uses configuration files installed by the service-packages in ext/services/$SERVICE_NAME-$VERSION. It uses per-service configuration files; first trying to source one of the following scripts:

services/$SERVICE_NAME-$VERSION/config.pre

services/$SERVICE_NAME/config.pre

ext/services/$SERVICE_NAME-$VERSION/config.pre

Then it tries to source one of the following (the same filename without .pre):

services/$SERVICE_NAME-$VERSION/config

services/$SERVICE_NAME/config

ext/services/$SERVICE_NAME-$VERSION/config

Then, it tries the same with .post appended to each:

services/$SERVICE_NAME-$VERSION/config.post

services/$SERVICE_NAME/config.post

ext/services/$SERVICE_NAME-$VERSION/config.post

Only the files in ext are provided by the package; the rest are for customizing the service in the project.

Please see the Section called Making Packages that Provide Services in Chapter 6 for more information about what variables may be set and how they affect servicectl's behavior.

Rules Governing the Packages' Behavior

This section describes how a properly-constructed package must behave in the Wigwam environment.

when servicectl start is done, the service should have bound to the listening port.

Chapter 4. Publication

With Wigwam, publication should not be scary. You should have been able to QA your staging server to the point were you are reasonably sure there are no bugs.

If there are problems once the package has been pushed out to the live server, it should be easy to revert back to the old version.

We recommend setting up staging clusters for debugging publication.

[TODO: explain how publishing works. we do the rsyncing because: we experienced too many hangs and CVS backoffs while each cluster machine waited on the others' locks, thus making publishing take much longer.]

Using pubtool

Table of Contents
pubtool -- publish content to clusters and tag versions of the project.

pubtool

Name

pubtool -- publish content to clusters and tag versions of the project.

Synopsis

pubtool tag [--cluster=CLUSTER | --tag-prefix=TAG-PREFIX | --tag=TAG]

pubtool changes [--cluster=CLUSTER | --tag-prefix=TAG-PREFIX | --tag=TAG]

pubtool diff [--cluster=CLUSTER | --tag-prefix=TAG-PREFIX | --tag=TAG] [--diff-args=DIFFARGS]

pubtool publish [CLUSTER] [--cluster=CLUSTER | --tag-prefix=TAG-PREFIX | --tag=TAG] [--master-dir=MASTER-DIR] [--force-checkout] [--cvs-only] [--output-style=STYLE]

pubtool tag

Make a sequenced tag that you can publish later.

Do this when you have tested the state on your developer or staging box and want to publish out to other machines.

By default, a tag is generated with the tag-prefix live. The tag is the tag-prefix concatenated with the date, formatted as YYYYMMDDHHMM.

You may specify a tag-prefix explicitly, using --tag-prefix, or you may use the tag-prefix for a given cluster, using --cluster.

If you don't want the date appended, you can use --tag to specify the exact tag name.

pubtool changes

Get a list of files that have been changed, added, or removed since the specified tag. Each file that differs from the tagged version is shown with an explanation of the difference:

added: The file was added to the project since TAG.
changed: The file has been changed since TAG.
removed: The file has been removed from the project since TAG.
UNKNOWN: The file is not in the project; it has never been checked into CVS.
NOT COMMITTED: The file has been added or changed locally, but the changes have not been committed to CVS.

UNKNOWN and NOT COMMITTED are in all capitals to warn you that if you tag and publish the project without checking in your changes to those files, the files will be different or missing when they are published. Before you publish from your staging server, it is a good idea to run pubtool changes to make sure that the site you are testing on the staging server is the same as the one you are about to publish.

By default, the latest tag with the tag-prefix for your playpen's cluster is used. You may specify an alternate tag-prefix explicitly using --tag-prefix, or you may use the latest tag with the tag-prefix for a given cluster using --cluster.

If you don't want the latest tag, you can use --tag to specify an exact tag.

pubtool diff

Get the differences since the specified tag using cvs diff.

If you don't want the latest tag, You can use --tag to specify an exact tag.

You can change the arguments passed to cvs diff with the --diff-args option. Otherwise --diff-args='-ub' will be used.

pubtool publish

Publish the site in a tagged state to the cluster CLUSTER. You usually do this after making a tag with pubtool tag --cluster CLUSTER.

Normally, the latest tag with the tag-prefix for CLUSTER is used. You may specify an alternate tag-prefix explicitly using --tag-prefix, or you may use the tag-prefix for another cluster, using --cluster.

If you don't want the latest tag, you can use --tag to specify an exact tag.

In order to inspect the tagged-site without affecting an existing copy of the project, a reference copy of the site in the tagged state will be checked out in $LOCAL_VAR/PROJECT_master, where PROJECT is the name of your project. You may override this location with the --master-dir=MASTER-DIR option.

By default, pubtool will build the new remote playpen on top of a copy of the existing playpen, if there is one, in order to save the time required to rebuild packages and recompile the project's sources. If you wish to force pubtool to rebuild the project and all its packages from the contents of cvs only, specify the --cvs-only option.

By default, if there is already a checkout of the project in the master directory, pubtool will use cvs update to update it to the tagged state. If you wish to force pubtool to remove the master directory and check out a fresh copy, specify the --force-checkout option.

pubtool publish uses a program called wparallelizer to perform commands in parallel on all the hosts in a cluster. You can choose the style in which wparallelizer will format the output of the remote commands with the --output-style option. To see the complete list of output styles, try running wparallelizer --help. The most commonly used styles are:

tiled: A full-screen text interface with each host's output displayed in its own window. This is the default.
line: Each line of output is printed with the name of the host before it.

If all goes well during publication, then the tag is added to a list of successfully published tags maintained in $PLAYPEN_ROOT/pub-record/published-tags.

Environment Variables

SSH_OPTIONS

Extra options to pass to ssh when running rsync and executing other commands on the cluster.

A useful value of SSH_OPTIONS is -o StrictHostKeyChecking=no, which makes ssh automatically add any host whose key it hasn't seen before to the user's .ssh/known_hosts file. Without this option, the ssh program will hang waiting someone to verify that you wish to add the new key to .ssh/known_hosts. Instead of using this slightly less secure flag, the release engineer may choose to ssh to each host in the cluster manually and confirm the new keys before publishing the first time.

How Publication Works

Publication does a few things:

makes the right packages available,
updates all the data,
builds any changed code in the project,
starts, stops, and restarts services as needed.

This is managed in phases:

prepare a staging area to rsync to each host on the cluster, using cvs update). We will also produce lists of files and packages that changed. This is in turn done in three parts: we copy some data specific to publication parallel to $PLAYPEN_ROOT on all the hosts (we append .pubdata to that path). Then we optionally copy the old contents of the non-ext portion of the playpen to minimize build times. Finally, we rsync the remaining changes across.
on each host, copy the old playpen to the new playpen to speed up the rsyncing.
rsync the files to each host.
on each host, build a new playpen root, without interrupting the existing playpen. This is done by publish/prepare-new-playpen.
on each host, stop the old services as needed, swap the playpens, adjust non-relocatable packages, and finally restart/start the needed services. This is done by publish/use-new-playpen.

These scripts communicate lists of files and services with each other by making a directory ext/wigwam-publish-data that will be rsyncd, then deleted.

All scripts run on the remote hosts during publication will have the environment variable WIGWAM_IS_PUBLISHING set to 1.

Chapter 5. Wigwam Walkthroughs

It is often easiest to learn by example, so in this chapter we invent several fictitious Wigwam users to show how we intend Wigwam to be used. These walkthroughs may be read in any order.

Study A: Simple Database-Driven Website

Suppose we wish to set up a webserver that will use CGI scripts to make pages from a MySQL database.

Study A: Starting the Project

First we make a new Wigwam project and install the appropriate services:
wigwam-bootstrap --project=project-name cd project-name cvs import project-name start wigwam cd .. rm -rf project-name cvs co project-name cd project-name ./autogen.sh . ./setup-env packagectl install service_static_apache - service_mysqld
The - indicate that the latest version of service_static_apache should be installed.

Now we need to configure the services; this will consist of setting environment variables for the cluster, the role and the project. First we recommend flushing out the developer role (and sometimes a developer cluster is useful...). First, mark that you are in that role:
echo developer >! etc/role
Run
servicectl check-config role --verbose
and it will tell you the environment variables you must set from that script, along with their descriptions if the package has descriptions:
developer: static_apache: needed OUTSIDE_HOSTNAME developer: static_apache: needed OUTSIDE_PORT developer: static_apache: needed STATIC_HTTPD_PORT developer: static_apache: needed SERVER_ADMIN developer: static_apache: needed LOG_DIR developer: mysqld: needed MYSQLD_DATA_DIR developer: mysqld: needed MYSQLD_PORT

To make things work, you must set these up in a cluster, role or project file. There are many possible ways, but sometimes we do it this way. Each port is relative to a PORT_PREFIX. Usually PORT_PREFIX will be defined on a cluster-wide basis, but for the developer role, it may as well be in the role's early configuration file, etc/roles/ROLE/config.pre:

The etc/roles/ROLE/config is sourced after etc/project.conf, so it cannot be used here, since we want to put the specific services ports in etc/project.conf.

or in etc/project.conf.pre.local. So:

          echo 'PORT_PREFIX=80' >> etc/roles/developer/config.pre

And add to etc/project.conf:

if test "x$PORT_PREFIX" = "x" ; then
  echo "$0: You must define PORT_PREFIX must before you source project.conf" 1>&2
  exit 1
fi
STATIC_HTTPD_PORT=${PORT_PREFIX}80
MYSQLD_PORT=${PORT_PREFIX}81

Then there are just three more variables to deal with, all of which we'll add into the developer role, in etc/roles/developer/config:

OUTSIDE_HOSTNAME=`hostname -f`
OUTSIDE_PORT=${PORT_PREFIX}80
LOG_DIR=$LOCAL_VAR/logs
MYSQLD_DATA_DIR=$LOCAL_VAR/mysqld
SERVER_ADMIN="$USER@$OUTSIDE_HOSTNAME"

Now you can being adding web pages in www/htdocs/.

Study A: Developing

Now we'll add some content to the webserver, and then start it. Start with a static webpage. Create the file www/htdocs/index.html:
<head> <title>Hello Wigwam</title> </head> <body> <h1> Get a string from mysql table and place here </h1> </body>
Now that we have some content, let's fire up the services which allow us to view this content via a web browser.
servicectl start static_apache mysqld

Study A: Staging

So after the static website is properly written and checked into CVS, it should be tested by staging it.

There are two roles to assign: the webserver and the database. If you are staging to a single server, you can use the developer role, or a similar role so that log files and so on can be properly redirected.

It is better to stage just as the final site will be staged: so pick a staging cluster of two machines, if possible. Make two roles:

webserver: A webserver (no database). The file etc/roles/webserver/config should set the variable playpen_services.
database: A database. etc/roles/database/config should set the variable playpen_services to mysqld; also MYSQLD_DATA_DIR should be set (use servicectl check-config --role webserver to test this).

Then run the pubtool tag to make a tagged version of the site. You will use the tag to publish.

Study A: Publishing

Publishing mostly requires configuring a cluster.

Assume that we are publishing on the vegas cluster. We make a file etc/clusters/vegas/hosts containing the host data:
mandalay,flamingo role=webserver playpen_root=/web/PROJECT username=PROJECT luxor role=database playpen_root=/web/PROJECT username=PROJECT
And a file etc/clusters/vegas/default-tag-prefix containing the line:
vegas

Then, once you are confident you have tested the site adequately, run:
pubtool tag --cluster vegas
from the playpen on the staging server. This instructs pubtool to make a new CVS tag beginning with the vegas cluster's default tag prefix "vegas". The tag will have the format vegas_YYYYMMDDHHMM

Then publish the site by running:
pubtool publish vegas
pubtool will find the latest tag beginning with the vegas cluster's default tag prefix (i.e., the tag you just created), and publish the site in the tagged state to the cluster.

Study B: Simple Apache-JServ Website

First download the wigwam bootstrap script:
wget http://downloads.sourceforge.net/wigwam/wigwam-bootstrap-3.0.30 chmod +x wigwam-bootstrap-3.0.30
This script can create a new wigwam project; to learn about more specific options, run:
./wigwam-bootstrap-3.0.30 --help
To use that script run,
./wigwam-bootstrap-3.0.30 \ --project=jm3.0 \ ---initialize=ApacheJServ,service_jserv_apache
The --project option tells the name of the project and the directory to create. --initialize tells the bootstrap script to add certain packages to it immediately. construct the project in as well as the name to give the project

Later, when checking the project out from CVS, you'll need to run:
./autogen.sh

Finally you'll need to configure the services:
./ext/bin/servicectl check-config role developer
This will report the needed environment variables that must be set for this configuration to run under the role developer.

To start all the services:
./ext/bin/servicectl start
Now create a bit of content to test your configuration with:
cd www/htdocs cat > index.jsp <% out.println("TESTING 1,2,3......"); %> ^D cat > index.html Hello World ^D

Chapter 6. Making Wigwam Packages

Wigwam packages consist of a source tarball and a collection of specially formatted files.

Managing Version Numbers of Wigwam Packages

The full name of a wigwam package is divided in three parts: PACKAGE-UPSTREAM_VERSION-RELEASE. We usually call UPSTREAM_VERSION-RELEASE just VERSION.

The UPSTREAM_VERSION number should exactly match the versioning system used by the original authors of the package. The RELEASE number should start at 1 for any given upstream version, then each update to the package which is based on the same upstream code should increment the release number.

For example, the first version of apache that we packaged was from apache_1.3.12.tar.gz, so the upstream-version was 1.3.12. The first Wigwam release of any package should have a release-number of 1, so the first version of apache in Wigwam 3.0 is 1.3.12-1. To correct a small packaging bug, 1.3.12-2 was released based on the same upstream code. Later, when we begin using Apache 2.0, we will make another package with the release-number set to 1; perhaps apache-2.0.0-1.

Files in a Wigwam Package

Each of these files begins with $PACKAGE-$VERSION.; here we just list the extensions:

archs

List of GNU-style architecture and package-architecture pairs. This lets a package specify exactly which architectures are supported without the package having to know about every GNU type.

For now, you may set WIGWAM_ARCHS_SUPPORTED to a list of package-architectures allowed; this way you can just say WIGWAM_ARCHS_SUPPORTED="i386" and use the standard intel versions; for example:
i686-pc-gnu-linux i386 i586-pc-gnu-linux i386 i486-pc-gnu-linux i386 i386-pc-gnu-linux i386

archfiles.$ARCH

List of architecture-dependent files. The $ARCH must correspond to the the right-hand column of the archs file. This lets a package specify exactly which architectures are supported without the package having to know about every GNU type.

For now, you may set WIGWAM_ARCHS_SUPPORTED to a list of package-architectures allowed; this way you can just say WIGWAM_ARCHS_SUPPORTED="i386" and use the standard intel versions.

files

List of files for this package that must be retrieved from the package archive. It should list every file in the source package archive for this package including itself.

This file is mandatory.

type

What type of package this is. This may be make, service, or raw.

options

Options specific to this package's build process, see the Section called Configuring the Build and Install Process.

md5sums

List of md5sums for each file listed in the .files list. This is used to verify that the package is downloaded from the package archive correctly.

This file is mandatory.

prepatch, postpatch

This are small scripts that will be run before and after the patches to the package's source are applied. They are rarely used: generally they are used when a file's permission needs changing, or when a file created by a patch needs non-default permissions.

tar.gz

the source code from the upstream maintainer, or from make dist from your CVS repository.

Though .tar.gz is probably the most common archive format, many others are supported.

Note that there is no required naming convention for upstream sources: you should use the exact name of the file as it is downloaded, unless its filename conflicts with a different file from another package. By common convention, the upstream sources are usually named PACKAGE-UPSTREAM_VERSION.tar.gz.

patch

Patches that should be applied to the upstream source code.

Patch files contain changes generated with the diff -u program. The build system automatically tries to guess the flags needed to patch the source code.

Any files ending in .patch.bz2, .patch.gz, and .patch are applied.

The packaging system tries to guess the right patch flags, but if it gets it wrong, you may need to use patch_flags_PATCH_FILE_CANONICALIZED where PATCH_FILE_CANONICALIZED is the patch's base filename with . converted to _ and - converted to _

predep, dep

Wigwam packages this package depends upon at various points in its lifetime. See the Section called Dependencies.

servicedep, filedep, packagedep

Files or packages that a service provided by a package depends upon. See the Section called Dependencies.

conflict

This is a list of packages which may not be installed concurrently with this package. Two packages conflict if either lists the other in its conflict file; there is no difference between A conflicts with B and B conflicts with A.

This file may list specific version numbers, using a SPACE character to separate the package name from the version number, or it may omit the version number to conflict with any version of that package.

provides

This is a list of packages which this package clones. These should act as reasonable substitutes for the packages they provide.

Each line of this file must start with a package name. After a single space character, there may also be a version number. If no version number is specified, the package can act as any version of the package it provides.

dirs

list of directories that must be available for this package to be installed.

installed_files

List of files that this package affects. This file may be automatically computed from your pacakge with the make-installed-files-list script in any package archive.

Each line in this file consists of an action, a space, and a filename. The action may be any of the following characters and should describe what happened to the file.

>: The file was created by installing this package.
<: The file was removed by installing this package.
~: The file was changed by installing this package.

Dependencies

In general, packages need other packages to operate correctly. The most common type is install dependence, where one package demands that another is installed before it can be installed.

Dependencies can describe which versions of a package are required too. We support intervals. If you use brackets, for example, [0.21,0.95], then the version numbers at the ends of the range are allowed. If you use parentheses, then the ends of the range are not valid for satisfying the dependence. You may also specify an exact version number.

Finally you may have a sequence of allowed ranges separated by semicolon (;). If any range matches, then the version is allowed.

Here are some examples:

[A,B]: Any version between A and B, including A and B, will satify the dependency.
[A,B): Any version between A and B, including A but not B, will satify the dependency.
(A,B]: Any version between A and B, not including A not including B, will satify the dependency.
(A,B): Any version between A and B, not including A or B, will satify the dependency.
(A,): Any version strictly greater than A will satify the dependency.
[A,]: Any version greater or equal to A will satify the dependency.
(,B): Any version strictly less than B will satify the dependency.
[,B]: Any version lesser or equal to B will satify the dependency.
B: Version B will satify the dependency.
A;B: Version A or version B will satify the dependency.

Each dependency type has its own file in the package:

predep

Packages that must be installed before this package can be downloaded or unpacked. For example, if wigwam-base gets extended to support another compression type, then packages which ship using that compression will have to predepend on the appropriate version of wigwam-base.

This is implemented by downloading predep file first after the .files file. Then before proceeding, installing all the predep packages. If wigwam-base was upgraded, the install system reexecs itself to use the new version.

dep

Packages that must be installed before this package can be built or installed.

This file consists of lines containing:
package_name package_version
If package_version is omitted or is the string any, then any version of that package will satisfy the dependency. If package_version is the string latest, then the latest version of package_name is required.

Use dep instead of predep wherever possible, because predep prevents us from being able to download all the packages and then analyze them. What this really means is that predep packages are not handled by the (TODO: unimplemented) planner.

For services there are additional kinds of dependencies:

servicedep

For each service, a list of other services that must be running before it can be started. The package should also specify the packages which provide these services in its .dep file.

servicedep will ignore failed dependencies. Its only function is to sort the services when servicectl start or servicectl stop is specified.

When the service is installed, this file gets copied to ext/services/SERVICE-VERSION/servicedep. It can be overridden by a project file services/SERVICE-VERSION/servicedep.

filedep

Many files affect the way a service runs, but often changing those files has no effect until the service is restarted.

The .filedep file contains a list of files which, if changed, will cause the service to be restarted when publishing. Paths are relative to $PLAYPEN_ROOT; shell-type wildcarding is allowed.

When the service is installed, this file gets copied to ext/services/SERVICE-VERSION/filedep. It can be overridden by a project file services/SERVICE-VERSION/filedep or supplemented by services/SERVICE-VERSION/filedep.additional. In both of these files, -VERSION may be omitted.

packagedep

Some services must be restarted after other packages are upgraded; in some cases they cannot have packages on which they depend gone for even a second, in others they will continue using the old package until they are restarted.

The .packagedep file contains a list of packages (without version numbers) which, if changed, will cause the service to be restarted when publishing.

When the service is installed, this file gets copied to ext/services/SERVICE-VERSION/packagedep. It can be overridden by a project file services/SERVICE-VERSION/packagedep or supplemented by services/SERVICE-VERSION/packagedep.additional. In both of these files, -VERSION may be omitted.

Configuring the Build and Install Process

Many package-types allow certain variables to be set in $PACKAGE-$VERSION.options. This file is sourced from a shell. The following variables are recognized by GNU and Perl style packages:

configure_options

Options to be given during source configuration, either passed to configure in the case of autoconf'd packages, or passed to perl Makefile.PL for perl "MakeMaker" packages.

configure_input

Passed as standard input to the source configuration process.

pre_configure_script

A script which gets eval'd before the source configuration is done by perl Makefile.PL or configure.

no_build_phase

If set to 1, this package has no meaningful build phase.

The pre_build_script and post_build_script are run whether or not no_build_phase is set.

pre_build_script

A script which gets eval'd before the build target is made.

post_build_script

A script which gets eval'd after the build target is made.

build_subdir

If building and installing should take place in a subdirectory, set that here. Sometimes packages put everything in src.

build_options

Options to be given during the make process. These options are repeated in the install phase.

build_only_options

Options to be given during the make process, but not during the install phase.

install_options

Options to be given during the install process.

no_install_phase

If set to 1, this package has no meaningful install phase.

The pre_install_script and post_install_script are run whether or not no_install_phase is set.

pre_install_script

A script which gets eval'd before the install target is made.

post_install_script

A script which gets eval'd after the install target is made.

All the standard build types uncompress any tarball present in the .files list into the package's build directory, $PLAYPEN_ROOT/ext/build/$PACKAGE-$VERSION/. For make-style packages, these tarballs should contain the source code from the original ("upstream") authors. For raw- and service-style packages, these tarballs are optional, and may contain support files, usually from the author of the package.

If a package's source files need modifications, for example, for portability or to make them better adapted to wigwam, you may include patches in the package. These should be normal patches, produced with diff -u. They will be automatically found and applied if they end with .patch, .patch.bz2 or .patch.gz.

Sometimes patching sources is not enough to fix them so they build. For example, sometimes a file's permission needs to be changed. To do that, one can put code in the .prepatch file:
chmod +x configure-helper-script chmod +w some-random-file

Makefile-style packages recognized the additional variables:

configure_style

The mechanism used to generate the source code configuration script.

perl: packages use ExtUtils::MakeMaker to produce a Makefile.
gnu: The package uses GNU autoconf or automake or otherwise obeys the GNU GNU coding standard.
prefix-only: The package has a configure script, by default named configure, but take only the --prefix option.
foreign: the package has a configure script, but doesn't take --prefix. So the configure_options variable should be set to the equivalent of --prefix.
none: the package has no source code configuration phase.

configure_script

A different script to run instead of ./configure or perl Makefile.PL.

The following extra configuration options are understood by raw or service packages:

dont_expand_tarballs: If set to 1, don't automatically unpack tarballs that are found in the package.

The following extra files are automatically recognized by raw and service style packages:

build

Contains a script to build the binaries and data from the upstream tarball and patches, etc. in ext/build/$package-$version.

install

Contains a script to install the binaries and data from the built package in ext/build/$package-$version to their installed locations in ext.

Service packages also install the file $package-$version.config into ext/services/$service-$version/config and optionally $package-$version.servicedep into ext/services/$service-$version/deps.

Please see the Section called Writing Custom Build and Install Scripts for more information.

Writing Custom Build and Install Scripts

Writing custom build and install scripts is necessary for most packages that don't use make, and it is the standard practice for service-style packages.

Sometimes minor customizations are all that is needed; you may use a make-style package, but set scripts in various environment variables: $pre_build_script, $post_build_script, $pre_install_script, $post_install_script. All the same rules apply to those scripts.

The following environment variables are set by the packaging system before running custom build and install scripts:

EXT_PACKAGE

The basename of the package, e.g. wigwam-base.

EXT_VERSION

The version number of the package, including the release number.

EXT_PKGBUILDDIR

The "build" directory for the package. This is just $PLAYPEN_ROOT/ext/build/$PACKAGE-$VERSION.

This is where tarballs are automatically unpacked. If there is only one tarball included with the package, and all of its contents are in a single subdirectory, then the subdirectory's contents will be moved into EXT_PKGBUILDDIR for convenience. Otherwise, the tarball(s) will simply be unpacked.

Of course, you may assume $PLAYPEN_ROOT is available too.

Making Non-Relocatable Packages

Most packages are relocatable: this means that once they are installed, you may merely move the $PLAYPEN_ROOT and update that environment version and the package will continue to function. Wigwam-base is an example of a relocatable package.

If a package isn't relocatable, it is a problem for publication, where we generally want to build everything in a temporary $PLAYPEN_ROOT then relocate that as late and fast as possible.

In order to solve that, we need a mechanism for adding hooks when a package is relocated; the external interface to this is:
packagectl new-playpen-root OLD-PLAYPEN-ROOT

Nonrelocatable packages need special treatment in the .options file. The following variables apply to all package styles:

is_not_freely_relocatable: All packages which require any action other than just moving the files must set this to 1.
is_completely_nonrelocatable: This package must be entirely rebuilt and reinstalled when the playpen is moved. This flag is highly discouraged.
relocate_build_script: This is a script that is run to relocate the build-directory.
relocate_install_script: This is a script that is run to relocate the installed image.

The various scripts may rely on the following shell variables:

is_built: Whether the package must be rebuilt.
is_installed: Whether the package must be reinstalled.
old_playpen_root: The previous $PLAYPEN_ROOT; the location from which we are switching.

As usual, the environment variables EXT_PACKAGE, EXT_VERSION, EXT_PKGBUILDDIR, EXT_DIR, and PLAYPEN_ROOT are set.

Writing Package Documentation

All packages should have documentation, eventually.

That documentation can be in one of two forms:

an .sgml in the package itself

This file should be a DocBook SGML chapter -- it will be checked into the wigwam-package-archive CVS tree.

Such documentation must be freestanding text -- if you need other files you must make a tarball out of them.

an .sgml in the upstream source

If the package contains its own SGML documentation, it should be referenced in its .options file, by

package_doc_dir: The package's documentation directory; all the data needed by the sgml file to build should be contained within this directory.
package_doc_file: The sgml file relative to the documentation directory.
package_doc_build_targets: Set this if you have toplevel build targets that should be made before the documentation can be extracted [this makes building the documentation repository much slower but is needed in some cases]
package_doc_skip_pod: Set this to 1 to omit POD files from the autogenerated documentation.
package_doc_extra_pod: Set this to a list of file in POD format that are not automatically found. (Either because they do not end in .pod or because package_doc_skip_pod is set to 1.)

Miscellenous Special Documentation

synopsis

If you include a file PACKAGE-VERSION.synopsis with the package (it must be listed in the .files list), then it will be interpreted as a one sentence description of the package. For example, the synopsis for wigwam says:
A framework for developing projects that provide services.
Alternately, the synopsis may be stored as project_doc_synopsis in the package's .options file.

description

If you include a file PACKAGE-VERSION.description with the package (it must be listed in the .files list), then it will be interpreted as brief, but complete, description of the package. It will usually be several paragraphs long.

Alternately, the description may be stored as project_doc_description in the package's .options file.

ChangeLog

We also support installing ChangeLog files, which are just lists of changes and in which version they were made. wigwam-base contains a ChangeLog which you may use as an example.

You may include a PACKAGE-VERSION.changelog file with your package, or it the changelog is contained in the tarball, you may use the package_doc_changelog_file shell variable to store the location of this file within the tarball.

NEWS

Put important news that should go on your autogenerated homepage in this file.

You may include a PACKAGE-VERSION.news file with your package, or it the NEWS file is contained in the tarball, you may use the package_doc_news_file shell variable to store the location of this file within the tarball.

Making Packages that Provide Services

Services when running provide functionality for users and other services. Most service packages contain only a few configuration files. The programs themselves should be packaged separately, and the service should depend upon the required programs' packages.

Services are really just normal packages. They are distinguished by the packaging system in several ways:

the package contains a file $PACKAGE-$VERSION.services which lists the services provided by this package; these are the names you'll use to change the package with servicectl
the package's build type defaults to service instead of configure.
the packaging system will run servicectl new-service service-name for each service in the $PACKAGE-$VERSION.services file.

By default, starting a service consists of starting a particular program. Usually this means making a symlink so that the program has the appropriate name in $0, so that it can be seen by top or killall. The scripts should make a pid file in the location specified by SERVICE_PIDFILE, which is set to LOCAL_VAR/run/SERVICE.pid. The output of the program will be redirected to SERVICE_OUTPUT_FILE which is set to LOCAL_VAR/run/SERVICE.output. The symlink itself should be make in LOCAL_VAR/linkfarm/.

The service is in charge of managing its own configuration, but in general, we recommend making a service_pre_start script that runs sub-conf on each configuration file. It take a path under etc and searches first in etc then in ext/etc, which allows users to override the configuration file by putting a copy in their project instead of using the package's version. sub generally takes strings like __ENVIRONMENT_VARIABLE__ and replaces that with the value of $ENVIRONMENT_VARIABLE.

The $PACKAGE-$VERSION.config file is a sh-script which is sourced by a subshell of servicectl. It is in the service package. It may set the following sh-variables. It should not export them. General running options:

service_program

The name of the binary to run. If it is not in $PLAYPEN_ROOT/bin or $PLAYPEN_ROOT/ext/bin, then it should be the path to a binary, relative to ext/.

service_run_name

The name to use as "argv[0]" or the first string in /proc/pid/cmdline or the string seen in top or ps.

The name of the service's package will be used if none is provided.

This is implemented by making symlinks under the directory $LOCAL_VAR/linkfarm, so if you specify service_no_symlink this will have no effect.

service_arguments

The flags to pass to the service when starting or stopping it.

service_pre_start

An optional command to run before starting or sending a SIGHUP to the service.

It is often more convenient to include a SERVICE-VERSION.pre-start file in your package.

service_post_start

An optional command to run after starting or sending a SIGHUP to the service.

It is often more convenient to include a SERVICE-VERSION.post-start file in your package.

service_pre_stop

An optional command to run before stopping the service.

It is often more convenient to include a SERVICE-VERSION.pre-stop file in your package.

service_post_stop

An optional command to run before stopping the service.

It is often more convenient to include a SERVICE-VERSION.post-stop file in your package.

service_stop_start_delay

An optional number of seconds to wait when restarting between stopping and starting the service. This only has an effect if the service does not handle SIGHUP and does not have a service_restart script.

In addition to the pre-start commands, some configuration files are also constructed immediately before starting the service. These are listed

service_subconfdir_conffiles

Configuration files which are run through sub-conf-dir. They are specified relative to either $PLAYPEN_ROOT/ext or $PLAYPEN_ROOT (for per-project overrides).

The output is placed in the same relative location, under LOCAL_VAR.

There are a few booleans which describe how to perform the restart action:

service_hupable: If set to 1, the service may be restarted by sending a SIGHUP.
service_restart_to_upgrade: If set to 1, the service must be restarted if any package it depends on has been upgraded. The default is 1, so you must set it to 0 explicitly if the service does not need to be restarted before upgrading.
service_stop_before_upgrading: If set to 1, the service must be stopped before it or anything on which it depends can be upgraded. The default is 1, so you must set it to 0 explicitly if the service does not need to be stopped before upgrading.
service_stop_before_upgrading_self: If set to 1, the service must be stopped before it is upgraded. The default is 1, so you must set it to 0 explicitly if the service does not need to be stopped before upgrading.
service_stop_before_changing_files

For ease of writing Java services, we support a few convenience variables, mostly to help location java libraries.

service_jar_dirs: A space-separated list of directories containing .jar files, all of which will be added to CLASSPATH before running the service. (Specified relative to ext.)
service_jar_files: A space-separated list of .jar files, all of which will be added to CLASSPATH before running the service. (Specified relative to ext.)

Other booleans describing the server's ability or behavior:

service_writes_pidfile

If set to 1, the service is expected to write its own pidfile. The $SERVICE_PIDFILE environment variable will be set to the full path of the pidfile the service must create before sourcing the .config or the start, stop or hup scripts.

service_no_pidfile

If set to 1, there should be no pidfile at all for this service. Such services must provide service_ping scripts.

service_no_symlink

If set to 1, the service cannot be invoked through a symlink; service_program must be invoked directly.

Service Runtime Symlinks

By default, services will be invoked through symlinks so that their run names can be changed if either PLAYPEN_TAG or service_run_name are set.

Note that this flag is automatically turned on if the program to run is the Java runtime (since the java program is a wrapper which cannot be a symlink).

service_required_envars

A list of environment variables that must be set before the service can be run.

service_optional_envars

A list of environment variables that may optionally be set but affect how the service runs.

There are other variables used to prompt the user about the meaning of the required environment variables. Each of these variables may be set for each variable listed in service_required_envars or service_optional_envars:

service_envar_description_ENVAR

A string describing the given environment variable.

service_envar_type_ENVAR

The type of value to which this environment variable may be set. This must be set to one of the following values:

simple_string

The value of this variable should be a nonempty string that does not contain whitespace.

string

The value of this variable should be a nonempty string that may contain whitespace.

filename

Like string, but intended for filenames.

integer

Any integer value.

port

A port number.

If the environment variable ABC is required and has type `port', then the value of the port will be computed in one of these ways:

If ABC is set, it will be used.
If PORT_PREFIX and ABC_OFFSET are set, then the value PORT_PREFIX + ABC_OFFSET will be used.
For now, I'll add a hack to assume untyped variables which end _PORT are of type `port', and print a warning.... Try to get the services switched over quick...

real

Any real number.

nonnegative_integer

Any positive integer or 0.

service_envar_constraints_ENVAR

Type-dependent constraints on a variable. For an numeric type, this may be a space-separated list of ranges: START-END. For a simple_string, this may be a space-separated list of allowed values.

service_envar_default_ENVAR

A default recommended value for this variable. (Used for creating example scripts).

Scripts which can brutally override the start, stop or restart behavior:

service_start, service_stop

We expect that the generic server starting and stopping mechanisms will be sufficient, but if they are not, you may set $service_start to the name of a script to run to start the service or service_stop to to a script to shut down the service.

These scripts are invoked with two arguments, the service name and version number. If the second argument is omitted, then the service is a local service. This should never happen to a packaged script.

By default, the service_start script is responsible for writing a $SERVICE_PIDFILE file. If you do not want to have a pidfile, you must use $service_no_pidfile, but you must be able to implement a $service_stop script without a pidfile.

service_restart

Provide a script which will restart the service. This will only be used if the service doesn't restart when sent a SIGHUP. If there is no restart script and SIGHUP doesn't cause a restart, then a "restart" request to servicectl, it will stop and then start the service.

Scripts the service can provide to test its aliveness. Testing the aliveness of a service:

service_ping: Provide a script which will test the service's aliveness. Without this we just ensure the process is still running, which may be wrong if an unclean shutdown occurred.
service_check: Provide a script which will test that the service is working, for example, fetching a known page from a webserver or even a 404 or some other known error.

A few packages need to set environment variables before running programs in that package (or in dependent packages). For that purpose, the package may include a $PACKAGE-$VERSION.profile script that will be automatically copied into ext/etc/profile. ( note: this only works for make style packages at the moment. raw and other packages styles will have to copy profile files in place themselves. ) This is sourced automatically by load-config.

Adding New Wigwam Packages

After making the wigwam package, you'll want to test it, and then later commit it to the global package archive so everyone can use it.

Testing the Package

This section describes how to test a Wigwam package and commit it to the global package archive.

The first thing you must do is test your newly made package. You'll need to make a personal package archive to do this. Use wigwam-bootstrap to do this:
wigwam-bootstrap --new-package-archive --project=my-archive
All the packages should be placed in the directory public/ within this new project, with directories for each packages, which contain directories for each released version. So you should make a directory public/PACKAGE/VERSION/, and put your packaging files into it.

Once the files for the package are in the archive, run projectctl build to fetch any needed files and to rebuild the package list. (You can check that this worked by looking in public/package-list.)

Committing to the Central Package Archive

Once you are done, you may commit your package into the CVS module wigwam-package-archive. From there it will be automatically installed in the global package archive (this is done at the top of every hour).

Appendix A. Standard Script Naming Conventions

Many scripts are installed by wigwam-base in ext/bin. If you change these scripts your changes cannot be conveniently committed to CVS, and they will be lost next time the package is upgraded.

But in some projects you may have special needs to change some scripts -- for example, if you need a non-standard configure option for a package, you may override ext/packages/$package-$version.options with packages/$package-$version.options or even packages/$package.options, which you can commit to CVS, and which won't be lost.

In addition to the "non-ext override" rule, there are other "hooks" you can use to run other scripts before or after the script in ext/. This section describes how these overrides work.

Most Wigwam packages install a number of scripts which are generated by the packaging. Such scripts should be installed in ext/bin. Project maintainers may put overriding commands in the non-ext version of the directory.

Note that by default run-configurable only works on sh-scripts because it sources your program from a subshell. To use it on a non-script, run it --non-sh, This will disable .presource from working and the presence of such a file will cause a non-fatal warning.

Scripts which might need customization should be wrapped in a call to ext/bin/run-configurable; which will optionally run a collection of user-hooks. This first option passed to run-configurable which does not begin with a - is the program name to run; this package's version should be in the ext/ directory, but that will be added by run-configurable. This argument will be called $base here. All the arguments after $base are passed verbatim to the program and to any other scripts that are run.

These are the various files run-configurable considers:

$base.local, $base, ext/$base: order in which to try the scripts; only the first script found is run. The $base.local script should not be committed to CVS.
$base.pre: if it exists, this script is run before the main script with the same arguments.
$base.presource: this script is sourced from the shell before running the main script. Doesn't work if the command is not a sh-script (i.e. if run-configuration was invoked with --non-sh.
$base.post: if it exists, this script is run after the main script with the same arguments.

So the command,
run-configurable bin/build/weird-script
then if $PLAYPEN_ROOT/bin/build/weird-script.local exists it will be used followed by $PLAYPEN_ROOT/bin/build/weird-script. If the version in ext/ is used, then $PLAYPEN_ROOT/bin/build/weird-script.presource. will first be sourced from the same shell as the one that will run the script from ext/. Also, if $PLAYPEN_ROOT/bin/build/weird-script.pre exists, it will be run first, and $PLAYPEN_ROOT/bin/build/weird-script.post will run after the main script successfully completes.

Appendix B. Files in Wigwam Base

Package Management Configuration

ext/etc/installed-packages

the packages (and their versions) that are installed in this Wigwam playpen.

etc/project-packages

the packages (and their versions) that are supposed to be installed to use this project.

etc/project-services

a newline separated list of packages (generally the executables are built from source material in the project, or are scripts that come with the project).

ext/etc/installed-services

a newline separated list of space-separated triplets:
SERVICE PACKAGE VERSION

etc/package-archives

the package archive or archives to use. This may be overridden by etc/package-archive.local.

ext/etc/installed-services

...

Wigwam Standard Programs

Wigwam comes with a large number of programs, mostly (or all) implemented as shell scripts. This section describes each of these programs, in rough groups by how they are used.

Publication Tools

ext/bin/pubtool: publish a version of the site to a live or staging server.
ext/bin/litemass [-h "hosts"] [-i identity-file] [-l username] [-r] [-v] [program] [args...]: Run the specified program in parallel on the specified hosts. Unless -v is turned on, the program's output is printed only if it fails.

Miscellaneous Utility Programs

ext/bin/run-configurable

run a script or program, allowing the standard user configuration and hooks.

ext/bin/wigdo [command]

run a command with all the Wigwam configuration (project, cluster, and role) exported to the environment. The command is passed to /bin/sh; it is probably best to always quote it:
wigdo 'echo LOCAL_VAR is $LOCAL_VAR'
so that shell metacharacters are preserved.

wparallelizer

Table of Contents
wparallelizer -- run a number of commands in parallel

wparallelizer

Name

wparallelizer -- run a number of commands in parallel

Synopsis

wparallelizer [--max NUM] [--names DELIMITER NAMES] [--output OUTPUT-MODE] [--commands COMMAND-FILE] [--error-table ERROR-TABLE] [--log LOG] [--0]

DESCRIPTION

This program runs a number of commands, possibly specifying a maximum number to run concurrently. The backend "rendering" engine is designed to allow convenient output mode customization.

Commands are read from scripts specified as parameters to the --commands flag. By default, we read newline separated commands (just like a normal script), or if --0 is specified, the command files will be parsed assuming NUL (character 0) is the delimiter. (This is useful if the commands may contain newlines.)

If multiple --commands or --message lines are given, the will be run in serial, that is, all the commands in a given command block are executed before the next command or message is processed.

A log of all the process output is optionally kept in file specified with the --log flag.

INTERFACE OPTIONS

The following output modes may be specified with the --output flag:

blob: Serialize the output of each process separately, in the same order in which the processes ended.
line: Each output line from each host is preserved on a separate line, with the host/process name prepended. Though the lines from all the hosts are printed without delay, it will not print will always wait for a full line from each host; each line from each host is print intact.
raw: No processing is done at all, all the outputs are scrambled together.
tiled: Make separate windows for the output of each process (using ncurses).

In addition to the output redirected from the subprocesses wparallelizer itself may be directed to print a summary of its operation. By default, it prints the exit statuses of failed processes (those which did not exit with status 0). You may chose a different summary type with --wrapup; here are the currently defined wrapup modes:

none: Do not output anything upon termination.
failed: Print information about processes which did not exit successfully. (This is the default)
all: Print information about all the processes that we ran.
stats: Show the exit statuses of all processes and the amount of data input and output by those processes.

ERROR HANDLING

wparallelizer is built to be general purpose, but it has compiled-in defaults that are only really appropriate for wigwam. We support a compile-time default and a runtime-choosable configuration of error messages, error id strings, and exit codes. The format of this file is rather strict; here is a two entry example:
{ 0, "success", "The process completed successfully" }, { 3, "tragedy" "You don't have write access to a directory required for publication" },
Please note that it is mandatory that the curly braces appear by themselves on a line, and also that it adheres to C-array declaration syntax.

wparallelizer defines some magic for making shell scripts more readable. If you invoke it with the --print-sh-error-table flag it will print a sh-script you can eval to import symbolic names for the variables; the symbolic names are the second field in the error-table, with errcode_ prepended; the error messages are likewise, but with errmessage_ prepended. For example the above error table would cause the following sh-script to be emitted:
errcode_success=0 errcode_success="The process completed successfully" errcode_tragedy=3 errmessage_tragedy="A tragedy occurred"

To change the compile-time default add CFLAGS=-DERROR_TABLE_FILENAME="filename" to your environment then upgrade -f wigwam-base. You, uh, probably want to use the runtime setting, WPARALLELIZER_ERROR_TABLE_FILE instead.

Programs Used Internally by pubtool

These are programs you will rarely need to run directly. Understanding their internals is marginally useful for obscure things, but mostly this section will be useful for those who have the misfortune of having to debug these tools.

ext/bin/get-last-tag: Print name of last publishing tag with given prefix. (Chiefly for use by pubtool.)
ext/bin/make-new-tag: ...

Scripts to Load Certain Configurations into sh-variables

ext/bin/load-config

Load playpen-specific configuration variables.

This script sets PLAYPEN_ROLE and PLAYPEN_CLUSTER (if applicable), sources all the environment setting scripts added by all packages, and sources the following scripts:

etc/clusters/$PLAYPEN_CLUSTER/config.pre
etc/roles/$PLAYPEN_ROLE/config.pre
ext/etc/project.defaults
etc/project.conf
etc/clusters/$PLAYPEN_CLUSTER/config
etc/roles/$PLAYPEN_ROLE/config

For each script, before it is sourced, the same filename with .pre.local is tried. After each script is sourced, the same filename with .post.local appended is tried. These are intended to be configurations and overrides which are not in CVS. In general, no .local files should be checked into CVS.)

sh-macros

Set up a bunch of macros that are mostly used for portability. Please see Appendix C.

Programs Used Internally by the Packaging System

Though these are not generally needed, they are documented for those who have to maintain Wigwam, or debug a packaging problem.

ext/bin/build/check-installed PACKAGE VERSION

Test whether PACKAGE is installed at least version VERSION.

ext/bin/build/compare-version-number ver_number1 ver_number2

Exits if the first version number is equal or greater than the second version number.

ext/bin/build/build-package PACKAGE VERSION

Optionally downloads and builds the specified package.

ext/bin/build/build-{raw,make}-style

Builds a package of the stated type. This script should always be run by build-package which will figure the type by looking at the $PACKAGE-$VERSION.type file. The default type is configure for packages conforming to the GNU standard. Perl style packages are those that have a Makefile.PL. raw packages have their own custom build script.

ext/bin/build/expand-sources TARBALL "PATCHES" OUTPUT_DIR

Expands TARBALL so its base file is in OUTPUT_DIR, then apply the patches in left-to-right order. OUTPUT_DIR must not exist when this script is run. PATCHES may be an empty string, but must be present even if it is empty.

ext/bin/build/forcible-mv-recursive SOURCE_DIR DEST_DIR

Moves the contents of SOURCE_DIR over DEST_DIR.

ext/bin/build/get-installed-version PACKAGE

Returns the version of PACKAGE that will be/is installed (it will be installed if update-packages is run successfully.)

ext/bin/build/guess-package-defaults

Given downloaded package files, determine the basic package info: name of the tarball, list of patches, architecture to use.

ext/bin/build/mark-install PACKAGE VERSION

Indicate to the packaging system that PACKAGE has been successfully installed in this playpen, at version VERSION.

ext/bin/build/mark-uninstall PACKAGE

Indicate to the packaging system that PACKAGE has been uninstalled.

ext/bin/build/propeller PROMPT LOGFILE [COMMAND ..]

Prints a 'propeller' to stderr while copying to the logfile.

If a command is given, it is executed and its standard error and output are used as input to the propeller. In this case, propeller will exit with the status of its subprocess.

You may set WIGWAM_PROPELLER_MODE to one of the following values:

Mode: stderr

Description: Copy standard output and error to the propeller's stderr.

Mode: propeller

Description: Normal mode of operation.

Mode: quiet

Description: Output nothing.

Mode: terse

Description: Output the start and end messages on the same line.

Mode: linewise

Description: Output the start and end messages on separate lines, with nothing in between.

Programs Used Internally by the Services System

Though these are not generally needed, they are documented for those who have to maintain Wigwam, or debug a service problem.

ext/bin/build/query-installed-service: From a service-name, this will tell you the package name and version.

Appendix C. Portable sh notes

Most of the configuration and scripting in wigwam-base is done in sh-script; this would work out well if sh were standard, but they differ.

Meticulous sh-scripters should always make sure that their scripts run correctly in ash, a minimal, posix 1003.2a-compliant shell. Below we will describe common problems with conforming to this shell, and other really obnoxious sh idiosyncrasies.

Here we collect a bunch of common idioms in sh-scripting, and introduce you to some macros in ext/bin/sh-macros.

Checking if an environment variable is empty

if test "x$ENV_VARIABLE" = "x" ; then ...

Checking if a file, directory exists

Use
test -r filename
to see if you can read a file or
test -w filename
to see if you have write permission, and
test -d directory
to see whether a directory exists.
test -e
isn't available on Solaris 7.

Checking if a file is executable

test -x
isn't on BSD 4.2, However, the sh-macros file defines test_x which you can use to test a file's executability (it falls back to test -r though).

Setting an environment variable only if it isn't set

... copy from autoconf docs ...

Appendix D. Setting up a Wigwam package archive

A Wigwam package archive is itself a Wigwam project.

Eventually you will want to create an empty package archive where you can install new packages or new versions of packages which aren't yet in the standard archive. To do this, follow the same steps you would perform when creating a new project, only invoke wigwam-bootstrap as follows:

wigwam-bootstrap --new-package-archive \ --project=my-package-archive \ --package-archive=ORIG-ARCHIVE cd my-package-archive ./autogen.sh source ./setup-env packagectl project build

The packagectl project build command will create necessary files in the public directory, even if you haven't yet added any packages to the public directory. Whenever you add packages to the public directory, you'll need to run packagectl project build to make them visible to projects which use this package archive.

You should update etc/package-archive in projects where you want to use this new archive. Just include a reference to the my-package-archive/public/ directory as well as listing any other package archives you wish to use.

Because most of the bulk of a Wigwam package archive is from the upstream packages, only locally developed packages and any files describing dependencies, compilation options, etc. are checked in to the the CVS repository.

You might want to create and maintain a local copy of the full package archive so that you have complete control, history, and backup of the packages your projects need.

Alternately, you may want your own copy of the archive if you find that your connectivity to the original package archive is slow or unreliable.

Finally, you might make use of a copy of the standard package archive when you are trying to upgrade versions of packages which are already present in the archive.

To build a local copy of the complete standard Wigwam package archive, check out the project and start by performing the steps you'd perform for any wigwam project:

       cvs -d:pserver:anonymous@cvs.wigwam.sourceforge.net:/cvsroot/wigwam checkout wigwam-package-archive
       cd wigwam-package-archive
       ./autogen.sh
       source ./setup-env
       packagectl project build

The project build command will download all the upstream packages for you and generate md5sums for them, and will generate the package_list and other support files necessary for the archive. Warning! This will take a long time and uses a lot of disk space.

After doing this, the complete package archive will be in $PLAYPEN_ROOT/public.

For example, if php 4.0.4 were released and the package archive already contained a package php-4.0.2-2, you could create the files necessary to include 4.0.4 in your new package archive by doing:
cheesy-upgrade php 4.0.2 2 4.0.4
After doing this, you should, of course, confirm that the options are still correct for the new package and that it actually compiles and functions when you try to use in a project.

Appendix E. Using C in a Wigwam Project

Projects often consist of custom servers or tools which must be very fast or more lowlevel than perl or sh. In that case, you may try C or java. This section is about using C (or C++) within a Wigwam project.

Another possibility, by the way, is developing your program to be a Wigwam package. That will make it easier to reuse the code in another project. But sometimes the C code is worthless without the supporting project. In that case, embedding the code directly in $PLAYPEN_ROOT/src can be helpful.

Here is how we advise proceeding:

Learn about automake and autoconf; these are the easiest way to build binaries from your source code.

Copy a autogen.sh in src/:
if test "x$PLAYPEN_ROOT" = "x"; then echo "You are NOT in a wigwam playpen." 1>&2 else CPPFLAGS="-I$PLAYPEN_ROOT/include -I$PLAYPEN_ROOT/ext/include" LIBS="-L$PLAYPEN_ROOT/lib -L$PLAYPEN_ROOT/ext/lib" export CPPFLAGS export LIBS echo "You are in a wigwam playpen." 1>&2 fi autoheader automake -a --foreign aclocal autoconf configure "$@"

Write some C code.
Write a configure.in and Makefile.am, as explained by the autoconf/automake documentation.

Run
./autogen.sh make
make it compile, then debug it.

Appendix F. External Programs Used by Wigwam

perl

See the perl online documentation; try running man perl. The same text is available at PERL: Home: Documentation.

awk

See the Single UNIX Specification, Version 2, awk.

sed

See the Single UNIX Specification, Version 2, sed.

sh

Please try to make your shell scripts as portable as possible. See the Single UNIX Specification, Version 2, sh.

cvs

The upstream documentation is texinfo and should be installed in some CVS package: try info cvs. Or see CVS - Table of Contents.

autoconf

A GNU program to generate configure scripts that test for system feature availability. The results of these tests are used to transform FILENAME.in to FILENAME by replacing doing substitutions like (in sed) s,@PERL@,/usr/local/bin/perl5.

automake

A program used with autoconf to produce Makefile.in files in each source directory each of which will be made into a Makefile by the configure script. It bases its work on the Makefile.am files.

You do not need to have autoconf or automake to use Wigwam, but you would need it for building Wigwam from CVS, and we recommended it if you are developing C code.

autoconf and automake come with documentation in the INFO format. It is available for download from ftp.gnu.org, but most systems come with it. It requires m4.

rsync

A program to synchronize the contents of two directories on two different machines. This program relies on SSH (or an equivalent) to connect securely between the machines.

Appendix G. Writing Documentation

We recommend using DocBook for writing your user documentation. The only exceptions are manpages for convenient executable references, and function references which are documented in the code.

Here are the recommended formats then, and some references to find out more information about them.

DocBook SGML: DocBook is just a stylesheet for SGML, which is a very general markup language. A good starting point for information is docbook.org.
POD (Perl's Plain Old Documentation): Convenient for documenting Perl modules since Perl supports inline pod documentation. Run man perlpod for more information.
Javadoc: An inline Java source code documentation format. See Sun's Javadoc page for more information.

Furthermore here are some user documentation recommendations:

Write in present tense.
Organize your documentation into sections by who would want to read it (maintainers? sysadmins?). There should always be an Introduction that describes the basic functionality, scope, and modules in your program.

Name

Synopsis

packagectl install

packagectl update-packages

packagectl update

packagectl update-local

packagectl update-packages

packagectl upgrade

packagectl project

packagectl info

Name

Synopsis

Description

servicectl start

servicectl stop

servicectl status

servicectl test

servicectl restart

servicectl check-config

servicectl new-service

Name

Synopsis

Description

Name

Synopsis

pubtool tag

pubtool changes

pubtool diff

pubtool publish

Environment Variables

Name

Synopsis

DESCRIPTION

INTERFACE OPTIONS

ERROR HANDLING

Notes