Hosted by Three Crickets

Sincerity
Packaging and Bootstrapping
For the JVM

Please note that Sincerity is still under development and that this documentation is incomplete.

Sincerity logo: jackrabbit and flowers

Tutorial

Install Sincerity

You need a JVM, at least version 6.
If you're an Ubuntu user, then use our repository! It would do everything for you.
Otherwise, download a Sincerity distribution. If you're download the Zip distribution, unpack the folder, and put it in any standard location, for example:
You can then run the "sincerity" script (Unix and Mac) from there or "sincerity.bat" (Windows).
You might want to add the Sincerity path to your system path, to allow for easy access from the command line. In Linux, you can do this by adding the following line to your user's .bashrc file:
PATH=$PATH:/opt/sincerity

Working with the Command Line

If you run Sincerity with a Graphical User Interface (GUI) using "sincerity gui". However, it's strongly recommended that you learn how to use the command line. Here are the main principles:
  1. All Sincerity commands exist within "plugins." The full name of a command is its plugin name, with a colon, and then the command name within the plugin. For example, "container:create" is the "create" command within the "container" plugin. Use "sincerity help" to list all available commands from all plugins. Many commands support command line arguments, both required and optional. See the command documentation for full details.
  2. As a short form, you can use only the command name. However, this will only work if there is no ambiguity, meaning that the same command does not exist in more than one plugin. For example, "create" will be equivalent to "container:create" if no other plugin has a "create" command. Also note that the full form of the "help" command is "help:help". (Plugin developers are encouraged not to use command names that would conflict with the core plugin commands, such as "create", "add", "install", etc.)
  3. Some Sincerity commands can only be run while pointing to a container. Generally, it's useful to run Sincerity when the current directory is somewhere in the container. There are a few rules to consider:
    1. Sincerity can only point to one container at a time.
    2. You can change the container or explicitly point to one using the "container:use" command.
    3. Otherwise, Sincerity will attempt to find for a container in the following order:
      1. The "sincerity.container.root" JVM property
      2. The "SINCERITY_CONTAINER" environment variable
      3. Search up filesystem tree from current path looking for a directory that has a ".sincerity" subdirectory
  4. Sincerity's current set of available plugins, which affects the set of available commands, is a combination of both the plugins available in the Sincerity installation as well as those available in the current container.
  5. You can chain commands together using ":". Command chains are used extensively in Sincerity.
    1. Note that a single command chain can change the current container multiple times. For example:
      sincerity use container1 : install : service start web-server : use container2 : log "test log message"
      
    2. Considering the above, note also that each time you switch container within a command chain the set of available plugins and commands changes, matching whatever is the current container at the time.
    3. Also keep in mind that a single command chain is always run within the same JVM. Sincerity achieves JVM classworld separation by swapping class loaders when it changes containers. If this behavior is not desired, you should avoid chaining and run your commands using separate "sincerity" command lines.
  6. If an argument begins with a "@" character, it will be interpreted as a shortcut, and searched for in the current container's "/configuration/sincerity/shortcuts.conf" file. If found, it will be expanded to the command defined there.
    1. Expansion to a command chain is allowed, as well as recursive use of shortcuts. For example:
      sincerity use container1 : @mv : start restlet
      
      Would expand to:
      sincerity use container1 : add mongovision : install : start restlet
      
      If the following entry is in "shortcuts.conf":
      mv = add mongovision : install
      
    2. Some commands support implicit use of shortcuts without requiring the "@" prefix. Specifically, the "dependencies:add" and "repositories:attach" commands will search for shortcuts with the "add#" and "attach#" prefixes respectively. For example:
      sincerity add mongovision
      
      Will expand to:
      sincerity attach three-crickets : add com.threecrickets.mongovision mongovision
      
      If the following entry is in "shortcuts.conf":
      add#mongovision = attach three-crickets : add com.threecrickets.mongovision mongovision
      
From here, you can continue reading about the core plugins to learn about all the essential commands.

Working with the Graphical User Interface (GUI)

Sincerity has a Swing-based GUI that displays information about your container and lets you perform operations on it. It can be used instead of the CLI, though each interface has its own strengths. The GUI is especially useful for displaying data, such as the dependency tree structure.
Sincerity provides the GUI frame, but the contents are provided by plugins. This means that the whole GUI would look differently according to whatever is the current container and what plugins it has installed. For this reason, when you change containers from within the GUI, it will restart.
If you run "sincerity" without any command, it will default to running "shell:console". For example:
sincerity use container1 : gui
A richer console, in which you can use full JavaScript, is available via the "jsshell:jsconsole" command.
If you are designing your own Sincerity plugin, it is strongly recommend that you include GUI support if appropriate via the optional gui() entry point.

Environment Variables

The "sincerity" script will try to find your operating system's default JVM and your Sincerity installation. You can modify its behavior using the following environment variables:

Components

Before detailing the core plugins and commands in the next chapter, it's important that you understand a few basic components:

Container

A set of files implementing a self-contained JVM-based execution environment managed by Sincerity and by you. The container has a root path, under which it may have a directory structure of any depth. Libraries, binary executables, configuration files, temporary work files and logs are all by default stored within the container.
Why such an emphasis on self-containment? One goal is for the container to be deployable anywhere as a whole, simply by copying the directory elsewhere. Another goal is for the container to be a useful playground: you can install and try out various applications and libraries without affecting your operating system. You can undo you work simply by deleting the container's directory.
It is possible and sometimes useful to break this principle of self-containment by using symbolic links.
Below are some a few standard container subdirectories used by the core plugins. Other plugins and skeletons may add more subdirectories.
/.sincerity/
Reserved for Sincerity's internal use. It's most essential use is to mark a directory as a container root.
/cache/
Files put here should be considered deletable without any negative effects.
Two subdirectories are most common: "/cache/sincerity/" is where Sincerity will store information about downloaded dependencies, and "/cache/javascript/" is where Scripturian will store its compiled JavaScript code.
/logs/
Files put here should be considered deletable without any negative effects. This is used by the logging plugin.
/configuration/
Container-wide configuration files for various libraries are found here. Note that generally Sincerity prefers "configuration by script," so that most of these files will be in JavaScript code. However, some libraries may require XML, property sheets, or other unfortunately idiosyncratic formats.
Various libraries will use their subdirectories here: for example, "/configuration/logging/" for the logging plugin.
/configuration/sincerity/
Here you can configure your container: repositories, dependencies, installed artifacts, and shortcuts. Note that you usually will not have to edit these files directory: many Sincerity core commands will manipulate these files for you.
For the format of "repositories.conf", see the Ivy documentation for resolvers. For the format of "dependencies.conf", see the Ivy documentation for dependencies.
/libraries/
Sincerity will install dependencies here, but you can also add your own files manually.
Note that the Sincerity installation also has a "/libraries/" subdirectory, which is considered in addition to the one found in your container.
/libraries/jars/
Sincerity will recursively add all Java archives (.jar files) here to the classpath. Those dependencies installed by Sincerity will follow the "/organization/name/version/name.jar" directory structure, for example: "org.slf4j/slf4j-api/1.6.6/slf4j-api.jar". It is not required that you follow the same structure for jars you install manually: all jars found under this directory will be added.
/libraries/classes/
Sincerity adds this path to the classpath, expecting to find JVM class files (".class"). The directory structure must be "/package/sub-package/.../classname.class". For example, the JVM class "org.myorg.Frame" would be in "/org/myorg/Frame.class".
/libraries/javascript/, /libraries/python/, etc.
These subdirectories are for libraries for specific programming languages to use directly. Note that these are slightly different from the "/libraries/scripturian/" subdirectory, which also contains programming language libraries, but is intended to use only from within Scripturian.
/libraries/scripturian/
Sincerity, as well as other products that use Scripturian, will look for executable documents here (and possibly in other places). Most libraries and frameworks will create their own subdirectory underneath. For example, Prudence libraries are under "/libraries/scripturian/prudence-scriptlet-resources/".
/libraries/scripturian/plugins/
This subdirectory is reserved for Sincerity plugins. Each document here represents a single plugin, and each plugin may implement any number of commands.
/libraries/scripturian/installers/
This subdirectory is reserved for Sincerity installers. Installers are run by the "artifacts:install" command, and are included in some dependencies as a way to execute arbitrary installation tasks. A common use case is for the install hook to manipulate the unpacked files in order to tailor them for the specific environment in which the container is running.
/libraries/web/
Files here are intended to be served over the web as static files, for example: images, HTML files, CSS, etc. Various web servers will look for files here (and possibly in other places). Various client-side web frameworks (such as jQuery, Ext JS) will thus install here, and be made available for various web servers you may have installed.
/programs/
The "delegate:start" command will look for Scripturian documents to run from here.
/executables/
The "delegate:execute" command will look for executables to run from here.
/reference/
Reference material, for use by humans or by software, will be available here.
/reference/documentation/
Here you'll find reference manuals and API documentation for installed dependencies. Files installed by Sincerity will be placed in "/organization/name/version/" directory structure, for example: "org.slf4j/slf4j-api/1.6.6/".
/reference/licenses/
Here you'll find licenses for installed dependencies. Files installed by Sincerity will be placed in "/organization/name/version/" directory structure, for example: "org.slf4j/slf4j-api/1.6.6/".
/reference/sources/
Here you'll find source code for installed dependencies. Files installed by Sincerity will be placed in "/organization/name/version/" directory structure, for example: "org.slf4j/slf4j-api/1.6.6/".

Dependency

A contained, versioned, installable set of files (called "artifacts"), which can in turn have its own list of dependencies. Dependencies are deployable software bundles, representing things like libraries, frameworks, platforms and skeletons, including complete applications and services.
All dependencies in Sincerity are identified by a two-part name, composed of a "group" prefix plus a unique "name" within the group, plus a version specifier. Different dependencies might have their own versioning schemes, but Sincerity is good at guessing these for the purposes of comparing versions.
Note that a dependency can also have none of its own files, and only a list of its own dependencies, allowing for a convenient shortcut for installing several dependencies together. These are sometimes called "meta-dependencies."
You can list all installed dependencies using the "dependencies:dependencies" command. However, note that dependencies are structured as a nested tree that may be better visualized using the GUI.

Artifact

There are files within a dependency. Sincerity supports a specific set of artifact types: JVM libraries (jars), language-specific libraries (Python eggs, Ruby gems, PHP packages, etc.), documentation bundles, source code bundles, software licenses, installers, dependency descriptors and more generic "packages" (see below).
You can list all installed artifacts using the "artifacts:artifacts" command.

Package

This is a special kind of Sincerity-specific jar artifact that can contain other files, and can additionally have special install/uninstall hooks.
Sincerity contains tools to help you easily create your own packages, as well as documentation about the package specification, so that you can manually create your own.
Packages are automatically unpacked using the "artifacts:install" command, but you can also explicitly unpack them using "packages:unpack".

Repository

A store for dependencies and their artifacts. Repositories usually contain indexes of available dependencies and versions. Some repositories also have friendly human-facing web frontends which you can use to search for dependencies. Sincerity supports several repository technologies, and can also help you deploy your own dependencies to them.
You can list all repositories attached to your container using the "repositories:repositories" command.

Working with a VCS

Because Sincerity containers are all in a single directory tree, it's very easy to use them with Version Control Systems (VCSes), such as git, Mercurial and Subversion.
One quick issue to note is that your "/.sincerity/" directory will often be empty, and many VCSes, such as Git, tend to ignore empty directories. To force Git to commit it, simply add a ".gitignore" file in that directory:
touch .sincerity/.gitignore
Otherwise, there are two recommended strategies for working with a VCS:

Strategy #1: Commit (Almost) Everything

This strategy is safest in terms of testing and debugging, because it guarantees that all developers and deployments are sharing the exact same files.
You might just want to make sure that you don't commit the deployment-specific directories. Here's an example of a ".gitignore":
/cache
/logs
The problem with this strategy—and it can be serious—is that distributed VCSes often require you to clone the repository with its entire history. Every time you change a large binary, it will increase the size of the repository. If this happens a lot, the repository can become quite unwieldy.
Some VCS have workarounds for this problem, though they would only work in environments where developers have access to these files via a shared directory. Consider, for example, git-annex for Git.

Strategy #2: Commit Only Your Work

This strategy makes good use of Sincerity and allows for compact repositories. The idea is that users of the repository will just have to run "sincerity install" to fill in all the missing files.
In order for this to work well, you will need some discipline. You will need to have your VCS explicitly ignore all files that are managed by Sincerity. A good way to do this a blanket ignore on all standard container and skeleton directories, and then add exceptions for files you add or change. Care must be taken during the commit phase to make sure that your changes have indeed been committed, and that you have not forgotten to add an exception. If you forget, your changes will not be committed and can be lost.
One small but important issue is that you want to make sure that "/configuration/sincerity/artifacts.conf" file is ignored. This file is managed by Sincerity specifically in order to keep track of files changed during "sincerity install" (and "sincerity unpack").
Here's an example .gitignore for a container based around a Prudence skeleton:
# Ignore everything by default, allowing Sincerity to manage it
/cache
/component
/configuration
/executables
/libraries
/logs
/programs
/reference
​
# Our applications
!/component/applications/myapp1
!/component/applications/myapp2
​
# Modifications to installed applications
!/component/applications/prudence-admin/routing.js
​
# Our shared libraries
!/libraries/scripturian/minjson.py
​
# Component modifications and additions
!/component/servers
!/component/services/database
​
# Logging configuration
!/configuration/logging/appenders/common-file.js
​
# Sincerity configuration
!/configuration/sincerity/repositories.conf
!/configuration/sincerity/dependencies.conf
!/configuration/sincerity/shortcuts.conf 
Note how we added exceptions for both new directories added to the container as well as changes to specific files. Also note that the "/configuration/sincerity/artifacts.conf" file is ignored, as required, due to the blanket ignore on "/configuration".
There are two possible disadvantages for this strategy:
First, unless you specify dependency versions precisely for all dependencies, every time a user runs "sincerity install" they may get different versions, and thus have a different container. For some testing strategies, this is a disadvantage. However, for more "agile" continuous build strategies, this can actually be seen as an advantage, as it makes sure that you are always at the cutting edge. As long as your tests are run before deployment, then this should not be a problem. However, it could still be a problem for coordinating debugging if multiple developers are working on the same VCS repository but are using different versions of dependencies. To work around this potential problem, you can of course maintain your own repository and coordinate its use with the development team, with the same care used for coordinating VCS repository use. Alternatively, for the particular problem of debugging, you can make sure to copy over files from the deployment in which the bug has been discovered, or possibly include a full "sincerity dependencies" dump with the bug report, allowing developers to precisely replicate its environment.
The second problem is that because you need to run "sincerity install", you would potentially be dependent on third-party repositories (Three Crickets, Maven Central, PyPI) to turn your VCS repository into a runnable system. A good solution is to use a repository proxy, such as Nexus, that would guarantee that you control access to all binaries within your organization, even if the third party repositories fail.

Working with Docker

Machine virtualization brought about a revolution in deployment strategies. And then came LXC, providing a more limited set of features via built-in isolation features in Linux: think chroot, but with filesystem and networking containment. LXC allows for much lighter containers as compared to virtualization.
So lightweight, in fact, that it makes sense to package and distribute applications via LXC. That's exactly what Docker does, by providing an easy-to-use set of tools, standardized packaging, repository management, and a curated catalog of ready-to-run base images. Many workload distribution systems, such as Mesos, support Docker packages, allowing you to deploy applications with exceptional flexibility, robustness, and economical utilization of resources.
(It's also interesting to see LXC encroaching into the data center market, which until now was dominated virtualization: LXD will allow you to manage a cloud of "machines" that are actually LXC containers, offering much greater density on existing hardware. It will even integrate with OpenStack, allowing data centers a smooth transition to this exciting technology.)

Running in Docker

Because Sincerity puts your entire container in one root directory, it's trivial to run your Sincerity container in a Docker image. In this example, we'll create a container with the Prudence example application, and then run it inside the ready-made "java" Docker image:
sincerity create /path/to/mycontainer : add prudence.example : install
​
sudo docker run --rm -it \
-v /path/to/mycontainer/:/opt/mycontainer/ \
-p 8080:8080 \
java:8u45-jre \
/opt/mycontainer/sincerity use /opt/mycontainer/ : start prudence
If you haven't used the "java:8u45-jre" Docker image yet, it will have to download it.
In this example, we've mapped our Sincerity container to "/opt/sincerity/" in the Docker image, and Prudence's default HTTP port to a port in the host, so that we could access the site at http://localhost:8080/. We've also enabled an interactive pseudo-TTY ("-it") so that we can press CTRL+C to quit.
The result seems identical to running "normally": and that's the beauty of Docker.
What good is this? Well, for one, it allows you to easily test your Sincerity container in various versions of the JVM without having to install them on your main operating system. But also, Docker can offer tighter security more easily than just, for example, running your Sincerity container under a custom user.

Packaging in Docker

Once you've tested your Sincerity container in Docker, it's time to package it for deployment.
First, create a "Dockerfile" in your container's directory. For our example:
FROM java:8u45-jre
MAINTAINER Three Crickets
ADD . /opt/mycontainer/
CMD /opt/mycontainer/sincerity use /opt/mycontainer/ : start prudence
EXPOSE 8080
You'll also want to create a ".dockerignore" file (which uses the same syntax as ".gitignore"). For our example:
/cache
/logs
Now we can build it:
sudo docker build -t threecrickets:mycontainer .
That's it! It was very fast, because Docker uses a transaction system: our new package is only a small diff over the original image. Running it is very similar to before:
sudo docker run -it -p 8080:8080 threecrickets:mycontainer
You can also run it in "detached" mode (like a daemon) using "docker run -d". Use "docker ps" to list existing running images, and "docker stop" to stop any.
To save the image into a self-contained, redistributable file:
sudo docker save threecrickets:mycontainer | bzip2 > mycontainer.tar.bz2
To load it:
cat mycontainer.tar.bz2 | bunzip2 | sudo docker load
Note that because the image is self-contained, the environment loading it does not need access to the repository where "java:8u45-jre" came from (it essentially includes the JVM). However, because all transactions have GUIDs, it would be identical to having retrieved "java:8u45-jre". So, if that environment were to be running 100 images based on "java:8u45-jre", it would only keep the actual installation once. (You can also "flatten" your image, as if it were a single commit, using "docker export".)
Also note that "save" does not keep the tags, though you can re-tag the image via its ID like so:
sudo docker images
...
sudo docker tag ... threecrickets:mycontainer
See the Docker documentation for more information about how to work with repositories.

The Sincerity Manual is provided for you under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The complete manual is available for download as a PDF.

Download manual as PDF Creative Commons License