Diligence is still under development and incomplete, and some this documentation is wrong. For a more comprehensive but experimental version download the Savory Framework, which was the preview release of Diligence.
REST Service
The REST Service makes it easy to create a RESTful API layer over your MongoDB database. It's powerful enough that it may be in itself the primary reason why you wish to use Diligence.
While there are tools to do this automatically—and the REST Service does have an automatic mode, too—the true power of this service is in its customizability. You can insert your own code anywhere in the resources to do special processing, for anything from data validation, through constraint enforcement, to security authorization and high-level business logic.
Moreover, the Prudence platform lets you access this RESTful layer internally, without any HTTP communication or serialization, so that you can use this layer as your primary data access layer API, both internally and for other services. There's no reason to create a separate API for internal vs. external use. This architecture also makes it trivial to separate your data processing nodes from your application logic nodes, should you ever want to do so.
Even without customization via code, out of the box you get the following features:
- The default format immediately supports Ext JS's RESTful data stores. Attach any MongoDB collection to an editable grid widget in a web browser! See the Sencha Integration manual for more information.
- Automatic content negotiation with support for JSON and XML formats, as well as a human-readable HTML format perfect for debugging via browsers. The HTML format even allows simple editing of your content. (Note, though, that if you want a full-fledged web frontend for your MongoDB data, you're better off with MongoVision, which is easily installable side-by-side with your Diligence application.)
- Pagination for traversing collections of any size.
- Choose which document fields you want to expose, and extract sub-documents from your main document.
- Apply straightforward "modes," which let you transform MongoDB's extended JSON format into simpler primitives. For example, "{$date: 1234}" would become "1234".
There are a lot of details below, but you shouldn't be intimidated by them. You do not have to learn every single feature of the REST Service in order to use it. In just a few lines of code, you can setup a whole RESTful layer automatically that will "just work" for many use cases.
Setup
Make sure to check out the API documentation for Diligence.REST.
Manual Setup
We'll start with manual configuration, because it will help you better understand how the REST Service works.
First, let's configure the URI-space in your application's "routing.js". Add the following to app.routes and app.dispatchers:
app.routes = { ... '/data/users/{id}/': '@users', '/data/users/': '@users.plural' } app.dispatchers = { ... javascript: '/manual-resources/' }
We can now configure our resources in "/libraries/manual-resources.js":
document.executeOnce('/diligence/service/rest/') resources = { ... users: new Diligence.REST.MongoDbResource({name: 'users'}), 'users.plural': new Diligence.REST.MongoDbResource({name: 'users', plural: true}) }
Automatic Setup
The REST Service can do all the above automatically for you, which is especially useful if you have lots of collections, or if you keep adding collections and want resources for them to be added automatically. Note that this automation does not occur dynamically while your application is running: you have to restart for this to work.
In your application's "routing.js".
MongoDB = null document.execute('/mongo-db/') document.executeOnce('/diligence/service/rest/') app.routes = { ... } Sincerity.Objects.merge(app.routes, Diligence.REST.createMongoDbRoutes({prefix: '/data/'}))
Important! The first two lines of code make sure that MongoDB is re-initialized before proceeding, so that we can be sure to avoid using the default MongoDB initialization in other applications. This is good practice when using Diligence in any initialization script.
In "/libraries/resources.js", we just need this:
document.executeOnce('/diligence/service/rest/') resources = { ... } Sincerity.Objects.merge(resources, Diligence.REST.createMongoDbResources())
You can also specify exactly which collections you want created:
Diligence.REST.createMongoDbResources({collections: ['users','notices','documents']})
Custom Queries
Sometimes you may be using a single MongoDB collection as a container for documents of several different types, and you would want them exposed as a separate URI-space.
The REST Service allows for this via a simple querying language. To illustrate it, lets first look at what the default query is for singular resources, if no query is provided by you:
resources = { ... users: new Diligence.REST.MongoDbResource({ name: 'users', query: {_id: {$oid: '{id}'}} }) } app.routes = { ... '/data/users/{id}/': {type: 'implicit', id: 'users'} }
The "query" key is in MongoDB's extended JSON format, and is used for the MongoDB "find" operation. The values are all cast using the conversation.locals, which, if you remember how to do Prudence routing, are extracted from the URI template. Let's look at this slowly:
-
If a "/data/users/123/" URI is accessed with a GET operation, the "123" will be extracted from the URI template. The effect will be as if we called:
conversation.locals.put('id', '123')
-
All the values in our resource's "query" value are cast using conversation.locals. So, our final query will be:
{_id: {$oid: '123'}}
-
The REST Service will use the above query for a "find" operation:
var data = collection.findOne({_id: {$oid: '123'}})
Knowing this, you can then set the "query" any way you like. You can use values extracted from conversation.locals, or any literal value. For example, let's create a URI-space for users of type "admin", to be accessed :
resources = { ... admins: new Diligence.REST.MongoDbResource({ name: 'users', query: {name: '{name}'}, {type: 'admin'}} }), 'admins.plural': new Diligence.REST.MongoDbResource({ name: 'users', query: {type: 'admin'}, plural: true }) } app.routes = { ... '/data/admins/{name}/': {type: 'implicit', id: 'admins'}, '/data/admins/': {type: 'implicit', id: 'admins.plural'} }
As a convenience, you can also add custom values to be cast using the "values" key. These will be merged with values from conversation.locals:
new Diligence.REST.MongoDbResource({ name: 'users', query: {name: '{name}'}, {type: '{type}'}}, values: {type: 'admin'} })
This allows for nice reusability when you create your own extended classes: you can share one query among many subclasses.
Custom Extraction
By default, the REST Service will extract and return the entire MongoDB document, but you can customize this quite powerfully, even to allow you to access sub-documents inside a document.
First off, you can simply choose the fields you want:
new Diligence.REST.MongoDbResource({ name: 'users', fields: ['name', 'email', 'address'] })
The "fields" key will be used at the level of MongoDB's driver, so that unused data won't even be retrieved from the database.
You can go further and extract sub-fields:
resources = { ... 'users.email': new Diligence.REST.MongoDbResource({ name: 'users', fields: 'email', extract: 'email' } }) app.routes = { ... '/data/users/{id}/email': {type: 'implicit', id: 'users.email'}, }
The result of a GET would be only a string of the email address. An example in JSON:
"myemail@mail.org"
Without the "extract", the representation would be this:
{ "_id": { "$oid": "4e057e94e799a23b0f581d7d" }, "email": "myemail@mail.org" }
Important! Not all client JSON parsers can deal with JSON data that is not a dict or an array. If you are extracting data that is not a dict or an array, you may need to implement your own special parsing.
With "extract" you can go further and even provide an array that will be extracted in order. For example:
resources = { ... 'users.groups': new Diligence.REST.MongoDbResource({ name: 'users', fields: 'authorization', extract: ['authorization', 'entities'] } }) app.routes = { ... '/data/users/{id}/groups': {type: 'implicit', id: 'users.groups'}, }
The above actually uses the data structure used by Diligence's Authorization Service to retrieve the security groups. The result of a GET would be an array. An example in JSON:
["users", "admins"]
Finally, you can do your own custom extraction, by providing a function:
new Diligence.REST.MongoDbResource({ name: 'users', fields: 'authorization', extract: function(doc) { return doc.authorization.entities.join(',') } })
Custom Modes
You can set up your own custom modes like so:
new Diligence.REST.MongoDbResource({ name: 'users', modes: { flat: function(data) { return Sincerity.Objects.flatten(data) } } })
See "Usage" below for information on how to use modes.
Overriding
There are two ways to override the default behavior: 1) inherit the Diligence.MongoDbResource class using the Sincerity.Classes API, or 2) monkey-patch the instances. The former method is more reusable, but the latter method works just as well and is easier if you just need to customize a single resource. Example of monkey-patching:
resources = { ... users: new Diligence.REST.MongoDbResource({name: 'users'}) } resources.users.doDelete = function(conversation) { ... // Call overridden method arguments.callee.overridden.call(this, conversation) }
Using this method you can even monkey-patch instances created automatically after a call to "Diligence.REST.createMongoDbResources()".
In-Memory Data
The REST Service does not have to use MongoDB to store data: it also supports storing data in memory, even shared memory distributed in the Prudence cluster.
This is useful if you don't need persistent storage in MongoDB (the data is considered volatile) and is also useful for creating mock data for testing. The URI-space otherwise behaves exactly the same as if it were attached to MongoDB collections. Performance, of course, should be better than if you were accessing MongoDB. On the other, your storage size is limited to your RAM. So, while this feature is not a replacement for using MongoDB, it can be quite useful in various scenarios.
Let's modify our example from above to use in-memory resources:
document.executeOnce('/sincerity/jvm/') var users = { '4e057e94e799a23b0f581d7d': { _id: '4e057e94e799a23b0f581d7d', name: 'newton', lastSeen: new Date() }, '4e057e94e799a23b0f581d7e': { _id: '4e057e94e799a23b0f581d7e', name: 'sagan', lastSeen: new Date() } } var usersMap = Sincerity.JVM.toMap(users, true) resources = { ... users: new Diligence.REST.InMemoryResource({name: 'users', documents: usersMap}), 'users.plural': new Diligence.REST.InMemoryResource({name: 'users', documents: usersMap, plural: true}) }
Note that we translated the "users" dict into a thread-safe JVM map. We could have also just sent the "users" dict directly to the "InMemoryResource" constructor, which can create the map for us. But, since we have two resources, the singular and the plural, and we want them to share the same map, we have created this map ourselves.
What if you're in a Prudence cluster, and want all nodes to share the same in-memory data? Let's modify our code:
resources = { ... users: new Diligence.REST.DistributedResource({name: 'users', documents: users}), 'users.plural': new Diligence.REST.DistributedResource({name: 'users', documents: users, plural: true}) }
The code is even simpler than the "InMemoryResource" code (no need to create "usersMap"), but requires some explanation:
- The "name" field will be used as the name of the Hazelcast map. You can configure this map by name in the Hazelcast configuration, otherwise it will use the Hazelcast defaults for new maps.
- The data from the "documents" field will be copied into the Hazelcast only once and only if the map is already empty. Thus, it should be thought of as your initialization data: the first time a resource is set up for that map, from anywhere in the cluster, this data will be copied in. From then on, for the life of the cluster, "documents" will be ignored. Thus, if you want to re-initialize the map, you will need to either restart your whole cluster, or programmatically set the data. (The Diligence Console would be very useful for that.)
- Note that we are serializing data using JSON into the distributed map. The performance hit should be minimal, but it's important to remember that only your data must be extended-JSON-compatible. (The "InMemoryResource" doesn't have this restriction.)
Usage
Resource Characteristics
All resources support the following URI query parameters:
- format: You can use this to specify the exact format you want, overriding any HTTP content negotiation. This is useful for testing and debugging, but can also help you in dealing with HTTP clients that can't easily set headers. Accepted values are "json", "xml" and "html". Note that when accessing resources internally, no serialization happens, and "format" is unnecessary.
- human: Setting this to "true" will further help your debugging, as it will return nicely indented, multiline JSON or XML representations.
-
mode: "Modes" are simple functions that are applied to all documents in order to transform the final representation. The REST Service comes with a few useful modes, but you can easily create your own, just make sure to hook them to the instance using the "modes" key. The query parameter value will be mapped to a key in this dict. Note that you can provide multiple "mode" values, in which case all mode functions will be called in order. Provided modes:
- primitive: This converts MongoDB extended values into simpler JSON structures. For example, "{timestamp: {$date: 12345}}" will become "{timestamp: 12345}".
- string: This converts all JSON values into strings. It's a good way to overcome various number accuracy issues, especially when dealing with PHP clients.
- stringid: Converts only the "_id" field to a string, in case it's a BSON ObjectId. Some clients, such as Ext JS, cannot deal with ID values that are dicts.
An example URI with all the above parameters:
/data/users/4e057e94e799a23b0f581d7d/?format=json&human=true&mode=primitive&mode=string
As for payloads, in POST and PUT operations, note that by default they must be in JSON, even if you are representing the result in XML or HTML. The reason is that there is no obvious way to translate XML to the final JSON format needed by MongoDB. If you do need to support XML payloads, you can override "handlePost" and "handlePut" to do this yourself according to your specifications.
Singular Resources
The REST Service will by default extract the "{id}" pattern in the URI into a MongoDB ObjectID for the document "_id" field. For example, if your route is "/data/users/{id}/", then "/data/users/4e057e94e799a23b0f581d7d/" would refer to the user document with that "_id."
Requests to the URI always return 404 if the document does not exist. Further notes:
- POST: All keys of the payload will be used for a "$set" in a MongoDB "findAndModify" operation, and the modified document will be returned. If you include an "_id" key in the payload it will be removed, because the ID in the URI takes precedence.
-
PUT: The payload will become a simple MongoDB "save" operation, which is an upsert, meaning it would either create a new resource or replace the existing one. If you include an "_id" key in the payload it will be removed, because the ID in the URI takes precedence. Note that if you want to create a new resource, it's up to you to make sure the the id is unique, otherwise you will get an HTTP 409 error (conflict). You can generate a unique ID by calling MongoDB.newId(). Example for generating a unique URI using templates:
'/data/users/{0}/'.cast(MongoDB.newId())
Plural Resources
The plural resource is a bit more complex. The returned representations include a "total" key, counting the size of the collection, and a "documents" key, containing an array of specific documents. For example:
{ "total": 1092, "documents": [ {"_id": {"$oid": "4e057f2ae799a23b0f581d7f" }, ... } ... ] }
The following additional query parameters are supported for pagination, controlling which documents are included in the "documents" array:
- start: The index from which to start collecting documents. By default it will be 0.
- limit: The maximum number of documents to return.
The "documents" array can definitely be empty if your "start" and "limit" values are not satisfied.
Further notes:
- POST: This lets you update many documents at once. Your payload should be an array of values that would be sent via the singular resource POST, as described above, however you must also include an "_id" for each value. The response will include all documents after their modification.
- PUT: This is how you add documents to your MongoDB collection. Simply provide an array of values, and they will become MongoDB "insert" operations. The response will include "_id" fields on all your documents, if you did not set them yourself.
- DELETE: This is a MongoDB "remove" operation, not a "drop".
Accessing Your Resources over the Web
All your resources support the HTML format, so you can easily access them via a web browser. For example, this link: http://localhost:8080/diligence-example/data/users/4e057e94e799a23b0f581d7d/.
This view supports simple editing of your resources: you can POST, PUT any resource using JSON or XML payloads, or DELETE them. It's a great way to test and debug your resources.
You can customize this view as you please: just create "/diligence/service/rest/singular.html" and "/diligence/service/rest/plural.html" files in your "/fragments/" directory. You can start with the default files under your container's "/libraries/prudence/" directory as a template.
Accessing Your Resources with the API
The Prudence.Resources API makes it very easy to access your resources, whether internally or on a different node. See the API documentation for full details, otherwise here we'll provide you with a quick tutorial for using it with the REST Service.
Let's start with the internal use case:
document.executeOnce('/prudence/resources/') var user = Prudence.Resources.request({ uri: '/data/users/4e057e94e799a23b0f581d7d/', internal: true }) print(user.name)
Again, we'll emphasize that when accessing the API internally neither HTTP nor serialization are involved. The data is never converted to JSON, instead it's extracted directly from MongoDB's BSON to JavaScript's internal data structure, exactly as if you were using the MongoDB API directly. There's obviously some overhead added by the Prudence platform and the REST Service, but it should be very minimal, especially when compared to the network fetch from MongoDB. In short, performance concerns should not stop you from using the REST Service in this fashion.
Accessing remote resources is almost identical, though obviously HTTP and JSON (or XML) are involved. As an example, we can try to access our local resource via HTTP:
var user = Prudence.Resources.request({ uri: 'http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/', mediaType: 'application/json' }) print(user.name)
Of course, the URI can point to anywhere on the network, or the Internet. Note that we had to explicitly specify our preferred media type, because our resource supports several different formats.
The API can be used for all REST methods:
var user = Prudence.Resources.request({ uri: '/data/users/4e057e94e799a23b0f581d7d/', internal: true, method: 'post', payload: { value: {email: 'newemail@mysite.org'} } })
Remotely, the REST methods are actual HTTP verbs:
var user = Prudence.Resources.request({ uri: 'http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/', mediaType: 'application/json' method: 'post', payload: { type: 'json', value: {email: 'newemail@mysite.org'} } })
We'll finish off this short tutorial by showing you that for every request you can also set query params:
var users = Prudence.Resources.request({ uri: '/data/users/', internal: true, query: { start: 5, limit: 3 } }) print(users[0].name)
Accessing Your Resources with cURL
cURL is an HTTP command line tool based on the cURL library, available for a great many Unix-like operating systems as well as Windows. It's especially useful for testing RESTful APIs. Here's a quick tutorial to get you started with using cURL with the REST Service.
First, a few GET commands to try:
curl "http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/?human=true" curl "http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/?format=xml&human=true" curl "http://localhost:8080/myapp/data/users/?limit=3&human=true"
You can send a payload using the "-d" switch, which also sets the HTTP verb to POST. For example, this will modify the email of a user:
curl -d '{"email":"newemail@mysite.org"}' "http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/?human=true"
When using "-d", you can also start your payload with "@" to signify that you want to send the contents of a file, in this case "data.json":
curl -d @data.json "http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/?human=true"
To set the HTTP verb explicitly, use "-X". Here we'll create a new user:
curl -X PUT -d @data.json "http://localhost:8080/myapp/data/users/?human=true"
And now we'll delete a user:
curl -X DELETE "http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/"
With the "-h" switch, you can also send HTTP headers in raw form:
curl -H "Accept: application/xml" "http://localhost:8080/myapp/data/users/4e057e94e799a23b0f581d7d/?human=true"
Finally, add the "-v" switch to print out the outgoing and incoming headers.
Extension
TODO
Extended MongoDbResource
Extending IterableResource
The Diligence Manual is provided for you under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The complete manual is available for download as a PDF.