Web Data

This chapter deals with sending and receiving data to and from the client (as well as external servers) via REST, focusing especially on the particulars for HTTP and HTML. It does not deal with storing data in backend databases.

Prudence is a minimalist RESTful platform, not a data-driven web framework, though such frameworks are built on top of it. Check out our Diligence, which is a full-blown framework based on Prudence and MongoDB. You may also be interested in the Model-View-Controller (MVC) chapter, which guides you through an approach to integrating data backends.

URLs

The simplest way in which a client sends data to the server is via the URL. The main part of the URL is parsed by Prudence and used for routing, but some of it is left for your own uses.

Generally, the whole or parts of the request URL can be accessed via the conversation.reference API.

Query Parameters

This is the matrix of parameters after the "?" in the URI.

For example, consider this URL:

http://mysite.org/myapp/user?name=Albert%20Einstein&enabled=true

Note the "%20" URI encoding for the space character. Query params will be automatically decoded by Prudence.

In JavaScript, you can use Prudence.Resources.getQuery API:

document.require('/prudence/resources/')

var query = Prudence.Resources.getQuery(conversation, {
	name: 'string',
	enabled: 'bool'
})

In the case of multiple params with the same name, the API would return the first param that matches the name. Otherwise, you can also retrieve all values into an array:

var query = Prudence.Resources.getQuery(conversation, {
	name: 'string[]',
	enabled: 'bool'
})

Low Level

For non-JavaScript you can use the lower-level conversation.query API:

var query = {
	name: conversation.query.get('name'),
	enabled: conversation.query.get('enabled') == 'true'
}

Use conversation.queryAll if you need to find multiple params with the same name.

Captured Segments

Variables in the URI template you configured in routing.js will be captured into conversation.locals. Note that you can also interpolate the captured variables into the target URI.

The Wildcard

If you've configured a URI template with a wildcard in routing.js, you can access the "*" value using conversation.wildcard. Note that you can also interpolate the wildcard into the target URI.

Fragments

This is whatever appears after the "#" in the URI. Note that for the web fragments are only used for response URLs: those sent from the server to the client. This is enforced: web browsers will normally strip fragments before sending URLs to the server, but the server can send them to web browsers. They are commonly used in HTML anchors:

<a name="top" /><h1>This is the top!</h1>
<p>Click <a href="#top">here</a> to go to the top</p>

But you can also use them in redirects:

conversation.redirectSeeOther(conversation.base + '#top')

Request Payloads

These are used in "POST" and "PUT" verbs.

In JavaScript, you can use Prudence.Resources.getEntity API to extract the data in various formats:

document.require('/prudence/resources/')

var data = Prudence.Resources.getEntity(conversation, 'json')

Otherwise, you can use the lower-level conversation.entity API:

document.require('/sincerity/json/')

var text = conversation.entity.text
var data = Sincerity.JSON.from(text)

Note that if the payload comes from a HTML "post" form, better APIs are available.

MIME Types

If you wish to support multiple request payload MIME types, be sure to check before retrieving:

var type = conversation.entity.mediaType.name
if (type == 'application/json') {
	var data = Prudence.Resources.getEntity(conversation, 'json')
	...
} else if (type == 'image/png') {
	var data = Prudence.Resources.getEntity(conversation, 'binary')
	...
}

Consumption

Note that you can only retrieve the request payload once. Once the data stream is consumed, its data resources are released. Thus, the following would result in an error:

print(conversation.entity.text)
print(conversation.entity.text)

The simple solution is retrieve once and store in a variable:

var text = conversation.entity.text
print(text)
print(text)

Parsing Formats

Which formats can Prudence parse, and how well?

This depends on which programming language you're using: for example, both Python and Ruby both come with basic JSON support in their standard libraries, and Python supports XML, as well. Sincerity provides JavaScript with support for both. Of course, you can install libraries that handle these and other formats, and even use JVM libraries.

For other formats, you may indeed need to add other libraries.

A decent starting point is Restlet's ecosystem of extensions, which can handle several data formats and conversions. However, these are likely more useful in pure Java Restlet programming, where they can plug into Restlet's sophisticated annotation-based conversion system. In Prudence, you will usually be applying any generic parsing library to the raw textual or binary data. Still, the Restlet extensions are useful for response payloads.

Cookies

Cookies represent a small client-side database, which the server can use to retrieve or store per-client data. Not all clients support cookies, and even those that do (most web browsers) might have the feature disabled, so it's not always a good idea to rely on cookies.

From the Client

Retrieve a specific cookie from those the client sent you according to its name using conversation.getCookie:

var session = conversation.getCookie('session')
if (null !== session) {
	print(session.value)
}

Or use conversation.cookies to iterate through all available cookies.

The following attributes are available:

name: (read only)
version: (integer) per a specific cookie
value: textual, or text-encoded binary data (note that most clients have strict limits on how much total data is allowed to be stored in all cookies per domain)
domain: the client should only use the cookie with this domain and its subdomains (web browsers will not let you set a cookie for a domain which is not the domain of the request or a subdomain of it)
path: the client should only use the cookie with URIs that begin with this path ("/", the default, would mean to use it with all URIs)

To the Client

You can ask that a client modify any of the cookies you've retrieved from it, upon a successful response, by calling the "save" method:

var session = conversation.getCookie('session')
if (null !== session) {
	session.value = 'newsession'
	session.save()
}

Ask the client to create a new cookie using conversation.createCookie:

var session = conversation.createCookie('session')
session.value = 'newsession'
session.save()

Note that createCookie will retrieve the cookie if it already exists.

When sending cookies to the client, you can set the following attributes in addition to those mentioned above, but note that you cannot retrieve them later:

maxAge: age in seconds, after which the client should delete the cookie; maxAge=0 deletes the cookie immediately, while maxAge=-1 (the default) asks the client to keep the cookie only for the duration of the "session" (this is defined by the client; for most web browsers this means that the cookie will be deleted when the browser is closed)
secure: true if the cookie is meant to be used only in secure connections (defaults to false)
accessRestricted: true if the cookie is meant to be used only in authenticated connections (defaults to false)
comment: some clients store this, some discard it

You can ask the client to delete a cookie by calling its "remove" method. This is identical to setting maxAge=0 and calling "save".

Security Concerns

You should never store any unencrypted secret data in cookies: though web browsers attempt to "sandbox" cookies, making sure that only the server ("domain") that stored them can retrieve them, they can be hijacked by other means. Better yet, don't store any secrets in cookies, even if encrypted, because even encryptions can be hacked. A cautious exception can be made for short-term secrets: for example, if you store a session ID in a cookie, make sure to expire it on the server so that it cannot be used later by a hacker.

A separate security concern for users is that cookies can be used to surreptitiously track user activity. This works because any resource on a web page—even an image hosted by an advertising company—can use cookies, and can also track your client's IP address. Using various heuristics it is possible to identify individual users and track parts of their browser history.

Because of these security concerns, it is recommended that you devise a "cookie policy" for users and make it public, assuming you require the use of cookies for your site. In particular, let users know which 3rd-party resources you are including in your web pages that may be storing cookies, and for what purpose.

Cookies are a security concern for you, too: you cannot expect all your clients to be standard, friendly web browsers. Clients might not be honoring your requests for cookie modifications, and might be sending you cookies that you did not ask them to store.

Be careful with cookies! They are a hacker's playground.

Custom Headers

The most commonly used request and response HTTP headers are supported by Prudence's standard APIs. For example: conversation.disposition, conversation.maxAge and conversation.client. However, Prudence also let's you use other headers, including your custom headers, via conversation.requestHeaders and conversation.responseHeaders. Note that you must use Prudence's standard APIs for headers if such exist: these APIs will only work for additional headers.

These APIs both return a Series object. (You usually won't need to access the elements directly, but in case you do: they are Header objects.) An example of fetching a request header:

var host = conversation.requestHeaders.getFirstValue('Host')

An example of setting a response header:

conversation.responseHeaders.set('X-Pingback', 'http://mysite.org/pingback/')

Redirection

Client-side redirection in HTTP is handled via response headers.

By Routing

If you need to constantly redirect a specific resource or a URI template, you should configure it in your routing.js, using the "redirect" route type:

app.routes = {
	...
	'/images/*': '>/media/{rw}'
}

Note that in this example we interpolated the wildcard.

By API

You can also redirect programmatically by using the conversation.redirectPermament, conversation.redirectSeeOther or conversation.redirectTemporary APIs:

conversation.redirectSeeOther(conversation.base + '/help/')

Note that if you redirect via API, the client will ignore the response payload if there is one.

In HTML

We're mentioning this here only for completion: via HTML, redirection is handled entirely in the web browser, with no data going to/from the server. A template resource example:

Go <a href="<%.%>/elsewhere/">elsewhere</a>.

Server-Side Redirection

In Prudence, this is called "capturing" and has particular use cases. (It can indeed be confusing that this functionality is often grouped together with client-side redirection.)

HTML Forms

HTML's "form" tag works in two very different modes, depending on the value of its "method" param:

"get": The form fields are all turned into query params and appended to the "action" URL. It is important to remember that an HTTP "GET" is idempotent and should not be used to store new data, but rather as a way to represent existing data in a particular way. Actually, "get" forms are not that useful, and are mostly an odd legacy from the early days of the World Wide Web, where "GET" was the only the supported HTTP verb on some platforms. See query parameters for handling.
"post": The form fields are actually encoded in the same way that query params are, but instead of being affixed to the URL, they are sent as the payload with an "application/x-www-form-urlencoded" MIME type. Though you can of course access this payload directly, it is recommended to use the specialized APIs detailed here.

Example form:

<form action="<%.%>/user/" method="post">
	<p>Name: <input type="text" name="name"></p>
	<p>Enabled: <input type="radio" name="enabled" value="true"></p>
	<p>Disabled: <input type="radio" name="enabled" value="false"></p>
	<p><button type="submit">Send</button></p>
</form>

In JavaScript, you can use Prudence.Resources.getForm API:

document.require('/prudence/resources/')

var form = Prudence.Resources.getForm(conversation, {
	name: 'string',
	enabled: 'bool'
})

In the case of multiple fields with the same name, the API would return the first fields that matches the name. Otherwise, you can also retrieve all values into an array:

var form = Prudence.Resources.getForm(conversation, {
	name: 'string[]',
	enabled: 'bool'
})

Low Level

For non-JavaScript you can use the lower-level conversation.form API family:

var form = {
	name: conversation.form.get('name'),
	enabled: conversation.form.get('enabled') == 'true'
}

Use conversation.formAll if you need to find multiple fields with the same name.

Accepting Uploads

HTML supports file uploads using forms and the "file" input type. However, the default "application/x-www-form-urlencoded" MIME type for forms will not be able to encode files, so you must change it to "multipart/form-data". For example:

<form action="<%.%>/user/" method="post" enctype="multipart/form-data">
	<p>Name: <input type="text" name="name"></p>
	<p>Upload your avatar (an image file): <input name="avatar" type="file" /></p>
	<p><button type="submit">Send</button></p>
</form>

Prudence has flexible support for handling uploads: you can configure them to be stored in memory, or to disk. See the application configuration guide.

You can access the uploaded data using the conversation.form API family. Here's a rather sophisticated example for displaying the uploaded file to the user:

<%
var name = conversation.form.get(name')
var tmpAvatar = conversation.form.get('avatar').file

// The metadata service can provide us with a default extension for the media type
var mediaType = conversation.form.get('avatar').mediaType
var extension = application.application.metadataService.getExtension(mediaType)

// We will put all avatars under the "/resources/avatars/" directory, so that they
// can be visible to the world
var avatars = new File(document.source.basePath, 'avatars')
avatars.mkdirs()
var avatar = new File(avatars, name + '.' + extension)

// Move the file to the new location
tmpAvatar.renameTo(avatar)
%>
<p>Here's the avatar you uploaded, <%= name %></p>
<img src="<%.%>/avatars/<%= avatar.name %>" />

Response Payloads

This section is mostly applicable to manual resources, although it can prove useful to affect the textual payloads of template resources. For static resources, the response payloads are of course the contents of the resource files.

Two important notes:

Prudence can automatically cache your response payloads. Upon a successful cache hit, Prudence will in fact bypass execution of (most of) your code.
If you are using your resources internally, it's possible to improve performance by avoiding serialization.

Textual and Binary Payloads

Template resources might seem to always return textual payloads. Actually, by default they will negotiate a compression format, which if selected will result in a binary: the compressed version of the text. But all of that is handled automatically by Prudence for that highly-optimized use case.

For manual resources, you can return any arbitrary payload by simply returning a value in handleGet, handlePost or handlePut. Both strings and JVM byte arrays are supported. A textual example:

function handleGet(conversation) {
	return 'My payload'
}

A binary example:

document.require('/sincerity/jvm/')

function handleGet(conversation) {
	var payload = Sincerity.JVM.newArray(10, 'byte')
	for (var i = 0; i < 10; i++) {
		payload[i] = i
	}
	return payload
}

Note that if you return a number, it will be treated specially as an HTTP status code. If you wish to return the number as the content of a textual payload, simply convert it to a string:

function handleGet(conversation) {
	return String(404)
}

If you wish to set both the payload and the status code, use an API for either one. Here well use the conversation.status API family. Note that if your status code is an error status code, you'll also want to bypass this error page using conversation.statusPassthrough:

function handleGet(conversation) {
	conversation.statusCode = 404
	conversation.statusPassthrough = true
	return 'Not found!'
}

Alternatively, we can use the conversation.setResponseText or conversation.setResponseBinary:

function handleGet(conversation) {
	conversation.setResponseText('Not found!', null, null, null)
	conversation.statusPassthrough
	return 404
}

Binary Streaming

Streaming using background tasks is not directly supported by Prudence as of version 2.0. However, this feature is planned for a future version, depending on support being added to Restlet.

Restlet Data Extensions

Instead of returning a string or a byte array, you can return an instance of any class inheriting from Representation. Restlet comes with a few basic classes to get you started. Here's a rather boring example:

function handleGet(conversation) {
	return new org.restlet.representation.StringRepresentation('My payload')
}

Where Restlet really shines is in its ecosystem of extensions, which can handle several data formats and conversions. For these extensions to work, you will need to install the appropriate library in your container's "/libraries/jars/" directory, as well as all dependent libraries. Please refer to the Restlet distribution for complete details.

Note that you can also set the response via the conversation.response.entity API:

var payload = new org.restlet.representation.StringRepresentation('My payload')
conversation.response.entity = payload

Or via the conversation.setResponseText API shortcut:

conversation.setResponseText('My payload', null, null, null)

Overriding the Negotiated Format

The response payload's MIME type and language have likely been selected for you automatically by Prudence, via HTTP content negotiation, based on the list of preferences you set up in handleInit. However, it's possible to override these values via the conversation.mediaType and conversation.language API families. This should be done sparingly: content negotiation is the preferred RESTful mechanism for determining the response format, and the negotiated values should be honored. However, it could be useful and even necessary to override it if you cannot use content negotiation, which might be the case if your clients don't support it, and yet you still want to support multiple formats.

In this example, we'll allow a "format=html" query param to override the negotiated MIME type:

function handleInit(conversation) {
	conversation.addMediaTypeByName('text/html')
	conversation.addMediaTypeByName('text/plain')
}

function handleGet(conversation) {
	if (conversation.query.get('format') == 'html') {
		conversation.mediaTypeName = 'text/html'
	}
	return conversation.mediaTypeName == 'text/html' ?
		'<html><body>My page</body></html>' :
		'My page'
}

An example of overriding the negotiated language:

function handleInit(conversation) {
	conversation.addMediaTypeByNameWithLanguage('text/html', 'en')
	conversation.addMediaTypeByNameWithLanguage('text/html', 'fr')
}

function handleGet(conversation) {
	if (conversation.query.get('language') == 'fr') {
		conversation.languageName = 'fr'
	}
	if (conversation.languageName == 'fr') {
		...
	}
	else {
		...
	}
}

Note that these APIs works just as well for template resources, though again content negotiation should be preferred.

Under the Hood

When the MIME type is "application/internal", Prudence is actually wrapping your return value in an InternalRepresentation. You can also construct it explicitly:

return new com.threecrickets.prudence.util.InternalRepresentation(data)

Note that, of course, if you return an instance of a class inheriting from Representation, Prudence will detect this and not wrap it again in an InternalRepresentation.

Browser Downloads

You can create browser-friendly downloadable responses using the conversation.disposition API. Here's an example using a manual resource:

function handleInit(conversation) {
	conversation.addMediaTypeByName('text/csv')
}

function handleGet(conversation) {
	var csv = 'Item,Cost,Sold,Profit\n'
	csv += 'Keyboard,$10.00,$16.00,$6.00\n'
	csv += 'Monitor,$80.00,$120.00,$40.00\n'
	csv += 'Mouse,$5.00,$7.00,$2.00\n'
	csv += ',,Total,$48.00\n'
	conversation.disposition.type = 'attachment'
	conversation.disposition.filename = 'bill.csv'
	return csv
}

Most web browsers would recognize the MIME type and ask the user if they would prefer to either download the file with the suggested "bill.csv" filename, or open it in a supporting application, such as a spreadsheet editor.

Note that the disposition is not cached. If you wish to use this feature, you need to disable caching on the particular resource.

External Requests

Prudence uses the Restlet library to serve RESTful resources, but can also use it to consume them. In fact, the client API nicely mirrors the server API.

Note that Prudence can also handle internal REST requests without going through HTTP or object serialization. There is an entire internal URI-space at your fingertips.

It's not a good idea to send an external request while handling a user request, because it could potentially cause a long delay and hold up the user thread. It would be better to use a background task. A possible exception is requests to servers that you control yourself, and that represent a subsystem of your application. In that case, you should still use short timeouts and fail quickly and gracefully.

For our examples, let's get information about the weather on Mars from MAAS.

In JavaScript, you can use the powerful Prudence.Resources.request API:

document.require('/prudence/resources/')
var weather = Prudence.Resources.request({
	uri: 'http://marsweather.ingenology.com/v1/latest/',
	mediaType: 'application/json'
})
if (null !== weather) {
	print('The max temperature on Mars today is ' + weather.report.max_temp + ' degrees')
}

The API will automatically convert the response according to the media type. In this case, we requested "application/json", so the textual response will be converted from JSON to JavaScript native data. The API will also automatically follow redirects.

Payloads sent to the server, for the "POST" and "PUT" verbs, are also automatically converted:

var newUser = Prudence.Resources.request({
	uri: 'http://mysite.org/user/newton/',
	method: 'put',
	mediaType: 'application/json',
	payload: {
		type: 'json',
		value: {
			name: 'Isaac',
			nicknames: ['Izzy', 'Zacky', 'Sir']
		}
	}
})

Read the API documentation carefully, as it supports many useful parameters.

Low Level

For non-JavaScript you can use the lower-level document.external API:

document.require('/sincerity/json/')
var resource = document.external('http://marsweather.ingenology.com/v1/latest/', 'application/json')
result = resource.get()
if (null !== result) {
	weather = Sincerity.JSON.from(result.text)
	print('The max temperature on Mars today is ' + weather.report.max_temp + ' degrees')
}

Timeout

Surprisingly, you cannot set the timeout per request, but instead you need to configure the timeout globally for the HTTP client. This is due to a limitation in Restlet that may be fixed in the future.

Secure Requests

These APIs support secure requests to "https:" servers. Such requests rely on the JVM's built-in authorization mechanism. Like most web browsers, the JVM recognizes the common Internet certificate authorities. This means that if you're using your own self-created keys, that don't use an approved certificate, you need to specify these keys via a "trust store" for the JVM. For an example, see secure servers in the configuration chapter.

RESTful Files

The same APIs can be used to easily access resources via the "file:" pseudo-protocol. Let's read a JSON file:

var data = Prudence.Resources.request({
	file: '/tmp/weather.json',
	mediaType: 'application/json'
})

The above is simply a shortcut to this:

var data = Prudence.Resources.request({
	uri: 'file:///tmp/weather.json',
	mediaType: 'application/json'
})

You can even "PUT" new file data, and "DELETE" files using this API.

The Prudence Manual is provided for you under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The complete manual is available for download as a PDF.

PrudenceScalable REST PlatformFor the JVM

Web Data

URLs

Query Parameters

Captured Segments

The Wildcard

Fragments

Request Payloads

MIME Types

Consumption

Parsing Formats

Cookies

From the Client

To the Client

Security Concerns

Custom Headers

Redirection

By Routing

By API

In HTML

Server-Side Redirection

HTML Forms

Accepting Uploads

Response Payloads

Textual and Binary Payloads

Restlet Data Extensions

Overriding the Negotiated Format

Browser Downloads

External Requests

Timeout

Secure Requests

RESTful Files

Prudence
Scalable REST Platform
For the JVM