Savory's Iterators Library

Iterators are used to traverse potentially very large series of entries without advance knowledge of how many entries are in the series. These are somewhat similar to JavaScript iterators, but more generic. A wrapper (Iterators.Generator) exists to let you use JavaScript iterators here.

Generally, any code that expects to traverse arrays can be made more scalable if iterators are used instead (and Iterators.Array lets you easily consume arrays). The problem with arrays is that all entries exist in memory at once: so that 1) the can take too much memory at once, and 2) they can waste valuable resources as we fill in the array entirely up-front, when we might only want to traverse a subset of entries.

The iterator API has been intentionally designed for compatibility with MongoDB.Cursor, so that a MongoDB cursor can be used anywhere an iterator is expected. For example, you can wrap a MongoDB cursor in a Iterators.Transfomer, include it in a Iterators.Chain, etc.

You must always close an iterator. The recommended semantics:

var iterator = ...
try {
	while (iterator.hasNext()) {
		var next = iterator.next()
	}
}
finally {
	iterator.close()
}

All the wrappers in the iterators library know to automatically close all the iterators they wrap, so you only every have to close the outermost iterator you are using.

An easy way to get a segment of an iterator's entries is to collapse it into an array. Just remember that the array should have a relatively small, manageable size, otherwise you're losing the advantages of iterators and can also risk running out of heap memory. Example:

var start = 70000
var limit = 500000
limit = Math.min(limit, 10000) // sensible max
var entries = Savory.Iterators.toArray(iterator, start, limit)

You can create your own iterator objects and have them join the fun as long as they agree with that protocol. Here are a bunch that come with Savory.

Refer to the Savory.Iterators API documentation for more details.

Array Wrapper

var	i1 = new Savory.Iterators.Array(['grapefruit', 'watermelon', 'durian'])

JVM Wrapper

var list = Savory.JVM.newList()
list.add('antelope')
list.add('zebra')
var i2 = new Savory.Iterators.JVM(list.iterator())

JavaScript Generator Function Wrapper

function serials() {
	for (var i = 0; i < 10; i++) {
		yield i
	}
}
var i3 = new Savory.Iterators.Generator(serials())

Fetcher Function Wrapper

function getNextThousand(options, index) {
	if (index == 5) {
		options.hasNext = false
		return
	}
	return (index + 1) * 1000
}
var i4 = new Savory.Iterators.Fetcher(getNextThousand)

Plain-Old MongoDB Cursors

var programs = new MongoDB.Collection('programs')
var i5 = programs.find({})

Iterator Chain

var i6 = new Savory.Iterators.Chain([i1, i2, i3, i4, i5])

Transformer

function putInBed(entry) {
	if (typeof entry == 'string') {
		return entry + ' in bed'
	}
	return entry
}
var i7 = new Savory.Iterators.Transformer(i6, putInBed)

Eliminator

function noOddNumbers(entry) {
	if ((typeof entry == 'number') && (entry % 2 == 1)) {
		return false
	}
	return true
}
var i8 = new Savory.Iterators.Eliminator(i7, noOddNumbers)

Buffer

var i9 = new Savory.Iterators.Buffer(i8, 5)