Lazy, composable, and modular JavaScript
ECMAScript 6, or the 6th edition of ECMA-262 standard, gives JavaScript developers new tools for writing more succinct and modular code. In this article, I’ll cover how we can use four of these features – iterables, generators, fat arrows, and for-of – in conjunction with higher-order functions, function composition, and lazy evaluation, to write cleaner and more modular JavaScript.
Before we dive in to a larger example, we’ll review some more general concepts.
Higher-order functions
A higher-order function is a normal function that satisfies at least one of the following conditions:
- It accepts one or more functions as parameters
- It returns a function
If you’ve ever written an event listener or used Array.prototype.map, then you’ve used a higher-order function.
Higher-order functions promote reusability and modularity by decoupling how an operation is applied from the operation itself.
For example, the function we pass to Array.prototype.map has no knowledge of collections or how to operate on them. All it knows is how to operate on a value. Therefore it can be reused in the context of both single values and collections.
Function composition
Function composition is the combination of simple functions to build more complicated ones. Given two functions f and g, compose(f, g) gives us a new function that returns f(g(x)). Also, since the result of composition is a function, it can be further composed or be passed to higher-order functions.
Let’s look at an example. We are tasked with writing a program that takes a file as input and returns an array of the number of vowel occurrences in each word on each line. There are many approaches to this problem. One such approach is to write a single big function that does it all.
The monolithic solution below uses the new for-of construct to loop over values in an iterable instead of the usual for loop. An iterable is a container that implements the iterator protocol and can yield values one by one (example: arrays, strings, generators, etc.).
function vowelCount(file) {
let contents = readFile(file)
let lines = contents.split('\n') // split content into array of lines
let result = [] // an array of arrays where each index maps to a line
// and each index within the inner array maps to the
// vowel count for a word on that line
for (let line of lines) {
let temp = []
let words = line.split(/\s+/)
for (let word of words) {
let vowelCount = 0
for (let char of word) {
if (
'a' === char || 'e' === char || 'i' === char || 'o' === char || 'u' === char
) vowelCount++
}
temp.push(vowelCount)
}
result.push(temp)
}
return result
}
The solution above is not extendable, not scalable and gives us no reusable components. An alternate approach is to use higher-order functions and composition.
// main.js
function vowelOccurrences(file) {
return map(words => map(vowelCount, words), listOfWordsByLine(read(file)))
}
function vowelCount(word) {
return reduce((prev, char) => {
if (
'a' === char || 'e' === char || 'i' === char || 'o' === char || 'u' === char
) return ++prev
else return prev
}, 0, word)
}
function listOfWordsByLine(string) {
return map(line => split(/\s+/, line), split('\n', string))
}
// reusable utils in util.js
function reduce(fn, accumulator, list) {
return [].reduce.call(list, fn, accumulator)
}
function map(fn, list) {
return [].map.call(list, fn)
}
function split(splitOn, string) {
return string.split(splitOn)
}
listOfWordsByLine returns an array of arrays where each element corresponds to an array of words that make up a line. For example:
let input = 'line one\nline two'
listOfWordsByLine(input) // [['line','one'],['line','two']]
In the code above, vowelCount counts the number of vowels in a word. vowelOccurrences uses vowelCount on the output of listOfWordsByLine to calculate the vowel count per line per word.
The second approach results in a few reusable functions that we can employ throughout our codebase and compose together to solve bigger problems. Thus higher-order functions and composition promote a bottom-up approach which can lead to succint and modular code.
Lazy evaluation
So what is lazy evaluation?
Lazy evaluation is a strategy where the evaluation of a piece of code is deferred until its results are needed.
In this article I’m going to focus on lazily consuming data and building lazy pipelines that have to be manually drained. I am not going to talk about how lazy evaluation is implemented at a language level (no graph reduction, normal form, etc.).
Let’s look at an example. Given a list of integers, square the elements of this list, and print the sum of the first four squared elements. To write a lazy implementation for this, we must first figure out when do we need to compute something. In our case, only when we want to sum the first four squared elements do we need to provide the squared elements. Therefore we can defer the squaring operation until we actually start summing. Armed with this knowledge, let’s implement a lazy solution.
let squareAndSum = (iterator, n) => {
let result = 0
while(n > 0) {
try {
result += Math.pow(iterator.next(), 2)
n--
}
catch(_) {
// length of list was lesser than `n` hence
// iterator.next threw to signal it's out of values
break
}
}
return result
}
let getIterator = (arr) => {
let i = 0
return {
next: function() {
if (i < arr.length) return arr[i++]
else throw new Error('Iteration done')
}
}
}
let squareAndSumFirst4 = (arr) => {
let iterator = getIterator(arr)
return squareAndSum(iterator, 4)
}
In the implementation, we start squaring elements only when the summing starts. Therefore, only those elements that are being summed are squared. This is achieved by controlling iteration and how values are yielded. A custom iteration protocol is implemented that yields elements one by one and signals when we have no more elements to yield. It is quite similar to what a lot of languages use. This protocol is encapsulated in an iterator object. The iterator object contains one function, next, which takes zero parameters. It yields the next element if there is one and throws otherwise.
The squareAndSum function takes as input an iterator object and n, the number of elements to sum. It pulls n values out of the iterator (by calling .next() n-times), squares and then sums them. getIterator gives us an iterator that wraps our list (which we call an iterable since we can iterate over it). squareAndSumFirst4 then uses getIterator and squareAndSum to give us the sum of first four numbers of the input list squared lazily. A nice side effect of using iterators is that it enables us to implement data structures that can yield infinite values.
Having to implement all of the above each time we require an iterator can be painful. Luckily, ES6 has introduced a simple way of writing iterators. They are called generators.
A generator is a pausable function that can yield values to its caller using the yield keyword multiple times during its execution. Generators, when invoked, return a Generator object. We can call next on the Generator object to get the next value. In JavaScript we create generators by defining a function with a *. Here’s an example.
// a generator that returns an infinite list of sequential
// integers starting from 0
// notice the "*" to tell the parser that this is a generator
function* numbers() {
let i = 0
yield 'starting infinite list of numbers'
while (true) yield i++
}
let n = numbers() // get an iterator from the generator
n.next() // {value: "starting infinite list of numbers", done: false}
n.next() // {value: 0, done: false}
n.next() // {value: 1, done: false}
// and so on..
A Generator object conforms to both the iterator and the iterable protocol. Hence it can be used with for-of to access the values it yields.
for (let n of numbers()) console.log(n) // will print infinite list of numbers
Now that we know a bit about lazy evaluation, higher-order functions, and function composition, let’s implement something to see how using these three approaches cleans up our code.
The problem
We are given a file that contains a username on each line. The file may potentially be larger than the RAM available. We are given a function that reads the next chunk from disk and gives us a chunk that ends with a newline. We are to get the usernames that start with “A” or “E” or “M.” We are then supposed to make requests with the usernames to http://jsonplaceholder.typicode.com/users?username=<username>. After this, we are to run a given set of four functions on the query response for the first four requests.
Sample file contents:
Bret
Antonette
Samantha
Karianne
Kamren
Leopoldo_Corkery
Elwyn.Skiles
Maxime_Nienow
Delphine
Moriah.Stanton
Let’s break up the problem into smaller chunks that we can write separate functions for. One approach would be to use the following functions:
- one that returns each username (
getNextUsername) - one to filter out usernames that begin with an “A”, “E” or “M” (
filterIfStartsWithAEOrM) - one that makes network requests and returns a promise (
makeRequest)
The functions above operate on values. We need a way of applying these functions to a list of values. We need higher-order functions that do the following:
- one that filters items from a list based on a predicate (
filter) - one that applies a function to every item in a list (
map) - one that applies functions from one iterable to data from another iterable (
zipWithwith a zipping function)
This whole approach can benefit by being lazy so that we dont make network requests for all of the usernames that match our criteria, but only for the first n where n is the number of functions that we have to run on the query responses.
We are given an array of functions that are to be run on the final responses and a function that gives us the next chunk lazily. We need a function that gives us usernames one by one lazily. To preserve laziness by controlling when values are yielded, let’s build our solutions using generators.
// functions that are run on the query response
let fnsToRunOnResponse = [f1, f2, f3, f4]
// mocks yielding the next chunk of data read from file
// the * denotes that this function is a generator in JavaScript
function* getNextChunk() {
yield 'Bret\nAntonette\nSamantha\nKarianne\nKamren\nLeopoldo_Corkery\nElwyn.Skiles\nMaxime_Nienow\nDelphine\nMoriah.Stanton\n'
}
// getNextUsername takes an iterator that yields the next chunk ending with a newline
// It itself returns an iterator that yields the usernames one at a time
function* getNextUsername(getNextChunk) {
for (let chunk of getNextChunk()) {
let lines = chunk.split('\n')
for (let l of lines) if (l !== '') yield l
}
}
Before writing the next bit of our solution, let’s have a look at ES6 Promises. A Promise is a placeholder for a future value of an incomplete operation. The ES6 Promise interface lets us define what should execute once the operation completes or fails. If the operation is successful, it invokes the success handler with the value of the operation. Otherwise, it invokes the failure handler with the error.
Coming back to our solution, let’s write the functions that operate on values. We need a function that returns true if a value meets our filter criteria and false otherwise. We also need a function that returns a URL when given a username. Lastly, we need a function that, when given a URL, makes a request and returns a promise for that request.
// this function returns true if the username meets our criteria
// and false otherwise
let filterIfStartsWithAEOrM = username => {
let firstChar = username[0]
return 'A' === firstChar || 'E' === firstChar || 'M' === firstChar
}
// makeRequest makes an ajax request to the URL and returns a promise
// it uses the new fetch api and fat arrows from ES6
// it's a normal function and not a generator
let makeRequest = url => fetch(url).then(response => response.json())
// makeUrl takes a username and generates a URL that we want to query
let makeUrl = username => 'http://jsonplaceholder.typicode.com/users?username=' + username
Now that we have functions that operate on values, we need functions that can apply these values to lazy lists of data. These are going to be higher-order functions. They should be lazy and should defer execution until they are explicitly asked to execute. This sounds like a good place to use generators, since we need values on demand.
// filter accepts a function (the predicate) that takes a value and returns a
// boolean and an iterator filter itself returns an that iterator yields the
// value iff the function when applied to the value returns true
function* filter(p, a) {
for (let x of a)
if (p(x)) yield x
}
// map takes a function and an iterator
// it returns a new iterator that yields the result of applying the function to each value
// in the iterator that was given to it originally
function* map(f, a) {
for (let x of a) yield f(x)
}
// zipWith takes a binary function and two iterators as input
// it returns an iterator which in turn applies the given function to values from each of
// iterators and yields the result.
function* zipWith(f, a, b) {
let aIterator = a[Symbol.iterator]()
let bIterator = b[Symbol.iterator]()
let aObj, bObj
while (true) {
aObj = aIterator.next()
if (aObj.done) break
bObj = bIterator.next()
if (bObj.done) break
yield f(aObj.value, bObj.value)
}
}
// execute makes a deferred iterator begin execution
// it basically calls `.next` on the iterator repeatedly
// till the iterator is done
function execute(iterator) {
for (x of iterator) ;; // drain the iterator
}
So now that we have the functions that we would need, let’s compose them to build our solution.
let filteredUsernames = filter(filterIfStartsWithAEOrM, getNextUsername(getNextChunk)
let urls = map(makeUrl, filteredUsernames)
let requestPromises = map(makeRequest, urls)
let applyFnToPromiseResponse = (fn, promise) => promise.then(response => fn(response))
let lazyComposedResult = zipWith(applyFnToPromiseResponse, fnsToRunOnResponse, requestPromises)
execute(lazyComposedResult)
lazyComposedResult is a lazy pipeline of composed function applications. No step in our pipeline will execute until we call execute on the final composed piece i.e., lazyComposedResult to start the process. Therefore, we will make only four network calls even though our result set post filtering might contain more than four values.
As a result we now have reusable functions that operate on values, reusable higher-order functions, and a way to compose all these together to write a succinct solution.
Epilogue
In this article, we defined higher-order functions, function composition and lazy evaluation. We then went through examples of each. Finally, we combined all the three approaches to write a lazy, modular and composable solution to a given problem.