In the Software University (SoftUni) we are developing a test automation system (online judge) for JavaScript code. It supports many languages and frameworks (C#, Java, JS, PHP, Python, SQL, C++, …). The interesting case now is with JS testing with Node.js. Students submit their JS code, the judge executes a few tests and responds with evaluation score (0…100):

Automated JS Testing with DOM and AJAX

Testing plain JS functions in the judge system works pretty well in production for many years, but now we are extending the system to support JS UI testing with DOM and AJAX requests, with frameworks like React and Angular. Students submit end-to-end JS single-page apps (ZIP file with many HTML and JS files) and the judge tests them automatically (without human touch). In its simplest form, testing JS code that invokes AJAX and changes the DOM works as follows:

1. Create a DOM tree with jsdom. Note that jsdom is not a fully functional browser. It is just a DOM implementation without navigation and is not 100% compatible with Chrome / Firefox. We use some hacks to provide objects like Option, HTMLLIElement and many others.
2. Put some HTML inside the jsdom’s document object, e.g. a form with a [Load] button.
3. Mock the XMLHttpRequest with Sinon, in order to intercepts all AJAX requests and return a fake HTTP response depending on the HTTP request content.
4. Execute the JS code, submitted by the student. It generally attaches event handlers to some DOM events, like click on the [Load] button, click on the [Login] link or submit a [Search] form. The event handlers in students’s code invoke an AJAX requests (asynchronously) and modify the DOM tree to display the results. In the general case student’s code may perform several AJAX requests and update the DOM tree several times. It is unknown when this process in completed (e.g. how many time it will take to perform the [Load] button’s action).
5. The tests in the judge system perform a DOM interaction, e.g. simulate a click on the [Load] button.
6. The judge system waits for the student’s JS code to complete and to modify the DOM tree, e.g. to display some data from a REST service.
7. The judge system checks whether the data in the DOM tree is correct (performs Mocha / Chai assertions).

Waiting for Asynchronous AJAX Requests + DOM Updates to Complete without having a Promise

The big problem here is at Step #6 from the above testing process:

How to wait for a JS event handler (e.g. button [Load] event handler) to complete all its asynchronous work (e.g. perform a sequence of AJAX request and render some HTML) before proceeding to test assertions?

It would be easy if the event handler returns some kind of promise, but this is not the case. We have a JS application holding some forms, buttons, links and even React components. We know that the app handles the buttons / links clicks and form submits somehow, but we don’t know anything about the code behind these events. The app code is black box. The only thing we know is that the application will invoke one or several AJAX requests, will asynchronously get some data from a REST service (which we mock with Sinon) and that the received data will be displayed in certain page element in the DOM. This could take 5 ms, or 50 ms or 3 seconds – we don’t know how many time. We don’t have a good indication when the operation is completed. Possible solutions:

• If we ask the students to return a promise from each event handler, this will significantly complicate theit work. It is unnatural for a button event handler to return a promise.
• If we use a timeout and perform the test assertions after 50 ms or 100 ms, this is not reliable. It may happen that the operating system is busy and these 50 or 100 ms are not enough. Additionally, this will make the testing slow.
• If we listen for DOM changes (e.g. with a MutationObserver), this is also unreliable, because clicking on the button could change the DOM many times. For example, the button event handle could load asynchronously some data with AJAX, then display it n the DOM, then make another asynchronous AJAX request, then display it in the DOM and thus the DOM will change several times.
• Node.js event-loop handles inspection – this is our reliable solution. Node.js provides an unofficial API for reading its event-loop: process._getActiveHandles(). It returns the list of active handles in Node.js in the event queue: unfinished timers, open sockets, servers listening for a connection, file system descriptiors, etc. You can list them using wtfnode.

The Solution: Waiting for Asynchronous Event Handlers to Complete Their Work

Finally, we have a stable solution how to wait for a event handler to complete all its asynchronous work. It relies on process._getActiveHandles() to take all active event-loop handlers, then takes all the timers and waits (with 5 ms polling) for all the timers to complete their work (except the timer used for polling).

This approach works reliably for server-side AJAX code testing with Node.js, jsdom + Sinon mocking. It waits reliably event handlers that perform multiple AJAX requests and update the DOM tree multiple times with internal delays. See the JavaScript code below for fully working example. The code below tests a function that is invoked when the [Load] button in a HTML form is clicked and asynchronously loads GitHub repositories and displays them 1000 times (to simulate a huge DOM update), then invokes another AJAX request to display a status from GitHub.

let expect = require('chai').expect;
let jsdom = require('jsdom');
let fs = require('fs');
let jquery = require('jquery');
let sinon = fs.readFileSync("/usr/local/lib/node_modules/sinon/pkg/sinon.js");

// ------------------- setup the jsdom environment -----------------------
let jsdomPromise = new Promise((resolve, reject) => {
    jsdom.env({
        html: '<html><body></body></html>',
        src: [sinon],
        done: function (errors, window) {
            global.window = window;
            global.document = window.document;
            global.$ = jquery(window);
            Object.getOwnPropertyNames(window)
                .filter((prop) => prop.toLowerCase().indexOf('html') >= 0)
                .forEach((prop) => global[prop] = window[prop]);
            resolve();
        }
    });
});

jsdomPromise.then(() => {
    // ------------------- setup the HTML page ---------------------------
    document.body.innerHTML =
        `<input type="text" id="username" value="testnakov" />
         <ul id="repos"></ul>
         <button id="btnLoadRepos">[Load]</button>`;

    // ---------- setup AJAX fake response mock (with Sinon) ----------------
    global.server = window.sinon.fakeServer.create();
    server.autoRespond = true;
    server.respondWith(/https:\/\/api.github.com\/users\/(\w+)\/repos/,
        function (xhr, name) {
            xhr.respond(200,
                {"Content-Type": "application/json"},
                '[{ "full_name": "TestRepo1", "html_url": "Some Url" }]');
        }
    );
    server.respondWith(/https:\/\/status.github.com\/api\/status.json/,
        function (xhr, name) {
            xhr.respond(200,
                {"Content-Type": "application/json"},
                '{"status":"good", "last_updated":"2016-11-25T12:57:24Z"}');
        }
    );


    // ------------------------ student's code ---------------------------
    function attachEvents() {
        $("#btnLoadRepos").click(loadRepos);

        function loadRepos() {
            $("#repos").empty();
            let url = "https://api.github.com/users/" +
                $("#username").val() + "/repos";
            setTimeout(function () {
                $.ajax({
                    url,
                    success: displayRepos,
                    error: displayError
                });
            }, 22);

            function displayRepos(repos) {
                console.log("AJAX request 1 success: " + JSON.stringify(repos));
                for (let i = 0; i < 10000; i++) {
                    for (let repo of repos) {
                        let link = $("<a>").text(repo.full_name);
                        link.attr("href", repo.html_url);
                        $("#repos").append($("<li>").append(link));
                    }
                }
                console.log("DOM modify part 1 finished.");
                setTimeout(function () {
                    $.ajax({
                        url: "https://status.github.com/api/status.json",
                        success: displayStatus,
                        error: displayError
                    });
                }, 33);

                function displayStatus(response) {
                    console.log("AJAX request 2 success: " + JSON.stringify(response));
                    $("#repos").append(
                        $("<div class='status'>").text(response.status));
                    console.log("DOM modify part 2 finished.");
                }
            }

            function displayError(err) {
                console.log("AJAX error: " + err);
                $("#repos").append($("<li>Error</li>"));
            }
        }
    }


    // ---------------------- run student's code -------------------------
    attachEvents();


    // ------------ perform a DOM interaction: click a button ------------
    $("#btnLoadRepos").trigger('click');


    // -------------- wait for student's code to complete ----------------
    let studentCodePromise = new Promise((resolve, reject) => {
        setTimeout(checkFinished, 5);
        function checkFinished() {
            let activeHandlers = process._getActiveHandles();
            let activeTimers = activeHandlers.filter(h =>
                Object.getPrototypeOf(h).constructor.name == "Timer");
            if (activeTimers.length == 1) {
                // Only one timer left active (invoker of the current function)
                resolve();
            }
            else
                setTimeout(checkFinished, 5);
        }
    });


    // ------------------- check student's code results ------------------
    studentCodePromise.then(function() {
        let repos = $("#repos li a");
        expect(repos.eq(0).text()).to.equal("TestRepo1", "Incorrect text");
        expect(repos.eq(0).attr('href')).to.equal("Some Url", "Incorrect link");
        let status = $("#repos .status");
        expect(status.eq(0).text()).to.equal("good", "Incorrect status");
        console.log("All tests passed.");

        // Restore the faked XMLHttpRequest created by Sinon
        server.restore();
    });
});

The output looks like this:

AJAX request 1 success: [{"full_name":"TestRepo1","html_url":"Some Url"}]
DOM modify part 1 finished.
AJAX request 2 success: {"status":"good","last_updated":"2016-11-25T12:57:24Z"}
DOM modify part 2 finished.
All tests passed.

All test assertions are executed after the button handler is completely finished its work (after 2 AJAX calls + several updates in the DOM tree).

Enjoy!