Enhancing AJAX Crawling capabilities of ZAProxy

Zed Attack Proxy (ZAP) is a tool for penetration testing of web applications that can be easily used by both security aware  an unaware engineers .

It has several features such as a spider to discover the pages of a given site, and a set of scanners to automatically and semi-automatically discover security flaws. You can find more information about its specifications it the ZAP wiki.


I recently started working on improving its AJAX crawling capabilities by developing a plugin to integrate the OWASP Ajax Crawling Tool and Crawljax.

I am focusing my work in two different lines.

• Building a plugin to facilitate the ajax spider integration in zaproxy.
• Improving the results of the spiders when targeting AJAX-based sites.

To accomplish the second one, I am testing ACT and Crawljax in a security testing framework called Wivet that builds a set of links in a tricky way to difficult spiders crawling those.

When I started testing crawljax, it was not able to crawl any link of wivet due to unsupported tags issues and the spider of ZAP only a 10% of the pages.

 Since then, I improved its results up to 72% of the links so far. However, I am still working on reducing that 28% of the most hard to crawl ones.

For instance, I modified the clickTag() method of the Crawler class to support meta refresh tags as follows:

if (eventable.getElement().getTag().toLowerCase().equals("meta")) {
    Pattern p = Pattern.compile("(\\d+);\\s+URL=(.*)");
    for (Attribute e : eventable.getElement().getAttributes()) {
        Matcher m = p.matcher(e.getValue());
        if (m.find()) {
            try {
                //time*1000 to convert seconds to ms
                Thread.sleep(Integer.parseInt(m.group(1))*1000);
            } catch (Exception e) {
                LOGGER.error(e);
            }
        }
    }
}
LOGGER.info("Executing " + eventable.getEventType() + " on element: " + eventable+ "; State: "+this.getStateMachine().getCurrentState().getName());
    if (this.fireEvent(eventable)) {
        ...

I also improved in crawljax appendFrameContent() of WebDriverBackedEmbeddedBrowser and getFramesCandidates() of CandidateElementExtractor to support frame tags among other enhancements.

Below, I show a video of a test I did with the different spiders targeting  my wivet test environment.

I used a tool called Istanbul to record the video, but it looks like the time between frames is kind of high and it is not possible to change it.