Thursday, June 21, 2012

Enhancing AJAX Crawling capabilities of ZAProxy

Zed Attack Proxy
Zed Attack Proxy (ZAP) is a tool for penetration testing of web applications that can be easily used by both security aware  an unaware engineers .

It has several features such as a spider to discover the pages of a given site, and a set of scanners to automatically and semi-automatically discover security flaws. You can find more information about its specifications it the ZAP wiki.

I recently started working on improving its AJAX crawling capabilities by developing a plugin to integrate the OWASP Ajax Crawling Tool and Crawljax.

I am focusing my work in two different lines.
  • Building a plugin to facilitate the ajax spider integration in zaproxy.
  • Improving the results of the spiders when targeting AJAX-based sites.

To accomplish the second one, I am testing ACT and Crawljax in a security testing framework called Wivet that builds a set of links in a tricky way to difficult spiders crawling those.

When I started testing crawljax, it was not able to crawl any link of wivet due to unsupported tags issues and the spider of ZAP only a 10% of the pages.

 Since then, I improved its results up to 72% of the links so far. However, I am still working on reducing that 28% of the most hard to crawl ones.

For instance, I modified the clickTag() method of the Crawler class to support meta refresh tags as follows:
if (eventable.getElement().getTag().toLowerCase().equals("meta")) {
    Pattern p = Pattern.compile("(\\d+);\\s+URL=(.*)");
    for (Attribute e : eventable.getElement().getAttributes()) {
        Matcher m = p.matcher(e.getValue());
        if (m.find()) {
            try {
                //time*1000 to convert seconds to ms
            } catch (Exception e) {
}"Executing " + eventable.getEventType() + " on element: " + eventable+ "; State: "+this.getStateMachine().getCurrentState().getName());
    if (this.fireEvent(eventable)) {

I also improved in crawljax appendFrameContent() of WebDriverBackedEmbeddedBrowser and getFramesCandidates() of CandidateElementExtractor to support frame tags among other enhancements.

Below, I show a video of a test I did with the different spiders targeting  my wivet test environment.

I used a tool called Istanbul to record the video, but it looks like the time between frames is kind of high and it is not possible to change it.


  1. Your Post Updated and tested, it's working perfectly fine. I love this blog. Online Florist Services in Gurgaon

  2. I advise you to lead your youtube channel. You can use this site to get more likes and views for your video

  3. And I recommend you to create and lead instagram blog. Now is very easy to buy instagram followers for your page

  4. Nicely written article. Thank you for this. The international tourist who wish to travel to Kenya should apply for Kenya visa application online. Applying online saves time and provide you the best services where you can check the Kenya visa documents requirement and other information very easily.