Thursday, 3 April 2014

Hacking ZAP #3 - Passive scan rules

Welcome to a series of blog posts aimed at helping you “hack the ZAP source code”.
The previous post in this series is: Hacking ZAP #2 - Getting Started

One of the easiest ways to enhance ZAP is to write new passive scan rules. 
Passive scan rules are used to warn the user of potential vulnerabilities that can be detected passively - they are not allowed to make any new requests or manipulate the requests or responses in any way.
They typically run against all of the requests and responses that flow through ZAP.
Passive rules run in separate background thread so that they have as little effect on performance as possible.

You can write passive scan rules dynamically using scripts, as we will see later in this series, but even then its very useful to understand some of the concepts and the underlying classes available to you.

Where to start

The easiest way to get started is to rip off an existing rule.
Passive scan rules can be found in 3 locations in the zap-extensions project, depending on their status:

There are also some simple examples that we will examine in more detail.
These are all in the alpha branch.

The main classes

The following classes are key to implementing passive scan rules

PluginPassiveScanner - this is the class that all passive rules must extend.
There are 2 key methods that you will need to implement:

public void scanHttpRequestSend(HttpMessage msg, int id)
This is called for every request. All details of the request are available via the 'msg' parameter, as detailed below.

public void scanHttpResponseReceive(HttpMessage msg, int id, Source source)
This is called for every response. All details of the request and response are available via the 'msg' parameter, as detailed below. The response is also available as a DOM structure via the ‘source’ parameter.

You can implement one or both of these methods depending on your requirements.
You can examine any part of the request and response in order to find potential vulnerabilities, but you must not change anything.

If you find a potential vulnerability then you can raise it via the method:
PassiveScanThread.raiseAlert(int id, Alert alert)

An HttpMessage is passed in to both of the ‘scan’ methods. This class has methods that allow you to access all aspects of the request and response, although the latter is obviously only available in scanHttpResponseReceive.
Some examples include:
  • msg.getRequestHeader().getMethod();
  • msg.getRequestHeader().getHttpCookies();
  • msg.getRequestHeader().getHeaders();
  • msg.getRequestHeader().getContentLength();
  • msg.getRequestBody().toString();
  • msg.getResponseHeader().getHeaders();
  • msg.getResponseHeader().getStatusCode();
  • msg.getResponseBody().toString();

A Source parameter is passed into scanHttpResponseReceive - this is a DOM representation of the response generated by the Jericho HTML parser.
See the Jericho documentation or the other scan rules for examples of how to access DOM elements.

The Alert class is used to represent potential vulnerabilities.
It supports the following fields:
  • pluginId        Used to identify the scanner, especially useful via the ZAP API
  • name           The summary displayed to the user
  • risk              An indication of how serious the issue is:
    • Alert.RISK_INFO               Informational (its not really a vulnerability)
    • Alert.RISK_LOW               A low level vulnerability
    • Alert.RISK_MEDIUM          A medium level vulnerability
    • Alert.RISK_HIGH               A high level vulnerability
  • reliability      An indication of how likely this is a real problem:
    • Alert.FALSE_POSITIVE    Should not be used - this is for the user to set
    • Alert.SUSPICIOUS            A lower level of confidence
    • Alert.WARNING                 A higher level of confidence
  • description  A more detailed description
  • uri                The URI affected
  • param          The name of the vulnerable parameter, if relevant
  • attack          The attack string used (not relevant for passive vulnerabilities)
  • otherInfo      Information that doesnt readily fit into any of the other fields
  • solution       Information about how to prevent the vulnerability
  • reference    A list of URLs giving more information about this type of vulnerability (separated by newline characters)
  • evidence     A string present in the request or response which can be used as evidence of the vulnerability - this will be highlighted when the related request or response is displayed
  • cweId          The CWE id
  • wascId        The WASC Threat Classification id

Simple example

The ExampleSimplePassiveScanner class implements a very simple passive scan rule.
As you will see, it just raises an alert randomly, so it isnt of any practical use.
However it does demonstrate a couple of useful features:

It uses the Vulnerabilities class to get the name, description, solution and references.
This class loads vulnerability details from the vulnerabilities.xml files included with ZAP.
There are actually a set of vulnerabilities.xml files as it is internationalized, so ZAP will read the localized version for the language the user has selected, defaulting back to English for any phrases that have not been translated.
This is therefore a quick and easy way to fill in these details, as long as the relevant vulnerability is included in that file.

It also uses the log4j Logger class to output debug messages. This is the recommended way of outputting such messages.

Note that the pluginId needs to be unique across all active and passive scan rules. The master list of ids is alert.xml

File based example

The ExampleFilePassiveScanner class implements a slightly more complex passive scan rule.
In this case it reads in a set of strings from a configuration file and checks for their presence in the response.
It could also use hardcoded strings, but the advantage of the approach taken is that a knowledgeable user could manually edit the file to meet their requirement.
The build process handles the file deployment, as described later.

This class also demonstrates a couple of other features:

Instead of using the Vulnerabilities class the code uses Constant.messages.getString(str)
All of the strings used in this way are defined in the Messages.properties file.
If you are just implementing the rule for your own benefit then you can hardcode the strings if you want, but internationalizing them is very simple and saves having to go back and change you code if you want to have your rule included in the ZAP Marketplace.

The code also makes use of the getLevel() method.
This returns an AlertThreshold class which indicates how strictly you should check for vulnerabilities.
The threshold returned can be one of:
  • LOW:         This indicates you should report more potential vulnerabilities, which might mean more false positives
  • MEDIUM:   This is the default level
  • HIGH:         This indicates you should report more fewer vulnerabilities, which might mean more false negatives
You do not have to use the threshold - especially as it might not be relevant for the vulnerability you are testing for, but it is also a useful way for the user to tune how the rules work and so its worth using if you can.

Building and deploying

All ZAP add-ons are build using Apache Ant.
For the alpha passive scan rules the build file is: branches/alpha/build/build.xml
All you need to do is run the deploy-pscanrulesAlpha target and the relevant add-on will be built and copied to the correct location, assuming you have a ZAP core project called ‘zaproxy’.
If you want to deploy to a different location then you can change it at the top of the file. 
ZAP automatically extracts the files specified in the manifest into a directory underneath the ZAP user directory.
An knowledgeable user can manually edit these files and any changes will take affect when ZAP is restarted.

Updating the help and manifest

There are a couple more things that you can do to finish off a new rule.

The first is to add a short description of the rule to the help file: pscanalpha.html

This is not really necessary unless you want to publish your rules.

The second is to include the new class along with any files it uses in the add-on manifest: ZapAddOn.xml
ZAP uses this file to identify any files that need to be extracted.
It also uses it when it downloads add-ons from the ZAP Marketplace, and adding your classes and files in here will allow ZAP to add, update and remove your rules dynamically.

A future post will cover how to contribute your code back to the ZAP community and progress it from alpha to beta and then release status.

The next post in this series will be: Hacking ZAP #4: Active scan rules

Thursday, 20 March 2014

Hacking ZAP #2 - Getting Started

Welcome to a series of blog posts aimed at helping you “hack the ZAP source code”.
The previous post in this series is: Hacking ZAP #1 - Why should you?

In order to change the ZAP source code you will need to set up a development environment.

We provide a full Eclipse workspace that is updated very regularly and is available to download from: http://sourceforge.net/projects/zaproxy/files/workspace/

However its worth understanding how the ZAP projects are structured, especially if you would like to use an alternative IDE.
 

Project structure


There are currently 2 main ZAP code projects, both on Google code, and ongoing development takes place both on the trunks and in branches: 

zaproxy 
zap-extensions 
There are more branches than shown, but you typically won’t need to use any of them.

We are discussing changing this project structure, as well as potentially moving to GitHub. This blog post will be updated to reflect any relevant changes.


There is also a zaproxy-test project, however the main part of the project contained unit tests which have now been migrated to a separate test folder within the other projects.

Setting up your IDE

If you choose not to use the Eclipse workspace then you can set up your IDE as follows:
  1. Import all of the projects you need, e.g.
  2. For each project, add all of the jars in the ‘lib’ directory to the classpath
You should update these projects at regular intervals in order to pick up the latest changes.

To run ZAP run
org.zaproxy.zap.ZAP.java

Each project also has a main Ant build file, build/build.xml, which we will examine in more detail later in the series.

There is more information about building ZAP on the
ZAP wiki
 
Note that if you just want to get going as quickly as possible you can just import the zaproxy trunk for core changes and/or the zap-extensions alpha branch for creating a new add-on.

The ZAP core

The zaproxy project is often called “the core”.

It has 2 main high level packages within the src folder:

  • org/parosproxy          The code inherited from the Paros project
  • org/zaproxy               New ZAP code
Other directories of note:
  • src/help                     The main ZAP help files
  • build                           Build related files
  • lib                              The jars ZAP relies on
  • test                            ZAP unit tests

We try to implement significant new features in the zap-extensions project as ‘add-ons’.

We do this for the following reasons:
  1. Core changes can only be delivered via ‘full’ ZAP releases. We typically only do these a few times a year.
  2. Add-ons can be released and updated as frequently as required. Users can dynamically install and update add-ons, often without having to even restart ZAP.
  3. Add-ons progress from alpha through beta to release, allowing users to understand how robust an add-on is likely to be. This allows developers to release early without worrying about breaking ZAP for everyone.
  4. Add-ons can still be included in ‘full’ ZAP releases - the WebSockets and Ajax Spider are 2 add-ons that we include by default.

Having said that, you may well find that changes you want to make can only be made in the core.
That is not a problem - you can make changes to the core - but these changes will probably not be available to users as quickly as those made to add-ons.

We also add new functionality to the core if we want it to be available for other add-ons to build on.


Its important to note that if you make changes to code in the Paros package then you must include a comment at the top of the file mentioning your change. This is required to satisfy the
Clarified Artistic License that Paros was released under.
We have a standard format for these comments
// ZAP: yyyy/mm/dd Issue xyz: Description of the changes
eg see CommandLine.java

If you make changes to the core then you typically just need to make them to the trunk.
There is a branch for every release. If your change fixes a significant issue then we may also want to apply it to the latest release branch. But you dont need to worry about that.

Add-ons, Extensions and Rules

There are many ways to extend ZAP programmatically.
Some of the main ways include:

  • Extensions, which are classes that extend the Extension.java class. They are a powerful way of adding functionality to ZAP, and much of the ‘core’ is actually implemented as extensions.
  • Active Scan rules, which are classes which extend Plugin.java, detect potential vulnerabilities and run as part of the Active Scanner. These rules typically make multiple requests on the target system. They only run when explicitly invoked by the user.
  • Passive Scan rules, which are classes which extend PluginPassiveScanner.java, detect potential vulnerabilities and run as part of the Passive Scanner. These rules cannot make any requests - all they can do is examine the requests and responses. They are typically invoked for every request that is proxied through ZAP.
Add-ons are packages which can include all of these components as well as ‘raw’ files. They are usually available on the ZAP Marketplace so that users can discover, download and install them from within ZAP. You can also install add-ons from your filestore via the “File / Load Add-on file…” menu.

New add-ons should be created in the alpha branch and only move to the beta and then release branches after they have been reviewed.


You can also create add-ons in your own repositories. If they are open source then they will still be eligible to be uploaded to the ZAP Marketplace, but it might take slightly longer to review them.

We will cover all of these components (and more) in more detail in a future posts.

If you have any questions about ZAP development then please ask them on the
ZAP Developer group.

The next post in this series is: Hacking ZAP #3: Passive scan rules

Monday, 10 March 2014

Hacking ZAP #1 - Why should you?

Welcome to a series of blog posts aimed at helping you “hack the ZAP source code”.

ZAP is an open source tool for finding vulnerabilities in web applications.
It is the most active OWASP project and is very community focused  - it probably has more contributors than any other web application security tool.
It is being continually enhanced and, unusually for a security tool, has been translated into over 25 languages thanks to over 70 translators.

This series is designed to help newcomers dive head-first into the ZAP source code.
However for this first blog post I thought I’d take a step back and give some reasons why you might want to change the ZAP source code in the first place.

You help others

ZAP was voted the most popular security tool of 2013 by ToolsWatch.org readers, competes with commercial security tools and in some cases exceeds their capabilities. For example ZAP was the first security tool to support Web Sockets, something few commercial tools support, even now.
By contributing to ZAP you help all of the people that are using it to make their applications more secure.

It’s a great way to learn

I’m a developer. I started ZAP as a way to teach myself about security, and I’m a big fan of learning by doing, or in this case, ‘coding’. If you want to understand a particular vulnerability properly then why not write some code that you think will detect it? By the end of that process you should understand the vulnerability much better!
It doesnt matter if you are completely new to security (as I was) or have been working on it for many years - there’s always more to learn.
I want ZAP to be a great teaching tool, and that’s why I encourage students to get involved with ZAP development.
We have had some great features implemented by students, both in their own time and as part of the Google Summer of Code. If you are a student and would like to contribute to ZAP (for example as part of your course work) then we’d be delighted to hear from you.
The ZAP developer group is available specifically to help people work on ZAP, and we always do our best to help newcomers find their feet.
And of course, having contributions to a major open source security project like ZAP on your CV or resume can be a real benefit when trying to break into the security industry.

You can make ZAP work your way

ZAP is open source, and one of the big advantages of open source is that you can change it to work exactly as you want.
You can keep your changes to yourself of course, but we realize that people want to work in different ways, and so we are always happy for people to add new options to ZAP so that people can choose exactly how it works for them.
And even if you want to take ZAP off in a direction we (the ZAP core developers) think is completely irrelevant to what we are trying to achieve, we’ll still happily work with you to try to make sure you can achieve what you want. Without breaking ZAP for everyone else of course.

It’s a powerful environment

Lets say you’ve come up with a neat way to find vulnerabilities, either in a completely automated way or with manual assistance.
A small stand alone tool is ok, but there are so many things you need to handle when using such a tool on real world application. Things like authentication, session handling and managing CSRF tokens.
ZAP provides a complete testing environment and already handles a whole range of problems that you are likely to encounter.
If it can’t handle a particular problem then let us know - we are constantly improving ZAP, and so we might already be working on it. And if we are not then we’ll add it to the list of enhancement requests - someone else reading this post might just decide to work on it.
One of the advantages of using an open source project like ZAP is that you can access all of the ZAP internals. We have no ‘secret sauce’ and so we can give you access to all of the ZAP functionality without having to limit you to a restricted API for commercial reasons.

It provides more exposure for your project

Maybe you’ve started a small open source project, it might even an OWASP one.
You may well find that it takes time to get people interested in it, particularly if its completely stand alone.
While ZAP is a great environment to build on, we realize that there are many reasons why people might not want to just build on top of ZAP.
However you can wrap your project in a ZAP add-on while keeping ownership and maintaining it separately from ZAP.
We are big fans of reuse, and encourage people to write new functionality in a way that allows it to be reused across a range of security tools, even our competitors. We practice what we preach, having done that for major new features in ZAP like Plug-n-Hack and Zest.
The ZAP marketplace allows users to browse, download and install add-ons from within ZAP.
Repackaging your tool as a ZAP add-on will introduce your project to people who might otherwise never have known about it or bothered to try it.

Profit???

ZAP is open source, and will stay open source.
You will never have to pay for a ‘pro’ version.
However we have no problem with people developing and selling closed source add-ons to ZAP.
If you think you can make money from selling ZAP add-ons then go for it.

The next post in this series is: Hacking ZAP #2 - Getting started