Wednesday, 27 May 2015

ZAP as a Service (ZaaS)

At OWASP AppSec EU in Amsterdam this year I announced ZAP as a Service (ZaaS).
The slides are here and the video will hopefully be available soon.

The idea behind this development is to enhance ZAP so that it can be run in a ‘server’ mode.
This is different to the current ‘daemon’ mode in that it will be designed to be a long running, highly scalable, distributed service accessed by multiple users with different roles.

ZaaS is definitely not ready for release yet - there will be loads of changes required to make this a reality, although some changes required for ZaaS have already been made.
However I decided to announce this as a future direction in order to stimulate discussions and hopefully encourage people to get involved.

And I want to stress that it is not a replacement for ‘desktop’ ZAP (as I’ve started calling it).
Desktop ZAP is an important focus for us, and it is the way we expect most people to use ZAP for the foreseeable future.
Instead its just a different way of running ZAP - theres going to be a lot of common code between desktop ZAP and ZaaS.

So how will ZaaS differ from desktop ZAP?

Database

The current HSQLDB is good for a desktop application as it requires no installation, but its not suitable for ZaaS.
In 2.4.0 we introduced a database independence layer so that alternative implementations can be supported, although the only implementation was the current hard coded HSQLDB option.
In the trunk theres now a generic SQL implementation, in which all of the SQL statements have been extracted into property files.
In theory any SQL db should be supportable, and working implementations of both HSQLDB and MySQL are provided.
Although MySQL is intended for daemon or ZaaS modes theres no reason why it cant also be used for the desktop, and in fact thats one of the options thats been tested.
Details of how to configure ZAP to use MySQL will be posted on the ZAP Developer Group soon.
One important aspect that has not been implemented yet is the ability to support multiple database instances in order to better segregate data.

Data Structures

While desktop ZAP uses the database very heavily, it still builds up some big data structures in memory. This is so that the Swing UI can react quickly to user events, such as scrolling through a long list.
Structures like the Sites tree and History table are held in memory and will constantly grow.
This is no good for a long running service, which is why we have introduced a ‘low memory’ option. When this option is used ZAP components that support it will not build up any significant data structures in memory, and those that dont support it will not be enabled.
This has been implemented for most of the core, but some of the add-ons (including some key active scan rules) still need to be changed.
Again, details of how to configure ZAP to use the low memory option will be posted on the ZAP Developer Group soon.

Processes and deployment

We will be restructuring the code so that it will be possible to run multiple ZAP processes across multiple machines.
Desktop ZAP will still be just one process, but we will add the option to run ZAP as multiple distributed processes which are likely to provide specific functionality.
For example we may well have ZAP ‘worker’ processes for long running tasks such as the spider and active scanner.
This work has not been started.

Users and roles

ZaaS will need to support multiple users with different roles. We want to be able to support a hierarchy of users, teams and companies all with access to their own data.
This work has not been started.

Access

In addition to the API we will need to implement a modern HTML5 interface for ZAP.
This is a very big task, and so we’ll need to initially target it at some very specific use cases before gradually expanding it to handle all of the ZAP functionality.
This work has not been started.

Application Lifetime

The target is for ZaaS to be capable of running in a five nines environment.
Achieving that level of uptime is not just a software problem, but even so ensuring that ZaaS can operate at that level is a huge challenge, especially for a product that at the moment typically only runs for a matter of hours.
The MySQL support, distributed architecture and low memory options are key here, but there are many other considerations such as ensuring there are no single points of failure and supporting rolling upgrades.
This is probably going to be the most challenging aspect of ZaaS, and one that several people have questioned.
However designing, implementing and in some cases even running highly available services is exactly what I was doing before I started working on ZAP. I actually started ZAP as a way to learn how to make the services I was working on as secure as possible!

Security

We take the security of Desktop ZAP very seriously, but the security of ZaaS will be even more critical.
Desktop ZAP is typically run on a single machine and only accessible from that machine.
ZaaS will have a remotely accessible HTML5 interface, and this significantly increases the attack service area.
As it will also contain details of the vulnerabilities of other services, it will be a very tempting target for attackers even if it is restricted to a company's internal network.
We will be considering security at all stages of ZaaS development and will ensure that it is thoroughly pentested by experienced security professionals not directly connected with the project.

License

One key thing that will not change between Desktop ZAP and ZaaS is the license.
ZaaS with be released under Apache V2, which means that anyone can use it for anything they like.
You will be able to set up a ZaaS instance for your company, and if you want to you could even set up ZaaS as an online service and charge money for it - thats completely permissible!

Development process

Although we have already implement features required for ZaaS there is still much work to do.
As mentioned at the start of this post, this announcement is to stimulate discussions and hopefully encourage people to get involved.
We are not able to give any idea of a possible release date for ZaaS at this stage, however we will aim to implement features in a way that ensures they can be used even without the full ZaaS solution.

We will be using the ZAP Development group for all of the ZaaS related discussions - please join in!

Wednesday, 3 September 2014

Alberto's GSoC 2014 Project for ZAP: SOAP Scanner Add-On

Hello everybody, my name is Alberto Verza, a 23 year student from Spain, and this summer I have participated in Google Summer of Code 2014. My project was the SOAP Scanner Add-On for ZAP, in which I worked during all the Program. Let me explain you the features it includes.

One of the interesting features this Add-On provides is WSDL file scanning. Until now, ZAP could find these kind of files and it could even search URLs inside them, but further petitions to these URLs had not a valid SOAP format specified by the WSDL file. With the SOAP Scanner Add-On, detected WSDL files are now read and SOAP petitions now follow the correct format. You can provide WSDL files in many ways: using the main window "quickscan" option, through proxy navigation, or even importing single files through tools menu.

However, SOAP Scanner Add-On functionality doesn't end here. It couldn't be called "scanner" if it wasn't capable of scanning vulnerabilities after all. That's why it has some SOAP dedicated scanners, which simulates specifical attack vectors for some known vulnerabilites like SOAP Action Spoofing or XML Injection, and it raises alerts when something unusual is coming from server's response. Moreover, the Add-On not only works with scanners made in this project, but also it is compatible with previous developed ones like SQL Injection Scanner or Cross-Site Scripting Scanner, among others.

SOAP Scanner Add-On has passed GSoC final evaluations in alpha state, so if you want to give it a try, you should take a look to the User Guide [1] first.

Finally, I want to stress that OWASP ZAP and SOAP Scanner Add-On are Open Source Software under Apache License 2.0, so contributions to the code are always welcome. If you have not worked for ZAP before, then ZAP Developers will be pleased to help anyone who wants to contribute through the Developers' Group [2], so don't hesitate to ask there.

I hope you find this useful :)

------------------------------------------------------------------------------------------------------------
[1] SOAP Scanner Add-On User Guide: https://docs.google.com/document/d/1yy7eZHP0mg46nHC7a2KaOfM08HNb84f5Kn5GF02-v9M/edit?usp=sharing

[2] ZAP Developers' group: https://groups.google.com/forum/#!forum/zaproxy-develop

Wednesday, 30 April 2014

Hacking ZAP #4 - Active scan rules

Welcome to a series of blog posts aimed at helping you “hack the ZAP source code”.
The previous post in this series is: Hacking ZAP #3 - Passive scan rules

Active scan rules are another relatively simple way to enhance ZAP.
Active scan rules attack the server, and therefore are only run when explicitly invoked by the user.
You should only use active scan rules against applications that you have permission to attack.

You can also write active scan rules dynamically using scripts, as we will see later in this series, but even then its very useful to understand some of the concepts underlying classes available to you.

Where to start

As with passive rules, the easiest way to get started is to rip off an existing rule.
Active scan rules can be found in 3 locations in the zap-extensions project, depending on their status:
There are also some simple examples that we will examine in more detail.
These are all in the alpha branch.

The main classes

Unlike passive scan rules there are different classes that you should extend depending on the type of rule you want to implement.

AbstractPlugin - you typically will not extend this class directly, but it provides key methods that you will need to use and abstract methods that will need to be implemented.

AbstractHostPlugin - extend this class if you want your code to be run once against the top node that the user scans. This is ideal for scanning things that are not ‘page’ related, such as the SSL certificate.
The key method you’ll need to implement is the void scan() method inherited from AbstractPlugin - this is where you perform your attacks.

AbstractAppPlugin - extend this class if you want your code to be run against every node (or page) being scanned. This is ideal for scanning elements that are not related to existing parameters, such as trying new debug flags.
The key method you’ll need to implement is the void scan() method inherited from AbstractPlugin - this is where you perform your attacks.

AbstractAppParamPlugin - extend this class if you want your code to run against every parameter in every node being scanned. This is ideal for scanning existing parameters.
The key method you’ll need to implement is void scan(HttpMessage msg, String param, String value) - this is where you attack the specified parameter.

Performing attacks

 

Unlike passive scan rules, active scan rules are expected to make requests to the server.

You should use the AbstractPlugin.getNewMsg() method to get hold of a new HttpMessage that you can use for your attack. If you make multiple requests then call getNewMsg() for each request. The message will be a copy of the original request but with an empty response. You can access the original request and response via the getBaseMsg() but you should not modify it.
You should use one of the AbstractPlugin.sendAndReceive(HttpMessage msg, …) methods to actually make the request. The variants allow you to choose whether the underlying code should handle redirects and/or anti CSRF tokens.
The sendAndReceive methods also handle user controlled features like authenticating as a specified user.
When you find potential issues you can raise them via one of the AbstractPlugin.bingo(..) methods.

Like passive scan rules, active scan rules support AlertThresholds which allow the user to indicate how strictly you should check for vulnerabilities.

Simple example


The ExampleSimpleActiveScanner class implements a very simple active scan rule.
As you will see, it just raises an alert randomly, so it isnt of any practical use.
Like the simple example passive scanner introduced in the previous post it uses the Vulnerabilities class for the documentation associated with the vulnerability.
Most of the methods should be self explanatory, but there are 2 that are worth explaining.
The getCategory() method is used to group related rules in the UI.
And the getRisk() method affects the order the rules are run - the rules which report higher risks are run before those that report lower risks.

File based example

The ExampleFileActiveScanner class implements a slightly more complex active scan rule which is equivalent to the example file passive scan rule introduced in the previous post.
This class introduces another feature, the attack strength, which allows the user to adjust the number of attacks each rule performs.
If you are implementing a rule just for your own use then you dont need to worry about this. However if you plan to publish it for others to use then you should consider supporting it.

The getAttackStrength() method returns an AttackStrength class which can be one of:
  • LOW:         Limit to around 6 requests per scan call
  • MEDIUM:    Limit to around 12 requests
  • HIGH:         Limit to around 24 requests
  • INSANE:    No limit, although 1000s wouldnt be a good idea

You should periodically check to see if the AbstractPlugin.isStop() method returns true - this indicates that the user has stopped the scan so your code should immediately return. The infrastructure will check this before invoking your code to scan a new target (eg page or parameter).

Advanced features

If your rule depends on another rule having been run then you need to specify that via the getDependency() method.
The persistent XSS rules do this, eg in TestPersistentXSSSpider and TestPersistentXSSAttack

If you need to know the exact type of parameter you are scanning, for example to ignore some types that are not relevant, then you can override the AbstractAppParamPlugin.scan(HttpMessage msg, NameValuePair originalParam) method and provide a null scan(HttpMessage msg, String param, String value) method. The NameValuePair class includes the parameter type.

Building and deploying

All ZAP add-ons are build using Apache Ant.
For the alpha active scan rules the build file is: branches/alpha/build/build.xml
All you need to do is run the deploy-ascanrulesAlpha target and the relevant add-on will be built and copied to the correct location, assuming you have a ZAP core project called ‘zaproxy’.
If you want to deploy to a different location then you can change it at the top of the file. 
ZAP automatically extracts the files specified in the manifest into a directory underneath the ZAP user directory.
An knowledgeable user can manually edit these files and any changes will take affect when ZAP is restarted.

Updating the help and manifest

As with passive scan rules its good to add a short description of the rule to the help file: ascanalpha.html and include the new class along with any files it uses in the add-on manifest: ZapAddOn.xml

The next post in this series will be: Hacking ZAP #5: Extensions

Thursday, 3 April 2014

Hacking ZAP #3 - Passive scan rules

Welcome to a series of blog posts aimed at helping you “hack the ZAP source code”.
The previous post in this series is: Hacking ZAP #2 - Getting Started

One of the easiest ways to enhance ZAP is to write new passive scan rules. 
Passive scan rules are used to warn the user of potential vulnerabilities that can be detected passively - they are not allowed to make any new requests or manipulate the requests or responses in any way.
They typically run against all of the requests and responses that flow through ZAP.
Passive rules run in separate background thread so that they have as little effect on performance as possible.

You can write passive scan rules dynamically using scripts, as we will see later in this series, but even then its very useful to understand some of the concepts and the underlying classes available to you.

Where to start

The easiest way to get started is to rip off an existing rule.
Passive scan rules can be found in 3 locations in the zap-extensions project, depending on their status:

There are also some simple examples that we will examine in more detail.
These are all in the alpha branch.

The main classes

The following classes are key to implementing passive scan rules


PluginPassiveScanner - this is the class that all passive rules must extend.
There are 2 key methods that you will need to implement:

public void scanHttpRequestSend(HttpMessage msg, int id)
This is called for every request. All details of the request are available via the 'msg' parameter, as detailed below.


public void scanHttpResponseReceive(HttpMessage msg, int id, Source source)
This is called for every response. All details of the request and response are available via the 'msg' parameter, as detailed below. The response is also available as a DOM structure via the ‘source’ parameter.


You can implement one or both of these methods depending on your requirements.
You can examine any part of the request and response in order to find potential vulnerabilities, but you must not change anything.


If you find a potential vulnerability then you can raise it via the method:
PassiveScanThread.raiseAlert(int id, Alert alert)


An HttpMessage is passed in to both of the ‘scan’ methods. This class has methods that allow you to access all aspects of the request and response, although the latter is obviously only available in scanHttpResponseReceive.
Some examples include:
  • msg.getRequestHeader().getMethod();
  • msg.getRequestHeader().getHttpCookies();
  • msg.getRequestHeader().getHeaders();
  • msg.getRequestHeader().getContentLength();
  • msg.getRequestBody().toString();
  • msg.getResponseHeader().getHeaders();
  • msg.getResponseHeader().getStatusCode();
  • msg.getResponseBody().toString();


A Source parameter is passed into scanHttpResponseReceive - this is a DOM representation of the response generated by the Jericho HTML parser.
See the Jericho documentation or the other scan rules for examples of how to access DOM elements.


The Alert class is used to represent potential vulnerabilities.
It supports the following fields:
  • pluginId        Used to identify the scanner, especially useful via the ZAP API
  • name           The summary displayed to the user
  • risk              An indication of how serious the issue is:
    • Alert.RISK_INFO               Informational (its not really a vulnerability)
    • Alert.RISK_LOW               A low level vulnerability
    • Alert.RISK_MEDIUM          A medium level vulnerability
    • Alert.RISK_HIGH               A high level vulnerability
  • reliability      An indication of how likely this is a real problem:
    • Alert.FALSE_POSITIVE    Should not be used - this is for the user to set
    • Alert.SUSPICIOUS            A lower level of confidence
    • Alert.WARNING                 A higher level of confidence
  • description  A more detailed description
  • uri                The URI affected
  • param          The name of the vulnerable parameter, if relevant
  • attack          The attack string used (not relevant for passive vulnerabilities)
  • otherInfo      Information that doesnt readily fit into any of the other fields
  • solution       Information about how to prevent the vulnerability
  • reference    A list of URLs giving more information about this type of vulnerability (separated by newline characters)
  • evidence     A string present in the request or response which can be used as evidence of the vulnerability - this will be highlighted when the related request or response is displayed
  • cweId          The CWE id
  • wascId        The WASC Threat Classification id

Simple example

The ExampleSimplePassiveScanner class implements a very simple passive scan rule.
As you will see, it just raises an alert randomly, so it isnt of any practical use.
However it does demonstrate a couple of useful features:


It uses the Vulnerabilities class to get the name, description, solution and references.
This class loads vulnerability details from the vulnerabilities.xml files included with ZAP.
There are actually a set of vulnerabilities.xml files as it is internationalized, so ZAP will read the localized version for the language the user has selected, defaulting back to English for any phrases that have not been translated.
This is therefore a quick and easy way to fill in these details, as long as the relevant vulnerability is included in that file.


It also uses the log4j Logger class to output debug messages. This is the recommended way of outputting such messages.


Note that the pluginId needs to be unique across all active and passive scan rules. The master list of ids is alert.xml

File based example


The ExampleFilePassiveScanner class implements a slightly more complex passive scan rule.
In this case it reads in a set of strings from a configuration file and checks for their presence in the response.
It could also use hardcoded strings, but the advantage of the approach taken is that a knowledgeable user could manually edit the file to meet their requirement.
The build process handles the file deployment, as described later.


This class also demonstrates a couple of other features:


Instead of using the Vulnerabilities class the code uses Constant.messages.getString(str)
All of the strings used in this way are defined in the Messages.properties file.
If you are just implementing the rule for your own benefit then you can hardcode the strings if you want, but internationalizing them is very simple and saves having to go back and change you code if you want to have your rule included in the ZAP Marketplace.


The code also makes use of the getLevel() method.
This returns an AlertThreshold class which indicates how strictly you should check for vulnerabilities.
The threshold returned can be one of:
  • LOW:         This indicates you should report more potential vulnerabilities, which might mean more false positives
  • MEDIUM:   This is the default level
  • HIGH:         This indicates you should report more fewer vulnerabilities, which might mean more false negatives
You do not have to use the threshold - especially as it might not be relevant for the vulnerability you are testing for, but it is also a useful way for the user to tune how the rules work and so its worth using if you can.

Building and deploying


All ZAP add-ons are build using Apache Ant.
For the alpha passive scan rules the build file is: branches/alpha/build/build.xml
All you need to do is run the deploy-pscanrulesAlpha target and the relevant add-on will be built and copied to the correct location, assuming you have a ZAP core project called ‘zaproxy’.
If you want to deploy to a different location then you can change it at the top of the file. 
ZAP automatically extracts the files specified in the manifest into a directory underneath the ZAP user directory.
An knowledgeable user can manually edit these files and any changes will take affect when ZAP is restarted.

Updating the help and manifest


There are a couple more things that you can do to finish off a new rule.


The first is to add a short description of the rule to the help file: pscanalpha.html

This is not really necessary unless you want to publish your rules.


The second is to include the new class along with any files it uses in the add-on manifest: ZapAddOn.xml
ZAP uses this file to identify any files that need to be extracted.
It also uses it when it downloads add-ons from the ZAP Marketplace, and adding your classes and files in here will allow ZAP to add, update and remove your rules dynamically.

A future post will cover how to contribute your code back to the ZAP community and progress it from alpha to beta and then release status.

The next post in this series is: Hacking ZAP #4: Active scan rules