Abbot Java GUI Test Framework
- Abbot helps you test your Java UI. It comprises Abbot, which lets you programmatically drive UI components, and Costello (built on Abbot) which allows you to easily launch, explore and control an application. The framework may be used with both scripts and compiled code.
This overview of the Abbot system will give you an idea of the different system components and how it is used. There is also some background concerning the issues involved with Java GUI testing.
You can see what Costello (the script editor) looks like or try it out via Java Web Start by using the links to the right. The screen shot shows a sample application, its UI hierarchy, and some component properties as displayed by Costello.
JUnit is used as the controlling harness for running tests and suites of tests, but scripts can easily be wrapped to run in other environments. Scripts may also be used to create a demo or tutorial for an application. You can drive the actual application and provide popup annotations while it is running.
- Anteater is a testing framework designed around Ant, from the Apache Jakarta Project. It provides an easy way to write tests for checking the functionality of a Web application or of an XML Web service.
The type of tests you can write using Anteater are:
Send a HTTP/HTTPS request to a Web server. When the response comes back, test that it meets certain criteria. You can check for HTTP headers and response codes, and validate the response body with regexp, XPath, Relax NG, or contentEquals tests, plus some binary formats. New tests can be easily added.
Listen for incoming HTTP requests at an given URL on the local machine. When a request comes on that URL, you can check its parameters and/or content, and send a response accordingly.
- Canoo WebTest is a free open source tool for automated testing of web applications.
It calls web pages and verifies results, giving comprehensive reports on success and failure. The White Paper provides an overview of the features and the design rationale. Detailed information is provided in the Syntax Reference and the Install and Troubleshooting guides.
While this site is for the core distribution, you will find extensions, supplementary tools and a latest breaking RSS news feed on the Canoo WebTest Community Site.
Cobertura is a free Java tool that calculates the percentage of code accessed
by tests. It can be used to identify which parts of your Java program are
lacking test coverage. It is based on jcoverage.
- Can be executed from ant or from the command line.
- Instruments Java bytecode after it has been compiled.
- Can generate reports in HTML or XML.
- Shows percent of lines coveraged and branches coveraged for each class,
package, and for the overall project.
- Shows the McCabe cyclomatic code complexity of each class, and the average
cyclomatic code complexity for each package, and for the overall product.
- Can sort HTML results by class name, percent of lines covered, percent of
branches covered, etc. And can sort in ascending or decending order.
- dbMonster is a tool which helps database application developers with tuning the structure of the database, tuning the usage of indexes, and testing the application performance under heavy database load. dbMonster generates as much random test data as you wish and puts it into SQL database. It provides a very pluggable interface and is trivial to use. dbMonster is written in Java.
Enterprise Web Test
- Enterprise Web Test allows Java programmers to write re-usable tests for web applications that, unlike HttpUnit, drive the actual web browser on the actual platform they intend to support. Tests can be leveraged for functional, stress, reliability.
- The Grinder is a JavaTM load-testing framework. It is freely available under a BSD-style open-source license.
The Grinder makes it easy to orchestrate the activities of a test script in many processes across many machines, using a graphical console application. Test scripts make use of client code embodied in Java plug-ins. Most users of The Grinder do not write plug-ins themselves, instead they use one of the supplied plug-ins. The Grinder comes with a mature plug-in for testing HTTP services, as well as a tool which allows HTTP scripts to be automatically recorded.
The Grinder was originally developed for the book Professional Java 2 Enterprise Edition with BEA WebLogic Server by Paco Gómez and Peter Zadrozny. Philip Aston took ownership of the code and reworked it to create The Grinder 2. Philip continues to enhance and maintain The Grinder, and welcomes all contributions. Recently Peter, Philip and Ted Osborne have published the book J2EE Performance Testing which makes extensive use of The Grinder
- Recognizing the importance of GUI testing, we have developed GUITAR, a GUI Testing Framework that presents a unified solution to the GUI testing problem. In the last four years, we have had considerable success in developing new technologies for GUI testing. Most of the results of our research have been published. I encourage you to visit our Publications Page. Our emphasis has been on developing new event-based tools and techniques for various phases of GUI testing
- Haste (High-level Automated System Test Environment) represents an approach to system testing that is philosophically consistent with standard XP unit testing practices. Test code runs in the same address space as the application under test, allowing for ready examination of application state. The fundamental Haste abstractions of Story, Step, and StoryBook provide a framework to implement system tests. Utility classes simplify test development. In addition to acting as XP acceptance tests, Haste tests aid source maintenance and extension, and can play an important role in a release process.
Jacareto is a capture & replay
tool for programs written in Java. You can capture actions on applications and
replay them later on (like macros). Jacareto can be used for many purposes:
- GUI tests
- Creation of animated demonstrations
- Creation of macros
- Qualitative and quantitative analyses of user behavior
There are two front ends included in Jacareto:
- CleverPHL is a GUI front end for Jacareto
- Picorder is a command line front end for Jacareto
- Jameleon is an acceptance-level automated testing tool that separates applications into features and allows those features to be tied together independently, creating test-cases. These test-cases can then be data-driven and executed against different environments. Even though it would be possible to write unit tests using Jameleon, Jameleon was designed with integration and acceptance-level testing in mind. Most bugs are found and fixed by good unit tests. However, this does not eliminate the need to test the application as a whole.
Jameleon was designed to test all sorts of applications. To make this possible, Jameleon was designed with a plug-in model. This means Jameleon was written to be independent of the application implmentation. To make life a little easier, a GUI for Jameleon is currently being worked on. It is therefore quite likely that a Swing plug-in will be implmented.
- Java PathFinder (JPF) is a system to verify executable Java bytecode programs. In its basic form, it is a Java Virtual Machine (JVM) that is used as an explicit state software model checker, systematically exploring all potential execution paths of a program to find violations of properties like deadlocks or unhandled exceptions. Unlike traditional debuggers, JPF reports the entire execution path that leads to a defect. JPF is especially well-suited to finding hard-to-test concurrency defects in multithreaded programs.
While software model checking in theory sounds like a safe and robust verification method, reality shows that it does not scale well. To make it practical, a model checker has to employ flexible heuristics and state abstractions. JPF is unique in terms of its configurability and extensibility, and hence is a good platform to explore new ways to improve scalability.
JPF is a pure Java application that can be run either as a standalone command line tool, or embedded into systems like development environments. It was mostly developed - and is still used - at the NASA
- JBlanket is a tool for assessing and improving method coverage of unit test cases. It is integrated with JUnit and Ant.
- JCrawler is an open-source (under the CPL) Stress-Testing Tool for web-applications. It comes with the crawling/exploratory feature. You can give JCrawler a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. The load parameters (hits/sec) are configurable
- Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
Apache JMeter features include:
- Can load and performance test HTTP and FTP servers as well as arbitrary
database queries (via JDBC)
- Complete portability and 100% Java purity .
- Full Swing and lightweight component support (precompiled JAR uses
- Full multithreading framework allows concurrent sampling by many
threads and simultaneous sampling of different functions by seperate thread
- Careful GUI design allows faster operation and more precise
- Caching and offline analysis/replaying of test results.
- Highly Extensible:
- Pluggable Samplers allow unlimited testing capabilities.
- Several load statistics may be choosen with pluggable timers .
- Data analysis and visualization plugins allow great
extendibility as well as personalization.
input to a test
- Scriptable Samplers (BeanShell is supported in version 1.9.2 and
- JOSIT (Java Observation Simulation Inspection Toolkit) is an open Application Programmer Interface for instrumenting applications written in the Java programming language.
JOSIT seamlessly integrates with any Java application written using Sun Microsystem standard library of Java graphical objects. JOSIT observes user actions, inspects the state of objects and scripts graphical events. Tools written in other programming languages may communicate with JOSIT via sockets.
Possible Applications of JOSIT include:
Logging of interface actions to:
Analyze efficiency of tasks
Support needed interface changes
Automate step by step instructions / User Manual creation
Real time user monitoring for:
Enabling Intelligent Tutoring systems
Intelligent Computer Aided Instruction (ICAI)
Computer Based Training (CBT)
Synchronization for application sharing
- JSpider is:
A highly configurable and customizable Web Spider engine.
Developed under the LGPL Open Source license
In 100% pure Java
You can use it to :
Check your site for errors (internal server errors, ...)
Outgoing and/or internal link checking
Analyze your site structure (creating a sitemap, ...)
Download complete web sites
Any task you want
- JSystem is a framework for writing and running automated tests, based on JUnit. Its main goal is to support automation of functional / system testing.
Features in the current release
- JSystem runner/scenario builder
- A new Swing runner that enables you to manage your test suites/scenarios,
execute scenarios, view and publish reports.
- Fixture Management
- Enables you to define a tree of fixtures, every fixture with setUp,
tearDown and failTearDown methods. Every test that extends SystemTestCase
can set the fixture it should work on top of. When a test is executed the
framework will navigate to the fixture that was set by the test. The
framework remembers the current position in the fixture tree. If two tests
run on top of the same fixture, and they are both executed as part of a
suite one after the other, the framework will navigate to position fixture
tree for the first test. Afterwards no fixture tree navigation will be
needed for the second. As part of this feature a fixture manager UI is
available. You can navigate the fixture tree manually.
- HTML results
- Enables extendable detail reporting service. Now it is possible to log
positive events from your tests and APIs. A hierarchal HTML view of the
results is available. You can use it to track complex functional/system
tests. You can extend it by writing you own reporting layer.
- SUT (System Under Test) independency
- A layer that enables a test to run on different setups. It can be seen as
a way to send parameters to tests.
- System object
- A convention to represent the setup/system you are working on with a
single object. The steps in the tests will be operations on the setup/system
- An extendable methodology able to analyze the test step result. You can
analyze text output (String), database output (ResultSet) and more.
- Latka is a functional (end-to-end) testing tool. It is implemented in Java, and uses an XML syntax to define a series of HTTP (or HTTPS) requests and a set of validations used to verify that the request was processed correctly.
Although Latka currently supports only HTTP and HTTPS request/response validations, it may be expanded to perform other sorts of functional testing as warranted.
- LogiTest is an open source application suite which can be used to test web sites and web applications.
Testing applications is crucial to providing users with the best and most stable application possible. Unfortunately there are very few applications at this time for testing Internet-based applications. This is the gap that the LogiTest suite fills.
The LogiTest suite includes the following tools:
The core application in the LogiTest suite. The LogiTest application is used to record and playback single tests. LogiTest can be used to create functional and regression test for web applications using a simple graphical user interface.
LogiMonitor is used to monitor web applications and send notifications or take actions when a web application fails.
- Marathon is a testing framework for GUI applications developed using Java/Swing. Marathon composes of recorder, runner and editor. The testscripts are composed of python code.
Marathon focuses on end-user testing. One need not know Java or Swing to record test scripts using Marathon. Marthon is kept intentionally simple. The aim to produce test scripts that are readable by everyone on the project. This includes the developers, testers and the customers. Marathon is flexible and provides facilities to extend the functionality available through python scripts or through writing ones own component resolvers.
MaxQ records you using a web site. It turns the links you click on and any
other input into a Python script that you can play back at any time. You might
use it to:
- Check that your web site still works (regression test).
- Check that your web site is producing valid HTML (using JTidy).
- Automatically extract information from, or take some action on, someones
else web site.
Key Benefits and Features
- Free and open source (see the license).
- Scripts are written in Jython, an
implementation of Python. Python
scripts are easy to understand, modify and extend.
- Works from the command line so you can run scripts unattended.
- Understands cookies.
- Written in Java so it runs anywhere.
- Easy to enhance because the source code is simple.
- Scripts can run as JUnit tests.
- Works behind proxy servers.
- Simplicity. MaxQ should be easy to understand and use, provided the user
has a basic understanding of programming concepts. The tool should operate
under the principle of least surprise.
- Practicality. QA engineers must actually be able to use the tool to test
real web sites.
- Pounder is a utility for automating Java GUI tests. It allows developers to dynamically load GUIs, record scripts, and then use those scripts in a test harness. Requires Java 1.4. It is licensed under the GNU Library General Public License.
Pounder differentiates itself from other utilities by allowing you to examine the results of a test run in source, while maintaining a separate GUI script that can be re-recorded if necessary. Browse the Howto and Internals pages for more details on what features are available, and how Pounder works.
- RUBiS is an auction site prototype modeled after eBay.com that is used to evaluate application design patterns and application servers performance scalability.
Our auction site benchmark implements the core functionality of an auction site: selling, browsing and bidding. We do not implement complementary services like instant messaging or newsgroups. We distinguish between three kinds of user sessions: visitor, buyer, and seller. For a visitor session, users need not register but are only allowed to browse. Buyer and seller sessions require registration. In addition to the functionality provided during visitor sessions, during a buyer session users can bid on items and consult a summary of their current bids, rating and comments left by other users. Seller sessions require a fee before a user is allowed to put up an item for sale. An auction starts immediately and lasts typically for no more than a week. The seller can specify a reserve (minimum) price for an item.
RUBiS is a free, open source initiative.
- The SLAMD Distributed Load Generation Engine (SLAMD) is a Java-based application designed for stress testing and performance analysis of network-based applications. It was originally developed by Sun Microsystems, Inc., but it has been released as an open source application under the Sun Public License, which is an OSI-approved open source license.
SLAMD was originally developed for the purpose of benchmarking and analyzing the performance of LDAP directory servers, and it is the most powerful and flexible tool available for this task. However, it is also well-suited for testing other kinds of network applications and has been used for things like Web servers and Web-based applications, relational databases, and mail servers. It can also be used for non-network based applications (and in fact, it is used for comparing things like CPU power and memory latency across a number of different kinds of systems), although its distributed nature makes it ideal for systems that can be accessed remotely.
- SlimDog offers a simple script based webapplication testing tool. It is based on httpunit. The tool offers a wide range of commands to work with forms, check the content of tables and navigation between HTML pages. Rather than writing long JUnit testcases or crucial XML files the users can write simple text scripts. Each line of the script file will contain one command which is a testnode. All commands inside one file will be processed as a testcase. The syntax of every command is simple and easy to learn. Several scripts can be combined to a testsuite. The results are written either to the console, a file or as a HTML page.
- Solex is a Web application testing tool built as a plug-in for the Eclipse IDE. It provides functions to record a client session, adjust it according to various parameters and replay it later typically in order to ensure non regression of the application behaviour (with stress testing capabilities being added at a later stage).
By recording, we mean that Solex acts as an HTTP proxy and records all HTTP requests and responses going through the wire between a Web client (eg. a Web browser) and a Web server. The task of replaying a scenario consists in sending the previously recorded and eventually customized HTTP requests to the server and asserting each response
- TestMaker is a test automation framework and utility to help you understand and solve functionality, scalability and performance of services.
TestMaker is a utility and framework to build test automation solutions. TestMaker is available for free under an open-source distribution license.
- UISpec4J is an Open Source functional and/or unit testing library for Swing-based Java applications, built on top of the JUnit test harness.
If you are writing a Swing application, you will appreciate UISpec4J above all for its simplicity: UISpec4Js APIs are designed to hide as much as possible the complexity of Swing, resulting in easy to write and easy to read test scripts. This is especially true when comparing UISpec4J tests with those produced using Swing or low-level, event-based testing libraries.
- API designed for readability: our APIs are designed to
make the test scripts as close as possible to human-readable text. We avoid
complex language construct such as casts or anonymous classes, and try to
use data structures well suited to the data to be checked (indented ASCII
for checking the hierarchical contents of a Tree, two-dimensional arrays for
checking the contents of a Table, etc.).
- Smart component search: the framework makes it possible
to find UI components without requiring that the developers assign names to
all of them. For instance, it is possible to find heavyweight components
such as trees or tables without specifying any name, and it is possible to
retrieve components such as buttons or check boxes using their displayed
- Rely on displayed information only: the library checks
the labels displayed by components such as tables, trees or lists,
instead of the string representation of their internal model data. To
achieve this, the UI components check the values shown by the Swing
- Invisible execution: the framework does not require that
the GUI be visible when the tests are being run. This is useful for instance
when running the tests suite on a "headless" integration server.
- No modifications needed in production code - you do not
need to ship UISpec4Js JAR with your deliverable product, and GUI tests can
be written without needing to adapt your application design.
- Extensibility: you can add your own custom UISpec4J
components to the library, for instance to support third-party Swing
- Smart color specification: colors are specified either
with a string representation (in hexadecimal format, as in HTML), or with a
human-readable name which is an approximation of the color - for instance,
AA1100 is still "red".
- The Open Web Application Security Project (OWASP) is dedicated to finding and fighting the causes of insecure software. Our open source projects and local chapters produce free, unbiased, open-source documentation, tools, and standards. The OWASP community also facilitates conferences, local chapters, articles, papers, and message forums. The OWASP Foundation, a not-for-profit charitable organization, ensures the ongoing availability and support for our work. Participation in OWASP is free and open to all, as are all the materials here.