aqua vs Xray: what you should pick for test management

Test management solutions are not necessarily standalone solutions. Some can be complementing other tools, such as Jira. Let’s see how one such Jira addon, Xray, fares against the standalone solution aqua.

aqua vs Xray

Deployment models

Update, 07.06.2022: the article has been amended to clarify that Xray users can rely on Jira for workflows, bulk editing, and dashboards. We are also happy to note that Xray have recently added the nested test cases functionality.

aqua has both Cloud and On-Premise offerings, the latter offering extra security and/or mandatory regulatory compliance. There is also some middle ground in hosting the test management solution on a private cloud server. Doing that decouples security and uptime risks from those of the vendor.

Being a Jira plugin, Xray’s deployment model matches what your Jira instance uses. It too can be vendor cloud, private cloud, or on-premise. Just remember that Atlassian stopped active development of On-Premise Jira and will pull the plug on customer support and security updates in 2024.

aqua deployment models

aqua deployment models

Deployment model

Xray

aqua

Vendor Cloud

Yes

Yes

Private Cloud

Yes

Yes

On-Premise

Yes*

Yes

*Jira discontinues On-Premise in 2024

User Management

Once again, Xray is bound to Jira’s limitations when it comes to user permissions. Jira offers 40 permissions that can be assigned to roles and even individuals. This pales in comparison with aqua, which offers over 100 permissions. It also stings that Xray can’t provide any QA-specific permissions, e.g. limiting developers’ input on test cases.

aqua's permission editor

aqua’s permission editor

Criteria

Xray

aqua

Default user roles

Yes

Yes

Custom roles

Yes

Yes

Permissions for roles

40

100+

Individual permissions

No

Yes

QA-specific permissions

No

Yes

SSO

Yes

Yes

Test Management

Xray does a pretty decent job of leveraging Jira’s functionality for testing purposes. There are however, some notable omissions, such as nested test cases and bulk editing of test cases. The latter can be extremely wasteful: is there much value from organising tests in test scenarios if you can’t edit them all at once?

Some limitations stem from Jira itself. Xray users can’t change share their Views of the server or individual projects. Change reversal is also unavailable.

Editing test case in Xray

Editing test case in Xray

Defining test steps in Xray

Defining test steps in Xray

Test management with aqua

Feature

Xray

aqua

Test cases

+

+

Test scenarios

+

+

Test reports

+

+

Execution history

+

+

Nested test cases

-

+

Bulk edits

Jira functionality

+

Scripts

-

+

Release planning

Versions, Agile sprints

Versions, Agile sprints

Labelling

+

+

Custom fields

+

+

Parametrisation

+

+

Filters

+

+

Folders

+

+

Views

+

+

Shared views

-

+

Workflows

Jira functionality

+

Logging

+

+

Item change history

+

+

Change reversal

-

+

Comments

+

+

Attachments

+

+

Mentions

+

+

Integrations & Automated Testing

Xray takes a DIY approach to test automation, which is much more barebones than a lot of test automation tools provide. The usual paths are either providing out-of-the-box integrations or modifying an open-source automation framework to offer first-party automation. It is also in good taste to provide REST API support so that customers can connect any third-party tool they like.

Defying the conventions, Xray merely supports several popular automation frameworks and thus challenges your testers to already know them. This may reduce the time spent on adopting new tools, but remember than REST API support — which Xray has — would’ve been sufficient for that. By contrast, aqua supports 10 automation tools out-of-the-box while also offering REST API.

Feature

Xray

aqua

Management Integrations

Yes (Jira)

Yes (Jira link, Jira sync)

Automation integrations

No

Yes (10 tools)

First-party automation

DIY frameworks

No

REST API

Yes

Yes

External versioning

Jira marketplace options

No

Auto-translation in workspace

No

Yes

Misc. integrations

Jira marketplace options

No

Full list of aqua integrations

Full list of aqua integrations

Dashboards

Dashboards are often an overlooked feature when picking a test management solution. They are, however, a very insightful tool for both QA specialists and their colleagues. The freedom to use data, enhance it with extra text/media, and visualise it improves testing processes as well as their understanding by other stakeholders.

Unfortunately, Xray seems too constrained to provide you with such freedom. Dashboards are limited to 6 widgets that span requirements coverage, test runs, and test evolutions. aqua, on the other hand, can use any data (including custom fields input), shape it into widgets within the dashboard, and even display KPI alerts for any metric.

Test runs summary in Xray

Test runs summary in Xray

aqua dashboard with KPI Alert at work

aqua dashboard with KPI Alert at work

Feature

Xray

aqua

Dashboards

Yes

Yes

Data

6 widgets or Jira functionality

Any data

Native custom fields data

No

Yes

Native graph types

Pie, bar chart

Pie, bar, stacked bar chart; table chart

KPI Alerts

No

Yes

Reporting

Just like with dashboards, Xray’s reporting is about working around limitations rather than facing none. You are initially offered 8 reports covering requirements (including requirement traceability matrix), test execution, and tests’ content. Anything else needs requires manually creating a template as a Word/Excel file. There is no native functionality to enhance reports with external text or imagery either.

Xray traceability report

Xray traceability report

aqua report example

aqua report example

Feature

Xray

aqua

Template library

Yes (5 report types)

Yes (7 sample reports)

Custom reports

Yes (Word/Excel templates)

Yes (native)

Pulling any data

-

+

Custom layout

-

+

External text & imagery

-

+

Drag-and-drop

-

+

Custom bands

-

+

Text formatting

-

+

Image formatting

-

+

Diagrams

+

+

Pivot tables

-

+

Scripts

-

+

Parametrisation

-

+

ALM

aqua is an Application Lifecycle Management solution. It can handle all of test case management, requirements management, defect management, and project management. Xray can’t match that without relying on Jira’s functionality as an issue tracker. Here, aqua is truly the best of two worlds: it provides native ALM functionality and too has a deep integration with Jira.

Feature

Xray

aqua

Native ALM

-

+

Test case management

+

+

Requirements management

Jira functionality

+

Defect management

Jira functionality

+

Software dev. methodology

Jira functionality

Agile, Waterfall, Extreme Programming

Workflows

+

+

Project management

Jira functionality

+

Jira+Xray project options

Jira+Xray project options

aqua project plan

aqua project plan

aqua workflow

aqua workflow

Conclusion

aqua is only slightly better at test management and user management, but it outperforms Xray at pretty much everything else with a huge margin. aqua’s dashboard and reporting functionalities are head and shoulders above the competitor, and it’s the same with test automation. Overall, aqua is a native ALM solution while Xray relies on Jira’s functionality — and it does so in more things than application lifecycle management.

Depending on your security protocol, longevity can be the biggest argument against Xray. While they have a desktop app for running tests, actual test management happens on Jira. Atlassian will be discontinuing the On-Premise version of Jira in 2024, putting users of third-party solutions for test management in a very tough position.

Try it yourself — 30-day aqua trial

GET TRIAL ACCESS
Related Articles

The need to implement QA at early development stages or execute it in parallel is…

photo
Olga Ryan
5 mins read

Part of the reason test management tools are popular is undoubtedly simplification find improvement of…

photo
photo
photo
Multiple Authors
10 mins read

Welcome to aqua’s product blog post unwrapping release 22.20 - 22.35. This product update covers…

photo
Kate Hornysh
10 mins read
Subscribe to aqua blog

Get the latest posts delivered right to your inbox