Why companies expand from Xray testing to aqua

Test management solutions are not necessarily standalone solutions. Some can be complementing other tools, such as Jira. Let’s see how one such Jira addon, Xray, fares against the standalone solution aqua.

aqua vs Xray

Deployment models

aqua has both Cloud and On-Premise offerings, the latter offering extra security and/or mandatory regulatory compliance. There is also some middle ground in hosting the test management solution on a private cloud server. Doing that decouples security and uptime risks from those of the vendor.

Being a Jira plugin, Xray’s deployment model matches what your Jira instance uses. It too can be vendor cloud, private cloud, or on-premise. Just remember that Atlassian stopped active development of On-Premise Jira and will pull the plug on customer support and security updates in 2024.

aqua deployment models

aqua deployment models

Deployment model

Xray

aqua

Vendor Cloud

Yes

Yes

Private Cloud

Yes

Yes

On-Premise

Yes*

Yes

*Jira discontinues On-Premise in 2024

User Management

Once again, Xray is bound to Jira’s limitations when it comes to user permissions. Jira offers 40 permissions that can be assigned to roles and even individuals. This pales in comparison with aqua, which offers over 100 permissions. It also stings that Xray can’t provide any QA-specific permissions, e.g. limiting developers’ input on test cases.

aqua's permission editor

aqua’s permission editor

Criteria

Xray

aqua

Default user roles

Yes

Yes

Custom roles

Yes

Yes

Permissions for roles

40

100+

Individual permissions

No

Yes

QA-specific permissions

No

Yes

SSO

Yes

Yes

Integrations & Automated Testing

Xray takes a DIY approach to test automation, which is much more barebones than a lot of test automation tools provide. The usual paths are either providing out-of-the-box integrations or modifying an open-source automation framework to offer first-party automation. It is also in good taste to provide REST API support so that customers can connect any third-party tool they like.

Defying the conventions, Xray merely supports several popular automation frameworks and thus challenges your testers to already know them. This may reduce the time spent on adopting new tools, but remember than REST API support — which Xray has — would’ve been sufficient for that. By contrast, aqua supports 10 automation tools out-of-the-box while also offering REST API.

aqua also brings a unique integration among test management solutions — a Chrome extension to run tests and immediately document them. aqua Test Recorder enables you to record interactions, edit captured data, leave comments, annotate screenshots, and then send everything to your company’s aqua server. This approach saves time on all the tab switching, and immediate documentation ensures the QA specialists share everything they observed.

For context, virtually all aqua competitors lack a first-party Chrome extension. They have to rely on third-party solutions that often cost extra money and lack native integrations with test management solutions.

Feature

Xray

aqua

Chrome extension

No

Yes

Management Integrations

Yes (Jira)

Yes (Jira link, Jira sync)

Automation integrations

No

Yes (10 tools)

First-party automation

DIY frameworks

No

REST API

Yes

Yes

External versioning

Jira marketplace options

No

Auto-translation in workspace

No

Yes

Misc. integrations

Jira marketplace options

No

Full list of aqua integrations

Full list of aqua integrations

Test Management

Xray does a pretty decent job of leveraging Jira’s functionality for testing purposes. There are however, some notable omissions, such as nested test cases and bulk editing of test cases. The latter can be extremely wasteful: is there much value from organising tests in test scenarios if you can’t edit them all at once?

Some limitations stem from Jira itself. Xray users can’t change share their Views of the server or individual projects. Change reversal is also unavailable.

Editing test case in Xray

Editing test case in Xray

Defining test steps in Xray

Defining test steps in Xray

Test management with aqua

Feature

Xray

aqua

Test cases

+

+

Test scenarios

+

+

Test reports

+

+

Execution history

+

+

Nested test cases

+

+

Bulk edits

Jira functionality

+

Scripts

-

+

Release planning

Versions, Agile sprints

Versions, Agile sprints

Labelling

+

+

Custom fields

+

+

Parametrisation

+

+

Filters

+

+

Folders

+

+

Views

+

+

Shared views

-

+

Workflows

Jira functionality

+

Logging

+

+

Item change history

+

+

Change reversal

-

+

Comments

+

+

Attachments

+

+

Mentions

+

+

Dependencies & Sustainability

As a Jira addon, Xray requires that you use Jira. This is not necessarily bad for companies that already use Jira, but it does mean putting all your eggs in the same basket. If Jira is down, it’s not only devs that struggle to work — QA people are blocked as well.

Performance-wise, there are plenty of reports that Jira gets slower as you add more plugins to it (even Atlassian admits that). This could mean that using a Jira plugin for QA would make the experience worse for both devs regularly using aqua and testers tethered to it. Independent solutions would have been preferable in that case, especially if the Jira instance is packed with plugins.

On the other hand, aqua can communicate and even fully sync with Jira while being a standalone tool. There is no intention to discontinue the On-Premise version as Atlassian plans to do with Jira from 2024. It does sound like the best of two worlds: you get to work with Jira when needed but are not dependent on it.

Criteria

Xray

aqua

Communicates with Jira

Yes (Jira plugin)

Yes (Jira plugin or database sync)

Works standalone

No

Yes

Planned On-Premise prospects past 2024

Tied to Jira*

Full support

*Jira discontinues On-Premise in 2024

Dashboards

Dashboards are often an overlooked feature when picking a test management solution. They are, however, a very insightful tool for both QA specialists and their colleagues. The freedom to use data, enhance it with extra text/media, and visualise it improves testing processes as well as their understanding by other stakeholders.

Unfortunately, Xray seems too constrained to provide you with such freedom. Dashboards are limited to 6 widgets that span requirements coverage, test runs, and test evolutions. aqua, on the other hand, can use any data (including custom fields input), shape it into widgets within the dashboard, and even display KPI alerts for any metric.

Test runs summary in Xray

Test runs summary in Xray

aqua dashboard with KPI Alert at work

aqua dashboard with KPI Alert at work

Feature

Xray

aqua

Dashboards

Yes

Yes

Data

6 widgets or Jira functionality

Any data

Native custom fields data

No

Yes

Native graph types

Pie, bar chart

Pie, bar, stacked bar chart; table chart

KPI Alerts

No

Yes

Reporting

Just like with dashboards, Xray’s reporting is about working around limitations rather than facing none. You are initially offered 8 reports covering requirements (including requirement traceability matrix), test execution, and tests’ content. Anything else needs requires manually creating a template as a Word/Excel file. There is no native functionality to enhance reports with external text or imagery either.

Xray traceability report

Xray traceability report

aqua report example

aqua report example

Feature

Xray

aqua

Template library

Yes (5 report types)

Yes (7 sample reports)

Custom reports

Yes (Word/Excel templates)

Yes (native)

Pulling any data

-

+

Custom layout

-

+

External text & imagery

-

+

Drag-and-drop

-

+

Custom bands

-

+

Text formatting

-

+

Image formatting

-

+

Diagrams

+

+

Pivot tables

-

+

Scripts

-

+

Parametrisation

-

+

ALM

aqua is an Application Lifecycle Management solution. It can handle all of test case management, requirements management, defect management, and project management. Xray can’t match that without relying on Jira’s functionality as an issue tracker. Here, aqua is truly the best of two worlds: it provides native ALM functionality and too has a deep integration with Jira.

Feature

Xray

aqua

Native ALM

-

+

Test case management

+

+

Requirements management

Jira functionality

+

Defect management

Jira functionality

+

Software dev. methodology

Jira functionality

Agile, Waterfall, Extreme Programming

Workflows

+

+

Project management

Jira functionality

+

Jira+Xray project options

Jira+Xray project options

aqua project plan

aqua project plan

aqua workflow

aqua workflow

Conclusion

aqua is only slightly better at test management and user management, but it outperforms Xray at pretty much everything else with a huge margin. aqua’s dashboard and reporting functionalities are head and shoulders above the competitor, and it’s the same with test automation. Overall, aqua is a native ALM solution while Xray relies on Jira’s functionality — and it does so in more things than application lifecycle management.

Depending on your security protocol, longevity can be the biggest argument against Xray. While they have a desktop app for running tests, actual test management happens on Jira. Atlassian will be discontinuing the On-Premise version of Jira in 2024, putting users of third-party solutions for test management in a very tough position.

Boost your company with aqua

START FOR FREE
Related Articles

Are you struggling to make your QA match the pace and workflows of your development?…

photo
Denis Matusovskiy
15 mins read

User interface testing is arguably the most impactful type of testing, certainly so for B2C…

photo
Robert Weingartz
9 mins read

In the fast-paced age we live in, slow or crumbling websites and applications often mean…

photo
Denis Matusovskiy
9 mins read