Why companies expand from Xray testing to aqua
Automation Management
27 mins read
July 9, 2024

Why companies expand from Xray testing to aqua

Test management solutions are not necessarily standalone solutions. Some can be complementing other tools, such as Jira. Letā€™s see how one such Jira addon, Xray, fares against the standalone solution aqua.

photo
photo
Denis Matusovskiy
Martin Koch

Deployment models

aqua has both Cloud and On-Premise offerings, the latter offering extra security and/or mandatory regulatory compliance. There is also some middle ground in hosting the test management solution on a private cloud server. Doing that decouples security and uptime risks from those of the vendor.

Being a Jira plugin, Xrayā€™s deployment model matches what your Jira instance uses. It too can be vendor cloud, private cloud, or on-premise. Just remember that Atlassian stopped active development of On-Premise Jira and will pull the plug on customer support and security updates in 2024.

aqua deployment models

aqua deployment models

Deployment model
Xray
aqua
Yes
Yes
Yes
Yes
Yes
Yes

*Jira discontinues On-Premise in 2024

User Management

Once again, Xray is bound to Jiraā€™s limitations when it comes to user permissions. Jira offers 40 permissions that can be assigned to roles and even individuals. This pales in comparison with aqua, which offers over 100 permissions. It also stings that Xray canā€™t provide any QA-specific permissions, e.g. limiting developersā€™ input on test cases.

aqua's permission editor

aqua’s permission editor

Criteria
Xray
aqua
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
No
Yes
Yes
Yes

Integrations & Automated Testing

Xray takes a DIY approach to test automation, which is much more barebones than a lot of test automation tools provide. The usual paths are either providing out-of-the-box integrations or modifying an open-source automation framework to offer first-party automation. It is also in good taste to provide REST API support so that customers can connect any third-party tool they like.

Defying the conventions, Xray merely supports several popular automation frameworks and thus challenges your testers to already know them. This may reduce the time spent on adopting new tools, but remember than REST API support ā€” which Xray has ā€” wouldā€™ve been sufficient for that. By contrast, aqua supports 10 automation tools out-of-the-box while also offering REST API.

aqua also brings a unique integration among test management solutions ā€” a Chrome extension to run tests and immediately document them. aqua Test Recorder enables you to record interactions, edit captured data, leave comments, annotate screenshots, and then send everything to your companyā€™s aqua server. This approach saves time on all the tab switching, and immediate documentation ensures the QA specialists share everything they observed.

For context, virtually all aqua competitors lack a first-party Chrome extension. They have to rely on third-party solutions that often cost extra money and lack native integrations with test management solutions.

Feature
Xray
aqua
No
Yes
Yes ( Jira )
Yes ( Jira link, Jira sync )
No
Yes ( 10 tools )
Yes ( DIY frameworks )
No
Yes
Yes
Yes ( Jira marketplace options )
No
No
Yes
Yes ( Jira marketplace options )
Yes
Full list of aqua integrations

Full list of aqua integrations

Test Management

Xray does a pretty decent job of leveraging Jiraā€™s functionality for testing purposes. There are however, some notable omissions, such as nested test cases and bulk editing of test cases. The latter can be extremely wasteful: is there much value from organising tests in test scenarios if you canā€™t edit them all at once?

Some limitations stem from Jira itself. Xray users canā€™t change share their Views of the server or individual projects. Change reversal is also unavailable.

Editing test case in Xray

Editing test case in Xray

Defining test steps in Xray

Defining test steps in Xray

 

Test management with aqua

Feature
Xray
aqua
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes

Dependencies & Sustainability

As a Jira addon, Xray requires that you use Jira. This is not necessarily bad for companies that already use Jira, but it does mean putting all your eggs in the same basket. If Jira is down, itā€™s not only devs that struggle to work ā€” QA people are blocked as well.

Performance-wise, there are plenty of reports that Jira gets slower as you add more plugins to it (even Atlassian admits that). This could mean that using a Jira plugin for QA would make the experience worse for both devs regularly using aqua and testers tethered to it. Independent solutions would have been preferable in that case, especially if the Jira instance is packed with plugins.

On the other hand, aqua can communicate and even fully sync with Jira while being a standalone tool. There is no intention to discontinue the On-Premise version as Atlassian plans to do with Jira from 2024. It does sound like the best of two worlds: you get to work with Jira when needed but are not dependent on it.

Criteria
Xray
aqua
Yes ( Jira plugin )
Yes ( Jira plugin or database sync )
No
Yes
Yes ( Tied to Jira* )
Yes ( Full support )

*Jira discontinues On-Premise in 2024

Dashboards

Dashboards are often an overlooked feature when picking a test management solution. They are, however, a very insightful tool for both QA specialists and their colleagues. The freedom to use data, enhance it with extra text/media, and visualise it improves testing processes as well as their understanding by other stakeholders.

Unfortunately, Xray seems too constrained to provide you with such freedom. Dashboards are limited to 6 widgets that span requirements coverage, test runs, and test evolutions. aqua, on the other hand, can use any data (including custom fields input), shape it into widgets within the dashboard, and even display KPI alerts for any metric.

Test runs summary in Xray

Test runs summary in Xray

aqua dashboard with KPI Alert at work

aqua dashboard with KPI Alert at work

Feature
Xray
aqua
Yes
Yes
No
No
No
Yes
Yes
Yes
No
Yes

Reporting

Just like with dashboards, Xrayā€™s reporting is about working around limitations rather than facing none. You are initially offered 8 reports covering requirements (including requirement traceability matrix), test execution, and testsā€™ content. Anything else needs requires manually creating a template as a Word/Excel file. There is no native functionality to enhance reports with external text or imagery either.

Xray traceability report

Xray traceability report

aqua report example

aqua report example

Feature
Xray
aqua
Yes
Yes
Yes
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
Yes
Yes
No
Yes
No
Yes
No
Yes

ALM

aqua is an Application Lifecycle Management solution. It can handle all of test case management, requirements management, defect management, and project management. Xray canā€™t match that without relying on Jiraā€™s functionality as an issue tracker. Here, aqua is truly the best of two worlds: it provides native ALM functionality and too has a deep integration with Jira.

Feature
Xray
aqua
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Jira+Xray project options

Jira+Xray project options

aqua project plan

aqua project plan

aqua workflow

aqua workflow

Conclusion

aqua is only slightly better at test management and user management, but it outperforms Xray at pretty much everything else with a huge margin. aquaā€™s dashboard and reporting functionalities are head and shoulders above the competitor, and itā€™s the same with test automation. Overall, aqua is a native ALM solution while Xray relies on Jiraā€™s functionality ā€” and it does so in more things than application lifecycle management.

Depending on your security protocol, longevity can be the biggest argument against Xray. While they have a desktop app for running tests, actual test management happens on Jira. Atlassian will be discontinuing the On-Premise version of Jira in 2024, putting users of third-party solutions for test management in a very tough position.

Boost your company with aqua

START FOR FREE
On this page:
See more
Speed up your releases x2 with aqua
Start for free
step
FAQ
What is XRay testing software used for?

XRay is testing software used for managing and tracking software testing efforts in the JIRA platform. It provides a suite of tools and features to help organisations plan, track, and report on their testing activities.

What are the differences between aqua and XRay?

aqua is a standalone ALM tool while Xray is a QA-only addon for Jira. Xray is bound to Jira-wide limitations (e.g. views functionality) while aqua is not. aqua has industry-first AI functionality for testing.

What are the benefits of aqua testing tool?

aqua has AI testing features, industry-leading reporting, and full customisation out of the box. It is designed to save time for both testers and their managers.

closed icon