Deployment models
aqua has both Cloud and On-Premise offerings, the latter offering extra security and/or mandatory regulatory compliance. There is also some middle ground in hosting the test management solution on a private cloud server. Doing that decouples security and uptime risks from those of the vendor.
Being a Jira plugin, Xray’s deployment model matches what your Jira instance uses. It too can be vendor cloud, private cloud, or on-premise. Just remember that Atlassian stopped active development of On-Premise Jira and will pull the plug on customer support and security updates in 2024.
aqua deployment models
Deployment model
Xray
aqua
Vendor Cloud
Yes
Yes
Private Cloud
Yes
Yes
On-Premise
Yes*
Yes
*Jira discontinues On-Premise in 2024
User Management
Once again, Xray is bound to Jira’s limitations when it comes to user permissions. Jira offers 40 permissions that can be assigned to roles and even individuals. This pales in comparison with aqua, which offers over 100 permissions. It also stings that Xray can’t provide any QA-specific permissions, e.g. limiting developers’ input on test cases.
aqua’s permission editor
Criteria
Xray
aqua
Default user roles
Yes
Yes
Custom roles
Yes
Yes
Permissions for roles
40
100+
Individual permissions
No
Yes
QA-specific permissions
No
Yes
SSO
Yes
Yes
Integrations & Automated Testing
Xray takes a DIY approach to test automation, which is much more barebones than a lot of test automation tools provide. The usual paths are either providing out-of-the-box integrations or modifying an open-source automation framework to offer first-party automation. It is also in good taste to provide REST API support so that customers can connect any third-party tool they like.
Defying the conventions, Xray merely supports several popular automation frameworks and thus challenges your testers to already know them. This may reduce the time spent on adopting new tools, but remember than REST API support — which Xray has — would’ve been sufficient for that. By contrast, aqua supports 10 automation tools out-of-the-box while also offering REST API.
aqua also brings a unique integration among test management solutions — a Chrome extension to run tests and immediately document them. aqua Test Recorder enables you to record interactions, edit captured data, leave comments, annotate screenshots, and then send everything to your company’s aqua server. This approach saves time on all the tab switching, and immediate documentation ensures the QA specialists share everything they observed.
For context, virtually all aqua competitors lack a first-party Chrome extension. They have to rely on third-party solutions that often cost extra money and lack native integrations with test management solutions.
Feature
Xray
aqua
Chrome extension
No
Yes
Management Integrations
Yes (Jira)
Yes (Jira link, Jira sync)
Automation integrations
No
Yes (10 tools)
First-party automation
DIY frameworks
No
REST API
Yes
Yes
External versioning
Jira marketplace options
No
Auto-translation in workspace
No
Yes
Misc. integrations
Jira marketplace options
No
Full list of aqua integrations
Test Management
Xray does a pretty decent job of leveraging Jira’s functionality for testing purposes. There are however, some notable omissions, such as nested test cases and bulk editing of test cases. The latter can be extremely wasteful: is there much value from organising tests in test scenarios if you can’t edit them all at once?
Some limitations stem from Jira itself. Xray users can’t change share their Views of the server or individual projects. Change reversal is also unavailable.
Editing test case in Xray
Defining test steps in Xray
Test management with aqua
Feature
Xray
aqua
Test cases
+
+
Test scenarios
+
+
Test reports
+
+
Execution history
+
+
Nested test cases
+
+
Bulk edits
Jira functionality
+
Scripts
-
+
Release planning
Versions, Agile sprints
Versions, Agile sprints
Labelling
+
+
Custom fields
+
+
Parametrisation
+
+
Filters
+
+
Folders
+
+
Views
+
+
Shared views
-
+
Workflows
Jira functionality
+
Logging
+
+
Item change history
+
+
Change reversal
-
+
Comments
+
+
Attachments
+
+
Mentions
+
+
Dependencies & Sustainability
As a Jira addon, Xray requires that you use Jira. This is not necessarily bad for companies that already use Jira, but it does mean putting all your eggs in the same basket. If Jira is down, it’s not only devs that struggle to work — QA people are blocked as well.
Performance-wise, there are plenty of reports that Jira gets slower as you add more plugins to it (even Atlassian admits that). This could mean that using a Jira plugin for QA would make the experience worse for both devs regularly using aqua and testers tethered to it. Independent solutions would have been preferable in that case, especially if the Jira instance is packed with plugins.
On the other hand, aqua can communicate and even fully sync with Jira while being a standalone tool. There is no intention to discontinue the On-Premise version as Atlassian plans to do with Jira from 2024. It does sound like the best of two worlds: you get to work with Jira when needed but are not dependent on it.
Criteria
Xray
aqua
Communicates with Jira
Yes (Jira plugin)
Yes (Jira plugin or database sync)
Works standalone
No
Yes
Planned On-Premise prospects past 2024
Tied to Jira*
Full support
*Jira discontinues On-Premise in 2024
Dashboards
Dashboards are often an overlooked feature when picking a test management solution. They are, however, a very insightful tool for both QA specialists and their colleagues. The freedom to use data, enhance it with extra text/media, and visualise it improves testing processes as well as their understanding by other stakeholders.
Unfortunately, Xray seems too constrained to provide you with such freedom. Dashboards are limited to 6 widgets that span requirements coverage, test runs, and test evolutions. aqua, on the other hand, can use any data (including custom fields input), shape it into widgets within the dashboard, and even display KPI alerts for any metric.
Test runs summary in Xray
aqua dashboard with KPI Alert at work
Feature
Xray
aqua
Dashboards
Yes
Yes
Data
6 widgets or Jira functionality
Any data
Native custom fields data
No
Yes
Native graph types
Pie, bar chart
Pie, bar, stacked bar chart; table chart
KPI Alerts
No
Yes
Reporting
Just like with dashboards, Xray’s reporting is about working around limitations rather than facing none. You are initially offered 8 reports covering requirements (including requirement traceability matrix), test execution, and tests’ content. Anything else needs requires manually creating a template as a Word/Excel file. There is no native functionality to enhance reports with external text or imagery either.
Xray traceability report
aqua report example
Feature
Xray
aqua
Template library
Yes (5 report types)
Yes (7 sample reports)
Custom reports
Yes (Word/Excel templates)
Yes (native)
Pulling any data
-
+
Custom layout
-
+
External text & imagery
-
+
Drag-and-drop
-
+
Custom bands
-
+
Text formatting
-
+
Image formatting
-
+
Diagrams
+
+
Pivot tables
-
+
Scripts
-
+
Parametrisation
-
+
ALM
aqua is an Application Lifecycle Management solution. It can handle all of test case management, requirements management, defect management, and project management. Xray can’t match that without relying on Jira’s functionality as an issue tracker. Here, aqua is truly the best of two worlds: it provides native ALM functionality and too has a deep integration with Jira.
Feature
Xray
aqua
Native ALM
-
+
Test case management
+
+
Requirements management
Jira functionality
+
Defect management
Jira functionality
+
Software dev. methodology
Jira functionality
Agile, Waterfall, Extreme Programming
Workflows
+
+
Project management
Jira functionality
+
Jira+Xray project options
aqua project plan
aqua workflow
Conclusion
aqua is only slightly better at test management and user management, but it outperforms Xray at pretty much everything else with a huge margin. aqua’s dashboard and reporting functionalities are head and shoulders above the competitor, and it’s the same with test automation. Overall, aqua is a native ALM solution while Xray relies on Jira’s functionality — and it does so in more things than application lifecycle management.
Depending on your security protocol, longevity can be the biggest argument against Xray. While they have a desktop app for running tests, actual test management happens on Jira. Atlassian will be discontinuing the On-Premise version of Jira in 2024, putting users of third-party solutions for test management in a very tough position.
