Deployment models
aqua has both Cloud and On-Premise offerings, the latter offering extra security and/or mandatory regulatory compliance. There is also some middle ground in hosting the test management solution on a private cloud server. Doing that decouples security and uptime risks from those of the vendor.
Being a Jira plugin, Xray’s deployment model matches what your Jira instance uses. It too can be vendor cloud, private cloud, or on-premise. Just remember that Atlassian stopped active development of On-Premise Jira and will pull the plug on customer support and security updates in 2024.
aqua deployment models
Vendor Cloud
Private Cloud
On-Premise
*Jira discontinues On-Premise in 2024
User Management
Once again, Xray is bound to Jira’s limitations when it comes to user permissions. Jira offers 40 permissions that can be assigned to roles and even individuals. This pales in comparison with aqua, which offers over 100 permissions. It also stings that Xray can’t provide any QA-specific permissions, e.g. limiting developers’ input on test cases.
aqua’s permission editor
Default user roles
Custom roles
Permissions for roles
Individual permissions
QA-specific permissions
SSO
Integrations & Automated Testing
Xray takes a DIY approach to test automation, which is much more barebones than a lot of test automation tools provide. The usual paths are either providing out-of-the-box integrations or modifying an open-source automation framework to offer first-party automation. It is also in good taste to provide REST API support so that customers can connect any third-party tool they like.
Defying the conventions, Xray merely supports several popular automation frameworks and thus challenges your testers to already know them. This may reduce the time spent on adopting new tools, but remember than REST API support — which Xray has — would’ve been sufficient for that. By contrast, aqua supports 10 automation tools out-of-the-box while also offering REST API.
aqua also brings a unique integration among test management solutions — a Chrome extension to run tests and immediately document them. aqua Test Recorder enables you to record interactions, edit captured data, leave comments, annotate screenshots, and then send everything to your company’s aqua server. This approach saves time on all the tab switching, and immediate documentation ensures the QA specialists share everything they observed.
For context, virtually all aqua competitors lack a first-party Chrome extension. They have to rely on third-party solutions that often cost extra money and lack native integrations with test management solutions.
Chrome extension
Management Integrations
Automation integrations
First-party automation
REST API
External versioning
Auto-translation in workspace
Misc. integrations
Full list of aqua integrations
Test Management
Xray does a pretty decent job of leveraging Jira’s functionality for testing purposes. There are however, some notable omissions, such as nested test cases and bulk editing of test cases. The latter can be extremely wasteful: is there much value from organising tests in test scenarios if you can’t edit them all at once?
Some limitations stem from Jira itself. Xray users can’t change share their Views of the server or individual projects. Change reversal is also unavailable.
Editing test case in Xray
Defining test steps in Xray
Test management with aqua
Test cases
Test scenarios
Test reports
Execution history
Nested test cases
Bulk edits
Scripts
Release planning
Labelling
Custom fields
Parametrisation
Filters
Folders
Views
Shared views
Workflows
Logging
Item change history
Change reversal
Comments
Attachments
Mentions
Dependencies & Sustainability
As a Jira addon, Xray requires that you use Jira. This is not necessarily bad for companies that already use Jira, but it does mean putting all your eggs in the same basket. If Jira is down, it’s not only devs that struggle to work — QA people are blocked as well.
Performance-wise, there are plenty of reports that Jira gets slower as you add more plugins to it (even Atlassian admits that). This could mean that using a Jira plugin for QA would make the experience worse for both devs regularly using aqua and testers tethered to it. Independent solutions would have been preferable in that case, especially if the Jira instance is packed with plugins.
On the other hand, aqua can communicate and even fully sync with Jira while being a standalone tool. There is no intention to discontinue the On-Premise version as Atlassian plans to do with Jira from 2024. It does sound like the best of two worlds: you get to work with Jira when needed but are not dependent on it.
Communicates with Jira
Works standalone
Planned On-Premise prospects past 2024
*Jira discontinues On-Premise in 2024
Dashboards
Dashboards are often an overlooked feature when picking a test management solution. They are, however, a very insightful tool for both QA specialists and their colleagues. The freedom to use data, enhance it with extra text/media, and visualise it improves testing processes as well as their understanding by other stakeholders.
Unfortunately, Xray seems too constrained to provide you with such freedom. Dashboards are limited to 6 widgets that span requirements coverage, test runs, and test evolutions. aqua, on the other hand, can use any data (including custom fields input), shape it into widgets within the dashboard, and even display KPI alerts for any metric.
Test runs summary in Xray
aqua dashboard with KPI Alert at work
Dashboards
Data
Native custom fields data
Native graph types
KPI Alerts
Reporting
Just like with dashboards, Xray’s reporting is about working around limitations rather than facing none. You are initially offered 8 reports covering requirements (including requirement traceability matrix), test execution, and tests’ content. Anything else needs requires manually creating a template as a Word/Excel file. There is no native functionality to enhance reports with external text or imagery either.
Xray traceability report
aqua report example
Template library
Custom reports
Pulling any data
Custom layout
External text & imagery
Drag-and-drop
Custom bands
Text formatting
Image formatting
Diagrams
Pivot tables
Scripts
Parametrisation
ALM
aqua is an Application Lifecycle Management solution. It can handle all of test case management, requirements management, defect management, and project management. Xray can’t match that without relying on Jira’s functionality as an issue tracker. Here, aqua is truly the best of two worlds: it provides native ALM functionality and too has a deep integration with Jira.
Native ALM
Test case management
Requirements management
Defect management
Software dev. methodology
Workflows
Project management
Jira+Xray project options
aqua project plan
aqua workflow
Conclusion
aqua is only slightly better at test management and user management, but it outperforms Xray at pretty much everything else with a huge margin. aqua’s dashboard and reporting functionalities are head and shoulders above the competitor, and it’s the same with test automation. Overall, aqua is a native ALM solution while Xray relies on Jira’s functionality — and it does so in more things than application lifecycle management.
Depending on your security protocol, longevity can be the biggest argument against Xray. While they have a desktop app for running tests, actual test management happens on Jira. Atlassian will be discontinuing the On-Premise version of Jira in 2024, putting users of third-party solutions for test management in a very tough position.
Boost your company with aqua
