Can a Jira plugin keep up with a dedicated solution?

Itā€™s tempting to manage testing in Jira, and the Xray plugin helps you do exactly that. Xray is good, but how much more will you get with a dedicated test management system like aqua? Find out below.

Key differences between aqua and Xray

aqua offers free AI-enhanced testing

aqua keeps On-Premise

aqua has better reporting

AI

AI is a rapidly developing tech, so we will look at cutting-edge tech rather than the baseline. Hereā€™s what you should expect from an AI-powered test management solution:

  • Analyse your workspace to get QA context
  • Create test cases, defects, and even requirements from a plain text description
  • Complete test case drafts
  • Prioritise tests
  • Remove duplicate tests and defects

As a standalone solution, aqua has the freedom to offer new functionality without any constraints. This includes AI test generation powered by OpenAIā€™s GPT model. aquaā€™s AI copilot is tuned to fit quality assurance much better than ChatGPT, and it also understands the context of your projects and existing tests. Xray has none of this.

Auto-train AI Copilot with your tests
No AI capabilities
Generate entire tests with AI
No AI capabilities
Complete test drafts
No AI capabilities
Prioritise tests
No AI capabilities
Remove duplicates
No AI capabilities
Auto-train AI Copilot with your tests
Generate entire tests with AI
Complete test drafts
Prioritise tests
Remove duplicates
No AI capabilities
No AI capabilities
No AI capabilities
No AI capabilities
No AI capabilities

Verdict: aqua wins (Xray has no AI-powered testing)

Test management

Test management is the main reason we are looking at these two tools. You should consider various aspects of handling test cases as well as traceability-minded features. These include:

  • Create, execute, share, and manage test cases in the workspace
  • Group tests into test scenarios, segment them into folders, assign labels
  • Quickly navigate the test suite, apply filters, use individual and shared views
  • See the entire execution history for a test case
  • Store a full log of changes for compliance
  • Visualise requirements coverage and dependencies between requirements, test cases, and defects

aqua and Xray are closely matched when it comes to test management. aqua, however, has the advantage of being an independent QA-first solution rather than a Jira plugin. This means custom QA-tuned workflows, the functionality to create nested test cases and include shared test steps. Some other features, such as workflows and execution history, just work better in a native QA tool like aqua.

  • Test cases, test scenarios, bulk test edits, custom fields
  • Execution history
  • Workflows
  • Quick navigation, filters, views
  • Item change history, logging
Shared test steps
Not available
Nested test cases
Not available
Change reversal
Not available
Shared views
Not available
  • Test cases, test scenarios, bulk test edits, custom fields
  • Execution history
  • Workflows
  • Quick navigation, filters, views
  • Item change history, logging
Shared test steps
Nested test cases
Change reversal
Shared views
Not available
Not available
Not available
Not available
Verdict: aqua wins by native test organisation features

Integrations

Most tools rely on automation via common third-party tools that QA specialists have been using for over a decade. The experience is much better when your test management solution has native integrations for industry-leading tools. REST API support is a must if you do not want to be at the vendorā€™s mercy for integrations.

  • First-party framework automation or third-party integrations with test automation tools
  • Native third-party integrations: Selenium, Jenkins, JMeter, SoapUI, Ranorex
  • Jira integration and/or Jira plugin
  • REST API support for integrating any tool

Xray takes a DIY approach to test automation. It gives you the space to set up various automation frameworks such as Cucumber and JUnit rather than an out-of-the-box solution. aqua, on the other hand, invites you to use industry-leading third-party solutions and offers native integrations with nearly a dozen of them. You can still use any frameworks including Cucumber and JUnit with REST API. The difference may seem small (you achieve the same goal), but it has a strong impact on modern QA workflows. With test automation becoming increasingly important, you want even less experienced specialists to be comfortable with it. Learning proven tools with large community support to use with aqua is much easier than taking up entire frameworks for Xray. The effort of learning a framework just to align with the DIY automation approach can go in vain, too. In February 2023, the last person working on Cucumber full-time was laid off. While open-source support wonā€™t stop there, this is a major setback for teams that built a long-term automation strategy around Cucumber.

  • REST API
  • Jira integration
Selenium, Jenkins, UFT, JMeter, SoapUI, Ranorex, Database MSSQL, Database Oracle for test automation
DIY test automation frameworks only
Bug reporting tool Capture
No native bug reporting integration
  • REST API
  • Jira integration
Selenium, Jenkins, UFT, JMeter, SoapUI, Ranorex, Database MSSQL, Database Oracle for test automation
Bug reporting tool Capture
DIY test automation frameworks only
No native bug reporting integration
Verdict: aqua wins by proper native QA integrations with more than Jira

Pricing flexibility

Pricing can vary a lot; you will find small differences that make no sense but also high fees that are completely justified. Here are some factors to consider:

  • Extra functionality beyond test management
  • Transparent pricing without locking essential features behind the most expensive plans
  • Custom enterprise plan
  • Price-to-volume scaling that suits your team size
  • No exorbitant setup and/or maintenance fee
  • Free licences for guests / non-tech specialists / manual testers

When considering Xrayā€™s pricing, you need to include pricing of Jira as well. You will need Jira Premium to match aquaā€™s project management and requirements functionality. Free licences, however, also play a big role. It hurts to pay the full fee just so all devs could access bug reports. It stings when there is no free read-only access for non-tech personnel, too. aqua offers free licences for both scenarios.

Jira license is not required
Jira license is required
Free licences for manual testers
Full-price licences only
Free licences for non-tech users
Full-price licences only
Jira license is not required
Free licences for manual testers
Free licences for non-tech users
Jira license is required
Full-price licences only
Full-price licences only

Verdict: aqua wins by free licences even for testers

Deployment Models

On-Premise deployment is a must in many industries, yet vendors wonā€™t explicitly say if they donā€™t offer it. Depending on your security policies and scale, deploying in a non-vendor Cloud is a beneficial option too.

  • Vendor Cloud
  • External Cloud: AWS, Azure, etc.
  • On-Premise

aqua supports both Cloud and On-Premise deployment. You can even choose to host your workspace on some other server rather than aquaā€™s. Xray offers the same, but with a major caveat: Jira is sunsetting the On-Premise offering on December 31, 2023. As a Jira plugin, Xray will suffer the same fate.

  • On-Premise
GDPR-compliant aqua-hosted German Cloud
Jira-hosted Cloud
Isolated Enterprise Custom Cloud (Any Azure data centre)
Custom Cloud
  • On-Premise
GDPR-compliant aqua-hosted German Cloud
Isolated Enterprise Custom Cloud (Any Azure data centre)
Jira-hosted Cloud
Custom Cloud
Verdict: aqua wins by maintaining On-Premise

Dashboards

QA dashboards serve two vital purposes. They help the QA team track their progress, but they also make other teams aware of potential bottlenecks. At a minimum, you should be able to include any data and share dashboards with the team.

  • Use all data of the workspace
  • Create private and shared dashboards
  • Choose between various types of graphs
  • Set up KPI/values alerts that arrive as email/app notifications

Dashboards are not a strong suite of Xray. You are limited to just 6 widgets that revolve around basic requirements coverage and test runs. aqua, on the other hand, can display any data and send you KPI alerts if a certain metric goes abnormally high (or low). These alerts shift the paradigm: dashboards look out for you rather than make you look at them.

Any data
6 widgets
KPI Alerts
No KPI Alerts
Any data
KPI Alerts
6 widgets
No KPI Alerts
Verdict: aqua wins by customisation and KPI Alerts

Reporting

Reporting is important for both internal and external stakeholders. The goal here is to get what you need with as little or as much effort as possible. When a tool offers both a template library and rich customisation, that is a good start.

  • Access vendor-provided template library
  • Create custom layouts and save them
  • Use and transform any data
  • Apply text & image formatting
  • Add external data and imagery
  • Employ scripts and parametrisation to auto-update reports

Xrayā€™s reporting is template-based as well. You can choose between 8 basic templates, and you can add any extra data only by making a Word/Excel template first. There is no native functionality to add external imagery and text as well.

  • Template library
  • Diagrams
Native custom reports
Manual Word/Excel templates only
Native use of any data
Manual Word/Excel templates only
Native text & image formatting
Manual Word/Excel templates only
Drag-and-drop
Not available
External text & imagery
Not available
Native custom layout
Manual Word/Excel templates only
Pivot tables
Not available
Scripts
Not available
Parametrisation
Not available
  • Template library
  • Diagrams
Native custom reports
Native use of any data
Native text & image formatting
Drag-and-drop
External text & imagery
Native custom layout
Pivot tables
Scripts
Parametrisation
Manual Word/Excel templates only
Manual Word/Excel templates only
Manual Word/Excel templates only
Not available
Not available
Manual Word/Excel templates only
Not available
Not available
Not available
Verdict: aqua wins by offering out-of-the-box custom reports

User management

Precise user management is essential when working on multiple projects and/or working with external specialists. It will also save you a lot of pain from running crowd testing in the same test management solution.

  • Set default roles
  • Create custom roles
  • Apply per-user permissions
  • Authenticate users via SSO

Xray is once again limited by Jiraā€™s functionality for user management. It does not introduce any QA-specific permissions. Just like Jira, Xray does not offer per-user permissions either. You may not need that flexibility now, but crowdtesting and new employee onboarding go much smoother with individual permissions.

  • SSO
  • Default user roles
  • Custom roles
Individual permissions
Role permissions only
QA-specific permissions
Regular Jira permissions only
  • SSO
  • Default user roles
  • Custom roles
Individual permissions
QA-specific permissions
Role permissions only
Regular Jira permissions only
Verdict: aqua wins by individual role permissions

ALM

This is not a hard requirement, but you may be interested in a test management solution that also handles the entire product lifecycle. This is a great money saver as you need licences from fewer vendors, and the synergy should save you some hours as well.

  • Test case management
  • Requirements management
  • Defect management
  • Project management

Jira is a project management solution that you expand with Xray to perform test management. aqua is a QA-minded application lifecycle management tool. This is a tie in our comparison of aqua and Xray testing tools: you get 4 solutions in 1 with both vendors.

  • Test case management
  • Defect management
  • Requirements management
  • Project management
  • Test case management
  • Defect management
  • Requirements management
  • Project management
Details verdict icon: Tie
Verdict: Tie

What people say

Here are a few things people like and dislike about both tools.

star star star star star

ā€œI was surprised to find such a comprehensive and mature tool for test management in the German market without having taken it seriously beforehand.ā€

Jƶrg GroƟmann

Head of Development at Bank 11

star star star

ā€œAccording to me, Xray is the best test management tool in Jira.The only thing bothers a lot is the order of test cases gets disorganised each time a test execution or test plan gets created.ā€

Dinesh P

Software Development Engineer at a small business (<50 emp.)

star star star star star

ā€œThe reporting is meaningful and provides a good basis for decisions. After the employees have used aqua, they recognize the added value very quickly.ā€

Thomas Haeske

Head of Organisation/IT at Berlin Hyp

star star star

ā€œ- Navigating test execution in test coverage could be simpler - No version control - No availability to link test execution to the sub-taskā€

A G2 Reviewer

Computer Software (50-1000 emp.)

star star star star star

ā€œManual test cases are easily automated with aqua. Seamless integration with test automation tools helps here.ā€

Jƶrn-Hendrick Sƶrensen

Test Manager at KBA

star star star

ā€œIt's a bit of a learning curve when getting started. It took our team about a year to realise that there are so many other tools to utilise, but didn't really understand how to get to that point. On top of that, Xray is different by JIRA account type, so the documentation is different. If you don't find the correct documentation that applies to your company's JIRA account (Cloud, Server, etc...), you might end up reading through the wrong one, and being utterly confused. ā€œ

Sarah P..

QA Engineer at a small business (<50 emp.)

star star star star star

ā€œI was surprised to find such a comprehensive and mature tool for test management in the German market without having taken it seriously beforehand.ā€

Jƶrg GroƟmann

Head of Development at Bank 11

star star star star star

ā€œThe reporting is meaningful and provides a good basis for decisions. After the employees have used aqua, they recognize the added value very quickly.ā€

Thomas Haeske

Head of Organisation/IT at Berlin Hyp

star star star star star

ā€œManual test cases are easily automated with aqua. Seamless integration with test automation tools helps here.ā€

Jƶrn-Hendrick Sƶrensen

Test Manager at KBA

star star star

ā€œAccording to me, Xray is the best test management tool in Jira.The only thing bothers a lot is the order of test cases gets disorganised each time a test execution or test plan gets created.ā€

Dinesh P

Software Development Engineer at a small business (<50 emp.)

star star star

ā€œ- Navigating test execution in test coverage could be simpler - No version control - No availability to link test execution to the sub-taskā€

A G2 Reviewer

Computer Software (50-1000 emp.)

star star star

ā€œIt's a bit of a learning curve when getting started. It took our team about a year to realise that there are so many other tools to utilise, but didn't really understand how to get to that point. On top of that, Xray is different by JIRA account type, so the documentation is different. If you don't find the correct documentation that applies to your company's JIRA account (Cloud, Server, etc...), you might end up reading through the wrong one, and being utterly confused. ā€œ

Sarah P..

QA Engineer at a small business (<50 emp.)

Final verdict

aqua is a test management system (or even a QA-first ALM) solution with a flexible and convenient 2-way Jira integration, while Xray is a plugin to shoehorn test management into Jira. aqua boasts several highly impactful features and lacks crucial limitations that Xray has. By the end of 2023, only aqua will be an option for teams that work On-Premise.

Bleeding-edge AI-powered testing functionality
Traditional QA only
10+ testing integrations & REST API
DIY frameworks + REST API
Autonomous solution
Relies on Jira to work
Native custom reports
Limited, Microsoft/Excel layout only
Individual permissions
Role permissions only
Continued On-Premise support
On-Premise discontinued by 2024
Bleeding-edge AI-powered testing functionality
10+ testing integrations & REST API
Autonomous solution
Native custom reports
Individual permissions
Continued On-Premise support
Traditional QA only
DIY frameworks + REST API
Relies on Jira to work
Limited, Microsoft/Excel layout only
Role permissions only
On-Premise discontinued by 2024