Can a Jira plugin keep up with a dedicated solution?

aqua and Xray are closely matched when it comes to test management. aqua, however, has the advantage of being an independent QA-first solution rather than a Jira plugin. This means custom QA-tuned workflows, the functionality to create nested test cases and include shared test steps. Some other features, such as workflows and execution history, just work better in a native QA tool like aqua.

Xray takes a DIY approach to test automation. It gives you the space to set up various automation frameworks such as Cucumber and JUnit rather than an out-of-the-box solution. aqua, on the other hand, invites you to use industry-leading third-party solutions and offers native integrations with nearly a dozen of them. You can still use any frameworks including Cucumber and JUnit with REST API. The difference may seem small (you achieve the same goal), but it has a strong impact on modern QA workflows. With test automation becoming increasingly important, you want even less experienced specialists to be comfortable with it. Learning proven tools with large community support to use with aqua is much easier than taking up entire frameworks for Xray. The effort of learning a framework just to align with the DIY automation approach can go in vain, too. In February 2023, the last person working on Cucumber full-time was laid off. While open-source support won’t stop there, this is a major setback for teams that built a long-term automation strategy around Cucumber.

When considering Xray’s pricing, you need to include pricing of Jira as well. You will need Jira Premium to match aqua’s project management and requirements functionality. Free licences, however, also play a big role. It hurts to pay the full fee just so all devs could access bug reports. It stings when there is no free read-only access for non-tech personnel, too. aqua offers free licences for both scenarios.

aqua supports both Cloud and On-Premise deployment. You can even choose to host your workspace on some other server rather than aqua’s. Xray offers the same, but with a major caveat: Jira is sunsetting the On-Premise offering on December 31, 2023. As a Jira plugin, Xray will suffer the same fate.

Dashboards are not a strong suite of Xray. You are limited to just 6 widgets that revolve around basic requirements coverage and test runs. aqua, on the other hand, can display any data and send you KPI alerts if a certain metric goes abnormally high (or low). These alerts shift the paradigm: dashboards look out for you rather than make you look at them.

Xray’s reporting is template-based as well. You can choose between 8 basic templates, and you can add any extra data only by making a Word/Excel template first. There is no native functionality to add external imagery and text as well.

Xray is once again limited by Jira’s functionality for user management. It does not introduce any QA-specific permissions. Just like Jira, Xray does not offer per-user permissions either. You may not need that flexibility now, but crowdtesting and new employee onboarding go much smoother with individual permissions.

Jira is a project management solution that you expand with Xray to perform test management. aqua is a QA-minded application lifecycle management tool. This is a tie in our comparison of aqua and Xray testing tools: you get 4 solutions in 1 with both vendors.