Agentic AI Test Automation for a PACS System

Each code change introduced into the PACS system required manual verification to confirm that the automated test suite remained accurate. While many changes were limited to interface-level updates — a renamed button, an added field, a modified workflow step — a meaningful share involved functional changes to existing behavior, and any of these could silently break a test or introduce a gap in coverage. Given the volume of change in a typical sprint, the team could not review every change with equal rigor, and the test suite gradually fell out of alignment between review cycles.

When a test failure did occur, diagnosing the root cause required significant manual effort: an engineer had to investigate the failure, determine whether it had occurred previously, and confirm whether the underlying requirement had changed. This knowledge was distributed across Jira tickets, historical defect reports, and individual recollection, rather than centralized in a system the test suite could reference directly. As a result, the same category of failure was sometimes re-investigated from first principles more than once, while genuinely new failures were occasionally dismissed as recurring, known issues.

NKORR built an agent that monitors the PACS application's code repository for changes and keeps the test suite aligned with it. The agent operates in four stages. It first identifies what has changed in the code since its last review. It then determines whether the test suite requires updates as a result, and updates or adds tests accordingly. It executes the full suite against the PACS application. Finally, it interprets the results and produces a concise report: what changed, what the agent did to the tests, which tests passed or failed, and whether the release is ready to proceed.

To find the affected tests, the agent reads the Git diff since its last review and maps the changed files to the features they touch. Each test is tagged with the feature or screen it covers, and the agent uses those tags, together with the existing test files, to decide which tests are impacted. The LLM makes the final call on what needs to change.

Behind the update step is a RAG knowledge store that holds past test failures and the requirements behind each feature. Each failure is saved as a record: the test, the feature area, the error message and key stack lines, the root cause, the fix, and the date. When a test fails, the agent retrieves the closest records, and the LLM checks whether the cause is actually the same, not just similarly worded. A confirmed match is reported with its earlier fix. Every new outcome is written back to the store.

The agent also labels each failure before acting on it: application defect, test defect, environment issue, or test-data issue. A locator that breaks right after a UI change points to a test defect. Many unrelated tests failing together, or the app not starting, point to the environment. Failures that depend on leftover or duplicate records point to test data. An assertion that fails against a stated requirement, with no test-side change, points to an application defect. The agent gives the label with its evidence, and low-confidence cases go to an engineer.

The agent writes new tests for new or changed functionality, but it never merges them itself. A small, low-risk fix, such as a locator that clearly points to the same element under a new name, is applied directly. Any new test, or any change to what a test checks, is opened as a pull request with the report as its description. CI re-runs the suite, a person reviews and merges, and the merged change is picked up on the next monitor pass. Figure 1 shows the flow, from a code change to an applied fix or a pull request.

Once this was running, test updates stopped waiting for a person to notice a change and go check it. Most small changes were picked up and reflected in the test suite the same day they were made. Failures that matched something the team had seen before were flagged as such right away, instead of being investigated from scratch again. New, genuinely unfamiliar failures stood out more clearly because they didn't match anything in the failure history. The team spent less time re-explaining old bugs and more time on failures that were actually new. Release reports gave a clear, evidence-based answer to whether a build was safe to ship, instead of a bare pass/fail count.