--- title: "Leveraging GitHub Copilot for Testing" canonical: "https://docs.getxray.app/space/XRAYCLOUD/1332707346/Leveraging%20GitHub%20Copilot%20for%20Testing" format: markdown --- > Macro (rw-ui-expands-macro) > > > Macro (rw-expand) > > > Macro (toc) # Introduction [GitHub Copilot](https://github.com/copilot) is an AI coding assistant and AI peer programmer from Microsoft. Copilot can be used from: - Within multiple IDEs [(such as VS Code)](https://code.visualstudio.com/docs/copilot/overview): to work with a local repository of code, having access to MCP servers. - The [GitHub Copilot website](https://code.visualstudio.com/docs/copilot/overview): to ask questions about repositories on GitHub or to delete background tasks to work on, for those repositories; the coding agent is preconfigured with some default MCP servers (e.g., to enable integration with Playwright, GitHub) and [can](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/extend-coding-agent-with-mcp) have access to further MCP servers. - The terminal using the [GitHub Copilot CLI](https://github.com/features/copilot/cli): to run a native agent in your terminal that works directly with your Issues and Pull Requests (PRs), execute subagents, and more, having access to MCP servers. - Automatically, on GitHub, to make code reviews or to implement certain tasks using the coding agent. ## Chat Modes Whenever interacting with Copilot, especially from within IDEs, there are several [chat modes](https://github.blog/ai-and-ml/github-copilot/copilot-ask-edit-and-agent-modes-what-they-do-and-when-to-use-them/) that provide different capabilities: - **Edit mode **(deprecated): to make granular edits, requiring their approval; this was available within IDEs. - **Ask mode**: to ask questions about your code or your repository. - **Agent mode**: an autonomous agent that can call tools and implement tasks/features/fixes; it provides some planning capabilities. - **Plan mode**: a tailored agent to research the task and create a detailed implementation plan before changes are made. ## GitHub Code Repositories For projects using GitHub code repositories, it’s possible to take advantage of two main features: - [Code review](https://docs.github.com/en/copilot/concepts/agents/code-review): feature available on GitHub repositories that reviews the code on PRs. It can delegate its findings to the coding agent, which in turn can open a new PR. - [Coding agent](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent): works autonomously in a GitHub Actions-powered environment to complete development tasks assigned through GitHub issues or GitHub Copilot Chat prompts, and creates pull requests with the results. It can be triggered from the repository page on GitHub, for example. The environment used by the coding agent can be [customized](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/customize-the-agent-environment) using a workflow file `.github/workflows/copilot-setup-steps.yml`; this can be required in order to run custom MCP servers. Besides the default MCP servers (GitHub, Playwright), [additional MCP servers can be available](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/extend-coding-agent-with-mcp) for the coding agents. > ℹ️ Copilot coding agent is distinct from the "agent mode" feature available in your IDE. Copilot coding agent works autonomously in a GitHub Actions-powered environment to complete development tasks assigned through GitHub issues or GitHub Copilot Chat prompts, and creates pull requests with the results. In contrast, agent mode in your IDE makes autonomous edits directly in your local development environment # Operations ## Tailoring Copilot AI features to Our Needs Even though Copilot can be used to address certain tasks using the defaults, we can, and should, customize it so that it provides better results tailored to our needs. The Copilot customization is possible using different customization capabilities explained ahead; all of these are materialized as files, usually in markdown format, stored in the repository. ### Custom Instructions [Custom instructions](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-custom-instructions) are a way to provide additional context to Copilot, with your preferences, tools you use, and all the specifics of the project at hand. It will tailor the responses of Copilot to better fit your needs. There are repository-wide instructions and path-specific instructions: - Repository-wide instructions apply to the whole repository, and are defined in a file `.github/copilot-instructions.md`. - Path-specific instructions apply just to the files that match given patterns, and are defined inside the `.github/instructions` directory and follow the `NAME.instructions.md` syntax. ### Skills [Skills](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/create-skills) are a way to modify Copilot’s behavior when it works on certain tasks. Skills are materialized as `SKILL.md` markdown files stored in `.github/skills` or `~/copilot/skills` directories, or any subfolder inside them, along with some additional resources. Skills are loaded dynamically and used whenever Copilot finds they’re applicable. ### Sub Agents A subagent is a separate AI agent instance that the main Copilot CLI agent launches to handle a specific task. For example, Copilot CLI may use a subagent when it determines that delegating part of the work to another agent is the most effective way to fulfill the user’s request. ### Custom Agents **Custom agents** give Copilot domain-specific expertise and define how it should approach tasks in that area. They act like “personas” that Copilot can adopt when handling certain kinds of work. It’s [possible to configure MCP servers](https://docs.github.com/en/copilot/concepts/agents/coding-agent/mcp-and-coding-agent#mcp-servers-for-custom-agents) in the custom agents definition file. > ℹ️ Copilot CLI has several built-in custom agents. For example, the `explore`, `task`, `research`, `code-review`, and `general-purpose` agents. You can define: > ℹ️ > ℹ️ - Your own custom agents to meet your specific needs. > ℹ️ - A custom agent in a Markdown file, as detailed in [Custom agents configuration](https://docs.github.com/en/copilot/reference/custom-agents-configuration). ### MCP Servers An **MCP server** is a service that allows AI applications, such as Copilot CLI, to connect to external data sources and tools. ### Tools A **tool** is an ability that Copilot uses to get something done, like searching files, viewing file contents, editing, running a task, or invoking a skill. Some tools are built in, and others can be added through MCP servers. ### Plugins A **plugin** is an installable package that brings a set of features to Copilot. It can bundle together various customization options, such as skills, custom agents, hooks, and MCP server configurations. ### Prompts [Prompt files](https://docs.github.com/en/copilot/tutorials/customization-library/prompt-files/your-first-prompt-file) are reusable prompts that can be easily called from an enabled AI to perform a certain task using `/` syntax. > ℹ️ Prompts are only available in GitHub Copilot Chat within IDEs, such as VS Code. ### Hooks [Hooks](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-hooks) let you run custom shell commands at key moments in an agent’s workflow, like when a session begins or ends, or right before and after a prompt is submitted or a tool is invoked. Hooks are configured as JSON files inside `.github/hooks` directory. ## GitHub Copilot and Xray As the team works on new features or bug fixes, GitHub Copilot can be a great companion to assist with programming, coding the features themselves, and also implementing the underlying Test automation code. Xray and Jira can be seen as a source for obtaining: - **The “requirements” specification**: as user stories, epics, or similar; sometimes, additional information may be available in linked Confluence pages. - **Items in the release scope**: Work items planned for a release. - **Items in the sprint scope:** Work items planned for a release. - **My items**: Work items assigned to me. - … We can also consider Xray and Jira as a target to: - Store Test results. - Create bug reports. - Create new feature ideas. - And more! There are many use cases that can be enabled by an integration between Copilot and Xray. Just to name a few: - Automating an existing manual Test. - Assessing progress on Testing and release readiness. - [Test Plan](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565153) progress. - Overall coverage status for a release. - Uncovered or failed requirements. - [Test Repository](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565166): organizing the Test Repository of some Space by some criteria. - Test Plan Board: organizing the Board of a Test Plan by some criteria. ### Integration with Xray It’s possible to integrate with Xray if using GitHub Copilot in: - [One of the IDEs](https://docs.github.com/en/copilot/how-tos/get-code-suggestions/get-ide-code-suggestions) that has the Copilot extension/plugin. - The [GitHub Copilot CLI](https://docs.github.com/en/copilot/concepts/agents/copilot-cli/about-copilot-cli). - [GitHub coding agent](https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/), usually on the GitHub website. > ✅ To enable the integration with Xray, while an official MCP server is not available, it’s possible to use an open-source MCP server that integrates with Xray Cloud through its GraphQL API: [mcp-graphql](https://github.com/blurrah/mcp-graphql). > ✅ > ✅ To integrate with Jira, it’s possible to use [Atlassian’s Rovo MCP server](https://support.atlassian.com/atlassian-rovo-mcp-server/docs/getting-started-with-the-atlassian-remote-mcp-server/) (this may not be needed for your use case, as much relevant information is available through Xray’s GraphQL API). ### Prerequisites and Setup The use cases shared in this tutorial require the use of specific MCP servers; their configuration is detailed ahead and depends on how we’ll interact with Copilot. The main prerequisite is **access to Copilot**. To use GitHub Copilot, you need either limited access through Copilot Free or full access through a paid Copilot plan. ### MCP Servers We’ll use a few MCP servers to be able to achieve the use cases detailed ahead; check the instructions on the respective repositories to know more about their capabilities, dependencies, and how to use them. We’ll make use of these MCP servers: - [mcp-graphql](https://github.com/blurrah/mcp-graphql) (**mandatory** for all use cases in this tutorial): to interact with Xray Cloud using its GraphQL API; this will be core for supporting the use cases in this tutorial. - [Atlassian Rovo MCP Server](https://support.atlassian.com/atlassian-rovo-mcp-server/docs/getting-started-with-the-atlassian-remote-mcp-server/) (optional). ### Sample Skills, Prompts, and Agents The skills, prompts, and custom agents used in this tutorial are available in a [GitHub repository](https://github.com/Xray-App/xray-code-snippets). As mentioned earlier, consider them as a source of inspiration so you can get some ideas, adapt them to your needs, and use them wisely, at your own risk. #### Installing Skills Skills can be installed manually by creating the proper folder and Markdown file in your local environment, but they are usually managed using [npx skills](https://github.com/vercel-labs/skills) instead. To list available skills: ``` npx skills add https://github.com/Xray-App/xray-code-snippets --list ``` ![Figure 1 - Available skills](media://e76a6fc4-967e-41cd-b63b-d5c9eb1b5b9e) To install a specific skill: ``` npx skills add https://github.com/Xray-App/xray-code-snippets --skill xray-data-extraction ``` #### Installing Custom Agents To install the provided sample agents, just copy the intended agent markdown file from the [repository](https://github.com/Xray-App/xray-code-snippets/tree/main/.github/agents) into your own `.github/agents` folder. ### Main Usage Scenarios #### Working from within IDEs To use Copilot from within VS Code, we need to install the [GitHub Copilot extension](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot), which actually installs 2 extensions: one for inline suggestions and the Copilot Chat panel. For other IDEs there are similar extensions/plugins. > ℹ️ For many interesting use cases of how to use Copilot right from your IDE, please check [https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/1255047170](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/1255047170). ### Using the GitHub Copilot Website Copilot is [accessible directly from a browser](https://github.com/copilot) and can be used to ask questions about your repository or to implement tasks by delegating them to the coding agent. If we aim just to ask questions about the code, as seen in the following example, there’s no need to configure MCP servers. #### Example: Information About Testing Frameworks Being Used As an example (Figure 2), we can use it (utilizing Ask mode) to have an understanding of the current frameworks being used for Testing. ![Figure 2 - Example](media://ed8d89b9-39de-4c2f-8d98-3faf5d468403) #### GitHub Copilot CLI [GitHub Copilot CLI](https://docs.github.com/en/copilot/concepts/agents/copilot-cli/about-copilot-cli) is a terminal-based AI agent that can answer questions, plan work, and complete tasks on your behalf. ``` $ copilot ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮ │ ╭─╮╭─╮ │ │ ╰─╯╰─╯ GitHub Copilot v1.0.10 │ │ █ ▘▝ █ Describe a task to get started. │ │ ▔▔▔▔ │ │ Tip: /allow-all Enable all permissions (tools, paths, and URLs) │ │ Copilot uses AI, so always check for mistakes. │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ● 💡 No copilot instructions found. Run /init to generate a copilot-instructions.md file for this project. ● Environment loaded: 1 MCP server, 3 plugins, 25 skills, 5 agents, Visual Studio Code connected ``` ##### Specific Configuration We can [add a new MCP server](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-mcp-servers) configuration to make it available to GitHub CLI, using `/mcp add` or by editing `~/.copilot/mcp-config.json`. ```json { "mcpServers": { "mcp-graphql": { "type": "stdio", "command": "npx", "tools": [ "*" ], "args": [ "mcp-graphql" ], "env": { "ENDPOINT": "https://xray.cloud.getxray.app/api/v2/graphql", "ALLOW_MUTATIONS": "true", "HEADERS": "{\"Authorization\":\"Bearer your_xray_auth_token\"}", "PATH": "/Users/xpto/.nvm/versions/node/v22.12.0/bin:/bin" } } } } ``` This requires that we have obtained the [Xray API](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565892) authentication elsewhere. As an example, right from your shell/terminal. ```shell curl -H "Content-Type: application/json" -X POST --data '{ "client_id": "32A27E69B0AC4E539C14010000000000","client_secret": "d62f81eb9ed859e11e54356dd8a00e4a5f0d0c2a2b52340776f6c70000000000" }' https://xray.cloud.getxray.app/api/v2/authenticate ``` ##### Examples ###### Example: Show Test Plan progress To obtain the overall Test Plan progress (i.e., its consolidated status), we can ask it directly, and GH CLI will load the necessary skills. Then, it will make the necessary interactions with [Xray’s GraphQL API](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565894) through the connected mcp-graphql MCP server. ``` ... ❯ show test plan progress of ST-3 ◐ The user wants to see test plan progress of ST-3. Let me use the xray-data-extraction skill or query GraphQL to get this information. ● skill(xray-data-extraction) ◐ The user wants to see test plan progress of ST-3. I need to read the reference files to understand how to get this data from Xray, then query the GraphQL API. ● Read obtaining-test-runs.md │ .agents/skills/xray-data-extraction/references/obtaining-test-runs.md └ 65 lines read ◐ I need to get test plan progress for ST-3. This is a test plan, not test executions. Let me check the other reference files to understand how to get test plan information. Let me also check if there's specific info about test plans. ... ``` ![Figure 3 - Test plan](media://4af70d01-9c73-44a3-9cf2-08152edb79dc) ### Coding Agent The coding agent can be triggered from GitHub Copilot website, from the Agents tab on the GitHub repository, or even from a Work item or PR. #### Specific Configuration To be able to use GitHub’s coding agent in its full potential, at a high level, we need to: - Configure MCP servers available for the coding agent. - Create an environment for the coding agent. - Define some variables/secrets for that environment. - Create a workflow to set up the development environment used by the coding agent. - Configure the allowlist for the URLs that the coding agent can have access to. > Macro (rw-ui-steps-macro) > > > Macro (rw-step) > > We start by configuring the MCP servers that we want to make available for the coding agent. We do that on **Settings > Copilot > Coding agent **(Figure 4). > > ![Figure 4 - Coding agent](media://66e057cb-bcda-42e2-ab34-2045e6d63449) > > > Macro (rw-step) > > Go to the **Model Context Protocol (MCP)** settings and add the MCP server configurations in the form of JSON content. > > ![Figure 5 - Workflow](media://03095dd3-b87b-44de-8928-251cd59d8005) > > ``` > { > "mcpServers": { > "atlassian-rovo-mcp": { > "command": "npx", > "type": "local", > "tools": ["*"], > "args": [ > "mcp-remote@latest", > "https://mcp.atlassian.com/v1/mcp", > "--header", > "Authorization: Basic $ATLASSIAN_API_KEY" > ], > "env": { > "ATLASSIAN_API_KEY": "$COPILOT_MCP_ATLASSIAN_API_KEY" > } > }, > > "mcp-graphql": { > "type": "stdio", > "command": "npx", > "args": [ > "mcp-graphql" > ], > "tools": ["*"], > "env": { > "ENDPOINT": "https://xray.cloud.getxray.app/api/v2/graphql", > "ALLOW_MUTATIONS": "true", > "HEADERS": "{\"Authorization\":\"Bearer $XRAY_AUTH_TOKEN\"}" > } > } > } > > } > ``` > > > Macro (rw-step) > > Create an environment named “copilot” as it will be used by the coding agent, so that we can assign it variables/secrets. > > ![Figure 6 - Copilot](media://574e044b-806c-4ab8-b795-eda7fec727b7) > > > Macro (rw-step) > > Add the secrets (e.g., for the Atlassian API token, Xray API credentials) on that environment. > > The secrets that we want to make directly available for the MCP servers (e.g., the Atlassian API token) need to have the “COPILOT_MCP” prefix. > > > Macro (rw-step) > > For the Xray related authentication, we could add a token directly (but we would need to obtain that elsewhere and update it frequently; instead, we can add secrets for the Xray API client ID and secret, respectively; Figure 7 - 1; 2). > > ![Figure 7 - Secret](media://48e4de2a-d392-48d2-bae8-13647cf08c84) > > > Macro (rw-step) > > To obtain the Xray authentication token, we will authenticate using the previous credentials on the `copilot-setup-steps.yml` workflow that is used to customize the coding agent’s environment. > > ``` > on: > workflow_dispatch: > permissions: > id-token: write > contents: read > jobs: > copilot-setup-steps: > runs-on: ubuntu-latest > permissions: > id-token: write > contents: read > environment: copilot > > steps: > # Checkout the repository to install dependencies (exp) > - name: Checkout code > uses: actions/checkout@v6.0.2 > - name: Authenticate with Xray > id: auth > run: | > RESPONSE=$(curl -s -X POST "https://xray.cloud.getxray.app/api/v2/authenticate" \ > -H "Content-Type: application/json" \ > -d '{ > "client_id": "'"${{ secrets.XRAYCLOUD_CLIENT_ID }}"'", > "client_secret": "'"${{ secrets.XRAYCLOUD_CLIENT_SECRET }}"'" > }') > > # Xray returns token as a JSON string (quoted), so strip quotes > TOKEN=$(echo $RESPONSE | tr -d '"') > echo "XRAY_AUTH_TOKEN=$TOKEN" >> $GITHUB_ENV > ``` > > > Macro (rw-step) > > On the Copilot coding agent settings, configure the allowlist (Figure 8) so it can reach relevant URLs, like the Xray API endpoint or Atlassian’s MCP endpoint. > > ![Figure 8 - Settings](media://666056f2-5252-4ab1-9d1e-210c48b9a520) ### Examples #### Example: Automate an Existing Manual Test in Xray In this use case, our goal is to create an automated Test based on an existing manual Test that already exists in Xray. This should follow the conventions and Testing libraries used in the Space. In this example (Figure 9), the existing manual Test in Xray has two steps. ![Figure 9 - Steps](media://29dec4cb-0997-4a65-abfe-8ac71e4a568d) > Macro (rw-ui-steps-macro) > > > Macro (rw-step) > > We can prompt the coding agent as follows. > > ``` > implement an automated test considering the steps of the manual test identified by the issue key ST-306 in Jira. That test is a Xray test, so use the proper MCP to get information. > ``` > > > Macro (rw-step) > > The coding agent begins by setting up a development environment, starting the MCP servers, and then use them to fetch the information from Xray (Figure 10). > > ![Figure 10 - Steps](media://d0fd4711-0237-4f46-9c30-54eaa71a9b6d) > > > Macro (rw-step) > > It proceeds by writing a Test using the annotation `@XrayTest`, which is provided by the `xray-junit-extensions` library being used in the code repository (Figures 11 and 12). > > ![Figure 11 - Code](media://410b5e12-8105-41d7-91ea-cda331770198) > > ![Figure 12 - Notation](media://7afb829a-98b9-42f6-928c-734cf887479c) > > > Macro (rw-step) > > A PR is created, and Tests are run, including the new one. > > ![Figure 13 - Test](media://0c461b13-ad60-4380-af33-333018550335) > > > Macro (rw-step) > > The PR is ready for review and can be merged, as existing checks, in this case, are all green (Figure 14 - 1). > > ![Figure 14 - PR](media://7a9ba854-3756-45ab-b1e3-aba9da938f56) ### Example: Overall Test Plan Progress This example (Figures 15 and 16) showcases a scenario where the user tries to obtain the progress of the Test Plan using the coding agent. Even though this may not be a realistic usage scenario by itself, it shows how it’s possible to get information and eventually use it to drive decisions by the coding agent. ``` show overall progress of Test Plan issue ST-3 based on the related test runs. Test Plan issue key ST-3 refers to the Jira issue key. Use the MCP tools to obtain informaton about the progress of that Test Plan. Don't make any changes in the code; just obtain the information using the MCP tools ``` ![Figure 15 - Test plan](media://6d66252a-6f06-4f5e-992b-e3b1255e4c92) ![Figure 16 - Overall](media://1cf01943-426a-4d03-971f-aaa6a9799afb) # References - [GitHub Copilot](https://docs.github.com/en/copilot) - [https://www.youtube.com/watch?v=SJqGYwRq0uc](https://www.youtube.com/watch?v=SJqGYwRq0uc) - [GitHub Copilot CLI](https://docs.github.com/en/copilot/concepts/agents/copilot-cli/about-copilot-cli) - [GitHub coding agent](https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/) - [about coding agent](https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent) - [Atlassian Rovo MCP Server](https://support.atlassian.com/atlassian-rovo-mcp-server/docs/getting-started-with-the-atlassian-remote-mcp-server/) (official) - [Comparing GitHub Copilot CLI customization features](https://docs.github.com/en/copilot/concepts/agents/copilot-cli/comparing-cli-features) - [Awesome GitHub Copilot resources](https://github.com/github/awesome-copilot) - [GitHub Agentic Workflows](https://github.github.com/gh-aw/introduction/overview/) > Macro (rw-ui-expands-macro) > > > Macro (rw-expand) > > If you have questions or technical issues, please [contact the Support team via the Customer Portal (Jira service management)](https://jira.getxray.app/servicedesk/customer/portal/2/user/login?destination=portal%2F2%2Fcreate%2F28) or [send us a message using the in-app chat](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44577312).