Thoughts on E2E testing for MCP servers

MCP is fundamentally different from API. Testing MCP servers will require new techniques.

What is End to End (E2E) testing?

End to end testing (E2E) is a testing method that simulates a real user flow to validate the correctness. For example, if you're building a sign up page, you'd set up your E2E test to fill out the form inputs, click submit, and assert that a user account was created. E2E testing is the purest form of testing: it ensures that the system works from and end user's environment.

There's an awesome article by Kent Dodds comparing unit tests, integration tests, and E2E tests and explaining the pyramid of tests. I highly recommend giving that a read. In regards to E2E testing, it is the highest confidence form of testing. If your E2E tests work, you can ensure that it'll work for your end users.

E2E testing for MCP servers

E2E testing for API servers is typical practice, where the E2E tests are testing a chain of API calls that simulate a real user flow. The same testing is needed for MCP servers where we set up an environment simulating an end user's environment and test popular user flows.

Whereas APIs are consumed by other APIs / web clients, MCP servers are consumed by LLMs and agents. End users are using MCP servers in MCP clients like Claude Desktop and Cursor. We need to simulate these environments in MCP E2E testing. This is where testing with Agents come in. We configure the agent to simulate an end user's environment. To build an E2E test for MCP servers, we connect the server to an agent and have the agent interact with the server. We have the agent run queries that real users would ask in chat and confirm whether or not the user flow ran correctly.

An example of running an E2E test for PayPal MCP:

1. Connect the PayPal MCP server to testing agent. To simulate Claude Desktop, we can configure the agent to use a Claude model with a default system prompt.

2. Query the agent to run a typical user query like "Create a refund for order ID 412"

3. Let the testing agent run the query.

4. Check the testing agents' tracing, make sure that it called the tool create_refund and successfully created a refund.

For step 4, we can have an LLM as a judge analyzing the testing agent's trace and check if the query was a success.

How we're building E2E tests at MCPJam

We're building MCPJam, an alternative to the MCP inspector - an open source testing and debugging tool for MCP servers. We started building E2E testing in the project and we're set to have a beta out for people to try sometime tomorrow. We're going to take the principles in this article to build the beta. We'd love to have the community test it out, critique our approach, and contribute!

If you like projects like this, please check out our repo and consider giving it a star! ⭐

https://github.com/MCPJam/inspector

We're also discussing our E2E testing approach on Discord!

https://discord.com/invite/JEnDtz8X6z