Solution Testing
What is Solution Testing?
Solution Testing involves testing an individual product from a customer’s point of view. Ideally, the product would be tested in the customer’s environment. In an Enterprise or Telecom, it would mean that the testing is done with the customers own live network elements or with other vendors’ network elements and real traffic, or both.
Solution Testing in the Vendor’s Network
In order to accept a vendor's new product or solution, a carrier would require a mutually agreed-upon set of solution tests. These solution tests are also required to be carried out in an environment that emulates that particular carrier's live network. Historically, this testing is done manually and is performed at the end of the Software Development Life Cycle (SDLC). To stay competitive, vendors have been moving this testing earlier in the SDLC. This shift-left philosophy has become an integral part of their continuous automated testing. The main challenge is executing complex test scenarios within complex test environments.
Solution Testing in the Carrier’s Network
Whether the vendor that sold the product has done Solution Testing or not, the carrier will likely test the product in a pre-production environment that closely resembles the live network. This can take many months for some carriers to pass the Solution Testing. In most cases, this testing is not automated. However, with the advent of CI/CD requirements in the vendor’s processes, carriers are now mandating that the vendor’s continuous testing environments be leveraged in the carrier’s pre-production environment and, in some cases, in their live networks.
Test Automation Orchestration for Solution Testing
Within Solution Testing, test cases are executed at different levels of complexity - from simple functional testing to more complex end-to-end testing. Solution tests are a perfect fit for a Test Automation Orchestration Framework feature. The key features are:
-
Third-Party Tool Integration - integrates with test tools that emulate nodes of the solution and test tools that generate data and traffic for the solution.
-
Server Pool Management - management of test servers for optimal solution utilisation.
-
Network Topology - the topological structure of a solution’s network that depicts the arrangement of elements to be tested.
-
Test Case Management - organises test cases within test suites and creates custom test plans for running solution tests.
-
​
-
Test Execution - runs solution tests at all levels such as Project, Test Plan, Test Suite, and Test Case.
-
Reporting - organises summary of solution testing objectives, activities, and results.
-
CI/CD Integration - a CI/CD pipeline tool (e.g., Jenkins) that builds code for the solution, runs solution tests (Continuous Improvement), and safely deploys new applications of the solution (Continuous Delivery).