智慧测试工具的引入切实解决了系统测试不透明、用例覆盖不充分等问题，触达了黑盒测试痛点，让测试人员克服了系统测试的盲目性，充分体现了智慧测试特点，测试人员根据未覆盖代码原因，可采取定向补测、定向风险分析、混沌测试等不同的处置措施，进一步提升了系统测试质量，降低应用系统投产风险，强化安全生产保障能力。
随着农业银行数字化转型的不断深入，项目在实施过程中出现研发周期短、迭代快、系统关联复杂度高、投产风险大等特点，微服务、分布式等架构技术也对测试工作提出更高要求，传统的黑盒测试无法获知测试用例执行后真正的代码覆盖情况，测试针对性难以把握，很难实现全面测试。目前业界虽然有部分开源软件可实现代码覆盖率采集和报告，但要真正在农业银行项目系统测试阶段落地，还存在覆盖率文件容易丢失、功能单一缺乏整合、手工操作过多过繁、便捷性和实用性差、测试流程如何融合、各模块覆盖率作业如何管理和展示等技术难题。此外，对不同种类的代码框架也需要在实践中探索和研究。系统测试痛点如图1所示。
基于以上情况，农业银行自主研发了一款系统测试代码覆盖率工具：智慧测试工具（Intelligent System Test Platform,ISTP），实现了代码覆盖率在系统测试阶段的落地，支持OCP/CCE云上Tomcat、云下Tomcat、云下Was等多种部署环境，解决了系统测试代码覆盖率报告持久化、应用集群部署模式下的报告合并、云上多POD报告合并、不同代码版本间覆盖率合并等诸多问题，为代码覆盖率在系统测试阶段落地扫清障碍。数字化转型网www.szhzxw.cn
一、ISTP初见
ISTP是基于Jenkins、JaCoCo、Diff-cover等软件二次开发打造的一站式JAVA代码全量/增量覆盖率采集、收集、报告展示和分析工具，可低成本融入系统测试流程，一次部署多次使用，零手工操作，工具集成环境界面一站式完成所有操作，便捷性、实用性和扩展性强，已在天研范围内100多个系统模块全面落地。智慧测试工具三层架构如图2所示，主要分为接入层、采集层和核心层，其中采集层为天研测试部自主研发模块，接入层和核心层在开源软件基础上二次开发实现。通过采集层的引入，即实现了核心层与接入层（通常为常备测试环境）的解耦，又实现了覆盖率文件持久化，解决了由于异常宕机或者重启导致的覆盖率文件丢失问题。
二、ISTP赋能测试
ISTP可以助力测试流程优化，为测试人员提供界面化的测试覆盖率情况。通过在系统首次准入阶段完成采集服务的申请及相关配置，即可完成一次部署多次使用，可追踪覆盖系统测试阶段针对所有开展的测试类型产生的代码路径，项目经理和测试经理等角色可以全程监控覆盖率情况，并及时开展覆盖率情况分析、定向补测和风险分析；在准出阶段，测试经理可根据全量及增量覆盖率报告判断是否满足准出要求。ISTP可以有效提升系统测试质量，降低生产残留缺陷，系统测试流程及操作步骤可参考图3和图4。
直观清晰的测试覆盖率报告可以极大提高分析效率，降低工具使用门槛，ISIP可清晰展示全量覆盖率报告、增量覆盖率报告、当前版本基线号、Git代码仓库地址和分支等各种信息。打开html文件，可以查看代码增量覆盖率情况详细报告，报告中包含增量代码覆盖率情况的整体信息，以及对应到每一个类文件的代码行覆盖情况。覆盖率报告展示如图5所示。
三、ISTP应用成效
目前，ISTP已在天研范围内全面落地，共接入系统模块100多个，搭建覆盖率构建流水线150多个，共开展115个批次项目测试覆盖率分析，其中实施定向补测79个批次，占比68.6%，每批次平均增补用例6.26个，代码覆盖率平均提高7.0%。项目实践中共发现测试场景遗漏、回归测试场景遗漏、定向风险分析、辅助版本验证4类典型场景，其中7个典型案例如图6所示。数字化转型网www.szhzxw.cn
1、测试场景遗漏
在图6的7个典型案例中，6个案例存在测试场景遗漏情况，遗漏测试场景的原因主要包括业务需求定义较宽泛导致测试遗漏、接口文档备选字段描述缺失导致遗漏、测试准入功能清单缺失部分功能导致测试遗漏、极限边界值校验场景遗漏、某功能联机场景测试了但批量场景遗漏、对异常情况的判断分支测试遗漏、测试人员技能差异导致测试遗漏等。ISTP的引入可较好评估测试充分性，清晰展示未覆盖代码情况，测试人员对未覆盖代码进行定向补测，进一步提升系统测试质量，减少生产残留缺陷。
2、回归测试场景遗漏
在图6的7个典型案例中，1个案例存在回归测试场景遗漏，该案例因优化代码圈复杂度，对存量功能代码进行改造，测试人员通过智慧测试工具可精准识别需要回归补测的业务功能，避免了回归测试场景遗漏，使增量代码覆盖率从54%提高到94%，进一步提升了系统测试质量，降低了投产风险。
3、定向风险分析
在图6的7个典型案例中，通过工具均可直接针对极限边界值校验、优化代码等场景进行定向风险分析，动态实时查看全量/增量覆盖率报告，实时跟踪测试用例执行情况，有效降低风险。
4、辅助版本验证数字化转型网www.szhzxw.cn
ISTP最初主要应用于系统测试阶段，但同样也可以适用于版本验证阶段，唯一的区别仅仅是常备测试环境差异。在版本验证阶段引入ISTP，赋能版本验证，通过版本号和代码覆盖报告可做实所测即所投，防止带病程序进入投产环境，进一步保障所测应用系统投产安全。
ISTP的引入切实解决了系统测试不透明、用例覆盖不充分等问题，触达了黑盒测试痛点，让测试人员克服了系统测试的盲目性，充分体现智慧测试特点，测试人员根据未覆盖代码原因，可采取定向补测、定向风险分析、混沌测试等不同的处置措施，进一步提升了系统测试质量，降低应用系统投产风险，强化安全生产保障能力。ISTP的建设符合农业银行信息科技发展战略，针对代码覆盖率领域待解决的管理或技术问题，开展了卓有成效的探索和实践，具有良好的实践指导或示范借鉴意义。
翻译：
The introduction of intelligent testing tools effectively solves the problems of opaque system testing and insufficient coverage of use cases, and reaches the pain point of black box testing, allowing testers to overcome the blindness of system testing and fully embodiments the characteristics of intelligent testing. According to the reasons of code not covered, testers can take different measures such as directional retesting, directional risk analysis and chaotic testing. The test quality of the system is further improved, the risk of application system production is reduced, and the ability of safety production is strengthened.
With the continuous deepening of the digital transformation of Agricultural Bank of China, features such as short research and development cycle, fast iteration, high complexity of system correlation, and high risk of production have emerged in the implementation of the project. The micro-service, distributed and other architectural technologies have also put forward higher requirements for testing. Traditional black box testing cannot know the real code coverage after the execution of test cases, and the pertinency of testing is difficult to grasp. Full testing is difficult to achieve. 数字化转型网www.szhzxw.cn
At present, although some open source software in the industry can realize code coverage collection and report, there are still some technical problems such as easy loss of coverage files, lack of integration of single functions, excessive and complicated manual operations, poor convenience and practicability, how to integrate test processes, and how to manage and display coverage operations of each module in order to truly put them into practice in the test phase of Agricultural Bank of China project system. In addition, different kinds of code frameworks need to be explored and studied in practice.
The system test pain points are shown in Figure 1.
Figure 1 System test pain points
Based on the above situation, Agricultural Bank of China independently developed a system test code coverage tool: Intelligent System Test Platform (ISTP) implements code coverage in system test phase, and supports OCP/CCE Tomcat on cloud, Tomcat under cloud, Was and other deployment environments. It solves many problems, such as persistence of system test code coverage report, report merging under application cluster deployment mode, multi-POD report merging on cloud, coverage merging between different code versions, etc., which clears away obstacles for code coverage to be implemented in system test phase.
ISTP
ISTP is a one-stop JAVA code full/incremental coverage collection, collection, report presentation and analysis tool based on Jenkins, JaCoCo, Diff-cover and other software secondary development. It can be integrated into the system test process at low cost, and can be deployed multiple times with zero manual operation. Tool integrated environment interface one-stop to complete all operations, convenience, practicability and scalability, has been in the scope of more than 100 system modules fully landed. The three-layer architecture of intelligent testing tool is shown in Figure 2, which is mainly divided into access layer, acquisition layer and core layer. 数字化转型网www.szhzxw.cn
The acquisition layer is the self-developed module of Tianyan Testing Department, and the access layer and core layer are developed and implemented on the basis of open source software. Through the introduction of the acquisition layer, the decoupling between the core layer and the access layer (usually the standing test environment) is realized, and the coverage file persistence is realized, which solves the problem of coverage file loss caused by abnormal downtime or restart.
Figure 2 ISTP three-tier architecture
ISTP enablement test
ISTP facilitates test process optimization and provides testers with an interface of test coverage. By completing the application for collection service and related configuration in the first access stage of the system, one deployment and multiple uses can be completed, and the code paths generated for all test types in the system test stage can be traced and covered. Project managers, test managers and other roles can monitor the coverage throughout the process and timely carry out coverage analysis, directional retesting and risk analysis. During the approval phase, the test manager can determine whether the approval requirements are met based on the full and incremental coverage reports. ISTP can effectively improve system test quality and reduce production residual defects. The system test process and operation procedure can be referred to Figure 3 and Figure 4.
Figure 3 Optimizing the system test process
Figure 4 Steps of optimizing system test
Intuitive and clear test coverage reports can greatly improve analysis efficiency and lower the threshold of tool use. ISIP can clearly display a variety of information such as full coverage reports, incremental coverage reports, current version baseline numbers, Git repository addresses and branches. Open the html file to view the detailed report on the incremental code coverage, which contains the overall information about the incremental code coverage, as well as the line coverage for each class file. The coverage report presentation is shown in Figure 5.数字化转型网www.szhzxw.cn
Figure 5 Coverage report presentation
Effectiveness of ISTP application
At present, ISTP has been fully implemented within the scope of Tianresearch. A total of more than 100 system modules have been connected, more than 150 coverage lines have been set up, a total of 115 batches of project test coverage analysis have been carried out, among which 79 batches of directional retesting have been carried out, accounting for 68.6%, an average of 6.26 additional use cases per batch have been added, and code coverage has increased by 7.0% on average. Four typical scenarios including test scenario omission, regression test scenario omission, directional risk analysis. And auxiliary version verification were found in the project practice, among which 7 typical cases are shown in Figure 6.
Figure 6 Case presentation
Missing test scenarios
Among the 7 typical cases in Figure 6, 6 cases have missing test scenarios. The reasons for missing test scenarios mainly include test omissions due to broad definition of business requirements, missing description of alternative fields in interface documents, missing partial functions of test access function list, missing limit boundary value verification scenarios, missing a function online scenario but batch scenario, missing branch tests for abnormal conditions, and skill differences of testers Resulting in test omissions, etc. The introduction of ISTP enables a better assessment of test adequacy, a clear display of uncovered code. And targeted retesting of uncovered code by testers to further improve system test quality and reduce residual defects in production.
Omission of regression test scenarios
Among the 7 typical cases in Figure 6, 1 case has regression test scene omission. In this case, due to the optimization of code cyclomatic complexity and the transformation of stock functional codes. Testers can accurately identify the business functions requiring regression retesting through intelligent testing tools, avoiding regression test scene omission and increasing the coverage rate of incremental code from 54% to 94%. The system test quality is further improved and the production risk is reduced.
Directional risk analysis
In the 7 typical cases shown in Figure 6, the tool can be used to conduct directional risk analysis directly for the scenes such as limit boundary value verification and code optimization, dynamically view the full/incremental coverage reports in real time. And track the execution of test cases in real time, effectively reducing risks.
Auxiliary version verification
ISTP is primarily used in the system test phase. But it can also be used in the version verification phase, with the only difference being the usual test environment. ISTP is introduced in the version verification stage to enable version verification. And the version number and code coverage report can be used to make real test and investment. Prevent the program with disease from entering the production environment. And further ensure the safety of the application system under test.
The introduction of ISTP effectively solves the problems of opaque system testing and insufficient coverage of use cases. And reaches the pain point of black box testing, allowing testers to overcome the blindness of system testing and fully embodies the characteristics of intelligent testing. According to the reasons of uncovered code. Testers can take different measures such as directional retesting, directional risk analysis and chaotic testing. The test quality of the system is further improved, the risk of application system production is reduced. And the ability of safety production is strengthened. The construction of ISTP is in line with the development strategy of information technology of Agricultural Bank of China. Aiming at the management or technical problems to be solved in the field of code coverage. ISTP has carried out fruitful exploration and practice, which has good practical guidance or demonstration significance.数字化转型网www.szhzxw.cn
本文由数字化转型网（www.szhzxw.cn）转载而成，来源：我们的开心；编辑/翻译：数字化转型网宁檬树。
免责声明: 本网站(http://www.szhzxw.cn/)内容主要来自原创、合作媒体供稿和第三方投稿，凡在本网站出现的信息，均仅供参考。本网站将尽力确保所提供信息的准确性及可靠性，但不保证有关资料的准确性及可靠性，读者在使用前请进一步核实，并对任何自主决定的行为负责。本网站对有关资料所引致的错误、不确或遗漏，概不负任何法律责任。
本网站刊载的所有内容(包括但不仅限文字、图片、LOGO、音频、视频、软件、程序等) 版权归原作者所有。任何单位或个人认为本网站中的内容可能涉嫌侵犯其知识产权或存在不实内容时，请及时通知本站，予以删除。
