Feature/parameterized tests with xray poc#535
Conversation
sebastianpfischer
left a comment
sebastianpfischer
left a comment
There was a problem hiding this comment.
Hi @DianeMornas , there are core design issues with this pull-request. Could you please get in touch with @Pog3k so that he setup a meeting with all of us to discuss it. Could you also please create a ticket associated to this pull-request and the needs?
Thank you in advance
sebastianpfischer
left a comment
sebastianpfischer
left a comment
There was a problem hiding this comment.
Like said above, we need to have a design discussion. In addition, tests, documentation and examples are missing
There was a problem hiding this comment.
The generation of the needed information should be created after the test was executed. If information needs to be fetchted during execution, most of the content should be part of the test_result module.
If we need to record now each line executed from a specific script, we need to take a look at it as well.
So I expect 2 parts: one in the test_result and maybe another here but more in a way we store information in a generic way that can per retrieve from the test_result itself
| @click.option( | ||
| "--xray-upload", | ||
| nargs=3, | ||
| type=click.Tuple([str, str, str]), | ||
| required=False, | ||
| help="Upload the test results to Xray. Requires the username, password and url as arguments.", | ||
| ) | ||
| @click.option( | ||
| "--test-execution-id", | ||
| nargs=1, | ||
| type=click.STRING, | ||
| required=False, | ||
| help="Test execution ID of Xray where to upload the test results. By default, a new test execution ticket is created except if the ID is given.", | ||
| ) |
There was a problem hiding this comment.
xray should not become an interface of pykiso. The upload should still be done via the tool.
| def generate_step_report_xray( | ||
| test_result: Union[BannerTestResult, XmlTestResult], | ||
| ) -> dict: | ||
| """Generate the step report dictionary for Xray | ||
|
|
||
| :param test_results: Result of tests to generate the report from | ||
| :return: The step report dictionary""" | ||
| return generate_all_step_report(test_result=test_result) | ||
|
|
||
|
|
||
| def generate_step_report_html( | ||
| test_result: Union[BannerTestResult, XmlTestResult], | ||
| output_file: str, | ||
| ) -> None: | ||
| """Generate the HTML step report based on Jinja2 template | ||
|
|
||
| :param test_result: Result of tests to generate the report from | ||
| :param output_file: Report output file path | ||
| """ | ||
|
|
||
| all_step_report = generate_all_step_report(test_result=test_result) |
There was a problem hiding this comment.
I think this is not the correct location for it
|
What I have seen in this POC: assert_step_report has been improved to collect additional test information, which will be used for the step report HTML and to provide all necessary details for the Xray report. In test_execution.py, additional functions related to Xray reports have been added. I think it would be cleaner if we moved these new functions to a separate file/module dedicated to Xray functions, perhaps something like tool/xray/xray_report.py. In assert_step_report.py, we are mixing step report functions with Xray again. I think it makes sense to move generate_step_report_xray to tool/xray/xray_report.py. The refactored step report logic is fine because both require the same dictionary, ALL_STEP_REPORT. I believe we need to modify the Pykiso interface because the information in the JUnit report is insufficient to capture all necessary details. See the improvements in assert_step_report. Somehow, we need make the informations stored ALL_STEP_REPORT available. Since this is quite generic, we could also introduce a flag like --kiso-report, which dumps the pickled dictionary or something similar. For the upload process, we could still use the external CLI, which has the benefit of allowing results to be uploaded at any time. This way, tests don’t need to be re-run if something fails. I'll be back on Monday to discuss this further. However, feel free to leave any comments in advance. |
It makes sense. But we should take a deeper look at the report mechanism all together. I believe this may need some major design decision. |
Codecov ReportAttention: Patch coverage is
📢 Thoughts on this report? Let us know! |
DianeMornas
force-pushed
the
feature/parameterized-tests-with-xray-poc
branch
from
9ac4717 to
2f27698
Compare
DianeMornas
force-pushed
the
feature/parameterized-tests-with-xray-poc
branch
from
dd276f8 to
c2ba76e
Compare
DianeMornas
force-pushed
the
feature/parameterized-tests-with-xray-poc
branch
from
c2ba76e to
654124b
Compare
3 participants
The goal of this POC is to change the way to upload test results to xray, before 2 steps were required first step run the test and generate the jjunit xml file at the same time then second step convert this junit into a junit compatible with xray to upload the test results. The parameterized were not supported. This way only worked with xray test ticket with a Generic type and not with Manual, Automated and automated Cucumber types.
The user still uses a decorator with the test_key ID of the xray test ticket
HTML step report generation has also been updated and support parameterized tests.