Carbontax

.gitattributes

@@ -0,0 +1 @@
+annotated_data/sustainability_reports.zip filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -50,7 +50,7 @@ coverage.xml
 .hypothesis/
 .pytest_cache/
 cover/
-
+apikey.ini 
 # Translations
 *.mo
 *.pot

01_rapp_holcim_dd_2023_vdef_bd_gpt-3.5-turbo_analysis.html

@@ -0,0 +1,38 @@
+<p>In governance:</p>
+<p>Q1: How does the company's board oversee climate-related risks and opportunities?</p>
+<p>Analysis1: The provided disclosure lacks specific information related to the board's oversight of climate-related risks and opportunities. While some general mentions of climate initiatives and actions are present, there is a notable absence of details regarding the board's processes, frequency of discussions, consideration of climate issues in decision-making, and monitoring of progress against goals. The disclosure primarily focuses on operational actions and achievements rather than providing a comprehensive overview of the board's role in overseeing climate-related matters.</p>
+<p>Score1: 20</p>
+<p>Q2: What is the role of management in assessing and managing climate-related risks and opportunities?</p>
+<p>Analysis2: The disclosure provided in the sustainability report lacks specific information regarding management's role in assessing and managing climate-related risks and opportunities. While some general references to risk management and stakeholder engagement are made, there is a lack of clarity on the assignment of responsibilities, organizational structure, processes for information flow, and monitoring mechanisms related to climate issues. The report focuses more on actions taken rather than the specific roles of management in addressing climate-related challenges."</p>
+<p>Score2: 25</p>
+<p>In strategy:</p>
+<p>Q3: What are the most relevant climate-related risks and opportunities that the organisation has identified over the short, medium, and long term? Are risks clearly associated with a horizon?</p>
+<p>Analysis3: The disclosure provided in the sustainability report partially addresses the requirements for identifying climate-related risks and opportunities over different time horizons. While some descriptions of risks and opportunities are present, there is a lack of clarity on the specific issues and processes used for determining material financial impacts. The report mentions some sectoral and geographical considerations but lacks comprehensive details on the time horizons and associated risks. More specific and detailed information is needed to fully meet the disclosure requirements.</p>
+<p>Score3: 50</p>
+<p>Q4: How do climate-related risks and opportunities impact the organisation's businesses strategy, economic and financial performance, and financial planning?</p>
+<p>Analysis4: The provided disclosure lacks specific details on how climate-related risks and opportunities impact the organization's business strategy, economic and financial performance, and financial planning. While some general mentions of climate change and sustainability initiatives are present, there is a lack of in-depth discussion on the identified impacts across various business aspects. The disclosure does not provide a comprehensive view of how climate-related issues are integrated into financial planning processes or how they influence financial performance.</p>
+<p>Score4: 25</p>
+<p>Q5: How resilient is the organisation's strategy when considering different climate-related scenarios, including a 2°C target or lower scenario? How resilient is the organisation's strategy when considering climate physical risks?</p>
+<p>Analysis5: The sustainability report provides some information on the organization's resilience to climate-related risks and opportunities, including a transition to a low-carbon economy and physical climate risks. It mentions strategies aligned with a 1.5°C trajectory and actions to address climate challenges. However, the report lacks specific details on the potential financial impacts, changes in strategies, and time horizons considered, which are crucial for a comprehensive assessment of resilience.</p>
+<p>Score5: 65</p>
+<p>In risk management:</p>
+<p>Q6: What processes does the organisation use to identify and assess climate-related risks?</p>
+<p>Analysis6: The disclosure provided in the sustainability report partially addresses the requirements for identifying and assessing climate-related risks. The report discusses processes for evaluating risks based on financial impact, temporality, and stakeholder involvement. However, it lacks specific details on how the organization determines the relative significance of climate-related risks compared to other risks, consideration of regulatory requirements, and definitions of risk terminology. More clarity and depth are needed to fully meet the disclosure requirements.</p>
+<p>Score6: 60</p>
+<p>Q7: How does the organisation manage climate-related risks?</p>
+<p>Analysis7: The disclosure provided in the sustainability report partially addresses the requirements for managing climate-related risks. The report discusses risk assessment, resilience evaluation, stakeholder engagement, and actions taken to address climate risks. However, it lacks specific details on how materiality determinations are made and how decisions are made to mitigate, transfer, accept, or control risks. More clarity and transparency are needed to fully meet the disclosure requirements.</p>
+<p>Score7: 65</p>
+<p>Q8: How are the processes for identifying, assessing, and managing climate-related risks integrated into the organisation's overall risk management?</p>
+<p>Analysis8: The disclosure provided in the sustainability report partially addresses the requirements for integrating climate-related risks into overall risk management. While some aspects such as risk assessment and stakeholder engagement are mentioned, there is a lack of specific details on how these processes are integrated into the organization's risk management framework. The report could benefit from more explicit descriptions and examples to demonstrate a comprehensive integration of climate-related risks into overall risk management.</p>
+<p>Score8: 60</p>
+<p>In metrics and targets:</p>
+<p>Q9: What metrics does the organisation use to assess climate-related risks and opportunities? How do the metrics help ensure that the performance is in line with its strategy and risk management process?</p>
+<p>Analysis9: The provided disclosure lacks specific details on the metrics used to assess climate-related risks and opportunities in line with the organization's strategy and risk management process. While some general discussions on risks and resilience are mentioned, there is a lack of clear presentation of key metrics, historical data, and forward-looking metrics. The disclosure does not fully meet the requirements outlined for high-quality disclosure on this critical element.</p>
+<p>Score9: 30</p>
+<p>Q10: Does the organisation disclose its Scope 1, Scope 2, and, if appropriate, Scope 3 greenhouse gas (GHG) emissions? What are the related risks and do they differ depending on the scope?</p>
+<p>Analysis10: The sustainability report provides some information related to Scope 1, Scope 2, and Scope 3 greenhouse gas emissions, but lacks clarity and specific details required for a comprehensive disclosure. While it mentions emissions percentages and some actions, it falls short in providing detailed calculations, historical data, and industry-specific efficiency ratios. The report also lacks a clear methodology description for emissions calculation. Overall, the disclosure is fragmented and lacks the necessary depth and consistency to fully meet the disclosure requirements."</p>
+<p>Score10: 40</p>
+<p>Q11: What targets does the organisation use to understand/quantify/benchmark climate-related risks and opportunities? How is the organization performing against these targets?</p>
+<p>Analysis11: The provided disclosure lacks specific details on the targets used by the organization to manage climate-related risks and opportunities and its performance against these targets. While some mentions of climate-related risks and actions are present, there is a lack of clarity on key aspects such as target types, time frames, base years, and key performance indicators. The disclosure focuses more on general risk assessments and societal impacts rather than detailed climate-related targets and performance metrics.</p>
+<p>Score11: 25</p>
+<p>Average score: 42.27272727272727</p>

01_rapp_holcim_dd_2023_vdef_bd_gpt-3.5-turbo_qa.html

@@ -0,0 +1,26 @@
+<p>In governance:</p>
+<p>Q1: How does the company's board oversee climate-related risks and opportunities?</p>
+<p>A1: Holcim's board oversees climate-related risks and opportunities by defining, evaluating, and prioritizing risks based on financial impact, stakeholder involvement, and probability. Regular exchanges with public authorities and internal stakeholders are conducted to address external stakeholder risks."</p>
+<p>Q2: What is the role of management in assessing and managing climate-related risks and opportunities?</p>
+<p>A2: Holcim's management assesses and manages climate-related risks by defining, evaluating, and prioritizing risks based on financial impact, stakeholders' involvement, and probability. They conduct self-assessments on natural disaster risks, aiming to enhance resilience and implement associated action plans ."</p>
+<p>In strategy:</p>
+<p>Q3: What are the most relevant climate-related risks and opportunities that the organisation has identified over the short, medium, and long term? Are risks clearly associated with a horizon?</p>
+<p>A3: Holcim has identified climate-related risks from extreme weather events and the need for water conservation. Opportunities include engaging employees in climate action. However, the report lacks specific details on risk quantification and mitigation strategies."</p>
+<p>Q4: How do climate-related risks and opportunities impact the organisation's businesses strategy, economic and financial performance, and financial planning?</p>
+<p>A4: Holcim acknowledges climate-related risks and opportunities, but lacks specific quantifiable data on financial impacts and planning. The report emphasizes societal engagement and resilience evaluations, but lacks detailed financial implications and strategies."</p>
+<p>Q5: How resilient is the organisation's strategy when considering different climate-related scenarios, including a 2°C target or lower scenario? How resilient is the organisation's strategy when considering climate physical risks?</p>
+<p>A5: Holcim's strategy shows resilience by aligning with a 1.5°C trajectory validated by SBTi for 2030 and 2050, emphasizing decarbonization and circular solutions. However, the report lacks specific quantifiable data on climate scenario planning and physical risk resilience."</p>
+<p>In risk management:</p>
+<p>Q6: What processes does the organisation use to identify and assess climate-related risks?</p>
+<p>A6: Holcim identifies and assesses climate-related risks by evaluating financial impact, timing, stakeholder involvement, and probability, conducting self-assessments on natural disaster risks including those from climate change, and aligning with international standards and sustainable development trends."</p>
+<p>Q7: How does the organisation manage climate-related risks?</p>
+<p>A7: Holcim manages climate-related risks by prioritizing stakeholder engagement, resilience assessments for natural disasters, and employee training on climate challenges. However, the report lacks specific data on risk mitigation actions and relies heavily on general statements."</p>
+<p>Q8: How are the processes for identifying, assessing, and managing climate-related risks integrated into the organisation's overall risk management?</p>
+<p>A8: The report outlines the integration of climate-related risks into overall risk management through stakeholder engagement, resilience assessments, and alignment with international standards. However, the lack of specific quantitative data raises concerns about the depth of integration and the potential for greenwashing."</p>
+<p>In metrics and targets:</p>
+<p>Q9: What metrics does the organisation use to assess climate-related risks and opportunities? How do the metrics help ensure that the performance is in line with its strategy and risk management process?</p>
+<p>A9: Holcim assesses climate-related risks based on financial impact, timing, scale, stakeholder involvement, and probability. They prioritize risks for external stakeholders through regular engagement with public authorities. However, the report lacks specific metrics on how these assessments align with the company's strategy and risk management process."</p>
+<p>Q10: Does the organisation disclose its Scope 1, Scope 2, and, if appropriate, Scope 3 greenhouse gas (GHG) emissions? What are the related risks and do they differ depending on the scope?</p>
+<p>A10: Holcim discloses its Scope 1, Scope 2, and Scope 3 greenhouse gas emissions. Scope 1 emissions are 75%, Scope 2 emissions are &lt;1%, and Scope 3 emissions are 24%. Risks include financial impact, stakeholder involvement, and regulatory changes."</p>
+<p>Q11: What targets does the organisation use to understand/quantify/benchmark climate-related risks and opportunities? How is the organization performing against these targets?</p>
+<p>A11: Holcim aims to limit global warming to 1.5°C by aligning with the Science Based Targets Initiative for 2030 and 2050. However, the report lacks specific details on current performance against these targets, raising concerns about transparency and accountability."</p>

README.md

@@ -1,2 +1,61 @@
 # chatreport
-Github implementation of https://reports.chatclimate.ai/
+Github implementation of CHATREPORT. This is a snapshot of the v1.0 backend of our [web application](https://reports.chatclimate.ai/) on 2023.07.28.
+
+## Directories
+- annotated_data: annotation results of our human evaluation
+  - chatgpt/gpt-4_results.json: ChatGPT/GPT-4's outputs on sampled reports.
+  - chatgpt/gpt-4_annotation_zip: our annotations on ChatGPT/GPT-4's outputs
+- generated_data
+  - analysis_reports.zip: a collection of 1015 analysis reports generated by CHATREPORT about NYSE companies.
+- code
+  - auto_prompt_engineer.py: the script for transferring experts' feedback to prompt engineering.
+  - app.py: the script handling application run
+  - document.py: the script handling PDF reading and retrieval
+  - reader.py: TCFD questions & conformity analyses
+  - user_qa.py: the script handling customized analyses through question answering
+
+## Usage
+Set up the environment
+```shell
+conda create --name chatreport python=3.10
+conda activate chatreport
+pip install -r requirements.txt
+```
+
+1. Modify `OPENAI_API_KEYS` in `apikey.ini`, you could fill in multiple api key like below
+```python
+OPENAI_API_KEYS = ['sk-XXX', 'sk-XXX']
+```
+
+2. Analyze a given report, for example: NYSE_SNE_2018.pdf
+```commandline
+python app.py --pdf_path NYSE_SNE_2018.pdf
+```
+- Analysis report will be stored at "NYSE_SNE_2018.html"
+- Answers to TCFD bullet points will be stored at "data/answers/NYSE_SNE_2018.json"
+- Assessments w.r.t. TCFD guidelines will be stored at "data/assessment/NYSE_SNE_2018.json"
+- Basic info will be stored at "data/basic_info/NYSE_SNE_2018.json"
+- The original PDF will be stored at "data/pdf/NYSE_SNE_2018.pdf"
+- Embedding/vector database will be stored at "data/vector_db/NYSE_SNE_2018/". When the report is queried again, there's no need to re-embed the vector DB.
+
+3. Conduct customized Question Answering
+```shell
+python app.py --pdf_path NYSE_SNE_2018.pdf --user_question "What is the level of cheap talk in the report?" --answer_length 50
+```
+- user_question takes the user's question
+- answer_length specifies the length of generation (we recommend 50)
+- Questions, answers, sources, pages, and answer length will be appended to "data/user_qa/NYSE_SNE_2018.jsonl" (lines of JSONs)
+
+## Citation
+Please cite our paper if you use CHATREPORT in your research.
+```bibtex
+@misc{ni2023chatreport,
+      title={CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools}, 
+      author={Jingwei Ni and Julia Bingler and Chiara Colesanti-Senni and Mathias Kraus and Glen Gostlow and Tobias Schimanski and Dominik Stammbach and Saeid Ashraf Vaghefi and Qian Wang and Nicolas Webersinke and Tobias Wekhof and Tingyu Yu and Markus Leippold},
+      year={2023},
+      eprint={2307.15770},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+

annotated_data/README.md

@@ -0,0 +1,33 @@
+## Zip files
+- chatgpt_annotation.zip contains the annotations on ChatGPT (June version) outputs about hallucinations on content/source dimensions.
+- gpt-4_annotation.zip contains the annotations on GPT-4 (June version) outputs about hallucinations on content/source dimensions.
+- sustainability_reports.zip contains the 10 sustainability reports that the data based on.
+
+## Structure of the JSON files:
+```
+{
+  "Report Names (e.g., NYSE_AIZ_2022)": {
+    "Question IDs (e.g., tcfd_1)": {
+      "question": "The TCFD question we asked about the sustainability report",
+      "answer": "The LLM’s answer to the question",
+      "sources": ["a list of source numbers that the LLM cited to answer the question"],
+      "relevant_chunks (i.e., evidence cited by the LLM)": {
+        "source_number_#1": "source #1 content",
+        "source_number_#2": "source #2 content",
+        ...
+      },
+      "irrelevant_chunks (i.e., evidence not cited by the LLM)": {
+        "source_number_#1": "source #1 content",
+        "source_number_#2": "source #2 content",
+        ...
+      }
+    },
+    "tcfd_2": ...,
+    "tcfd_3": ...,
+    ...
+    "tcfd_11": ...,      
+  }
+  "NYSE_BV_2022": ...
+  ...
+}
+```