Pdf parsing

chunker_test_results.txt

@@ -0,0 +1,55 @@
+============================= test session starts ==============================
+platform darwin -- Python 3.10.12, pytest-8.4.2, pluggy-1.6.0 -- /Users/diyakothari/Desktop/Wattbot/venv/bin/python
+cachedir: .pytest_cache
+rootdir: /Users/diyakothari/Desktop/Wattbot
+plugins: cov-7.0.0
+collecting ... collected 24 items
+
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_text_basic PASSED [  4%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_text_overlap PASSED [  8%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_text_empty PASSED [ 12%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_text_shorter_than_chunk_size PASSED [ 16%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_text_exact_chunk_size PASSED [ 20%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_text_whitespace_handling PASSED [ 25%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_custom_chunk_size PASSED [ 29%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_custom_overlap PASSED [ 33%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_zero_overlap PASSED [ 37%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_large_overlap PASSED [ 41%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_multiline_text PASSED [ 45%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_extract_text_from_pdf_real_file PASSED [ 50%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_extract_text_from_pdf_returns_string PASSED [ 54%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_extract_text_from_pdf_mock PASSED [ 58%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_extract_tables_from_pdf_success PASSED [ 62%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_extract_tables_from_pdf_failure PASSED [ 66%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_extract_tables_both_flavors PASSED [ 70%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_pdf_real_file PASSED [ 75%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_pdf_creates_text_chunks PASSED [ 79%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_pdf_creates_table_chunks PASSED [ 83%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_save_chunks_to_json PASSED [ 87%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_save_chunks_creates_directory PASSED [ 91%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_init_default_parameters PASSED [ 95%]
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_init_custom_parameters PASSED [100%]
+
+=============================== warnings summary ===============================
+venv/lib/python3.10/site-packages/PyPDF2/__init__.py:21
+  /Users/diyakothari/Desktop/Wattbot/venv/lib/python3.10/site-packages/PyPDF2/__init__.py:21: DeprecationWarning: PyPDF2 is deprecated. Please move to the pypdf library instead.
+    warnings.warn(
+
+venv/lib/python3.10/site-packages/pypdf/_crypt_providers/_cryptography.py:32
+  /Users/diyakothari/Desktop/Wattbot/venv/lib/python3.10/site-packages/pypdf/_crypt_providers/_cryptography.py:32: CryptographyDeprecationWarning: ARC4 has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.ARC4 and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
+    from cryptography.hazmat.primitives.ciphers.algorithms import AES, ARC4
+
+wattbot2025/tests/test_chunker.py::TestDocumentChunker::test_chunk_pdf_real_file
+  /Users/diyakothari/Desktop/Wattbot/venv/lib/python3.10/site-packages/camelot/parsers/base.py:238: UserWarning: No tables found in table area (97.691, 188.51954679999997, 515.7449530889999, 485.81292525945946)
+    cols, rows, v_s, h_s = self._generate_columns_and_rows(bbox, user_cols)
+
+-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
+================================ tests coverage ================================
+______________ coverage: platform darwin, python 3.10.12-final-0 _______________
+
+Name                              Stmts   Miss  Cover   Missing
+---------------------------------------------------------------
+wattbot2025/src/data/chunker.py      60      7    88%   82-90
+---------------------------------------------------------------
+TOTAL                                60      7    88%
+======================== 24 passed, 3 warnings in 9.82s ========================

wattbot2025/data/ranked/amazon2023_chunks_ranked.json

@@ -0,0 +1,122 @@
+[
+  {
+    "rank": 1,
+    "score": 9.132716887768051,
+    "content": "ctively contribute more than 50% of emissions globally \nto Amazon’s Scope 3 footprint, to provide a plan for how \nthey will decarbonize their operations and demonstrate real \nprogress over time. We will prioritize our business toward \nthose who provide their plans and results on their path to \nnet-zero carbon emissions. In addition, we also launched our \n“Amazon Sustainability Exchange”—a free, pu",
+    "type": "text"
+  },
+  {
+    "rank": 2,
+    "score": 9.096112610125404,
+    "content": "lity Report Appendix People\nOverview Environment Value Chain\nWater Waste and Circularity Packaging Carbon-Free  Energy Carbon\n\nOur Progress\nAmazon’s Carbon Footprint\nIn 2023, our absolute carbon emission s decreased by 3%.2  \nThis overall decrease was driven by an 11% reduction in \nemissions from electricity (Scope 2) and a 5% decrease in \nindirect and supply chain emissions (Scope 3). We had a 7%",
+    "type": "text"
+  },
+  {
+    "rank": 3,
+    "score": 9.08335271425718,
+    "content": "ic tons of CO2e in 2023—equivalent to the\",\n,to develop and scale innovations that reduce carbon,,\n,,,baseline understanding of carbon emissions from industrial\nsystems in new stores in North America starting in 2025.,,\"carbon emissions generated from driving over 11,100 cars in\",\n,\"emissions associated with the construction, operation, and\",,\n,,,\"equipment, paving the way to ultimately identifyin",
+    "type": "table"
+  },
+  {
+    "rank": 4,
+    "score": 8.707361573036623,
+    "content": "uts us in a unique,,,\n,such as nuclear.,,\nposition to be a leader in decarbonization strategies. We,,,\nhave an opportunity to demonstrate how achieving net-,•  We engage with suppliers to help reduce emissions from,,\n\"zero carbon emissions is possible across many sectors, while\",activities beyond our direct operations. We encourage,,\ncreating solutions that benefit our business as well as the,\"the",
+    "type": "table"
+  },
+  {
+    "rank": 5,
+    "score": 8.18430570245479,
+    "content": "hanical, electrical, \nand plumbing equipment. Using data that is new to the \nindustry, Technical Memorandum 65.3 will enable Amazon, \nour supply chain partners, and our peers to establish a \nbaselin e understanding of carbon emissions from industrial \nequipment, paving the way to ultimately identifying \nadditional carbon reduction opportunities.\nServers and Hardware\nAs the world’s most comprehensi",
+    "type": "text"
+  },
+  {
+    "rank": 6,
+    "score": 7.96014875338234,
+    "content": "ur goal to reach net-zero carbon emissions,\"these alternatives where possible, based on a number of\",,\n,,\"vehicle deployment and infrastructure, advance the\",\n,,,\"our business, we are also investing in carbon neutralization\"\n\"by 2040, 10 years ahead of the Paris Agreement—and have\",\"factors including cost, emissions reduction potential, and\",,\n,,\"deployment of carbon-free energy, modernize the gri",
+    "type": "table"
+  },
+  {
+    "rank": 7,
+    "score": 7.9320909100196095,
+    "content": "ig goals and work backward to achieve,\n,\"many business units and subsidiaries including AWS, Devices,\"\n\"them, such as The Climate Pledge, our goal to reach net-\",\n,\"Fresh, Whole Foods Market, Amazon Private Brands, Twitch,\"\n\"zero carbon emissions by 2040, 10 years ahead of the\",\n,\"MGM Studios, and Ring.\"\nParis Agreement. We apply that same tenacity to how we,\naddress some of the world’s biggest en",
+    "type": "table"
+  },
+  {
+    "rank": 8,
+    "score": 7.890755003738783,
+    "content": "uting chip \nefficiency, adding Low Power Mode to devices, and \ninstalling energy-efficient lighting and HVAC solutions \nin buildings. • We select lower-carbon alternatives,  such as lower-\ncarbon concrete and steel in construction, and lower-\nemission fuels and vehicles in transportation. We use \nthese alternatives where possible, based on a number of \nfactors including cost, emissions reduction p",
+    "type": "text"
+  },
+  {
+    "rank": 9,
+    "score": 7.868317211700303,
+    "content": "to set goals to decarbonize their own operations, \nand working with suppliers on initiatives to reduce their \ngreenhouse gas (GHG) emissions. We use broad carbon-free energy options to support our \ncontinued growth, enabling us to deploy and grow new \ntechnologies such as artificial intelligence (AI). By scaling \ncarbon-free energy, we aim to make Amazon a more resilient \nand more sustainable busi",
+    "type": "text"
+  },
+  {
+    "rank": 10,
+    "score": 7.861685173369221,
+    "content": "to help reduce emissions from \nactivities beyond our direct operations. We encourage \nthem to set credible decarbonization goals, publicly share \nprogress, and implement carbon reduction strategies \nthroughout their operations and supply chains —and \nwe are providing support to help our supply chain \ntake action.1 \nIn addition to decarbonizing our own business, we are helping \ndrive progress acros",
+    "type": "text"
+  },
+  {
+    "rank": 11,
+    "score": 7.661873940434669,
+    "content": "0,1,2\nAddressing Health Equity,,\n,,\nAWS is harnessing the power of the cloud to advance health,Funding Nature in Our Communities,\nequity globally. Through the AWS Health Equity Initiative,,\n,\"We use nature-based solutions \n to mitigate carbon emissions outside of our value\",\n\"(HEI), AWS has pledged to provide up to $60 million in\",,\n,chain and supplement the carbon reduction efforts we’re driving ",
+    "type": "table"
+  },
+  {
+    "rank": 12,
+    "score": 7.623157662949613,
+    "content": "tions Guiding Principles Reporting Framework \n(UNGPRF). \nLearn more in our 2023 Sustainability Reporting  \nFramework Summary  How to Navigate This Report\nLook for these symbols throughout the report:\n A link that directs you to a website\n A link within the report\n A link to a downloadIntroductionAbout Amazon\nAmazon is a global company with approximately  \n1.5 million full- and part-time employees ",
+    "type": "text"
+  },
+  {
+    "rank": 13,
+    "score": 7.6049343072448545,
+    "content": "le Supply Chain Human Rights Community Impact Supplier DiversityLooking Forward\nAs we move forward, Amazon aims to continue supporting \nthe communities where we operate, focusing on the areas \nwhere we can make the biggest change: education, food \nsecurity, disaster response, affordable housing equity, and \nhealth equity. Our goal is to keep using our infrastructure, \ntechnology, and passion for i",
+    "type": "text"
+  },
+  {
+    "rank": 14,
+    "score": 7.4998172150174,
+    "content": "ustainability goals Our carbon neutralization approach focuses on three \nactions outside of our value chain. Based on climate science, \nwe know that these areas have a significant unmet need for \ninvestment and can deliver critical mitigation benefits: \n1. Reducing deforestation\n2. Advancing the removal of carbon from the atmosphere \nwith nature-based solutions\n3. Scaling up carbon removal technol",
+    "type": "text"
+  },
+  {
+    "rank": 15,
+    "score": 7.466290500718765,
+    "content": "2015 through 2022, but with the inclusion of the additional \nprograms, actual savings was more than 3 million metric tons in 2022 \nand more than 4 million in 2023.\n 5 BloombergNEF.\n 6 Renewable natural gas (RNG) is created by decomposing organic waste \nmaterials anaerobically (without oxygen).\n 7 Electric vehicles include vans, four-wheel vehicles, three-wheel vehicles, \ntwo-wheel e-bikes, and e-m",
+    "type": "text"
+  },
+  {
+    "rank": 16,
+    "score": 6.878379361978933,
+    "content": "(RNG).6\",,\n,,,Last Mile Electric Delivery Vehicles\n,,\"(SFC) Exchange Network, a nonprofit organization whose\",\n,\"In 2023, we piloted the use of hydrogen fuel cell vehicles\",mission is to accelerate the reduction of logistics emissions,Increasing the number of EVs in Amazon’s delivery fleet is an\n,(FCVs) in Europe and Japan. These initiatives will provide us,\"by fostering collaboration. As part of ",
+    "type": "table"
+  },
+  {
+    "rank": 17,
+    "score": 6.861801886743519,
+    "content": "emissions by 2040, 10 years ahead of the Paris \nAgreement. We are continually working to reduce \nemissions throughout our business, as well \nas partnering across our supply chain and the \nindustries in which we operate to share and scale \nwhat we’ve learned.Goal\nInspire and empower others to sign The Climate Pledge \nand join us on a mission to reach net-zero carbon \nemissions by 2040\n473\nSignatori",
+    "type": "text"
+  },
+  {
+    "rank": 18,
+    "score": 6.77550292060147,
+    "content": "leted and operational. We aim to reduce \nembodied carbon in building construction by using lower-\nemission concrete, lower-emission steel, and mass timber. In \n2023, 29 Amazon building projects were constructed with \nlower-carbon concrete and steel, and collectively reduced \nembodied carbon by 79,500 metric tons of CO2e, equivalent \nto the emissions generated by 17,200 cars driven for a year.\nBeca",
+    "type": "text"
+  },
+  {
+    "rank": 19,
+    "score": 6.726239559833433,
+    "content": "taking action to achieve our,,,\"wind farm in Mississippi, and becoming the first corporate\"\n,\"initiatives, and public policy advocacy to advance access\",,\ngoals in the following ways:,,,",
+    "type": "table"
+  },
+  {
+    "rank": 20,
+    "score": 6.473402333685977,
+    "content": "r total carbon footprint. This decrease resulted from \nreductions related to building construction, leased buildings \nand equipment, and third-party transportation, as more \ngoods were shipped by Amazon’s own logistics providers \nversus third-party providers than in 2022.\nBuilding construction is a significant driver of carbon \nemissions in many supply chains due to the associated \nembodied carbon",
+    "type": "text"
+  }
+]