- About NEMSIS Dataset
- Project Purpose
- How to Use This Repo
- Data Processing
- Data Analysis
- Findings
- Outcome Indicator
- Further Discussion
- Attribution
- Contributors
NEMSIS
NEMSIS is the National Emergency Medical Services Information System.
It is a public repository for Emergency Medical Services (EMS) data from the USA.
NEMSIS is a collaborative system used to improve prehospital patient care through the standardization, aggregation, and utilization of point-of-care EMS data at local, state, and national levels.
We seek to replicate the data analysis conducted in the article Differences in Out-of-Hospital Cardiac Arrest Management and Outcomes across Urban, Suburban, and Rural Settings but with a more recent dataset (2022 vs 2018). This analysis primarily aimed to describe treatment patterns and the odds of a favorable patient outcome (e.g., return of spontaneous circulation (ROSC) or being presumptively alive at the end of the incident) among rural OHCA patients in the U.S. by comparing the difference in outcomes between rural and non-rural settings cardiac arrest patients. In this analysis, we will also be investigating different factors in 911 Call times based on Urbanicity, including:
- Epinephrine Usage
- CPR Administered Before EMS Arrival
- Gender
- ROSC Achievement Rate
- Longer-Term End-of-Event Outcome (Presumptively Alive or Dead)
Working with this dataset has posed serious challenges, many of which we tried to mitigate using these steps. Because of the complexity of the work, this repository is divided into two main directories. There is a data-processing directory and a data-analysis directory. In addition, the docs directory includes some background research resources.
If you are interested in working with NEMSIS data for a different research question, once you have the data, you can use the guidelines included in this repo to produce a tailored NEMSIS database. You can find more on our data-processing logic and what the code is doing here.
The 2022 NEMSIS dataset was retrieved here from the national database. You must request the research dataset, "ASCII 2022.zip", through the NEMSIS Help Desk, but be forewarned that the dataset is large (16.78 GB zipped, 162 GB unzipped).
Once the NEMSIS zip file is acquired, place it in './data-processing/data/raw/'. Once you have the zip in the correct folder, make the database. Directions for this can be found here.You will have to be in the data-processing folder of this repo for these directions to work.
Note: This makes a 126GB SQLite database and takes significant time.
Once you have the database, you can query it based on criteria of interest using SQLite3 syntax. If you would like to work with the same cases that we selected for our analysis, go to root directory, project-ErinP123, and reproduce that pickle file using:
make pickle
This also takes ~1 hour but gives you a reasonably sized (58MB) file. To produce the reduced set of cases, the code is querying the data based on criteria from the referenced study. The list of features being extracted is based on features needed to conduct the analysis from the referenced study. In addition, we retained any features needed to generate the end of life outcome based on this study (used by the reference study to generate their secondary outcome for analysis).
For more information on our filtering standards, refer to them here.
The data analysis component of this project replicates the methodology of the referenced 2018 study while incorporating the 2022 dataset and additional features of interest to stakeholders. This project uses a series of 2 sided t-tests and chi-square tests to test statistical differences in factors based on the urbanicity of the 911 call.
You will have to be in the data-analysis folder of this repo for these directions to work.
You can reproduce the tests run by using:
make analysis
We did test an alternative analysis method that relies on just-in-time sql queries of the the database rather than the generation of the full cardiac_joins.pickle. You can see this in table1.py. To run table1.py you must have made the rural and urban PcrKey tables. See those instructions in the under optional step 5 of the database build. This method takes less time at the outset as you save the time of building a large pickle file. However, it does take a similar amount of time when you account for running each query. This could be addressed through database optimization methods. This would be an interesting avenue to explore fo next steps.
There are small differences between these two analyses methods. We attribute these to the data cleaning process required to use the data features in the python analysis code, cleaning that results in dropping some cases. But these differences are another avenue worth exploring. We present the results from the python analysis script below since this included more tests and is easily reproduced by the make command.
Our filtering resulted in 124,317 cases compared to 64,489 cases in the 2018 study. The 2018 NEMSIS dataset has about half as many events as the 2022 dataset (22.5 million in 2018 to 2022's 54 million). Keep in mind, that not all cases will have the data for each analysis test, so total cases per test will be less than 124,317.
Total Included
All incidents: 124317 || Rural: 11301, 9.09% || Urban: 113016, 90.91%
-
Urbanicity and Response Times
Method: Using a two-sided t-test sourced from pre-authored Python libraries, we compared the mean age and EMS response times between rural and urban/suburban settings.
Average Time of response for Urban Areas 7.23 minutes
Average Age for Rural Areas 9.73 minutes
Time in min P Score 0.0000
Rural settings were associated with significantly longer EMS response times than urban/suburban settings, replicating a key finding from the 2018 study.
- Urbanicity and Mean Age Using our method of filtering there does look like theres a significant difference in pscore
Average Age for Urban Areas 63.69
Average Age for Rural Areas 64.29
Age In Year P Score 0.0003
-
Epinephrine Administration
Method: The paper used a chi-square test to assess the relationship between urbanicity and epinephrine administration. We also used a proportional Z-test to see if there was a difference in the results.
These will need to be double checked as our codes could differ or not be inclusive. The majority of used codes came from RxNorm.
Epinephrine Use Rates by Urbanicity:
sum count Percent
Urbanicity
Rural 3445 9213 37.392815
Urban 38781 95179 40.745332
Chi-Square Test Results:
Chi-Square Statistic: 39.05, p-value: 0.0000
Two-Sample Proportion Z-Test Results:
Z-Statistic: 6.26, p-value: 0.0000
Just as in the 2018 study, there is a statistically significant difference in whether or not Epinephrine was used on a patient, with rural patients getting it less.
When analyzing the dose amount, we had to take a few liberties with the data by excluding doses that were not standardizable and excluding outliers. This showed there was a slight difference in dosage levels, although more testing would have to be done to prove that it was statistically significant.
Average EpiDose for Urban Areas (mg): 2.21
Average EpiDose for Rural Areas(mg): 1.04
p-value for Dose Difference: 0.0524
While trying to decipher the paper, we also ran a multivariable logistic regression to see whether ROSC was achieved. Since we were most interested in Epinephrine usage, we ran it based on Urbanicity and found significant results
====================================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------------
const -0.3369 0.025 -13.329 0.000 -0.386 -0.287
Epi -0.2516 0.015 -17.127 0.000 -0.280 -0.223
Urbanicity_Urban 0.2983 0.026 11.543 0.000 0.248 0.349
====================================================================================
This shows that Urbanicity paired with epi usage (this is a binary Y/N for receiving Epinephrine) was significantly important in whether a Patient achieved ROSC.
- Other Factors While we focused the majority of our analysis on the use of epinephrine, we were able to compare other factors that could differ based on urbanicity.
- Shockable Rhythm
Similarly to the 2018 paper, we found there was no statistical difference in whether a patient had a shockable rhythm or not. Both Urban and Rural patients had about a 25% chance of having a shockable rhythm, which is slightly higher than the percentages in the paper.
Chi-Square Test Results For Having Shockable Rhythm:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
eArrest_11
False 8784 88799 97583
True 2401 24137 26538
All 11185 112936 124121
Contingency Table(Percents):
Urbanicity Rural Urban All
eArrest_11
False 78.53 78.63 78.62
True 21.47 21.37 21.38
Chi-Square Stat : 0.0535
p-score: 0.9996
- CPR Administerd Prior to EMS arrival
This factor differs from the results in the 2018 study. In that study, there was no significant difference in this factor. We need to verify further that our filtering criteria align with the criteria used in the 2018 study.
Chi-Square Test Results for CPR Prior to EMS Arrival:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
eArrest_05
False 4525 46723 51248
True 6073 57664 63737
All 10598 104387 114985
Contingency Table(Percents):
Urbanicity Rural Urban All
eArrest_05
False 42.7 44.76 44.57
True 57.3 55.24 55.43
Chi-Square Stat : 16.57
p-score : 0.0023
- Male Population
This result is consistent with the 2018 paper in both significance and percentages. There is no statistical significance in differences between the percentage of male patients in Rural and urban areas.
Chi-Square Test Results for Males:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
ePatient_13
False 4260 43150 47410
True 6896 69588 76484
All 11156 112738 123894
Contingency Table(Percents):
Urbanicity Rural Urban All
ePatient_13
False 38.19 38.27 38.27
True 61.81 61.73 61.73
Chi-Square Stat: 0.0339
p-score : 0.9999
- Having Achieved ROSC
The significance of the differences in the proportion that achieved ROSC in rural versus urban settings matches the 2018 study. Still, the percentage of patients that receive ROSC in both settings is higher than the study found. This is further evidence that we should confirm our filtering criteria to compare our results to the 2018 study results more confidently.
Chi-Square Test Results For Having Achieved ROSC:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
eArrest_12
False 6267 51753 58020
True 3992 47555 51547
All 10259 99308 109567
Contingency Table(Percents):
Urbanicity Rural Urban All
eArrest_12
False 61.09 52.11 52.95
True 38.91 47.89 47.05
Chi-Square Stat : 300.5933
p-score: 0.0000
- Mechanical CPR Use
Similar to the paper, we saw a statistically significant difference in Rural and Urban use of Mechanical CPR, with Rural use being higher. Although the percentage is consistent with the paper, it does seem much lower.
Chi-Square Test Results For Having Mechanical CPR:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
eArrest_09
False 8250 86570 94820
True 1912 17181 19093
All 10162 103751 113913
Contingency Table(Percents):
Urbanicity Rural Urban All
eArrest_09
False 81.18 83.44 83.24
True 18.82 16.56 16.76
Chi-Square Stat : 33.7443
p-score: 0.0000
- Witnessed by Medical Provider These numbers are insignificant, and there is no statistical difference between Urbanicity and whether EMS witnessed the event, with ~21% of cases in both Urban and Rural settings being seen.
Chi-Square Test Results For Having been witnessed by a medical provider:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
eArrest_04
False 8121 82143 90264
True 2851 27950 30801
All 10972 110093 121065
Contingency Table(Percents):
Urbanicity Rural Urban All
eArrest_04
False 74.02 74.61 74.56
True 25.98 25.39 25.44
Chi-Square Stat : 1.8728
p-score: 0.7591
- Revieved Advanced Life Support These results are consistent with the 2018 paper. There is a significant difference in Urbanicity groups receiving Advanced life support, with the Urban settings getting it slightly more of the time.
Chi-Square Test Results For Advanced Life Support:
Contingency Table(raw numbers):
Urbanicity Rural Urban All
eResponse_15
False 1513 11562 13075
True 9672 101374 111046
All 11185 112936 124121
Contingency Table(Percents):
Urbanicity Rural Urban All
eResponse_15
False 13.53 10.24 10.53
True 86.47 89.76 89.47
Chi-Square Stat : 116.8421
p-score: 0.0000
- To make the amount of epinephrine administered standardized, I dropped a few odd cases such as "Other," "Keep Vein Open," and "Milliequivalents," as well as cases that measured per hour. These comprised a handful of cases (<20 cases per measurement) and accounted for approximately 2.46% of the overall analyzed data.
- After seeing the dosages, I noticed one extreme outlier (this could have been a clerical error as I don't think anybody got 10000 mg of epinephrine). We only kept the data within three standard deviations of the mean for rural and urban datasets; this changed the overall amount of urban dosages substantially (from 12.3mg to 2.21mg)
- Analysis was only run on chi-square tests with codes, and all NV codes were excluded (codes "7701001", "7701003"), although we did not blanket call this statement as when you dropped rows with ANY value, you lost nearly all of your data set. This accounts for the different totals in the chi-square tests.
- Since this is healthcare data, I did not think it was smart and could have been potentially dangerous to impute data, so we dropped values we did not know the answer to.
- We found difference in significance for a few features significantly age and CPR Adminsitered. This could be due to the difference in filtering, or the vast difference in degrees of freedom as our data set has ~2x the sample size.
In addition to the previous analyses performed, this project aimed to:
- Replicate the longer-term End-of-Event Outcome Indicator imputation method (indicates whether a patient is presumptively alive or dead) to use with patient care reports (PCR events) that are missing definitive outcomes of alive or dead after an OHCA event by using other available medical data.
- Compare the likelihood of patients being "presumptively alive or dead" in rural, suburban, wilderness and urban settings.
- Use multivariable logistic regression to evaluate the association between the end-of-event outcome indicator and previously evaluated factors such as Urbanicity, Patient Age and Gender, EMS Response Times, Level of Care Provided, Epinephrine Administration and Arrest Characteristics
- Perform sensitivity testing of the indicator imputation method by reporting True Positive Rates ("dead" prediction accuracy) and True Negative Rates ("alive" prediction accuracy) to ensure the robustness of the analyses.
However, by the end of the semester, we were still troubleshooting step 1 and will need to pass the completion of steps 1-4 to the inheritors of this project. Our Python script replication of the indicator via outcome_predictor.py, using the sample NEMSIS dataset pickle file, cardiac_joins.pickle in data-processing\data\processed as input, produced the following results.
- 53.45% of the total were indicated as dead
- 31.57% as alive
- 14.98% as having conflicting results (indicated as both alive and dead)
Chi-Square Test Results For End of Event:
There were 18590 unresolved cases out of 124121 total cases, 14.98%
Contingency Table(raw numbers):
Urbanicity Rural Urban All
binary_outcome
Alive 3069 36119 39188
Dead 6476 59867 66343
All 9545 95986 105531
Contingency Table(Percents):
Urbanicity Rural Urban All
binary_outcome
Alive 32.15 37.63 37.13
Dead 67.85 62.37 62.87
Chi-Square Stat : 111.5372
p-score: 0.0000
In the (Peters et al, 2023), they found 14,704 (22.8%) of total cases were presumed alive, 1,128 (20.1%) for rural and 13,111 (23.0%) for urban. Given the challenges replicating the filtering criteria, there are a number of reasons that our numbers could be divergent. We do think that this imputation script needs additional attention. In addtion, given that the original authors fo this imputation method found cardiac events particularly rife with conflicting outcomes, there may be a better imputation method to use in the case of cardiac events.
For further details regarding the background, challenges, and recommendations for the continuation of this part of the project, go to outcome_indicator.md
To reproduce the imputation results, use:
make impute
Note: to review the imputation results, we recommend uncommenting code provided in the script to output to a csv file before using the command.
It is very challenging to determine the exact query used in the study based on how it is described in the article (for example, cases involving trauma/injury need to be filtered out, and many features in the database contain codes that might indicate trauma/injury). Determining filtering criteria has proven to be one of the most challenging aspects of this project, even with correspondence with the article authors and conversations with the project stakeholders.
A useful next step is to use the database to reproduced a more well defined article methodology. The outcome indicator offers one such example.
In addition to the size of the dataset, the data itself is challenging to work with. As noted, this is made worse when authors do not specify their methodology in a way that supports replication. Even with that level of detail, as noted by Miller et al (2021),
Finally, the end-of-event indicator determinations depend on the documentation practices of the responding EMS systems and the attending EMS professionals, with many missing and sometimes contradictory entries. The NEMSIS dataset also does not delineate chart entries recorded in real time, versus those recorded after the conclusion of the event. This highlights the need for careful data entry by EMS clinicians and attentive data cleaning by data coordinators, but it is a commonly reported limitation of NEMSIS-based studies (12–14).(Miller et al, 2021)
Hopefully they will continue to refine the data collection and entry methods. With so much data available, it seems likely that many import innovations in emergency response medicine can be gleaned. We hope the stakeholders and future student groups find this work helpful in progressing toward that end.
It is worth discussing the importance of having content experts and digging in and exploring the research base to help understand what results make sense. Engaging subject matter experts who can speak to use of and intpretation of the features and codes would be extremely valuable.
See our list of many Attributions here
The Roux Institute at Northeastern University, DS5110 Fall 2024 Class:
Erin Pryor | Christine Voyer | Zachary Merriam With direct guidance from Professor Phillip Bogden