Bug: Differing results with same inputs

[Gorgyrella]

Dear @sipemu, first, thank you very much for providing inexpensive access to this great application, which I am currently leveraging for my research at the University of St.Gallen.

While re-running the very same R code after a 40-day break, I suddenly get slightly different results than initially. I suspect that this is due to a change in how the API calculates ARs. When comparing the two analysis report files (as attached), I noticed that the more recent report now mentions the familiar "The event window seems to include an unexpected number of days the firm's stock has not been traded on" warning for firms which did not show this warning before. On the mathematical end, the values of the Actual Stock Return, Actual Market Return, and Expected Stock Return all changed for the affected firms, which in turn affects all subsequent calculations.

Please note that I am running the same versions of Mac OS and R Studio as well as the very same input files. The only difference on my end is that my API key expired, so I purchased a new one. I checked all public documentation but did not find information about a recent update. Hence, I am contacting you here. I am happy to provide the relevant code snippets or input files if necessary.

Any indication on why this change in results might have occurred is much apprciated!

230726_analysis_report.csv
230830_analysis_report.csv

[EST-M1]

Dear Gorgyrella, could you please share your input files with ***@***.***? There was no change in the AR calculation as such - we have, however, two general modes: log returns and normal returns. Are all events affected or only some? We're happy to explore your input files and come back to you with an answer after a review. Am Do., 31. Aug. 2023 um 17:19 Uhr schrieb Gorgyrella < ***@***.***>:

…

Dear @sipemu <https://github.com/sipemu>, first, thank you very much for providing inexpensive access to this great application, which I am currently leveraging for my research at the University of St.Gallen. While re-running the very same R code after a 40-day break, I suddenly get slightly different results than initially. I suspect that this is due to a change in how the API calculates ARs. When comparing the two analysis report files (as attached), I noticed that the more recent report now mentions the familiar *"The event window seems to include an unexpected number of days the firm's stock has not been traded on"* warning for firms which did not show this warning before. On the mathematical end, the values of the *Actual Stock Return*, *Actual Market Return*, and *Expected Stock Return* all changed for the affected firms, which in turn affects all subsequent calculations. Please note that I am running the same versions of Mac OS and R Studio as well as the very same input files. The only difference on my end is that my API key expired, so I purchased a new one. I checked all public documentation but did not find information about a recent update. Hence, I am contacting you here. I am happy to provide the relevant code snippets or input files if necessary. Any indication on why this change in results might have occurred is much apprciated! 230726_analysis_report.csv <https://github.com/EventStudyTools/api-wrapper.r/files/12487557/230726_analysis_report.csv> 230830_analysis_report.csv <https://github.com/EventStudyTools/api-wrapper.r/files/12487559/230830_analysis_report.csv> — Reply to this email directly, view it on GitHub <#29>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUF5QWRTBT7K3S3EFQXFXFDXYCTOZANCNFSM6AAAAAA4GH3UTI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Gorgyrella]

Thank you for the swift reply! The input files are too large to be sent per e-mail, so please find them attached here.

01_RequestFile.csv
02_FirmData.csv
03_MarketData.csv

I am consistently using the simple returns model throughout my analysis, by specifying the following two commands before running my event study:

1. returnEstParams <- ARCApplicationInput$new()
2. returnEstParams$setReturnType("simple")

Not all events are affected, only some of them. Specifically, the results for the following event dates have changed: 08.10.05, 08.01.05, 25.06.06, 15.04.07, 25.07.10, 27.08.11, 27.06.15, 05.03.16, 22.03.20

Your help is highly appreciated!

[adrian-ford]

Interestingly all weekend dates. Suggest check setNonTradingDays parameter whether “earlier” or “later” in each case?On 1 Sep 2023, at 08:54, Gorgyrella ***@***.***> wrote: Thank you for the swift reply! The input files are too large to be sent per e-mail, so please find the attached here. 01_RequestFile.csv 02_FirmData.csv 03_MarketData.csv I am consistently using the simple returns model throughout my analysis, by specifying the following two commands before running my event study: returnEstParams <- ARCApplicationInput$new() returnEstParams$setReturnType("simple") Not all events are affected, only some of them. Specifically, the results for the following event dates have changed: 08.10.05, 08.01.05, 25.06.06, 15.04.07, 25.07.10, 27.08.11, 27.06.15, 05.03.16, 22.03.20 Your help is highly appreciated! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Gorgyrella]

@adrian-ford I consistently set them to "later" in all my analyses. In general, I specify the following commands before running my analyses:

1. returnEstParams <- ARCApplicationInput$new()
2. returnEstParams$setReturnType("simple")
3. returnEstParams$setBenchmarkModel("mm")
4. returnEstParams$setNonTradingDays("later")
5. returnEstParams$setResultFileType("csv")
6. returnEstParams$setTestStatistics(c("art", "cart", "aaraptlz", "caaraptlz", "aargrankt", "caargrankt"))

[adrian-ford]

suggest manually adjust the dates in the file to next trading day for these examples as a test and see if it matches previous - would show it’s purely a date handling issue rather than calculation On 1 Sep 2023, at 09:25, Gorgyrella ***@***.***> wrote: @adrian-ford I consistently set them to "later" in all my analyses. In general, I specify the following commands before running my analyses: returnEstParams <- ARCApplicationInput$new() returnEstParams$setReturnType("simple") returnEstParams$setBenchmarkModel("mm") returnEstParams$setNonTradingDays("later") returnEstParams$setResultFileType("csv") returnEstParams$setTestStatistics(c("art", "cart", "aaraptlz", "caaraptlz", "aargrankt", "caargrankt")) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[Gorgyrella]

@adrian-ford Thank you for the suggestion! When adjusting the event dates in the 01_RequestFile.csv manually, around 46% of firm CARs are inexplicably different from the results I got yesterday, although only marginally in a -0.0146 to 0.0168 range. The overall CAAR result from today's manually adjusted model (-0.0523) is virtually the same as yesterday's CAAR result from the automatically adjusted one (-0.0522). While this raises yet another question why these two models themselves differ, the initial issue why these results deviate from the old model 40 days ago (with a CAAR of -0.0503) remains. All three models use the same event dates, whether manually or automatically adjusted as well as the same input files. Hence, the differing results are really concerning to me. Any other inputs or ideas?

[adrian-ford]

Did manually adjusting the dates remove the event window unexpected type errors which you got second time around but not originally?On 1 Sep 2023, at 18:14, Gorgyrella ***@***.***> wrote: @adrian-ford Thank you for the suggestion! When adjusting the event dates in the 01_RequestFile.csv manually, around 46% of firm CARs are inexplicably different from the results I got yesterday, although only marginally in a -0.0146 to 0.0168 range. The overall CAAR result from today's manually adjusted model (-0.0523) is virtually the same as yesterday's CAAR result from the automatically adjusted one (-0.0522). While this raises yet another question why these two models themselves differ, the initial issue why these results deviate from the old model 40 days ago (with a CAAR of -0.0503) remains. All three models use the same event dates, whether manually or automatically adjusted as well as the same input files. Hence, the differing results are really concerning to me. Any other inputs or ideas? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[Gorgyrella]

@adrian-ford I've just noticed that I've made a small mistake when describing the part with the error message. In fact, the initial model's analysis report from 40 days ago used to have a lot of these unexpected error messages, which then disappeared in the newer model of yesterday. There is no difference in occurrence of this error message between today's model, where I manually adjusted the event dates and the one of yesterday. So even though I slightly misspecified initially, the question remains why the quantitative results changed over time and how that might be related to the disappearance of the error message.

[adrian-ford]

Thanks for the update. As all the ones with differences were weekend event dates, it does point to a date processing issue as this seems too unlikely to be a coincidence, yet the evidence seems inconsistent. Have to wait for the EST support team investigation outcome I guess. On 1 Sep 2023, at 22:15, Gorgyrella ***@***.***> wrote: @adrian-ford I've just noticed that I've made a small mistake when describing the part with the error message. In fact, the initial model's analysis report from 40 days ago used to have a lot of these unexpected error messages, which then disappeared in the newer model of yesterday. There is no difference in occurrence of this error message between today's model, where I manually adjusted the event dates and the one of yesterday. So even though I slightly misspecified initially, the question remains why the quantitative results changed over time and how that might be related to the disappearance of the error message. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[EST-M1]

Hi together, so here's the issue with the marginal differences in results between automatic vs. manual shifted non-trading days: the estimation window was not shifted when moving the event date by 1 day (if event day is Sunday) or by 2 days (if event day is Saturday) or by more days (in case there were additional holidays on the days before/after the weekend). This will be fixed by a release this afternoon - Please find attached input files with which to test a Saturday/Sunday case. Should work for the big sample equally. Regarding the other question on potential changes between earlier analyses and today, The only change that took place in the last weeks was an increase of memory to the server - which does not appear as a very compelling reason to me, but given your firm data file is very large, memory limitations may have impacted the date shifter for non-trading days. My recommendation would be to use the results where results for automatic vs. manual date shifts are identical, these are correct. Am Fr., 1. Sept. 2023 um 23:38 Uhr schrieb Adrian Ford < ***@***.***>:

…

Thanks for the update. As all the ones with differences were weekend event dates, it does point to a date processing issue as this seems too unlikely to be a coincidence, yet the evidence seems inconsistent. Have to wait for the EST support team investigation outcome I guess. On 1 Sep 2023, at 22:15, Gorgyrella ***@***.***> wrote: @adrian-ford I've just noticed that I've made a small mistake when describing the part with the error message. In fact, the initial model's analysis report from 40 days ago used to have a lot of these unexpected error messages, which then disappeared in the newer model of yesterday. There is no difference in occurrence of this error message between today's model, where I manually adjusted the event dates and the one of yesterday. So even though I slightly misspecified initially, the question remains why the quantitative results changed over time and how that might be related to the disappearance of the error message. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUF5QWUFT6JCCCTH5IBUFE3XYJITZANCNFSM6AAAAAA4GH3UTI> . You are receiving this because you commented.Message ID: ***@***.***>

[Gorgyrella]

@EST-M1 thank you very much for the explanation. The input files with which to test a Saturday/Sunday case were not attached to the e-mail, could you please re-send them or upload them on GitHub directly? Regarding your comment on server memory limitations: I also had the same issue that the older and newer results suddenly differed in another analysis which uses a sample only about 15% the size of the case we are currently discussing, so I am also not too sure if the server memory explains this. In any case, can you tell what the current server limitations are, so I can decide if I should run my analysis sequentially? And is using the ARC calculator directly on your website subject to the same server limitations?

[EST-M1]

Here you go:
01 RequestFile_test.csv
02 FirmData_test.csv
03 MarketData_test.csv

There are no explicit memory limits, our server has certain RAM that is consumed by analyses, single or parallel runs - depending on current usage. We increased it recently.

GUI has an upload limit due to browser timeouts, see: https://www.eventstudytools.com/instructions-axc
"General notes: A size limit of approximately 5MB applies to each input file - you can zip individual files before uploading. Additionally, the GUI can process up to 1k events in a single event study, the API handles up to 4k events. The preferred decimal separator for closing prices is the dot. Once you uploaded the data, be patient as the calculation may require up to several minutes depending on your sample size."