draft nuv dark monitor. help. by cmagness · Pull Request #156 · spacetelescope/cosmo

cmagness · 2020-02-06T21:29:40Z

it doesn't break!!! but i think it's clunky and not sure what is happening at the plotting stage. everything seems to be fine up to there.

eventually will close #79
Resolves #157

…anything else useful

codecov · 2020-02-06T21:35:01Z

Codecov Report

Merging #156 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #156   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          18       18           
  Lines        1476     1476           
=======================================
  Hits         1368     1368           
  Misses        108      108

Impacted Files	Coverage Δ
cosmo/filesystem.py	`88.38% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bf5484...57ef9d2. Read the comment docs.

jwhite3 · 2020-02-11T19:53:52Z

+    run = 'monthly'
+
+    def get_data(self):
+        # access data, perform any filtering required for analysis


Suggested change

# access data, perform any filtering required for analysis

"""access data, perform any filtering required for analysis"""

jwhite3 · 2020-02-11T19:54:16Z

+        return data
+
+    def calculate_dark_rate(self, dataframe, xlim, ylim):
+        # calculate dark rate for one exposure, with a dataframe with TIME,


Suggested change

# calculate dark rate for one exposure, with a dataframe with TIME,

"""calculate dark rate for one exposure, with a dataframe with TIME, XCORR, and YCORR values"""

jwhite3 · 2020-02-11T20:08:14Z

+    # TODO: update docs
+    # docs = "https://spacetelescope.github.io/cosmo/monitors.html#<darks>"
+    output = COS_MONITORING
+


Suggested change

# Plot stuff

x = 'TIME'

y= 'DARK_RATE'

plot_type = 'scatter'

^^ If you want to use the built-in basic plotting

https://spacetelescope.github.io/monitor-framework/creating_monitors.html#defining-a-new-monitor

…ing calculated wrong

jwhite3 · 2020-03-16T18:39:10Z

 .DS_Store
+
+# Misc
+testoutputs/


Was this something for you in particular? If so, I'd be careful about adding it here since .gitignore could grow quite a bit depending on what folks are doing.

Instead, try using the the unit tests! You could even update some of the "cleanup" fixtures to remove "output" on the start of tests rather than the end so that you can inspect the test artifacts. If you'd like some suggestions or help with this, let me know!

yes, actually that would be helpful! but i was planning to remove that stuff before merging it in, it's just for ease of directory access right now

jwhite3 · 2020-03-16T18:45:04Z

+class NUVDarkDataModel(BaseDataModel):
+    """Datamodel for NUV Dark files."""
+    files_source = FILES_SOURCE
+    subdir_pattern = '?????'


Doesn't look like you're using subdir_pattern elsewhere, so I think there are a couple of options:

Get rid of this attribute

Use find_files (which uses subdir_pattern)

I'd recommend the 2nd option just because it might make "forward compatibility" easier (i.e. if dark datasets aren't manually identified in this way in the future). I think @dzhuliya has an example of using find_files on the fuv dark branch

yes i want to use find_files but for some reason it wasn't working for me so i was planning to update that. i'll look at dzhuliya's branch

jwhite3 · 2020-03-16T18:53:30Z

+
+        return dark_rate_array, dec_year_array
+
+    def track(self):


The track method should return some sort of quantity (scalar, dataframe, etc), not execute plotting

yep that is the end goal, don't worry. once the histogram is working correctly it will get the rates from the histogram and return those

jwhite3 · 2020-03-16T18:59:02Z

+        fig = plt.figure(figsize=(12, 9))
+        bin_size = 1e-8
+        n_bins = int((dark_counts.max() - dark_counts.min()) / bin_size)
+        ax = fig.add_subplot(2, 1, 1)
+        ax.hist(dark_counts, bins=n_bins, align='mid', histtype='stepfilled')
+        counts, bins = np.histogram(dark_counts, bins=100)
+        cuml_dist = np.cumsum(counts)
+        count_99 = abs(cuml_dist / float(cuml_dist.max()) - .99).argmin()
+        count_95 = abs(cuml_dist / float(cuml_dist.max()) - .95).argmin()
+
+        mean = dark_counts.mean()
+        med = np.median(dark_counts)
+        std = dark_counts.std()
+        mean_obj = ax.axvline(x=mean, lw=2, ls='--', color='r', label='Mean ')
+        med_obj = ax.axvline(x=med, lw=2, ls='-', color='r', label='Median')
+        two_sig = ax.axvline(x=med + (2 * std), lw=2, ls='-', color='gold')
+        three_sig = ax.axvline(x=med + (3 * std), lw=2, ls='-',
+                               color='DarkOrange')
+        dist_95 = ax.axvline(x=bins[count_95], lw=2, ls='-',
+                             color='LightGreen')
+        dist_99 = ax.axvline(x=bins[count_99], lw=2, ls='-', color='DarkGreen')
+
+        ax.grid(True, which='both')
+        ax.set_title('Histogram of Dark Rates', fontsize=15, fontweight='bold')
+        ax.set_ylabel('Frequency', fontsize=15, fontweight='bold')
+        ax.set_xlabel('Counts/pix/sec', fontsize=15, fontweight='bold')
+        ax.set_xlim(dark_counts.min(), dark_counts.max())
+        ax.xaxis.set_major_formatter(FormatStrFormatter('%3.2e'))
+
+        ax = fig.add_subplot(2, 1, 2)
+        # log_bins = np.logspace(np.log10(dark.min()), np.log10(dark.max()),
+        # 100)
+        ax.hist(dark_counts, bins=n_bins, align='mid', log=True,
+                histtype='stepfilled')
+
+        ax.axvline(x=mean, lw=2, ls='--', color='r', label='Mean')
+        ax.axvline(x=med, lw=2, ls='-', color='r', label='Median')
+        ax.axvline(x=med + (2 * std), lw=2, ls='-', color='gold')
+        ax.axvline(x=med + (3 * std), lw=2, ls='-', color='DarkOrange')
+        ax.axvline(x=bins[count_95], lw=2, ls='-', color='LightGreen')
+        ax.axvline(x=bins[count_99], lw=2, ls='-', color='DarkGreen')
+
+        # ax.set_xscale('log')
+        ax.grid(True, which='both')
+        ax.set_ylabel('Log Frequency', fontsize=15, fontweight='bold')
+        ax.set_xlabel('Counts/pix/sec', fontsize=15, fontweight='bold')
+        ax.set_xlim(dark_counts.min(), dark_counts.max())
+        ax.xaxis.set_major_formatter(FormatStrFormatter('%3.2e'))
+
+        fig.legend([med_obj, mean_obj, two_sig, three_sig, dist_95, dist_99],
+                   ['Median: {0:.2e}'.format(med),
+                    'Mean: {0:.2e}'.format(mean),
+                    r'2$\sigma$: {0:.2e}'.format(med + (2 * std)),
+                    r'3$\sigma$: {0:.2e}'.format(med + (3 * std)),
+                    r'95$\%$: {0:.2e}'.format(bins[count_95]),
+                    r'99$\%$: {0:.2e}'.format(bins[count_99])], shadow=True,
+                   numpoints=1, bbox_to_anchor=[0.8, 0.8])


Just to reiterate, plotting should not be in track, but also, plotly should be used for plotting if possible by creating traces and whatnot and then adding them to the figure attribute:

def plot(self): ... # Make plotly traces here; let's say you have a list called "traces" self.figure.add_traces(traces)

yeah that is the end goal as well, but marc wanted the plots replicated for now

jwhite3 · 2020-03-16T19:01:53Z

+        for index, row in plotdf.iterrows():
+            all_dec_year = list(
+                itertools.chain(all_dec_year, row["DECIMAL_YEAR"]))
+            all_dark_rates = list(
+                itertools.chain(all_dark_rates, row["DARK_RATE"]))


This bit might be what's causing the funkiness in your plots perhaps?

jwhite3 · 2020-03-16T19:42:19Z

+        if self.data is None:
+            self.data = self.get_data()


I see where you're going with this, and it's not a bad idea, but the precedent with the other monitors has been to not enable "asynchronous" steps in the sense that if you want to call individual steps, you still need to call the other steps that it depends on.

If you enabled this here, it should be enabled for all of the other monitors as well so there's no confusion on expected behavior between monitors.

draft of the nuv dark monitor. it doesn't break. not sure if it does …

67f5cc8

…anything else useful

cmagness added the New Feature label Feb 6, 2020

cmagness assigned jwhite3 Feb 6, 2020

jwhite3 reviewed Feb 11, 2020

View reviewed changes

jwhite3 added this to the Add Dark Monitors milestone Feb 14, 2020

this is NOT FUNCTIONAL but i am pushing it so @jwhite can review. thx.

8b215b0

jwhite3 reviewed Mar 10, 2020

View reviewed changes

Comment thread cosmo/monitors/data_models.py

jwhite3 reviewed Mar 10, 2020

View reviewed changes

Comment thread cosmo/monitors/nuv_dark_monitor.py

Comment thread cosmo/monitors/nuv_dark_monitor.py Outdated

Comment thread cosmo/monitors/nuv_dark_monitor.py

Comment thread cosmo/monitors/nuv_dark_monitor.py Outdated

this is a disaster and i need help to figure out why the rates are be…

57ef9d2

…ing calculated wrong

jwhite3 reviewed Mar 16, 2020

View reviewed changes

Comment thread cosmo/monitors/nuv_dark_monitor.py

jwhite3 reviewed Mar 16, 2020

View reviewed changes

Comment thread cosmo/monitors/nuv_dark_monitor.py

jwhite3 reviewed Mar 16, 2020

View reviewed changes

Comment thread cosmo/monitors/nuv_dark_monitor.py

jwhite3 reviewed Mar 16, 2020

View reviewed changes

	# access data, perform any filtering required for analysis
	"""access data, perform any filtering required for analysis"""

	# calculate dark rate for one exposure, with a dataframe with TIME,
	"""calculate dark rate for one exposure, with a dataframe with TIME, XCORR, and YCORR values"""

+    # Plot stuff
+    x = 'TIME'
+    y= 'DARK_RATE'
+    plot_type = 'scatter'

Conversation

cmagness commented Feb 6, 2020 • edited by jwhite3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Feb 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwhite3 Feb 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwhite3 Feb 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cmagness commented Feb 6, 2020 •

edited by jwhite3

Loading

codecov Bot commented Feb 6, 2020 •

edited

Loading

jwhite3 Feb 11, 2020 •

edited

Loading

jwhite3 Feb 11, 2020 •

edited

Loading