call_dots confounded by the duplicates along the tile borders

[sergpolly]

More weird stuff:

cooler output ...
Traceback (most recent call last):
  File "/home/venevs/miniconda3/envs/dots-latest/bin/cooltools", line 11, in <module>
    sys.exit(cli())
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooltools/cli/call_dots.py", line 246, in call_dots
    nproc, score_dump_mode, verbose)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooltools/dotfinder.py", line 1150, in scoring_step
    mode = 'w')
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooler/create/_create.py", line 921, in create_cooler
    max_merge=max_merge)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooler/create/_create.py", line 632, in create_from_unordered
    columns=columns, dtypes=dtypes, mode='a', **kwargs)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooler/create/_create.py", line 565, in create
    file_path, target, meta.columns, iterable, h5opts, lock)
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooler/create/_create.py", line 204, in write_pixels
    for i, chunk in enumerate(iterable):
  File "/home/venevs/miniconda3/envs/dots-latest/lib/python3.6/site-packages/cooler/create/_ingest.py", line 307, in _validate_pixels
    raise BadInputError("Found duplicate pixels:\n{}".format(err.head().to_csv(sep='\t')))
cooler.create._ingest.BadInputError: Found duplicate pixels:
	bin1_id	bin2_id	count	la_exp.donut.value	la_exp.vertical.value	la_exp.horizontal.value	la_exp.lowleft.value
75	76	999	0	0.039334712950361124	0.02985299926634868	0.18436449063546803	0.0679983868254756
76	76	1000	0	0.08116557025783917	0.0	0.1444649071338037	0.07124153316898331
152	77	999	0	0.05743539640390035	0.03396130943199788	0.17071727978971427	0.0779715844835965
153	77	1000	0	0.09102873515462423	0.0	0.11910133333345246	0.08039903033544311
230	78	999	0	0.06473282768473614	0.018054547860104672	0.0799101745120326	0.0745334029293596

bash latest_pipe_dots.sh ESC-R1 U54-H1ESC4DN-FA-DSG-MNase_hg19 8  688.31s user 114.51s system 999% cpu 1:20.28 total

Originally posted by @sergpolly in #51 (comment)

[sergpolly]

investigation into efficient intermediate scores dump #51 (comment) lead to the discovery of the duplicated scores pixels reports - i.e. not careful enough specification if the tile boundaries ...

Here is an illustration of the geometric place of the duplicated pixels for a slice of the chr1:

I thought I carefully avoided those in the beginning and tested for that - apparently it was not enough.
all of the current call_dots applications might be affected - cause having duplicates skews the multiple hypothesis testing - not by much but anyways

[sergpolly]

likely it's the number of NaNs allowed ...
when you allow too many - pixels scored on the border end up on the list:

[golobor]

@sergpolly - may be move to discussions?