Skip to content

Accelerate write - #46

Merged
xumi1993 merged 5 commits into
develfrom
devel_jc
Aug 3, 2026
Merged

xumi1993 merged 5 commits into
develfrom
devel_jc

Conversation

@JingChen-Thu

Copy link
Copy Markdown
Contributor

Accelerate sr.write:

1m20s -> 1.6 s

by preparing text info before loop.

Frequently using 'itertuples' is computationally expensive.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes SrcRec.write() in pytomoatt by avoiding per-source DataFrame filtering/itertuples() over receiver data, aiming to significantly reduce runtime when writing large src_rec files.

Changes:

  • Precomputes receiver/common-source/common-receiver output lines grouped by src_index to avoid repeated DataFrame filtering inside the source loop.
  • Switches source iteration to itertuples(name=None) and writes prebuilt receiver blocks via dictionary lookups.
Suppressed comments (2)

pytomoatt/src_rec.py:532

  • Common-source receiver record formatting reduces weight to 3 decimals (%7.3f), which changes persisted values compared to the previous 4-decimal formatting. If weights are meant to be preserved across write→read, keep the previous precision.
                    "%7d %7d %6s %9.4f %9.4f %9.4f %7d %6s %9.4f %9.4f %9.4f %s %8.4f %7.3f\n" % row

pytomoatt/src_rec.py:559

  • Common-receiver receiver record formatting reduces weight to 3 decimals (%7.3f), which changes persisted values compared to the previous 4-decimal formatting. If weights are meant to be preserved across write→read, keep the previous precision.
                    "%7d %7d %6s %9.4f %9.4f %9.4f %7d %6s %9.4f %9.4f %9.4f %s %8.4f %7.3f\n" % row

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pytomoatt/src_rec.py Outdated
+ origin_time.nanosecond * 1e-9
)
output.write("{:d} {} {} {} {} {} {} {:.4f} {:.4f} {:.4f} {:.4f} {} {} {:.4f}\n".format(
output.write("%7d %6d %2d %2d %2d %2d %5.2f %9.4f %9.4f %9.4f %5.2f %7d %s %7.3f\n" % (
Comment thread pytomoatt/src_rec.py Outdated
Comment on lines 566 to 570
for src in tqdm.tqdm(
src_points.itertuples(),
src_points.itertuples(name=None),
total=src_points.shape[0],
desc="Writing src_rec file",
):
Comment thread pytomoatt/src_rec.py Outdated
):
src_index, rec_index, staname, stla, stlo, stel, phase, tt, weight = row
rec_lines_by_src.setdefault(src_index, []).append(
"%7d %7d %6s %9.4f %9.4f %9.4f %s %8.4f %7.3f\n" % (
Comment thread pytomoatt/src_rec.py Outdated
+ origin_time.nanosecond * 1e-9
)
output.write("{:d} {} {} {} {} {} {} {:.4f} {:.4f} {:.4f} {:.4f} {} {} {:.4f}\n".format(
output.write("%7d %6d %2d %2d %2d %2d %5.2f %9.4f %9.4f %9.4f %5.2f %7d %s %7.3f\n" % (
@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@xumi1993

xumi1993 commented Aug 3, 2026

Copy link
Copy Markdown
Member

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

Copilot AI commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the...

All issues from the review have been addressed in commit $(git rev-parse --short HEAD):

  1. Weight precision (lines 495, 532, 559): Restored %7.4f (4 decimal places) for receiver weight fields.
  2. Source header format (line 585): Restored the strftime("%Y_%m_%d_%H_%M_%S.%f") approach to preserve full microsecond precision for second, and {:.4f} for mag and weight.
  3. Column order dependency (lines 566–570): Replaced positional src[1], src[2], … indexing with an explicit column list (_src_cols) passed to itertuples, so the mapping is stable regardless of DataFrame column reordering.
  4. Regression test: Added test_write_roundtrip_preserves_microsecond_and_weight_precision to assert that write→read roundtrips preserve microsecond origin-time precision and 4-decimal weight precision for both source and receiver records.

@xumi1993
xumi1993 merged commit a24d38b into devel Aug 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants