Skip to content

Macro Synteny Paf microservice #1113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
matthewwiese wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Macro Synteny Paf microservice #1113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c7e146d
Add macro-synteny-paf microservice
svengato Jul 8, 2024
1d7d7d0
Fix markdown format of macro-synteny-paf README.md
svengato Jul 9, 2024
74a5bdf
Get data from other microservices instead of direct Redis database calls
svengato Jul 30, 2024
17222f1
Look up chromosome names from genome prefixes
svengato Sep 5, 2024
cdcff28
Add caching, move gene families and block position lookup to macro-sy…
matthewwiese Oct 13, 2025
10af9af
Add option to process with chromosome name instead of gene families
matthewwiese Oct 13, 2025
60777d8
Upgrade grpc
matthewwiese Oct 13, 2025
8e5ddaf
Merge branch 'main' into macro-synteny-paf
matthewwiese Oct 13, 2025
c3ce88c
Fix dependency reversion after merging main
matthewwiese Oct 13, 2025
1395b52
Remove gzip compression but still cache results
matthewwiese Oct 17, 2025
0163480
Fix cache_key scope, add logging
matthewwiese Oct 17, 2025
ece0b91
Bump grpcio to 1.76.0
matthewwiese Oct 21, 2025
e122dc6
Improve performance of string handling
matthewwiese Oct 21, 2025
5f0b354
Remove LRU cache, use f-strings
matthewwiese Oct 21, 2025
e69b502
Add json data return format
matthewwiese Oct 24, 2025
b62e654
Adjust Redis Pipeline to improve performance of _getTargets()
matthewwiese Oct 30, 2025
a0aa053
Reimplement second Pipeline to count genes, keep batching approach
matthewwiese Oct 31, 2025
dde3eba
Use both terminal genes instead of just block.i
matthewwiese Nov 4, 2025
8e651d3
Handle both dict and grpc
matthewwiese Nov 14, 2025
697c01a
Add testing MVP
matthewwiese Nov 14, 2025
d8f58d5
Aggregate results to improve performance by reducing data transfer
matthewwiese Nov 21, 2025
4166773
Implement identity metric for use in JBrowse Dynamic Synteny plugin
matthewwiese Jan 14, 2026
4b8bb96
Implement 'regular' anchors by returning full DAG path nodes instead …
matthewwiese Jan 21, 2026
471201e
Update loader to use pyranges1 instead of gffutils
matthewwiese Jan 28, 2026
4e5a86b
Skip/filter self identity synteny when using regular anchors
matthewwiese Feb 12, 2026
8129497
Update pkg_resources to importlib.resources for Python 3.13+, bump gr…
matthewwiese Feb 12, 2026
6c4b3f6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 6, 2026
bfb0353
Fix pyranges1 import
matthewwiese Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Testing
In each microservice's directory, run:

```
./run-tests.sh
```

It expects [gcv-docker-compose]() to be the parent directory. Wherever that repo is on your machine:

```
GCV_DOCKER_COMPOSE=/home/mrbean/github/gcv-docker-compose ./run-tests.sh
```

File structure overview:

```
<some-service>/
├── tests/
│ ├── test_<some-class>.py
│ └── conftest.py # Fixtures
├── pytest.ini # Configuration
├── requirements-test.txt # Test dependencies
└── Dockerfile.test # Docker image for running tests
```
6 changes: 2 additions & 4 deletions chromosome/chromosome/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

import os
import sys
from importlib.resources import files as resource_files

import pkg_resources
import setuptools


Expand Down Expand Up @@ -61,9 +61,7 @@ def build_package_protos(self):
if filename.endswith(".proto"):
proto_files.append(os.path.abspath(os.path.join(root, filename)))

well_known_protos_include = pkg_resources.resource_filename(
"grpc_tools", "_proto"
)
well_known_protos_include = str(resource_files("grpc_tools").joinpath("_proto"))

for proto_file in proto_files:
command = [
Expand Down
22 changes: 12 additions & 10 deletions chromosome/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,43 +6,45 @@
#
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.12.15
aiohttp==3.13.0
# via
# aiohttp-cors
# chromosome (setup.py)
aiohttp-cors==0.8.1
# via chromosome (setup.py)
aiosignal==1.4.0
# via aiohttp
attrs==25.3.0
attrs==25.4.0
# via aiohttp
frozenlist==1.7.0
frozenlist==1.8.0
# via
# aiohttp
# aiosignal
grpcio==1.74.0
grpcio>=1.78.0
# via
# chromosome (setup.py)
# grpcio-tools
grpcio-tools==1.74.0
grpcio-tools>=1.78.0
# via chromosome (setup.py)
idna==3.10
idna==3.11
# via yarl
multidict==6.6.4
multidict==6.7.0
# via
# aiohttp
# yarl
propcache==0.3.2
propcache==0.4.1
# via
# aiohttp
# yarl
protobuf==6.32.0
protobuf==6.32.1
# via grpcio-tools
redis==6.4.0
# via chromosome (setup.py)
typing-extensions==4.15.0
# via grpcio
uvloop==0.21.0
# via chromosome (setup.py)
yarl==1.20.1
yarl==1.22.0
# via aiohttp

# The following packages are considered to be unsafe in a requirements file:
Expand Down
6 changes: 2 additions & 4 deletions chromosome_region/chromosome_region/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

import os
import sys
from importlib.resources import files as resource_files

import pkg_resources
import setuptools


Expand Down Expand Up @@ -61,9 +61,7 @@ def build_package_protos(self):
if filename.endswith(".proto"):
proto_files.append(os.path.abspath(os.path.join(root, filename)))

well_known_protos_include = pkg_resources.resource_filename(
"grpc_tools", "_proto"
)
well_known_protos_include = str(resource_files("grpc_tools").joinpath("_proto"))

for proto_file in proto_files:
command = [
Expand Down
22 changes: 12 additions & 10 deletions chromosome_region/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,43 +6,45 @@
#
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.12.15
aiohttp==3.13.0
# via
# aiohttp-cors
# chromosome_region (setup.py)
aiohttp-cors==0.8.1
# via chromosome_region (setup.py)
aiosignal==1.4.0
# via aiohttp
attrs==25.3.0
attrs==25.4.0
# via aiohttp
frozenlist==1.7.0
frozenlist==1.8.0
# via
# aiohttp
# aiosignal
grpcio==1.74.0
grpcio>=1.78.0
# via
# chromosome_region (setup.py)
# grpcio-tools
grpcio-tools==1.74.0
grpcio-tools>=1.78.0
# via chromosome_region (setup.py)
idna==3.10
idna==3.11
# via yarl
multidict==6.6.4
multidict==6.7.0
# via
# aiohttp
# yarl
propcache==0.3.2
propcache==0.4.1
# via
# aiohttp
# yarl
protobuf==6.32.0
protobuf==6.32.1
# via grpcio-tools
redis==6.4.0
# via chromosome_region (setup.py)
typing-extensions==4.15.0
# via grpcio
uvloop==0.21.0
# via chromosome_region (setup.py)
yarl==1.20.1
yarl==1.22.0
# via aiohttp

# The following packages are considered to be unsafe in a requirements file:
Expand Down
6 changes: 2 additions & 4 deletions chromosome_search/chromosome_search/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

import os
import sys
from importlib.resources import files as resource_files

import pkg_resources
import setuptools


Expand Down Expand Up @@ -61,9 +61,7 @@ def build_package_protos(self):
if filename.endswith(".proto"):
proto_files.append(os.path.abspath(os.path.join(root, filename)))

well_known_protos_include = pkg_resources.resource_filename(
"grpc_tools", "_proto"
)
well_known_protos_include = str(resource_files("grpc_tools").joinpath("_proto"))

for proto_file in proto_files:
command = [
Expand Down
22 changes: 12 additions & 10 deletions chromosome_search/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,43 +6,45 @@
#
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.12.15
aiohttp==3.13.0
# via
# aiohttp-cors
# chromosome_search (setup.py)
aiohttp-cors==0.8.1
# via chromosome_search (setup.py)
aiosignal==1.4.0
# via aiohttp
attrs==25.3.0
attrs==25.4.0
# via aiohttp
frozenlist==1.7.0
frozenlist==1.8.0
# via
# aiohttp
# aiosignal
grpcio==1.74.0
grpcio>=1.78.0
# via
# chromosome_search (setup.py)
# grpcio-tools
grpcio-tools==1.74.0
grpcio-tools>=1.78.0
# via chromosome_search (setup.py)
idna==3.10
idna==3.11
# via yarl
multidict==6.6.4
multidict==6.7.0
# via
# aiohttp
# yarl
propcache==0.3.2
propcache==0.4.1
# via
# aiohttp
# yarl
protobuf==6.32.0
protobuf==6.32.1
# via grpcio-tools
redis==6.4.0
# via chromosome_search (setup.py)
typing-extensions==4.15.0
# via grpcio
uvloop==0.21.0
# via chromosome_search (setup.py)
yarl==1.20.1
yarl==1.22.0
# via aiohttp

# The following packages are considered to be unsafe in a requirements file:
Expand Down
6 changes: 2 additions & 4 deletions gene_search/gene_search/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

import os
import sys
from importlib.resources import files as resource_files

import pkg_resources
import setuptools


Expand Down Expand Up @@ -61,9 +61,7 @@ def build_package_protos(self):
if filename.endswith(".proto"):
proto_files.append(os.path.abspath(os.path.join(root, filename)))

well_known_protos_include = pkg_resources.resource_filename(
"grpc_tools", "_proto"
)
well_known_protos_include = str(resource_files("grpc_tools").joinpath("_proto"))

for proto_file in proto_files:
command = [
Expand Down
22 changes: 12 additions & 10 deletions gene_search/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,43 +6,45 @@
#
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.12.15
aiohttp==3.13.0
# via
# aiohttp-cors
# gene_search (setup.py)
aiohttp-cors==0.8.1
# via gene_search (setup.py)
aiosignal==1.4.0
# via aiohttp
attrs==25.3.0
attrs==25.4.0
# via aiohttp
frozenlist==1.7.0
frozenlist==1.8.0
# via
# aiohttp
# aiosignal
grpcio==1.74.0
grpcio>=1.78.0
# via
# gene_search (setup.py)
# grpcio-tools
grpcio-tools==1.74.0
grpcio-tools>=1.78.0
# via gene_search (setup.py)
idna==3.10
idna==3.11
# via yarl
multidict==6.6.4
multidict==6.7.0
# via
# aiohttp
# yarl
propcache==0.3.2
propcache==0.4.1
# via
# aiohttp
# yarl
protobuf==6.32.0
protobuf==6.32.1
# via grpcio-tools
redis==6.4.0
# via gene_search (setup.py)
typing-extensions==4.15.0
# via grpcio
uvloop==0.21.0
# via gene_search (setup.py)
yarl==1.20.1
yarl==1.22.0
# via aiohttp

# The following packages are considered to be unsafe in a requirements file:
Expand Down
6 changes: 2 additions & 4 deletions genes/genes/commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

import os
import sys
from importlib.resources import files as resource_files

import pkg_resources
import setuptools


Expand Down Expand Up @@ -61,9 +61,7 @@ def build_package_protos(self):
if filename.endswith(".proto"):
proto_files.append(os.path.abspath(os.path.join(root, filename)))

well_known_protos_include = pkg_resources.resource_filename(
"grpc_tools", "_proto"
)
well_known_protos_include = str(resource_files("grpc_tools").joinpath("_proto"))

for proto_file in proto_files:
command = [
Expand Down
Loading