Replies: 20 comments 3 replies
-
|
I think you're running into #97 Suggest trying the unreleased main branch and then: |
Beta Was this translation helpful? Give feedback.
-
|
@tae898 would love to see numbers before and after |
Beta Was this translation helpful? Give feedback.
-
|
This was a 4M node, 35M edges graph. Or roughly half the size of your graph. Imported in < 10 secs. |
Beta Was this translation helpful? Give feedback.
-
|
Do i have to run the main branch or is it already pushed to pypi? |
Beta Was this translation helpful? Give feedback.
-
|
They get pushed on a nightly basis: https://test.pypi.org/project/ladybug/#history Once #487 lands, it should appear in the next nightly build. |
Beta Was this translation helpful? Give feedback.
-
Does it only import slowly on tag v0.16.1? If the tag v0.15 is faster? |
Beta Was this translation helpful? Give feedback.
-
try the latest release 0.17.0 |
Beta Was this translation helpful? Give feedback.
-
Here is my scenario: CREATE NODE TABLE IF NOT EXISTS Assignment (id STRING,fullScore DOUBLE,deadline STRING,assignmentId INT64,name STRING,description STRING,title STRING,publishTime STRING, PRIMARY KEY(id)); CREATE NODE TABLE IF NOT EXISTS Section (id STRING,year INT64,semesterType STRING,enrolledCount INT64,name STRING,classTime STRING,classroom STRING,capacity INT64,sectionId INT64,semester STRING, PRIMARY KEY(id)); CREATE REL TABLE IF NOT EXISTS Assignment_belongsToSection_Section(FROMAssignmentTOSection, MANY_MANY); Test Data & Scenarios merge test: I would like to inquire about feasible optimizations and tuning solutions. Potential optimization directions include table schema adjustment, Ladybug configuration tuning and Cypher statement optimization. Since duplicate data exists in the dataset, we are required to use either MERGE or UNWIND + MERGE syntax for data insertion. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Another short term possibility: ingest to duckdb, dedup, export parquet and ingest that. |
Beta Was this translation helpful? Give feedback.
-
I used Ladybug's 'copy from' method to write data. The total data volume is around 2 billion points and 6 billion points, simulating a customer usage scenario of 10 million points per day. Among these, there are 2.5 million points and 7.5 million edges related to the points. Data is written daily, with each batch containing 100,000 records. I'm using the Python API with max_db_size set to 8T. During continuous writing, when I've written 500 million points and 1.5 billion edges, my memory usage has already reached 80GB. Even though I call db.close after writing each day's 10 million records, the memory keeps rising, and write performance drops significantly. Do you have any best practice suggestions? How can I better use Ladybug, and what strategies are there to handle memory spikes? Note that my data isn't copied all at once but is copied daily according to the business scenario, 10 million records each day. 
|
Beta Was this translation helpful? Give feedback.
-
|
Did you remember to disable hash index? Can you share
From the largest import that succeeded? |
Beta Was this translation helpful? Give feedback.
-
I remember you mentioned disabling the hash index, but if I turn it off, how can I still use the hash index for fast queries? Does this disable just turn it off without checking, or does it mean the hash index isn't created at all? in this PR 487 CALL enable_default_hash_index=false; Because in my scenario I'm reading while writing, and not just writing everything and then reading the data, if I disable before creating the table, can the hash index be recreated? In the method like above, what command do I need to send to re-enable the hash index? Or, can I skip this operation, and the table will still use the hash index by default when querying? Does it rebuild the hash index in memory on the first query? Would that be more time-consuming? Kind of like how icebug-disk works? For ladybug, after mounting icebug-disk, the first startup needs to rebuild the hash index, so queries take longer. Am I understanding this correctly? |
Beta Was this translation helpful? Give feedback.
-
|
Hash index has a number of problems: a) It's space inefficient. See https://gist.github.com/adsharma/c2da6591fb0888c1ff3de11c08db63a6 Nothing is going to auto create hash indexes. Monitor query performance to see if indexes are needed in the first place. |
Beta Was this translation helpful? Give feedback.
-
|
The reason you're seeing 80GB of memory is likely because you're using a giant transaction which has all the rows. Try something like: |
Beta Was this translation helpful? Give feedback.
-
actually I use it like this: `import ladybug as lb db = lb.Database("mydb.lbug") init_str = """ conn.execute(init_str) loop cycle: { ` |
Beta Was this translation helpful? Give feedback.
-
Yes. You won't have an index on node tables created from that point on. And no need to run
There's a smaller ART index you can create if you really think query performance is bad. Indexes updated docs -> https://docs.ladybugdb.com/cypher/indexes/
Yes. Take a look at https://docs.ladybugdb.com/import/csv/ Btw how large are the CSV files?. Also this |
Beta Was this translation helpful? Give feedback.
-
|
Some generic advice from Gemini Indexing Strategies for Columnar Tables with Integer KeysIn a columnar table, a traditional hash index is rarely optimal. You should instead rely on the storage engine's native sort keys, zone maps, or compression algorithms for maximum performance. Columnar databases (like ClickHouse, Snowflake, or Amazon Redshift) process data in blocks of columns rather than row-by-row, which fundamentally changes how indexes operate. [1, 2] Why Hash Indexes Struggle on Columnar Integer Keys
Recommended Alternatives for Columnar Integer KeysBecause columnar tables handle data differently, you should use architectural indexing rather than secondary index structures:
Summary of the Best ApproachIf your integer key is the primary identifier used to filter queries, you should set it as the Sort Key or Primary Key during table creation. This forces physical ordering on disk, giving you the near-instant point-lookup speed of a hash index while preserving the blistering range-scan capabilities native to columnar architecture. [5] |
Beta Was this translation helpful? Give feedback.
-
|
If you look at how postgres optimizes the bulk COPY operation:
We are different of course because of the columnar table org and CSR based REL tables. But many things are common. Note that ART indexes are optimized with respect to WAL (they don't write every index mutation to WAL), but not optimized with respect to amount of memory used. You'll need a large (64GB) machine to create an index on a "name" column on billion rows. Memory size scales linearly with table size. |
Beta Was this translation helpful? Give feedback.
-
The total number of rows in the overall CSV data is 2 billion points and 10 billion edges, similar to LDBC's SF1000. But the node CSV isn't a single file, and the rel CSV isn't a single file either. The data will be split into batches of 100k per CSV file, so there will be a lot of CSV files. |
Beta Was this translation helpful? Give feedback.
-
|
So you have two concerns here:
Concrete suggestions: Memory usage:
Too. large = GBs of memory Write performance:
I think you've got plenty of suggestions here to work on. |
Beta Was this translation helpful? Give feedback.
-
|
I thought disable hash index is not suite my test case , the table already has data. I can not import next csv data. 
|
Beta Was this translation helpful? Give feedback.
-
|
At the same time, I noticed another issue when using copy from. Since Ladybug's BufferManager uses MMAP to manage memory, during writing the following can happen: '8T mmap -> occupies virtual address space; So if I keep writing large amounts of data for a long time without doing a db.close, memory keeps increasing and cannot be unmapped. That’s why I periodically do db.close to release mmap frames. Do you have any good suggestions regarding this issue? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Ladybug version
v0.16.1
What operating system are you using?
Ubuntu 22.04
What happened?
Summary
Large bulk ingest is still very slow for our workload even when using bulk
COPY-style loading through the Python API, rather than single-row transactional inserts.I know there is already related context in:
But this report is specifically about large-scale bulk load performance, not per-row or non-batched insert overhead.
Environment
0.16.13.12.13Linux x86_64What we are doing
This is not single-row insert mode.
We:
COPY ... FROM.COPY ... FROM.In our harness, node tables are copied directly from CSV, and relationship CSVs are rewritten into chunks and then loaded via
COPY.Dataset shape
Synthetic property graph:
Observed behavior
Large ingest takes roughly 19 to 20 hours before analytical queries even begin.
Representative ingest times from repeated runs:
68,619,567 ms(~19.06 h)69,380,471 ms(~19.27 h)72,656,042 ms(~20.18 h)So in practice, just preparing the large dataset is already prohibitively expensive for benchmarking and evaluation workflows.
Why this seems distinct from #302
Issue #302 discusses poor performance for non-batched / transactional single-row inserts.
Our case is different:
That makes it feel like there may still be a separate bottleneck in the current bulk-load path at larger scales.
Minimal reproduction shape
The exact harness is custom, but the important part is:
If useful, I can provide a more minimal standalone repro script that just focuses on schema creation + CSV
COPYload without the rest of the benchmark machinery.Expected behavior
Bulk ingest should be substantially faster for this scale, or at least there should be a documented fast-ingest path that makes this practical.
Actual behavior
Large bulk ingest completes, but takes around 19 to 20 hours, which is too slow for practical large-scale benchmark preparation.
Additional note
I am intentionally keeping this issue focused on ingest time only.
In separate large runs, we also saw later query-execution instability / process kills during heavy OLAP traversal queries, but I do not want to mix that into this report unless you think the two are likely connected.
Are there known steps to reproduce?
No response
Beta Was this translation helpful? Give feedback.
All reactions