Backend optimizations for Neo4j

[pdonorio]

We are trying to get the most tight integration with graphdb as possible in the basic blueprint of this framework. We are using neomodel python library to map Graph nodes&relationships to Python Classes.

Also great news, neomodel will support the latest neo4j 3.0 native python driver, see the issue i opened here. Until then we will stick with version 2.3.3.

Models should be made customizable, for the moment the base version is in some kind of limbo, between backend and vanilla code.

On the backend side some things should be done to verify what is the best way to handle connections and models injections. I expect this to be on @mdantonio shoulders, and @dpaoletti could help to get some understanding of the graphdb as well.

[mdantonio]

neomodel will support the latest neo4j 3.0 native python driver

Very good news... we expect a huge performance boost

Speaking about neo4j optimizations I start by linking this little guide to tune neo4j: performance tuning

[pdonorio]

this little guide to tune neo4j

It seems like the ulimit is a problem here too 😨

[pdonorio]

Neo4j 3 is available with neomodel.
How can we test it?

[pdonorio]

How can we test it?

We just need as root inside the backend container:

pip install --upgrade git+git://github.com/robinedwards/neomodel.git@HEAD#egg=neomodel-dev

and change the main variable for connection into

os.environ['NEO4J_BOLT_URL'] = "bolt://neo4j:chooseapassword@gdb"
# note: no port and the rest

[mdantonio]

To upgrade from neo4j 2.x to 3.0:

https://neo4j.com/guides/upgrade/

To upgrade using the docker image:

http://neo4j.com/docs/operations-manual/current/deployment/single-instance/docker/

• edit docker compose file to upgrade neo4j version
• in docker compose add the env var: NEO4J_dbms_allowFormatMigration: true
• start the server and wait for the graph conversion
• remove the env var
• restart again

Now i'm unable to test it because:

The Compose file ... is invalid because: NEO4J_dbms_allowFormatMigration contains true, which is an invalid type, it should be a string, number, or null

with: docker-compose version 1.8.1, build 878cff1

with NEO4J_dbms_allowFormatMigration: 'true' is ignored by neo4j
with NEO4J_dbms_allowFormatMigration: 1 gives an error

Any idea?

[mdantonio]

with NEO4J_dbms_allowFormatMigration: 'true' is ignored by neo4j

I found the real probelm.

The migration flag is not ignored, is the old graphdb that is not recognized.
In the old data folder we have a three sub folders: dbms, graph.db and log
After the startup of neo4j3 a fourth folder is created: database with a graph.db created into
I created a database folder and moved the graph.db
Now the old graph is recognized

What is really strange is that the migration process not start at all. But the graph works. The migration is not required? And why? I cannot find any guide saying "no migration is required from 2.3.3 to 3.x" Everywhere "the migration is required from 2.x to 3.x"

[mdantonio]

I found a problem with the id property.

In the previous neomodel version the internal node id was accessed as node._id
So we added to models an id String property containing an uuid

Now the internal id is accessed as node.id and we have a collision

We have to rename all id properties to uid (both in models and in base.py: getJsonResponse starting at line 508)

[mdantonio]

An other difference is found in the docker image env variables

NEO4J_CACHE_MEMORY became NEO4J_dbms_memory_pagecache_size
NEO4J_HEAP_MEMORY became NEO4J_dbms_memory_heap_maxSize

NEO4J_AUTH does not change

[mdantonio]

At the moment i'm unable to perform more tests

So the final migration guide is:

• change docker compose to upgrade neo4j version to 3.0.7
• change docker compose to fix CACHE and HEAP env var
• create a databases folder into the datadir, move the graph.db folder into databases
• in graph.py remove s.environ["NEO4J_REST_URL"] = bla bla
• in graph.py add os.environ["NEO4J_BOLT_URL"] = "bolt://%s:%s@%s" % (USER, PW, HOST)
• in vanilla/neo4j.py change models, rename id properties to uid
• in base.py change getJsonResponse to use uid property
• convert all nodes to use uid instead of id MATCH (n) SET n.uid = id
• start the whole stack in debug mode
• enter backend container and install neomodel 3 pip install --upgrade git+git://github.com/robinedwards/neomodel.git@9672e285a6ba4ca8bc926dae7586f7d88ef3d9d2
• start backend and frontend and everything should work

[mdantonio]

Created a branch in mdantonio/http-api-base

New migration guide

Migrate DB:

• create a databases folder into the datadir
• move the graph.db folder into databases
• convert all nodes to use uid instead of id MATCH (n) WHERE not exists(n.uuid) and exists(n.id) SET n.uuid = n.id

Migrate source code

• in backend: git checkout neo4j3
• in vanilla/neo4j.py change models, rename id properties to uuid
• start the whole stack in debug mode
• enter backend container and install neomodel 3 pip install --upgrade git+git://github.com/robinedwards/neomodel.git@9672e285a6ba4ca8bc926dae7586f7d88ef3d9d2
• start backend and frontend and everything should work

[mdantonio]

Install last neomodel tag:

pip install --upgrade git+git://github.com/robinedwards/neomodel.git@3.0.3

To connect from python add in graph.py:

from neomodel import config
config.DATABASE_URL = "bolt://%s:%s@%s" % (USER, PW, HOST)