Unipop Elasticsearch plugin working only on 10000 docuements

[gaurav6041]

I have an index where one field has has a value of department in which the employee is present in.
Below is present Unipop elasticsearch configurations

{
  "class": "org.unipop.elastic.ElasticSourceProvider",
  "addresses": "http://192.168.10.121:6968",
  "vertices": [
    {
      "index": "employee_records",
      "id": "@department",
      "label": "hastags",
      "properties": {
        "value": "@department"
      }
    },
    {
      "index": "employee_records",
      "id": "@unique",
      "label": "unique",
      "properties": {
        "value": "@unique"
      }
    }
  ],
  "edges": [
    {
      "index": "employee_records",
      "id": "@_id",
      "label": "connects",
      "properties": {},
      "outVertex": {
        "ref": true,
        "id": "@unique",
        "label": "unique"
      },
      "inVertex": {
        "ref": true,
        "id": "@department",
        "label": "hastags"
      }
    }
  ]
}

And below are results
{v[Tech]=3159, v[Analysis]=3726, v[Admin]=2111, v[Support]=1003}

And Here is my gremlin query => g.V().outE('connects').inV().groupCount()
There are total of 65000 employee records. But the above result totals to 9999
In my above configurations i have mapped department of employees with employee id.
So count of all the departments should total out to 9999 whereas it should be 65000.

This means that unipop is picking only 10000 documents from elasticsearch.
This might me due to limit of elasticsearch of fetching 10000 documents in one request and scroll api is not been used.

[randanman]

@babiy8 @eyalmar100 didn't you recently add support for the scroll API?

[gaurav6041]

The problem is still there of only 10000 documents being fetched from elasticsearch. Can anyone help?

[randanman]

@HenShalom is this bug happening to you as well?

[eyalmar100]

Hi
My name is Eyal, and I'm one of the developers of Unipop.
Do you want to send me all the necessary details of your project to my email ?
I will try to reconstruct the issue and see why its not working for you.
My email is : eyalmar100@gmail.com

[eyalmar100]

This is a long shot, but in the DocumentController class there is field :
.. private final int maxLimit = 10000;
try to change it and see what happens ..

[seanbarzilay]

The maxLimit field is the maximum number of documents elasticsearch can fetch and to my knowledge we haven't implemented the scroll API yet.

…

On Wed, Sep 5, 2018, 5:26 PM eyalmar100 ***@***.***> wrote: This is a long shot, but in the DocumentController class there is field : .. private final int maxLimit = 10000; try to change it and see what happens .. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#135 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK8elAx95dWAWUnERupyTdH7OvdgItmyks5uX98SgaJpZM4WPWlq> .

[randanman]

@eyalmar100 thats not a good idea, and not a scalable solution.
I know for a fact that you worked on solving this, because I sat with you and @babiy8 to solve the scroll API issues. What happened with that? did you push the needed fixes?

[eyalmar100]

I didn't know why/what this field refers to .. now I know after @seanbarzilay answered .
Sometime its good to have some remarks/documentation ..
Anyway, we started to implement the scroll API , but did not finish it yet ( the scroll API is based on 'polling' mechanism) , and we have to think how to interact/ cooperate with the 'client' to make it work right.
We will finish it soon.

[seanbarzilay]

@eyalmar100 If you want we can talk, before I left I had an idea on how to implement the scroll API.

[eyalmar100]

@seanbarzilay - Yes . I will be happy to talk :)
my mail is : eyalmar100@gmail.com
you can send me a message with your contacts information ( phone , etc .)
Thanks

[randanman]

@eyalmar100 @babiy8 thats very unfortunate. I believe we wrote most of the code when I sat with you. You guys said you'll finish it and push it. What happened?

[gaurav6041]

Hi All
Can you tell me till when the fixes would be done.
I wish to explore it further and do performance testing on large datasets

[eyalmar100]

Hן
I'm exploring it , and hope it will be fixed during next week .