ReportTypes unhandled error

[ctalbotSFDC]

Mine dies with this message:

2020-06-08T14:59:57.285Z [mdata-indexer] info: [00D0r0000008cvQEAQ] Completed indexing PublicGroups
mdata-worker_1 | logFileName:logs/reporttypes.log
mdata-worker_1 | 2020-06-08T14:59:57.985Z [reporttypes] debug: [00D0r0000008cvQEAQ] Fetched report type categories 13
mdata-worker_1 | (node:1) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'length' of undefined
mdata-worker_1 | at ReportTypes.createReportTypes (/app/metadata/ReportTypes.js:46:39)
mdata-worker_1 | at runMicrotasks ()
mdata-worker_1 | at processTicksAndRejections (internal/process/task_queues.js:97:5)
mdata-worker_1 | at async ReportTypes.process (/app/metadata/ReportTypes.js:101:9)
mdata-worker_1 | at async MdataIndexer.startIndexing (/app/MdataIndexer.js:111:17)
mdata-worker_1 | at async handleMdataIndexRequest (/app/mdata-worker.js:45:5)
mdata-worker_1 | at async Command.callback (/app/mdata-worker.js:27:13)
mdata-worker_1 | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
mdata-worker_1 | (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[ctalbotSFDC]

Fixed by updating the reporttypes.js file to have an if(results.records) { } around the for loop on line 46

[rtanikella]

@ctalbotSFDC Please query your resulting graph DB to confirm: MATCH (o:CustomObject)--(ReportType) RETURN count(l); If it returns a value greater than 1 then I'm I would be interested in knowing how that failed when you tried my suggestion to replace results.records with just results in that loop.

I just pulled the latest from Anand's repo, but I haven't checked to see if anything has changed there that might override my thinking on this, but if I'm understanding the code correctly (chime in @anandbn, please) this loop is matching plural names of custom objects to their singular names, which is later used to create the edge between the record type node and the custom object node. My guess is that skipping that loop because result.records was null or empty will not cause any run-time failure because that plural-to-singular map is intact, but with only a single entry. But as a result an interesting portion of your graph is missing.

The underlying code in the stack is SFConnectionUtils.js queryAll(), which in turn calls the JSforce query() and queryMore(). These return an object that has a records array in addition to other items. But the SFConnectionUtils.js queryAll() returns only that array. ReportType.js is written as if it is consuming the entire object returned from JSforce's query*(), which is what I thought the root cause was.

[ctalbotSFDC]

here's the error I get without my if statement & using your diifferent for variable:

mdata-worker_1  | 2020-06-08T17:33:04.677Z [mdata-indexer] info: [00D0r0000008cvQEAQ] Completed indexing PublicGroups
mdata-worker_1  | logFileName:logs/reporttypes.log
mdata-worker_1  | 2020-06-08T17:33:05.305Z [reporttypes] debug: [00D0r0000008cvQEAQ] Fetched report type categories 13
mdata-worker_1  | (node:1) UnhandledPromiseRejectionWarning: TypeError: Cannot read property '0' of undefined
mdata-worker_1  |     at ReportTypes.createReportTypes (/app/metadata/ReportTypes.js:48:106)
mdata-worker_1  |     at runMicrotasks (<anonymous>)
mdata-worker_1  |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
mdata-worker_1  |     at async ReportTypes.process (/app/metadata/ReportTypes.js:103:9)
mdata-worker_1  |     at async MdataIndexer.startIndexing (/app/MdataIndexer.js:111:17)
mdata-worker_1  |     at async handleMdataIndexRequest (/app/mdata-worker.js:45:5)
mdata-worker_1  |     at async Command.callback (/app/mdata-worker.js:27:13)
mdata-worker_1  | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
mdata-worker_1  | (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I do get reporttypes 320 when I put my if statement around -- however, I get weird other messages throughout which say object is null & can't find corresponding items:

mdata-worker_1  | 2020-06-08T17:41:14.214Z [reporttypes] debug: [00D0r0000008cvQEAQ] Contacts with Assets - Could not find object API name for :Contacts
mdata-worker_1  | 2020-06-08T17:41:14.214Z [reporttypes] debug: [00D0r0000008cvQEAQ] Contacts with Assets - Could not find object API name for :Assets
mdata-worker_1  | 2020-06-08T17:41:14.215Z [neo4jutils] debug: Executing query :MATCH (n:ReportType {name:$findByKey}) return n with params:
mdata-worker_1  | {
mdata-worker_1  |     "findByKey": "Contacts with Assets"
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T17:41:14.218Z [neo4jutils] debug: Found existing ReportType for name=Contacts with Assets
mdata-worker_1  | 2020-06-08T17:41:14.218Z [neo4jutils] debug: Executing query :MATCH (n:ReportType {name:$findByKey}) set n=$props return n with params:
mdata-worker_1  | {
mdata-worker_1  |     "findByKey": "Contacts with Assets",
mdata-worker_1  |     "props": {
mdata-worker_1  |         "chart": null,
mdata-worker_1  |         "currency": null,
mdata-worker_1  |         "dashboardSetting": null,
mdata-worker_1  |         "description": null,
mdata-worker_1  |         "developerName": null,
mdata-worker_1  |         "division": null,
mdata-worker_1  |         "folderId": "0053h000000VNnWAAW",
mdata-worker_1  |         "hasDetailRows": true,
mdata-worker_1  |         "hasRecordCount": true,
mdata-worker_1  |         "id": null,
mdata-worker_1  |         "name": "Contacts with Assets",
mdata-worker_1  |         "reportBooleanFilter": null,
mdata-worker_1  |         "reportFormat": "TABULAR",
mdata-worker_1  |         "scope": "my",
mdata-worker_1  |         "showGrandTotal": true,
mdata-worker_1  |         "showSubtotals": true,
mdata-worker_1  |         "standardFilters": null,
mdata-worker_1  |         "supportsRoleHierarchy": false,
mdata-worker_1  |         "userOrHierarchyFilterId": null,
mdata-worker_1  |         "category": "Accounts & Contacts",
mdata-worker_1  |         "type": "ContactAsset",
mdata-worker_1  |         "orgId": "00D0r0000008cvQEAQ"
mdata-worker_1  |     }
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T17:41:14.222Z [reporttypes] debug: [00D0r0000008cvQEAQ] Created 0 nodes
mdata-worker_1  | 2020-06-08T17:41:14.222Z [reporttypes] debug: [00D0r0000008cvQEAQ] Completed 7 of 321
mdata-worker_1  | 2020-06-08T17:41:14.810Z [reporttypes] debug: [00D0r0000008cvQEAQ] Account History - Could not find object API name for :Account History
mdata-worker_1  | 2020-06-08T17:41:14.810Z [neo4jutils] debug: Executing query :MATCH (n:ReportType {name:$findByKey}) return n with params:
mdata-worker_1  | {
mdata-worker_1  |     "findByKey": "Account History"
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T17:41:14.813Z [neo4jutils] debug: Found existing ReportType for name=Account History
mdata-worker_1  | 2020-06-08T17:41:14.813Z [neo4jutils] debug: Executing query :MATCH (n:ReportType {name:$findByKey}) set n=$props return n with params:
mdata-worker_1  | {
mdata-worker_1  |     "findByKey": "Account History",
mdata-worker_1  |     "props": {
mdata-worker_1  |         "chart": null,
mdata-worker_1  |         "currency": null,
mdata-worker_1  |         "dashboardSetting": null,
mdata-worker_1  |         "description": null,
mdata-worker_1  |         "developerName": null,
mdata-worker_1  |         "division": null,
mdata-worker_1  |         "folderId": "0053h000000VNnWAAW",
mdata-worker_1  |         "hasDetailRows": true,
mdata-worker_1  |         "hasRecordCount": true,
mdata-worker_1  |         "id": null,
mdata-worker_1  |         "name": "Account History",
mdata-worker_1  |         "reportBooleanFilter": null,
mdata-worker_1  |         "reportFormat": "TABULAR",
mdata-worker_1  |         "scope": "user",
mdata-worker_1  |         "showGrandTotal": true,
mdata-worker_1  |         "showSubtotals": true,
mdata-worker_1  |         "standardFilters": null,
mdata-worker_1  |         "supportsRoleHierarchy": false,
mdata-worker_1  |         "userOrHierarchyFilterId": null,
mdata-worker_1  |         "category": "Accounts & Contacts",
mdata-worker_1  |         "type": "AccountAuditHistory",
mdata-worker_1  |         "orgId": "00D0r0000008cvQEAQ"
mdata-worker_1  |     }
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T17:41:14.817Z [reporttypes] debug: [00D0r0000008cvQEAQ] Created 0 nodes
mdata-worker_1  | 2020-06-08T17:41:14.817Z [reporttypes] debug: [00D0r0000008cvQEAQ] Completed 8 of 321

and

mdata-worker_1  | 2020-06-08T23:13:46.333Z [reports] debug: [00D0r0000008cvQEAQ] 00O3h000000rTTvEAM - Could not find custom Field: Object: null, Field:OWNER
mdata-worker_1  | 2020-06-08T23:13:46.333Z [neo4jutils] debug: Executing query :match p=allShortestPaths((obj:CustomObject {name:$start})-[:BelongsTo|RefersTo*0..3]-(fld:CustomField {name:$end}))RETURN reduce(theNames =[], n IN nodes(p)| theNames + coalesce(n.name,"")) AS theNames,reduce(theNames =[], n IN nodes(p)| theNames + coalesce(n.Id,"")) AS theIds with params:
mdata-worker_1  | {
mdata-worker_1  |     "start": "Case",
mdata-worker_1  |     "end": "First_Name__c"
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T23:13:46.339Z [reports] debug: [00D0r0000008cvQEAQ] Found Field with Id:00N3h000000XAkVEAW for report column:Case.First_Name__c
mdata-worker_1  | 2020-06-08T23:13:46.340Z [neo4jutils] debug: Executing query :MATCH (f)-[rel:Contains {name:$findByKey}]->(o) return rel with params:
mdata-worker_1  | {
mdata-worker_1  |     "findByKey": "TeleMed - SightCall Sessions Report.Field.Case.First_Name__c"
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T23:13:46.351Z [neo4jutils] debug: Found existing Contains for name=TeleMed - SightCall Sessions Report.Field.Case.First_Name__c
mdata-worker_1  | 2020-06-08T23:13:46.351Z [neo4jutils] debug: Executing query :MATCH (f)-[rel:Contains {name:$findByKey}]->(o) set rel=$props return rel with params:
mdata-worker_1  | {
mdata-worker_1  |     "findByKey": "TeleMed - SightCall Sessions Report.Field.Case.First_Name__c",
mdata-worker_1  |     "props": {
mdata-worker_1  |         "name": "TeleMed - SightCall Sessions Report.Field.Case.First_Name__c",
mdata-worker_1  |         "orgId": "00D0r0000008cvQEAQ"
mdata-worker_1  |     }
mdata-worker_1  | }
mdata-worker_1  | 2020-06-08T23:13:46.356Z [reports] debug: [00D0r0000008cvQEAQ] 00O3h000000rTTvEAM - Created 0 relationships

[ctalbotSFDC]

@rtanikella result from your query (changed the L to a 1):

╒══════════╕
│"count(1)"│
╞══════════╡
│1006.0 │
└──────────┘

[ctalbotSFDC]

This is what my ReportTypes.js file looks like now:

    async createReportTypes(reportTypes){
        await super.updateMetadataStatus('In Progress', {   type: 'ReportTypes',  totalTypes: reportTypes.length});
        let results = await this.conn.queryAll("select QualifiedApiName,PluralLabel from EntityDefinition");
        let pluralObjNameToAPIName = {};
        if(results.records) {
        	for(var i=0;i<results.records.length;i++){
            	this.logger.debug('['+this.conn.userInfo.organization_id + '] Plural Label:'+results.records[i].PluralLabel+' --> '+results.records[i].QualifiedApiName);
            	pluralObjNameToAPIName[results.records[i].PluralLabel]=results.records[i].QualifiedApiName;
        	}
        }

[rtanikella]

@ctalbotSFDC Ah, it makes sense that you'd get 320 nodes. The query I asked you to run is wrong on two counts: First, it should have read MATCH (o:CustomObject)-[r]-(t:ReportType) RETURN count(r); second, the createReportTypes() routine would still create the nodes for the report types, just not the relationships to the custom objects (save for Activity). This is because the custom objects (save for Activity) are absent from the pluralObjNameToAPIName map because your condition skipped populating the map.

If I run it with the following, skipping the loop if results.records is falsy as you suggest:

if(results.records){
    for(var i=0;i<results.length;i++){
        this.logger.debug('['+this.conn.userInfo.organization_id + '] Plural Label:'+results[i].PluralLabel+' --> '+results[i].QualifiedApiName);
        pluralObjNameToAPIName[results[i].PluralLabel]=results[i].QualifiedApiName;
    }
}

I get these:
MATCH (o:ReportType) RETURN count(o); returns 91
MATCH (o:CustomObject) RETURN count(o); returns 39
MATCH (o:CustomObject)-[r]-(t:ReportType) RETURN count(r); returns 0

If I remove the condition you suggested and make sure to replace all five occurrences within createReportTypes() of results.records with results as follows:

for(var i=0;i<results.length;i++){
    this.logger.debug('['+this.conn.userInfo.organization_id + '] Plural Label:'+results[i].PluralLabel+' --> '+results[i].QualifiedApiName);
    pluralObjNameToAPIName[results[i].PluralLabel]=results[i].QualifiedApiName;
}

I get these:
MATCH (o:ReportType) RETURN count(o); returns 91
MATCH (o:CustomObject) RETURN count(o); returns 39
MATCH (o:CustomObject)-[r]-(t:ReportType) RETURN count(r); returns 76

Your original post indicated the results.records issue on line 46. The subsequent trace you shared indicates the error on line 48. Assuming the line numbers haven't shifted with any edits you made, I'm guessing not all occurrences were modified.

[ctalbotSFDC]

yeah - I had some extra debug lines in my file, but I didn't update the code the same way you mentioned... looks like it fixed the issue... but I do get a lot of the could not find field stuff... so maybe something with the fields earlier on was an issue too....

mdata-worker_1  | 2020-06-09T14:05:57.769Z [reports] debug: [00D0r0000008cvQEAQ] 00O3h000000GsXYEA0 - Could not find custom Field: Object: Case, Field:TYPE
mdata-worker_1  | 2020-06-09T14:05:57.770Z [reports] debug: [00D0r0000008cvQEAQ] Found Field with Id:Case.Age for report column:AGE

[rtanikella]

@ctalbotSFDC I believe some of those are just informational from the upsert routine. Basically a way of indicating that the indexer will need to insert a new node instead of upsert an existing one. @anandbn please confirm.

[ctalbotSFDC]

on prior runs, it was

Could not find custom Field: Object: null

However, it looks like there aren't any with the Object: null on them anymore - so guess fixing that code fixed that too (that was my bigger concern)

Thanks!

[anandbn]

wow @rtanikella @ctalbotSFDC .. you guys are pretty active on Mdata :) .. thanks. Report Types has a core issue around queryAll so if you have a lot of report types that part of the code does error out. Based on the discussion, did the issue get resolved?

[ctalbotSFDC]

Yes - we got this one resolved