To report an issue programmatically, use the Create Data Issue endpoint of the REST API.
+History requests in live algorithms still use the historical datasets, so report issues with them as described above. The real-time stream comes from the data provider you select when you deploy, so the Data Issues page doesn't track it. If you encounter an issue with real-time data, first confirm that the data provider supports the asset. We recommend the QuantConnect data provider as the first choice. If the issue persists, open a support ticket and attach the project and the live deployment.
\ No newline at end of file diff --git a/01 Cloud Platform/99 API Reference/00.json b/01 Cloud Platform/99 API Reference/00.json index 2fc301b471..bfb0ced664 100644 --- a/01 Cloud Platform/99 API Reference/00.json +++ b/01 Cloud Platform/99 API Reference/00.json @@ -18,6 +18,7 @@ "11" : "Generate reports for backtests to evaluate performance", "12" : "Read account information", "13" : "Read LEAN engine version", + "15" : "Report and list data issues", "99" : "Examples of using the API with Jupyter Notebooks" } } diff --git a/01 Cloud Platform/99 API Reference/15 Data Issues/01 Create Data Issue/01 Introduction.html b/01 Cloud Platform/99 API Reference/15 Data Issues/01 Create Data Issue/01 Introduction.html new file mode 100644 index 0000000000..82b89ab54d --- /dev/null +++ b/01 Cloud Platform/99 API Reference/15 Data Issues/01 Create Data Issue/01 Introduction.html @@ -0,0 +1,5 @@ + + ++Report an issue with a dataset on the Data Issues page. +
diff --git a/01 Cloud Platform/99 API Reference/15 Data Issues/01 Create Data Issue/02 Description.html b/01 Cloud Platform/99 API Reference/15 Data Issues/01 Create Data Issue/02 Description.html new file mode 100644 index 0000000000..580252d679 --- /dev/null +++ b/01 Cloud Platform/99 API Reference/15 Data Issues/01 Create Data Issue/02 Description.html @@ -0,0 +1,5 @@ + + +
+Every report must include a reproducible case, so our Data Team can reproduce the issue. Attach the project of a backtest or notebook that shows the issue with the select-project property and the backtest with the select-backtest property, or include a code snippet in the content property. Follow the best practices when you write the report. Before you report an issue, use the /community/data-library/issues/list endpoint to check if an open issue already covers the same ticker and resolution.
+
+The dataset, time window, and description of the issue. The /community/data-library/issues/create API accepts requests in the following format:
+
CreateDataIssueRequest Model - Request body to report a data issue. |
+|
|---|---|
| ticker | string Ticker of the security with the issue. |
+
| security | string Enum Asset class of the dataset. Options : ['Equity', 'Equity Option', 'Index', 'Index Option', 'Future', 'Future Option', 'Forex', 'CFD', 'Crypto', 'Crypto Future'] |
+
| market | string Market of the dataset. The valid markets depend on the asset class, as shown in the market dropdown on the Data Issues page. |
+
| resolution | string Enum Resolution of the dataset. The valid resolutions depend on the asset class and market. Options : ['Tick', 'Second', 'Minute', 'Hour', 'Daily', 'Coarse Fundamental', 'Fine Fundamental', 'Margin'] |
+
| from-date | string First date of the issue, in YYYY-MM-DD format. |
+
| to-date | string Last date of the issue, in YYYY-MM-DD format. The end of the time window must be after the start. |
+
| from-time | string Time of day the issue starts, in HH:MM format. Required for Second and Minute resolution when the time window is shorter than 7 days. |
+
| to-time | string Time of day the issue ends, in HH:MM format. Required for Second and Minute resolution when the time window is shorter than 7 days. |
+
| corrupted-files | string Enum Include to report files that can't be read. Include at least one issue type. Options : ['Corrupted Files'] |
+
| missing-points | string Enum Include to report missing data points. Options : ['Missing Points'] |
+
| missing-files | string Enum Include to report missing files. Options : ['Missing Files'] |
+
| wrong-values | string Enum Include to report wrong values. Options : ['Wrong Values'] |
+
| listing-error | string Enum Include to report a listing error. Options : ['Listing Error'] |
+
| split-error | string Enum Include to report a split error. Options : ['Split Error'] |
+
| dividend-error | string Enum Include to report a dividend error. Options : ['Dividend Error'] |
+
| name-mapping-error | string Enum Include to report a ticker mapping error. Options : ['Name Mapping Error'] |
+
| content | string HTML description of the issue. |
+
| select-project | integer Id of one of your projects to attach to the issue, so the Data team can see the code that shows the issue. To get your project Ids, use the /projects/read endpoint. |
+
| select-backtest | string Id of a backtest of the attached project to attach to the issue. Requires select-project. To get the backtest Ids of a project, use the /backtests/list endpoint. |
+
| Example | +
+
+{
+ "ticker": "SPY",
+ "security": "Equity",
+ "market": "USA",
+ "resolution": "Tick",
+ "from-date": "2024-08-01",
+ "to-date": "2024-08-02",
+ "from-time": "09:30",
+ "to-time": "16:00",
+ "corrupted-files": "Corrupted Files",
+ "missing-points": "Missing Points",
+ "missing-files": "Missing Files",
+ "wrong-values": "Wrong Values",
+ "listing-error": "Listing Error",
+ "split-error": "Split Error",
+ "dividend-error": "Dividend Error",
+ "name-mapping-error": "Name Mapping Error",
+ "content": "SPY data is missing on 2024-08-01.",
+ "select-project": 0,
+ "select-backtest": "string"
+}
+ |
+
+The /community/data-library/issues/create API provides a response in the following format:
+
CreateDataIssueResponse Model - Response from reporting a data issue. |
+|
|---|---|
| discussionId | integer Id of the new data issue. The issue page is https://www.quantconnect.com/datasets/issue/{discussionId}. |
+
| success | boolean Indicate if the API request was successful. |
+
| errors | string Array List of errors with the API call. |
+
| Example | +
+
+{
+ "discussionId": 0,
+ "success": true,
+ "errors": [
+ "string"
+ ]
+}
+ |
+
UnauthorizedError Model - Unauthorized response from the API. Key is missing, invalid, or timestamp is too old for hash. |
+|
|---|---|
| www_authenticate | string Header |
+
+List the issues reported on the Data Issues page. +
diff --git a/01 Cloud Platform/99 API Reference/15 Data Issues/02 List Data Issues/02 Description.html b/01 Cloud Platform/99 API Reference/15 Data Issues/02 List Data Issues/02 Description.html new file mode 100644 index 0000000000..65ab015b4b --- /dev/null +++ b/01 Cloud Platform/99 API Reference/15 Data Issues/02 List Data Issues/02 Description.html @@ -0,0 +1,5 @@ + + +
+The results are paginated. Use the start and end properties to select a range of issues and the resultsFound property of the response to get the total number of issues that match the filters. To get all the issues in one request, set end to a value greater than or equal to resultsFound.
+
+Filters and range of the data issues to list. The /community/data-library/issues/list API accepts requests in the following format:
+
ListDataIssuesRequest Model - Request body to list data issues. |
+|
|---|---|
| status | string Enum Status group of the issues to list. Open issues are In queue, In progress, Waiting Review, or Waiting for Data Vendor. Closed issues are Resolved, Closed, or Cannot Be Resolved. The default value is open. Options : ['open', 'closed'] |
+
| start | integer Index of the first issue to return. The default value is 0. |
+
| end | integer Index after the last issue to return. The default value is 20. |
+
| search | string Ticker to filter the issues by. The ticker must match exactly, but the match isn't case-sensitive. |
+
| subscribed | boolean If true, list only the issues you're subscribed to. |
+
| Example | +
+
+{
+ "status": "open",
+ "start": 0,
+ "end": 20,
+ "search": "SPY",
+ "subscribed": true
+}
+ |
+
+The /community/data-library/issues/list API provides a response in the following format:
+
DataIssueListResponse Model - Response from listing data issues. |
+|
|---|---|
| results | DataIssue Array Data issues in the requested range. |
+
| resultsFound | integer Number of issues that match the filters. |
+
| openIssuesCount | integer Number of open issues on the Data Issues page. |
+
| closedIssuesCount | integer Number of closed issues on the Data Issues page. |
+
| success | boolean Indicate if the API request was successful. |
+
| errors | string Array List of errors with the API call. |
+
| Example | +
+
+{
+ "results": [
+ {
+ "discussionId": 0,
+ "ticker": "string",
+ "security": "string",
+ "market": "string",
+ "resolution": "string",
+ "from": "2024-08-01 00:00:00",
+ "to": "2024-08-02 00:00:00",
+ "status": "In queue",
+ "corruptedFile": 0,
+ "missingPoints": 0,
+ "missingFile": 0,
+ "wrongValues": 0,
+ "listingError": 0,
+ "splitError": 0,
+ "dividendError": 0,
+ "mappingError": 0,
+ "reported": 0,
+ "subscribed": true,
+ "userName": "string",
+ "userPublicId": "string",
+ "userProfile": "string"
+ }
+ ],
+ "resultsFound": 0,
+ "openIssuesCount": 0,
+ "closedIssuesCount": 0,
+ "success": true,
+ "errors": [
+ "string"
+ ]
+}
+ |
+
DataIssue Model - An issue reported on the Data Issues page. |
+|
|---|---|
| discussionId | integer Id of the data issue. The issue page is https://www.quantconnect.com/datasets/issue/{discussionId}. |
+
| ticker | string Ticker of the security with the issue. |
+
| security | string Asset class of the dataset. |
+
| market | string Market of the dataset. |
+
| resolution | string Resolution of the dataset. |
+
| from | string Start of the issue, in YYYY-MM-DD HH:MM:SS format. |
+
| to | string End of the issue, in YYYY-MM-DD HH:MM:SS format. |
+
| status | string Enum Status of the issue. Options : ['In queue', 'In progress', 'Waiting Review', 'Waiting for Data Vendor', 'Resolved', 'Closed', 'Cannot Be Resolved'] |
+
| corruptedFile | integer 1 if the issue reports corrupted files, otherwise 0. |
+
| missingPoints | integer 1 if the issue reports missing data points, otherwise 0. |
+
| missingFile | integer 1 if the issue reports missing files, otherwise 0. |
+
| wrongValues | integer 1 if the issue reports wrong values, otherwise 0. |
+
| listingError | integer 1 if the issue reports a listing error, otherwise 0. |
+
| splitError | integer 1 if the issue reports a split error, otherwise 0. |
+
| dividendError | integer 1 if the issue reports a dividend error, otherwise 0. |
+
| mappingError | integer 1 if the issue reports a ticker mapping error, otherwise 0. |
+
| reported | integer Unix timestamp of when the issue was reported. |
+
| subscribed | boolean True if you're subscribed to the issue. |
+
| userName | string Name of the member who reported the issue. |
+
| userPublicId | string Public Id of the member who reported the issue. |
+
| userProfile | string URL of the profile image of the member who reported the issue. |
+
| Example | +
+
+{
+ "discussionId": 0,
+ "ticker": "string",
+ "security": "string",
+ "market": "string",
+ "resolution": "string",
+ "from": "2024-08-01 00:00:00",
+ "to": "2024-08-02 00:00:00",
+ "status": "In queue",
+ "corruptedFile": 0,
+ "missingPoints": 0,
+ "missingFile": 0,
+ "wrongValues": 0,
+ "listingError": 0,
+ "splitError": 0,
+ "dividendError": 0,
+ "mappingError": 0,
+ "reported": 0,
+ "subscribed": true,
+ "userName": "string",
+ "userPublicId": "string",
+ "userProfile": "string"
+}
+ |
+
UnauthorizedError Model - Unauthorized response from the API. Key is missing, invalid, or timestamp is too old for hash. |
+|
|---|---|
| www_authenticate | string Header |
+
select-project property and the backtest with the select-backtest property, or include a code snippet in the content property. Follow the best practices when you write the report. Before you report an issue, use the /community/data-library/issues/list endpoint to check if an open issue already covers the same ticker and resolution.
+ requestBody:
+ description: The dataset, time window, and description of the issue.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDataIssueRequest'
+ required: true
+ responses:
+ "200":
+ description: The Id of the new data issue.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/CreateDataIssueResponse'
+ "401":
+ $ref: '#/components/responses/UnauthorizedErrorResponse'
+ operationId: createDataIssue
+ /community/data-library/issues/list:
+ post:
+ tags:
+ - 01 Cloud Platform
+ - 99 API Reference
+ - 15 Data Issues
+ - 02 List Data Issues
+ summary: List the issues reported on the Data Issues page.
+ description: The results are paginated. Use the start and end properties to select a range of issues and the resultsFound property of the response to get the total number of issues that match the filters. To get all the issues in one request, set end to a value greater than or equal to resultsFound.
+ requestBody:
+ description: Filters and range of the data issues to list.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ListDataIssuesRequest'
+ required: true
+ responses:
+ "200":
+ description: The data issues that match the request.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/DataIssueListResponse'
+ "401":
+ $ref: '#/components/responses/UnauthorizedErrorResponse'
+ operationId: listDataIssues
components:
@@ -8892,6 +8944,260 @@ components:
conversation:
$ref: '#/components/schemas/AgentConversation'
description: Response wrapping the conversation transcript of an agent deployment.
+ CreateDataIssueRequest:
+ type: object
+ required:
+ - ticker
+ - security
+ - market
+ - resolution
+ - from-date
+ - to-date
+ - content
+ properties:
+ ticker:
+ type: string
+ description: Ticker of the security with the issue.
+ example: SPY
+ security:
+ type: string
+ description: Asset class of the dataset.
+ enum:
+ - Equity
+ - Equity Option
+ - Index
+ - Index Option
+ - Future
+ - Future Option
+ - Forex
+ - CFD
+ - Crypto
+ - Crypto Future
+ market:
+ type: string
+ description: Market of the dataset. The valid markets depend on the asset class, as shown in the market dropdown on the Data Issues page.
+ example: USA
+ resolution:
+ type: string
+ description: Resolution of the dataset. The valid resolutions depend on the asset class and market.
+ enum:
+ - Tick
+ - Second
+ - Minute
+ - Hour
+ - Daily
+ - Coarse Fundamental
+ - Fine Fundamental
+ - Margin
+ from-date:
+ type: string
+ description: First date of the issue, in YYYY-MM-DD format.
+ example: "2024-08-01"
+ to-date:
+ type: string
+ description: Last date of the issue, in YYYY-MM-DD format. The end of the time window must be after the start.
+ example: "2024-08-02"
+ from-time:
+ type: string
+ description: Time of day the issue starts, in HH:MM format. Required for Second and Minute resolution when the time window is shorter than 7 days.
+ example: "09:30"
+ to-time:
+ type: string
+ description: Time of day the issue ends, in HH:MM format. Required for Second and Minute resolution when the time window is shorter than 7 days.
+ example: "16:00"
+ corrupted-files:
+ type: string
+ description: Include to report files that can't be read. Include at least one issue type.
+ enum:
+ - Corrupted Files
+ missing-points:
+ type: string
+ description: Include to report missing data points.
+ enum:
+ - Missing Points
+ missing-files:
+ type: string
+ description: Include to report missing files.
+ enum:
+ - Missing Files
+ wrong-values:
+ type: string
+ description: Include to report wrong values.
+ enum:
+ - Wrong Values
+ listing-error:
+ type: string
+ description: Include to report a listing error.
+ enum:
+ - Listing Error
+ split-error:
+ type: string
+ description: Include to report a split error.
+ enum:
+ - Split Error
+ dividend-error:
+ type: string
+ description: Include to report a dividend error.
+ enum:
+ - Dividend Error
+ name-mapping-error:
+ type: string
+ description: Include to report a ticker mapping error.
+ enum:
+ - Name Mapping Error
+ content:
+ type: string
+ description: HTML description of the issue.
+ example: SPY data is missing on 2024-08-01.
+ select-project:
+ type: integer
+ description: Id of one of your projects to attach to the issue, so the Data team can see the code that shows the issue. To get your project Ids, use the /projects/read endpoint.
+ select-backtest:
+ type: string
+ description: Id of a backtest of the attached project to attach to the issue. Requires select-project. To get the backtest Ids of a project, use the /backtests/list endpoint.
+ description: Request body to report a data issue.
+ CreateDataIssueResponse:
+ type: object
+ properties:
+ discussionId:
+ type: integer
+ description: Id of the new data issue. The issue page is https://www.quantconnect.com/datasets/issue/{discussionId}.
+ success:
+ type: boolean
+ description: Indicate if the API request was successful.
+ errors:
+ type: array
+ description: List of errors with the API call.
+ items:
+ type: string
+ description: Response from reporting a data issue.
+ ListDataIssuesRequest:
+ type: object
+ properties:
+ status:
+ type: string
+ description: Status group of the issues to list. Open issues are In queue, In progress, Waiting Review, or Waiting for Data Vendor. Closed issues are Resolved, Closed, or Cannot Be Resolved. The default value is open.
+ enum:
+ - open
+ - closed
+ start:
+ type: integer
+ description: Index of the first issue to return. The default value is 0.
+ end:
+ type: integer
+ description: Index after the last issue to return. The default value is 20.
+ example: 20
+ search:
+ type: string
+ description: Ticker to filter the issues by. The ticker must match exactly, but the match isn't case-sensitive.
+ example: SPY
+ subscribed:
+ type: boolean
+ description: If true, list only the issues you're subscribed to.
+ description: Request body to list data issues.
+ DataIssueListResponse:
+ type: object
+ properties:
+ results:
+ type: array
+ description: Data issues in the requested range.
+ items:
+ $ref: '#/components/schemas/DataIssue'
+ resultsFound:
+ type: integer
+ description: Number of issues that match the filters.
+ openIssuesCount:
+ type: integer
+ description: Number of open issues on the Data Issues page.
+ closedIssuesCount:
+ type: integer
+ description: Number of closed issues on the Data Issues page.
+ success:
+ type: boolean
+ description: Indicate if the API request was successful.
+ errors:
+ type: array
+ description: List of errors with the API call.
+ items:
+ type: string
+ description: Response from listing data issues.
+ DataIssue:
+ type: object
+ properties:
+ discussionId:
+ type: integer
+ description: Id of the data issue. The issue page is https://www.quantconnect.com/datasets/issue/{discussionId}.
+ ticker:
+ type: string
+ description: Ticker of the security with the issue.
+ security:
+ type: string
+ description: Asset class of the dataset.
+ market:
+ type: string
+ description: Market of the dataset.
+ resolution:
+ type: string
+ description: Resolution of the dataset.
+ from:
+ type: string
+ description: Start of the issue, in YYYY-MM-DD HH:MM:SS format.
+ example: "2024-08-01 00:00:00"
+ to:
+ type: string
+ description: End of the issue, in YYYY-MM-DD HH:MM:SS format.
+ example: "2024-08-02 00:00:00"
+ status:
+ type: string
+ description: Status of the issue.
+ enum:
+ - In queue
+ - In progress
+ - Waiting Review
+ - Waiting for Data Vendor
+ - Resolved
+ - Closed
+ - Cannot Be Resolved
+ corruptedFile:
+ type: integer
+ description: 1 if the issue reports corrupted files, otherwise 0.
+ missingPoints:
+ type: integer
+ description: 1 if the issue reports missing data points, otherwise 0.
+ missingFile:
+ type: integer
+ description: 1 if the issue reports missing files, otherwise 0.
+ wrongValues:
+ type: integer
+ description: 1 if the issue reports wrong values, otherwise 0.
+ listingError:
+ type: integer
+ description: 1 if the issue reports a listing error, otherwise 0.
+ splitError:
+ type: integer
+ description: 1 if the issue reports a split error, otherwise 0.
+ dividendError:
+ type: integer
+ description: 1 if the issue reports a dividend error, otherwise 0.
+ mappingError:
+ type: integer
+ description: 1 if the issue reports a ticker mapping error, otherwise 0.
+ reported:
+ type: integer
+ description: Unix timestamp of when the issue was reported.
+ subscribed:
+ type: boolean
+ description: True if you're subscribed to the issue.
+ userName:
+ type: string
+ description: Name of the member who reported the issue.
+ userPublicId:
+ type: string
+ description: Public Id of the member who reported the issue.
+ userProfile:
+ type: string
+ description: URL of the profile image of the member who reported the issue.
+ description: An issue reported on the Data Issues page.
responses:
BacktestResponse:
description: Backtest read response packet.
diff --git a/Resources/qc-api/examples/data-issues.php b/Resources/qc-api/examples/data-issues.php
new file mode 100644
index 0000000000..e75ad0a4b8
--- /dev/null
+++ b/Resources/qc-api/examples/data-issues.php
@@ -0,0 +1,57 @@
+The following example demonstrates listing the open data issues for a ticker and reporting a new issue with an attached project and backtest when none exists through the cloud API.
+ +
+
+### List Data Issues
+# Send a POST request to the /community/data-library/issues/list endpoint to find open issues for SPY
+response = post(f'{BASE_URL}/community/data-library/issues/list', headers=get_headers(), json={
+ "status": "open", # open or closed
+ "start": 0, # Index of the first issue to return
+ "end": 20, # Index after the last issue to return
+ "search": "SPY" # Exact ticker match
+})
+# Parse the JSON response into python managable dict
+result = response.json()
+# Keep the open issues that already cover SPY minute data
+existing = [i for i in result.get('results', []) if i['resolution'] == 'Minute']
+# Check if the request was successful and print the issues
+if result['success']:
+ print("Open Data Issues:")
+ print(result)
+
+### List Backtests
+# Send a POST request to the /backtests/list endpoint to find a backtest that shows the issue
+project_id = 12345678 # Replace with the Id of your project
+response = post(f'{BASE_URL}/backtests/list', headers=get_headers(), json={
+ "projectId": project_id,
+ "includeStatistics": False
+})
+# Parse the JSON response into python managable dict
+result = response.json()
+# Pick the first backtest of the project to attach to the issue
+backtest_id = result['backtests'][0]['backtestId'] if result.get('backtests') else None
+
+### Create Data Issue (only when no open issue covers it)
+if not existing:
+ # Send a POST request to the /community/data-library/issues/create endpoint to report the issue
+ response = post(f'{BASE_URL}/community/data-library/issues/create', headers=get_headers(), json={
+ "ticker": "SPY",
+ "security": "Equity",
+ "market": "USA",
+ "resolution": "Minute",
+ "from-date": "2024-08-01",
+ "to-date": "2024-08-01",
+ "from-time": "09:30", # Minute data over less than 7 days needs times
+ "to-time": "16:00",
+ "missing-points": "Missing Points", # At least one issue type
+ "content": "SPY minute bars are missing on 2024-08-01 between 09:30 and 16:00.",
+ "select-project": project_id, # Optional. Project that shows the issue
+ "select-backtest": backtest_id # Optional. Backtest of that project
+ })
+ # Parse the JSON response into python managable dict
+ result = response.json()
+ # Check if the request was successful and print the link to the issue
+ if result['success']:
+ print(f"https://www.quantconnect.com/datasets/issue/{result['discussionId']}")
+