"\xAD" from ASCII-8BIT to UTF-8 ERROR when requested GET

[AshwiniDoddamaniFluke]

`RestCore: spent 7.070209 Requested GET https://streaming.preprod.connect.fluke.com/iot-service/history/reports/download/b9498f13-62cf-44ba-9a1e-1eb2b4444d22?deviceType=fc3540&frequency=TWELVE_HOUR&sessionStartTime=1499763413506&from=1501668000000&reportType=xlsx&requesterTimeZone=GMT%2B0530&app_platform=website&app_vers=272

"\xAD" from ASCII-8BIT to UTF-8

/.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/core_ext/object/json.rb:34:in encode' /.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/core_ext/object/json.rb:34:in to_json'
/.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/core_ext/object/json.rb:34:in to_json' /.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/json/encoding.rb:55:in to_json'
/.rvm/gems/ruby-2.3.3/gems/json-1.8.6/lib/json/common.rb:223:in generate' /.rvm/gems/ruby-2.3.3/gems/json-1.8.6/lib/json/common.rb:223:in generate'
/.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/json/encoding.rb:99:in stringify' /.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/json/encoding.rb:33:in encode'
/.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/json/encoding.rb:20:in encode' /.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/core_ext/object/json.rb:37:in to_json'
/.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/action_controller/metal/renderers.rb:159:in block in <module:Renderers>' /.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/action_controller/metal/renderers.rb:152:in block in _render_to_body_with_renderer'
/.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/set.rb:306:in each_key' /.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/set.rb:306:in each'
/.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/action_controller/metal/renderers.rb:148:in _render_to_body_with_renderer' /.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/action_controller/metal/renderers.rb:144:in render_to_body'
/.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/abstract_controller/rendering.rb:26:in render' /.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/action_controller/metal/rendering.rb:36:in render'
/.rvm/gems/ruby-2.3.3/gems/actionpack-5.0.0/lib/action_controller/metal/instrumentation.rb:44:in block (2 levels) in render' /.rvm/gems/ruby-2.3.3/gems/activesupport-5.0.0/lib/active_support/core_ext/benchmark.rb:12:in block in ms'`

[godfat]

Anymore information? Like the full backtrace, the original responses, etc.

[AshwiniDoddamaniFluke]

`Requested GET https: //streaming.preprod.connect.fluke.com/iot-service/history/reports/download/b9498f13-62cf-44ba-9a1e-1eb2b4444d22?deviceType=fc3540&frequency=TWELVE_HOUR&sessionStartTime=1499763413506&from=1502877600000&reportType=xlsx&requesterTimeZone=GMT%2B0530&app_platform=website&app_vers=270

RestBuilder::Promise: ERROR: invalid byte sequence in UTF - 8

from["/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/util/json.rb:75:in sub'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/util/json.rb:75:in strip_bom'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/util/json.rb:66:in normalize'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/json_response.rb:34:in process'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/json_response.rb:26:in block in call'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/common_logger.rb:13:in block in call'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/cache.rb:34:in block in app_call'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in block in resolve'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in each'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in inject'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:118:in resolve'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:109:in fulfilling'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:84:in block in fulfill'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:84:in synchronize'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:84:in fulfill'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:136:in protected_yield'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/promise_pool-0.9.0/lib/promise_pool/promise.rb:49:in block in defer'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-builder-0.9.1/lib/rest-builder/engine.rb:28:in call'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/auth_basic.rb:10:in call'", "/Users/ashwini/no-rocket-science/sw-web/lib/fluke/token.rb:24:in call'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/cache.rb:29:in app_call'", "/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/cache.rb:25:in call'",
"/Users/ashwini/.rvm/gems/ruby-2.3.3/gems/rest-core-4.0.1/lib/rest-core/middleware/common_logger.rb:12:in call'",

[godfat]

This means that the response was claiming to be UTF-8 encoded however it's not. Do you have the original response available?

Suppose this is really the case, what could we do here? We can't reliably parse the JSON without knowing the correct encoding. We could somehow try to fix the response, but that would cause data loss and might not give desired result.

[SunnyKukkarFluke]

Hi @godfat,

We are getting an xlsx file as response of this API. Here are the response headers which we are getting in postman:

connection →keep-alive
content-disposition →attachment; filename=FC3540Report1503054716877.xlsx
content-length →189062
content-type →application/xlsx
date →Fri, 18 Aug 2017 11:11:57 GMT
expires →: 0
x-application-context →application:7300

[godfat]

We need the response body to figure out the encoding issue. However on the other hand, it seems like the response was actually an XLSX file, which is not JSON, therefore you shouldn't use JsonResponse middleware.

What's the actually code you're using? If you're using Universal, you should disable :json_response.

[SunnyKukkarFluke]

godfat,

Once we are calling get api of rest core using "rest_core_obj.request_full(data)" to download XLSX.
I am also surprised even if in response header the content-type is mentioned as XLSX, why its encoding as json.

[godfat]

@SunnyKukkarFluke You could try:

rest_core_obj.request_full(data.merge(:json_response => false))

The built-in middleware was not designed to download anything, so if you want to do different stuffs with the same client object, then you should write your own middleware.