Consider the following test program
#!/usr/bin/node
const request = require("sync-request");
function getDataFromUrl(src) {
const res = request("GET", src, {
headers: {
"User-Agent": "sync-request/6.1.0"
}});
return res.getBody("utf8");
};
result = getDataFromUrl(process.argv.slice(2)[0]);
console.log(result);
It simply does a request to the url passed via command line argument setting the header User-Agent to sync-request/6.1.0 and prints the result.
If i run it to check the header it sends to the URL https://headers.cloxy.net/request.php everything looks ok. But when I try the URL http://headers.cloxy.net/request.php (without https) it doesn't work as expected. This is because the url with http does a re-direct to the url with https and then sync-request doesn't set the user-agent header on the final url when following the redirection from the original url.
Example:
$ node test.js https://headers.cloxy.net/request.php
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Show My HTTP Request Headers</title>
</head>
<body>
<h1>Your Request HTTP Headers</h1>
<ul>
<li>Connection: close</li>
<li>Host: headers.cloxy.net</li>
<li>Accept-Encoding: gzip,deflate</li>
<li>User-Agent: sync-request/6.1.0</li>
</ul>
<hr />
<p>Powered by:</p>
<p><a href="https://peername.com/" target="_blank"><img src="/peername.png" alt="PeerName Blockchain-based domains" title="Blockchain-based domains" /></a></p>
</body>
</html>
$ node test.js http://headers.cloxy.net/request.php
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Show My HTTP Request Headers</title>
</head>
<body>
<h1>Your Request HTTP Headers</h1>
<ul>
<li>Connection: close</li>
<li>Host: headers.cloxy.net</li>
</ul>
<hr />
<p>Powered by:</p>
<p><a href="https://peername.com/" target="_blank"><img src="/peername.png" alt="PeerName Blockchain-based domains" title="Blockchain-based domains" /></a></p>
</body>
</html>
Note how in this second request, following the redirect, it not only dropped the User-Agent header but also The Accept-Encoding one
The url at the http site does a standard 301 redirect to the site with https as shown below
$ curl -i http://headers.cloxy.net/request.php
HTTP/1.1 301 Moved Permanently
Date: Tue, 03 Nov 2020 02:28:49 GMT
Server: Apache
Location: https://headers.cloxy.net/request.php
Cache-Control: max-age=3600
Expires: Tue, 03 Nov 2020 03:28:49 GMT
Content-Length: 245
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://headers.cloxy.net/request.php">here</a>.</p>
</body></html>
Consider the following test program
It simply does a request to the url passed via command line argument setting the header
User-Agenttosync-request/6.1.0and prints the result.If i run it to check the header it sends to the URL https://headers.cloxy.net/request.php everything looks ok. But when I try the URL http://headers.cloxy.net/request.php (without https) it doesn't work as expected. This is because the url with
httpdoes a re-direct to the url withhttpsand then sync-request doesn't set the user-agent header on the final url when following the redirection from the original url.Example:
Note how in this second request, following the redirect, it not only dropped the
User-Agentheader but also TheAccept-EncodingoneThe url at the
httpsite does a standard301redirect to the site withhttpsas shown below