Register Log In

Forums Bug Reports $urlget content-length header

Print Thread

$urlget content-length header #271532 11/04/23 04:55 AM
Joined: Jul 2006 Posts: 4,030 France W Wims OP Hoopy frood
OP Wims Hoopy frood W Joined: Jul 2006 Posts: 4,030 France	1) $urlget removes the Content-length header from the $urlget().reply value 2) as a result of 1) $urlget does not report the value of the Content length header via $urlget().size, this value is 0 and it does not create/populate the output filename (if a binvar is used as the output, the binvar is filled correctly, but content length still missing and size still 0) `//.remove test.mrc \| alias urlgetfail if (content-length !isin $!urlget($1).reply) echo -ag no content length size: $!urlget($1).size state $!urlget($1).state content $!exists(test.mrc) $!lof(test.mrc) reply $!urlget($1).reply $(\|) else echo -ag state $!urlget($1).state $(\|) alias urlgetfail \| noop $urlget(https://www.mirc.com/versions.txt,gfik,test.mrc,urlgetfail)` #mircscripting @ irc.swiftirc.net == the best mIRC help channel

Re: $urlget content-length header Wims #271533 11/04/23 12:55 PM
Joined: Dec 2002 Posts: 5,292 London, UK Khaled Hoopy frood
Khaled Hoopy frood Joined: Dec 2002 Posts: 5,292 London, UK	Thanks for your bug report. The most likely reason for this is that the content-length is not always included in the header reply from the server. Did you check to see if the content-length is actually in the header? eg. by opening the link in a browser and using developer tools to see the header details?

Re: $urlget content-length header Khaled #271534 11/04/23 01:06 PM
Joined: Jul 2006 Posts: 4,030 France W Wims OP Hoopy frood
OP Wims Hoopy frood W Joined: Jul 2006 Posts: 4,030 France	It's not possible for the content length header not to be provided by an http server for this situation, no need to check. What was tried was multiple http servers and content length is always removed. #mircscripting @ irc.swiftirc.net == the best mIRC help channel

$urlget content-length header #271535 11/04/23 01:48 PM
Joined: Jul 2006 Posts: 4,030 France W Wims OP Hoopy frood
OP Wims Hoopy frood W Joined: Jul 2006 Posts: 4,030 France	Sorry about that previous post, I mistested, you're right, with chunk encoding, no content length header is required, which is what www.mirc.com is doing. But wrong example then, the issue is still present when chunk encoding is not used, here is a better example: `//alias urlgetfail if (content-length !isin $!urlget($1).reply) echo -ag no content length: $!v2 $(\|) alias urlgetfail \| noop $urlget(https://raw.githubusercontent.com/SReject/JSON-For-Mirc/master/src/JSON%20For%20mIRC.mrc,gfik,test.mrc,urlgetfail)` vs `alias urlgetfail2 sockclose urlgetfail \| sockopen -e urlgetfail raw.githubusercontent.com 443 on :sockopen:urlgetfail:sockwrite urlgetfail $+(GET /SReject/JSON-For-Mirc/master/src/JSON%20For%20mIRC.mrc HTTP/1.1,$crlf,Host: raw.githubusercontent.com,$crlf,User-Agent: mIRC,$crlf,$crlf) on :sockread:urlgetfail:var %a \| sockread %a \| if (!$sockbr) return \| if (%a == $null) sockclose urlgetfail \| else echo -ag headers: %a` Originally Posted by /sockopen headers: HTTP/1.1 200 OK headers: Connection: keep-alive headers: Content-Length: 42128 headers: Cache-Control: max-age=300 headers: Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox headers: Content-Type: text/plain; charset=utf-8 headers: ETag: "a98867cff9c5baf701be4c97ea1b39d354e2704dcaccc274cb3bfa9d4212bcff" headers: Strict-Transport-Security: max-age=31536000 headers: X-Content-Type-Options: nosniff headers: X-Frame-Options: deny headers: X-XSS-Protection: 1; mode=block headers: X-GitHub-Request-Id: F598:1C3F:9B750:D7823:643563E3 headers: Accept-Ranges: bytes headers: Date: Tue, 11 Apr 2023 13:43:00 GMT headers: Via: 1.1 varnish headers: X-Served-By: cache-mrs10579-MRS headers: X-Cache: MISS headers: X-Cache-Hits: 0 headers: X-Timer: S1681220580.807887,VS0,VE265 headers: Vary: Authorization,Accept-Encoding,Origin headers: Access-Control-Allow-Origin: * headers: X-Fastly-Request-ID: 45c8a7c842858d548578fec2ad9d088b94b45cd5 headers: Expires: Tue, 11 Apr 2023 13:48:00 GMT headers: Source-Age: 0 Originally Posted by $urlget no content length: HTTP/1.1 200 OK Connection: keep-alive Cache-Control: max-age=300 Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox Content-Type: text/plain; charset=utf-8 ETag: W/"e3990ff20cc6506d6eb2c5acd28024d73beda98f037186e4b1cabe933637a433" Strict-Transport-Security: max-age=31536000 X-Content-Type-Options: nosniff X-Frame-Options: deny X-XSS-Protection: 1; mode=block X-GitHub-Request-Id: FE10:3D5E:5056C3:639E88:64356057 Accept-Ranges: bytes Date: Tue, 11 Apr 2023 13:44:19 GMT Via: 1.1 varnish X-Served-By: cache-mrs10549-MRS X-Cache: HIT X-Cache-Hits: 2 X-Timer: S1681220659.094595,VS0,VE0 Vary: Authorization,Origin Access-Control-Allow-Origin: * X-Fastly-Request-ID: 9ed29e68eac9b65651f7e947a25f9ce5414fa99d Expires: Tue, 11 Apr 2023 13:49:19 GMT Source-Age: 285 #mircscripting @ irc.swiftirc.net == the best mIRC help channel

Re: $urlget content-length header Wims #271541 12/04/23 08:25 AM
Joined: Dec 2002 Posts: 5,292 London, UK Khaled Hoopy frood
Khaled Hoopy frood Joined: Dec 2002 Posts: 5,292 London, UK	No problem, it does actually look like the WinInet API removes the Content-Length from the reply header when the client enables gzip/deflate decoding. As far as I can tell, there is no way to prevent this, as it is part of the API, so you would need to use .size in this context.

Re: $urlget content-length header Khaled #271549 12/04/23 05:03 PM
Joined: Jul 2006 Posts: 4,030 France W Wims OP Hoopy frood
OP Wims Hoopy frood W Joined: Jul 2006 Posts: 4,030 France	Quote so you would need to use .size in this context But this is part of my report Khaled, I cannot use .size because it returns 0 because the API is removing the content length. Another thing is that the output file is not created in this case. It seems rather strange that the API would be so that you cannot get the length #mircscripting @ irc.swiftirc.net == the best mIRC help channel

Re: $urlget content-length header Wims #271550 13/04/23 07:53 AM
Joined: Dec 2002 Posts: 5,292 London, UK Khaled Hoopy frood
Khaled Hoopy frood Joined: Dec 2002 Posts: 5,292 London, UK	Quote But this is part of my report Khaled, I cannot use .size because it returns 0 because the API is removing the content length. Another thing is that the output file is not created in this case. It seems rather strange that the API would be so that you cannot get the length The output file is not created? That is odd. When I run your above scripts, the file is downloaded correctly. Can you please test this again? As for .size, sorry, I meant .rcvd, which returns the total bytes received. Quote It seems rather strange that the API would be so that you cannot get the length If you find a way around it, let me know. As far as I can tell, it is an API bug. The API that enables gzip/deflate decoding in WinINet, but disables content-length in the header, is InternetSetOption() with the INTERNET_OPTION_HTTP_DECODING option. If INTERNET_OPTION_HTTP_DECODING is not set, content-length is present as expected. Remember that content-length is the size of the transfer, so if the transfer encoding is gzip/deflate, it will be the compressed transfer size, not the size of the received file, which is in .rcvd.

Link Copied to Clipboard