|
|
Joined: Dec 2002
Posts: 252
Fjord artisan
|
OP
Fjord artisan
Joined: Dec 2002
Posts: 252 |
I'm a few updates behind, currently testing on mIRC 7.67, but I don't see any changes to urlget besides the new "e" switch, so Unsure if this affects any newer versions, but it does for anything 7.67 and below. I'm using a web-api that requires basic auth. The "username" is my email, and that's where the issue occurs. I'll do a mock-example below to illustrate, as I can't exactly paste a working example without including my userinfo. so We'll just use "localhost" I first tried something along the lines of: http://user@gmail.com:pass@localhost/?test=true NOTE: link works fine in all browsers tested, "URL bar" aslo. 200 OK and some return data from the API call. noop $urlget(%URI,gbi,&Update,Handle_Update) results:
* url: (Redacted)
* redirect:
* method: get
* type: binvar
* target: &Update
* alias: Handle_Update
* id: 1026
* state: fail
* size: 1622
* resume: 0
* rcvd: 0
* time: 484
* reply: HTTP/1.1 404 Not Found
Cross-Origin-Resource-Policy: cross-origin
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Date: Tue, 09 Jan 2024 01:20:48 GMT
Server: sffe
Content-Length: 1622
X-XSS-Protection: 0
This totally breaks and connects to gmail.com, makes my request, and naturally gets a 404 not found... Somehow ignores the beginning and afterward of the URI extracting only the scheme (http) and the host to be "gmail.com" losing the actual host+port, yet somehow manages to retain the proper path to request... Doing some reading: according to rfc3986 A URI is composed of 5 parts: <scheme>://<authority>/<path>?<query>#<fragment> These "tokens" such as <authority> have subsections to be broken down even further such as: <userinfo>@<host>:<port> each "token" must follow layout rules, there's reserved characters and unreserved characters. all "reserved" characters should be replaced by it's corresponding percent-encoded equivalent So I decided to try this URI http://user%40gmail.com:pass@localhost/?test=true NOTE: link works fine in all browsers tested, "URL bar" aslo. 200 OK and some return data from the API call. noop $urlget(%URI,gbi,&Update,Handle_Update) results:
* url: (Redacted)
* redirect:
* method: get
* type: binvar
* target: &Update
* alias: Handle_Update
* id: 1027
* state: fail
* size: 0
* resume: 0
* rcvd: 0
* time: 1328
* reply: HTTP/1.1 401 Unauthorized
Server: nginx
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, private
Date: Tue, 09 Jan 2024 01:23:22 GMT
WWW-Authenticate: Basic realm="Update API"
and again a fail, but this time it connected to the proper place! woohoo! I'm not 100% sure but I assume what happens is the <userinfo> is mime encoded and sent as-is without percent-decoding which sends: dXNlciU0MGdtYWlsLmNvbTpwYXNz decoded: user%40gmail.com:pass and NOT what it should: dXNlckBnbWFpbC5jb206cGFzcw== decoded: user@gmail.com:pass for debug purposes here's the callback alias alias Handle_Update {
var %id = $1 , %BV = $urlget(%id).target
echo -ai2 * url: $urlget(%id).url
echo -ai2 * redirect: $urlget(%id).redirect
echo -ai2 * method: $urlget(%id).method
echo -ai2 * type: $urlget(%id).type
echo -ai2 * target: $urlget(%id).target
echo -ai2 * alias: $urlget(%id).alias
echo -ai2 * id: $urlget(%id).id
echo -ai2 * state: $urlget(%id).state
echo -ai2 * size: $urlget(%id).size
echo -ai2 * resume: $urlget(%id).resume
echo -ai2 * rcvd: $urlget(%id).rcvd
echo -ai2 * time: $urlget(%id).time
echo -ai2 * reply: $urlget(%id).reply
if ($bvar(%BV,0)) { echo -ai2 * Data : $bvar(%BV,1-).text }
}
For now I have scripted a rather sizeable almost complete drop-in replacement for $urlget() called $wget() to do this with raw sockets. I can include if you'd like so you can test noop $urlget() vs noop $wget() as their syntax and usage is virtually identical. Might post it anyways after some more beta testing. May be handy for those on older versions of windows as well that can't use WinINet.
|
|
|
|
|
Joined: Dec 2002
Posts: 5,424
Hoopy frood
|
|
Hoopy frood
Joined: Dec 2002
Posts: 5,424 |
Thanks for your bug report. mIRC v7.67 was released in 2021 :-) Please test this with the latest version of mIRC and let us know if you are still seeing the same issue.
|
|
|
|
|
Joined: Dec 2002
Posts: 252
Fjord artisan
|
|
OP
Fjord artisan
Joined: Dec 2002
Posts: 252 |
Most recent version confirmed. I've setup some tests here to illustrate the issue. After the code snippet some examples with output illustrating the issue will follow. I firmly believe there's a parser issue here when there's multiple @ symbols in a URI, Demonstrated on case 2, to show it not going to the specified host, but rather the second half of the username if it contains an "@".
alias testUrlGet {
if (!$sock(phttpd)) { pseudohttpd }
var %user = user@localhost
var %pass = pass
var %host = localhost
var %path = /some/where
var %query = some=additional&data=here
var %URI = $+(http://,%user,:,%pass,@,%host,%path,?,%query)
echo -s *** Connecting to: %URI
noop $urlget(%URI,gbi,&Bool,process_UrlGetReturn)
}
alias process_UrlGetReturn {
var %id = $1 , %BV = $urlget(%id).target
echo -si2 * url: $urlget(%id).url
echo -si2 * redirect: $urlget(%id).redirect
echo -si2 * method: $urlget(%id).method
echo -si2 * type: $urlget(%id).type
echo -si2 * target: $urlget(%id).target
echo -si2 * alias: $urlget(%id).alias
echo -si2 * id: $urlget(%id).id
echo -si2 * state: $urlget(%id).state
echo -si2 * size: $urlget(%id).size
echo -si2 * resume: $urlget(%id).resume
echo -si2 * rcvd: $urlget(%id).rcvd
echo -si2 * time: $urlget(%id).time
echo -si2 * reply: $urlget(%id).reply
if ($bvar(%BV,0)) { echo -si2 * Data : $bvar(%BV,1-).text }
}
;========================================================================================
; Fake HTTP for testing...
;========================================================================================
alias pseudohttpd {
if (!$window(@PseudoHttpd)) { window -dk @PseudoHttpd -1 -1 640 480 }
if (!$sock(phttpd)) { socklisten phttpd 80 }
phttpd.debug *** Listening on 80
}
alias phttpd.debug { echo -i5 @PseudoHttpd $1- }
on *:socklisten:phttpd: {
var %x = 1 , %sock = phttp. $+ %x
while ($sock($+(phttp.,%x))) { var %x = %x + 1 , %sock = phttp. $+ %x }
sockaccept %sock
}
on *:sockread:phttp.*: {
var %in
if ($sockerr > 0) { return }
while ($sock($sockname).rq) {
sockread -n %in
if (!$sockbr) { break }
if (%in != $null) { phttpd.debug -> $+([,$sockname,]) %in }
if ($regex(%in,/^GET\s/i)) { sockmark $sockname 1 }
if ($regex(%in,/^Authorization\:\sBasic (.*)/i)) { sockmark $sockname $iif($decode($regml(1),m) === user@localhost:pass,3,2) }
}
if ($sock($sockname).mark) {
var %mark = $v1
if (%mark == 3) { var %status = 200 , %stext = OK , %msg = Success! }
else { var %status = 401 , %stext = Unauthorized , %msg = Access to this resource is denied, your client has not supplied the correct authentication. }
bset -t &out 1 $+(HTTP/1.1 %status %stext,$crlf,Date: $asctime($calc($gmt - $daylight),ddd $+ $chr(44) dd mmm yyyy HH:nn:ss ) GMT,$crlf,Server: localhost,$crlf,Host: localhost,$crlf,WWW-Authenticate: basic realm="mIRC",$crlf,Connection: close,$crlf)
sockwrite $sockname &out
bset -ct &out 1 $+(Content-type: text/html,$crlf,$crlf,<h1> %status %stext </h1><hr> %msg)
sockwrite $sockname &out
phttpd.debug <- $+([,$sockname,]) %status
}
}
on *:sockwrite:phttp.*: { if (!$sock($sockname).sq) { sockclose $sockname } }
on *:close:@PseudoHttpd: { sockclose phttp* }
Without any modifications: /testurlget Results:
*** Connecting to: http://user@localhost:pass@localhost/some/where?some=additional&data=here
* url: http://user@localhost:pass@localhost/some/where?some=additional&data=here
* redirect:
* method: get
* type: binvar
* target: &Bool
* alias: process_UrlGetReturn
* id: 1031
* state: fail
* size: 0
* resume: 0
* rcvd: 0
* time: 1000
* reply: HTTP/1.1 401 Unauthorized
Date: Wed, 10 Jan 2024 00:08:19 GMT
Server: localhost
Host: localhost
WWW-Authenticate: basic realm="mIRC"
Connection: close
Content-type: text/html
HTTPD Window:
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
<- [phttp.1] 401
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
-> [phttp.1] Authorization: Basic dXNlciU0MGxvY2FsaG9zdDpwYXNz
<- [phttp.1] 401
Through whatever reason this one half worked! We notice that it connects, tries to get, gets a 401, connects again and tries to auth. somewhere dunno if it's WinINet or in mIRC, it tried to auth with: Encoded: dXNlciU0MGxvY2FsaG9zdDpwYXNz Decoded: user%40localhost:pass so somewhere in this chain it did get percent encoded, but the stored encoded value was never "decoded" before being mimed and sent as a literal... Therefore the base64 is wrong. It gets more strange! close or clear the @PseudoHttpd (which shuts down the fake server, or if you cleared it just clears the old log) Now let's modify the %user on line 4: let's make it user@gmail.com /testurlget Results:
*** Connecting to: http://user@gmail.com:pass@localhost/some/where?some=additional&data=here
* url: http://user@gmail.com:pass@localhost/some/where?some=additional&data=here
* redirect:
* method: get
* type: binvar
* target: &Bool
* alias: process_UrlGetReturn
* id: 1032
* state: fail
* size: 1601
* resume: 0
* rcvd: 0
* time: 468
* reply: HTTP/1.1 404 Not Found
Cross-Origin-Resource-Policy: cross-origin
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Date: Wed, 10 Jan 2024 00:14:53 GMT
Server: sffe
Content-Length: 1601
X-XSS-Protection: 0
HTTPD Window: none! We connected to gmail.com NOT localhost! Maybe we need to percent encode user:pass? @ = character 64, which is 40 in hex, let's use on line 4: user%40localhost maybe somewhere between mirc and wininet the string to base64 encode gets percent-encoded and decoded whatever.. let's just try it out... /testurlget Results:
*** Connecting to: http://user%40localhost:pass@localhost/some/where?some=additional&data=here
* url: http://user%40localhost:pass@localhost/some/where?some=additional&data=here
* redirect:
* method: get
* type: binvar
* target: &Bool
* alias: process_UrlGetReturn
* id: 1033
* state: fail
* size: 0
* resume: 0
* rcvd: 0
* time: 2734
* reply: HTTP/1.1 401 Unauthorized
Date: Wed, 10 Jan 2024 00:17:17 GMT
Server: localhost
Host: localhost
WWW-Authenticate: basic realm="mIRC"
Connection: close
Content-type: text/html
HTTPD Window:
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
<- [phttp.1] 401
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
-> [phttp.1] Authorization: Basic dXNlciU0MGxvY2FsaG9zdDpwYXNz
<- [phttp.1] 401
we got back an auth! what is it? dXNlciU0MGxvY2FsaG9zdDpwYXNz == user%40localhost:pass ok so we're back to where we started...
|
|
|
|
|
Joined: Dec 2002
Posts: 5,424
Hoopy frood
|
|
Hoopy frood
Joined: Dec 2002
Posts: 5,424 |
Thanks for confirming. As far as I can tell, the reason for this is that RFC3986 does not permit using an @ sign (or any of the other reserved characters that can delimit a URI) in either the username or the password. Both need to be separately percent-encoded by the scripter, and then separated by a ":" colon, before being used in the URI passed to $urlget(), in order for the URI to be parsed correctly.
In this case, it looks like mIRC needs to assume that the scripter always percent-encodes the username and password, and it should therefore always percent-decode the username and password after it has parsed the URI, before passing them to WinINET. That should resolve the issue. This change will be in the next beta.
The only possible backward-compatibility issue is if a scripter is already using username:password and one of these contains a combination of characters that looks percent-encoded. Hmm.
It may also be worth adding a percent encoding/decoding switch to the $encode()/$decode() identifiers in this case.
Note: RFC3986 indicates that using user:pass in the URI is deprecated, however since mIRC is parsing the URI independently of WinINET, it should continue to work in future.
Last edited by Khaled; 11/01/24 05:12 PM.
|
|
|
|
|
