mIRC Home    About    Download    Register    News    Help

Print Thread
Joined: Dec 2002
Posts: 252
T
Talon Offline OP
Fjord artisan
OP Offline
Fjord artisan
T
Joined: Dec 2002
Posts: 252
I'm a few updates behind, currently testing on mIRC 7.67, but I don't see any changes to urlget besides the new "e" switch, so Unsure if this affects any newer versions, but it does for anything 7.67 and below.

I'm using a web-api that requires basic auth. The "username" is my email, and that's where the issue occurs. I'll do a mock-example below to illustrate, as I can't exactly paste a working example without including my userinfo. so We'll just use "localhost"

I first tried something along the lines of:
http://user@gmail.com:pass@localhost/?test=true

NOTE: link works fine in all browsers tested, "URL bar" aslo. 200 OK and some return data from the API call.

Code
noop $urlget(%URI,gbi,&Update,Handle_Update)

results:
Code
* url: (Redacted)
* redirect:
* method: get
* type: binvar
* target: &Update
* alias: Handle_Update
* id: 1026
* state: fail
* size: 1622
* resume: 0
* rcvd: 0
* time: 484
* reply: HTTP/1.1 404 Not Found
  Cross-Origin-Resource-Policy: cross-origin
  Content-Type: text/html; charset=UTF-8
  X-Content-Type-Options: nosniff
  Date: Tue, 09 Jan 2024 01:20:48 GMT
  Server: sffe
  Content-Length: 1622
  X-XSS-Protection: 0

This totally breaks and connects to gmail.com, makes my request, and naturally gets a 404 not found...
Somehow ignores the beginning and afterward of the URI extracting only the scheme (http) and the host to be "gmail.com" losing the actual host+port, yet somehow manages to retain the proper path to request...


Doing some reading: according to rfc3986 A URI is composed of 5 parts:
<scheme>://<authority>/<path>?<query>#<fragment>

These "tokens" such as <authority> have subsections to be broken down even further such as:
<userinfo>@<host>:<port>

each "token" must follow layout rules, there's reserved characters and unreserved characters. all "reserved" characters should be replaced by it's corresponding percent-encoded equivalent


So I decided to try this URI

http://user%40gmail.com:pass@localhost/?test=true

NOTE: link works fine in all browsers tested, "URL bar" aslo. 200 OK and some return data from the API call.

Code
noop $urlget(%URI,gbi,&Update,Handle_Update)

results:
Code
* url: (Redacted)
* redirect:
* method: get
* type: binvar
* target: &Update
* alias: Handle_Update
* id: 1027
* state: fail
* size: 0
* resume: 0
* rcvd: 0
* time: 1328
* reply: HTTP/1.1 401 Unauthorized
   Server: nginx
   Content-Type: text/plain; charset=UTF-8
   Transfer-Encoding: chunked
   Connection: keep-alive
   Cache-Control: no-cache, private
   Date: Tue, 09 Jan 2024 01:23:22 GMT
   WWW-Authenticate: Basic realm="Update API"

and again a fail, but this time it connected to the proper place! woohoo!

I'm not 100% sure but I assume what happens is the <userinfo> is mime encoded and sent as-is without percent-decoding which sends:
dXNlciU0MGdtYWlsLmNvbTpwYXNz
decoded: user%40gmail.com:pass

and NOT what it should:

dXNlckBnbWFpbC5jb206cGFzcw==
decoded: user@gmail.com:pass

for debug purposes here's the callback alias
Code
alias Handle_Update {
  var %id = $1 , %BV = $urlget(%id).target

  echo -ai2 * url: $urlget(%id).url
  echo -ai2 * redirect: $urlget(%id).redirect
  echo -ai2 * method: $urlget(%id).method
  echo -ai2 * type: $urlget(%id).type
  echo -ai2 * target: $urlget(%id).target
  echo -ai2 * alias: $urlget(%id).alias
  echo -ai2 * id: $urlget(%id).id
  echo -ai2 * state: $urlget(%id).state
  echo -ai2 * size: $urlget(%id).size
  echo -ai2 * resume: $urlget(%id).resume
  echo -ai2 * rcvd: $urlget(%id).rcvd
  echo -ai2 * time: $urlget(%id).time
  echo -ai2 * reply: $urlget(%id).reply

  if ($bvar(%BV,0)) { echo -ai2 * Data : $bvar(%BV,1-).text }
}

For now I have scripted a rather sizeable almost complete drop-in replacement for $urlget() called $wget() to do this with raw sockets. I can include if you'd like so you can test noop $urlget() vs noop $wget() as their syntax and usage is virtually identical. Might post it anyways after some more beta testing. May be handy for those on older versions of windows as well that can't use WinINet.

Joined: Dec 2002
Posts: 5,404
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 5,404
Thanks for your bug report. mIRC v7.67 was released in 2021 :-) Please test this with the latest version of mIRC and let us know if you are still seeing the same issue.

Joined: Dec 2002
Posts: 252
T
Talon Offline OP
Fjord artisan
OP Offline
Fjord artisan
T
Joined: Dec 2002
Posts: 252
Most recent version confirmed.

I've setup some tests here to illustrate the issue. After the code snippet some examples with output illustrating the issue will follow.

I firmly believe there's a parser issue here when there's multiple @ symbols in a URI, Demonstrated on case 2, to show it not going to the specified host, but rather the second half of the username if it contains an "@".

Code
alias testUrlGet {
  if (!$sock(phttpd)) { pseudohttpd }
  var %user = user@localhost
  var %pass = pass
  var %host = localhost
  var %path = /some/where
  var %query = some=additional&data=here
  var %URI = $+(http://,%user,:,%pass,@,%host,%path,?,%query)

  echo -s *** Connecting to: %URI
  noop $urlget(%URI,gbi,&Bool,process_UrlGetReturn)
}
alias process_UrlGetReturn {
  var %id = $1 , %BV = $urlget(%id).target

  echo -si2 * url: $urlget(%id).url
  echo -si2 * redirect: $urlget(%id).redirect
  echo -si2 * method: $urlget(%id).method
  echo -si2 * type: $urlget(%id).type
  echo -si2 * target: $urlget(%id).target
  echo -si2 * alias: $urlget(%id).alias
  echo -si2 * id: $urlget(%id).id
  echo -si2 * state: $urlget(%id).state
  echo -si2 * size: $urlget(%id).size
  echo -si2 * resume: $urlget(%id).resume
  echo -si2 * rcvd: $urlget(%id).rcvd
  echo -si2 * time: $urlget(%id).time
  echo -si2 * reply: $urlget(%id).reply

  if ($bvar(%BV,0)) { echo -si2 * Data : $bvar(%BV,1-).text }
}

;========================================================================================
; Fake HTTP for testing...
;========================================================================================

alias pseudohttpd { 
  if (!$window(@PseudoHttpd)) { window -dk @PseudoHttpd -1 -1 640 480 }
  if (!$sock(phttpd)) { socklisten phttpd 80 }
  phttpd.debug *** Listening on 80
}
alias phttpd.debug { echo -i5 @PseudoHttpd $1- }

on *:socklisten:phttpd: {
  var %x = 1 , %sock = phttp. $+ %x
  while ($sock($+(phttp.,%x))) { var %x = %x + 1 , %sock = phttp. $+ %x }
  sockaccept %sock
}
on *:sockread:phttp.*: {
  var %in
  if ($sockerr > 0) { return }
  while ($sock($sockname).rq) {
    sockread -n %in
    if (!$sockbr) { break }
    if (%in != $null) { phttpd.debug -> $+([,$sockname,]) %in }
    if ($regex(%in,/^GET\s/i)) { sockmark $sockname 1 }
    if ($regex(%in,/^Authorization\:\sBasic (.*)/i)) { sockmark $sockname $iif($decode($regml(1),m) === user@localhost:pass,3,2) }
  }
  if ($sock($sockname).mark) {
    var %mark = $v1
    if (%mark == 3) { var %status = 200 , %stext = OK , %msg = Success! }
    else { var %status = 401 , %stext = Unauthorized , %msg = Access to this resource is denied, your client has not supplied the correct authentication. }
    bset -t &out 1 $+(HTTP/1.1 %status %stext,$crlf,Date: $asctime($calc($gmt - $daylight),ddd $+ $chr(44) dd mmm yyyy HH:nn:ss ) GMT,$crlf,Server: localhost,$crlf,Host: localhost,$crlf,WWW-Authenticate: basic realm="mIRC",$crlf,Connection: close,$crlf)
    sockwrite $sockname &out
    bset -ct &out 1 $+(Content-type: text/html,$crlf,$crlf,<h1> %status %stext </h1><hr> %msg)
    sockwrite $sockname &out
    phttpd.debug <- $+([,$sockname,]) %status
  }
}
on *:sockwrite:phttp.*: { if (!$sock($sockname).sq) { sockclose $sockname } }
on *:close:@PseudoHttpd: { sockclose phttp* }

Without any modifications:
/testurlget

Results:
Code
*** Connecting to: http://user@localhost:pass@localhost/some/where?some=additional&data=here
* url: http://user@localhost:pass@localhost/some/where?some=additional&data=here
* redirect:
* method: get
* type: binvar
* target: &Bool
* alias: process_UrlGetReturn
* id: 1031
* state: fail
* size: 0
* resume: 0
* rcvd: 0
* time: 1000
* reply: HTTP/1.1 401 Unauthorized
Date: Wed, 10 Jan 2024 00:08:19 GMT
Server: localhost
Host: localhost
WWW-Authenticate: basic realm="mIRC"
Connection: close
Content-type: text/html

HTTPD Window:
Code
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
<- [phttp.1] 401
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
-> [phttp.1] Authorization: Basic dXNlciU0MGxvY2FsaG9zdDpwYXNz
<- [phttp.1] 401

Through whatever reason this one half worked!
We notice that it connects, tries to get, gets a 401, connects again and tries to auth.

somewhere dunno if it's WinINet or in mIRC, it tried to auth with:
Encoded: dXNlciU0MGxvY2FsaG9zdDpwYXNz
Decoded: user%40localhost:pass

so somewhere in this chain it did get percent encoded, but the stored encoded value was never "decoded" before being mimed and sent as a literal... Therefore the base64 is wrong.




It gets more strange!

close or clear the @PseudoHttpd (which shuts down the fake server, or if you cleared it just clears the old log)
Now let's modify the %user on line 4: let's make it user@gmail.com
/testurlget

Results:
Code
*** Connecting to: http://user@gmail.com:pass@localhost/some/where?some=additional&data=here
* url: http://user@gmail.com:pass@localhost/some/where?some=additional&data=here
* redirect:
* method: get
* type: binvar
* target: &Bool
* alias: process_UrlGetReturn
* id: 1032
* state: fail
* size: 1601
* resume: 0
* rcvd: 0
* time: 468
* reply: HTTP/1.1 404 Not Found
Cross-Origin-Resource-Policy: cross-origin
Content-Type: text/html; charset=UTF-8
X-Content-Type-Options: nosniff
Date: Wed, 10 Jan 2024 00:14:53 GMT
Server: sffe
Content-Length: 1601
X-XSS-Protection: 0

HTTPD Window:
Code
none! We connected to gmail.com NOT localhost!

Maybe we need to percent encode user:pass? @ = character 64, which is 40 in hex, let's use on line 4: user%40localhost maybe somewhere between mirc and wininet the string to base64 encode gets percent-encoded and decoded whatever.. let's just try it out...

/testurlget

Results:
Code
*** Connecting to: http://user%40localhost:pass@localhost/some/where?some=additional&data=here
* url: http://user%40localhost:pass@localhost/some/where?some=additional&data=here
* redirect:
* method: get
* type: binvar
* target: &Bool
* alias: process_UrlGetReturn
* id: 1033
* state: fail
* size: 0
* resume: 0
* rcvd: 0
* time: 2734
* reply: HTTP/1.1 401 Unauthorized
Date: Wed, 10 Jan 2024 00:17:17 GMT
Server: localhost
Host: localhost
WWW-Authenticate: basic realm="mIRC"
Connection: close
Content-type: text/html

HTTPD Window:
Code
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
<- [phttp.1] 401
-> [phttp.1] GET /some/where?some=additional&data=here HTTP/1.1
-> [phttp.1] Accept: */*
-> [phttp.1] Accept-Encoding: gzip, deflate
-> [phttp.1] User-Agent: mIRC
-> [phttp.1] Host: localhost
-> [phttp.1] Connection: Keep-Alive
-> [phttp.1] Cache-Control: no-cache
-> [phttp.1] Authorization: Basic dXNlciU0MGxvY2FsaG9zdDpwYXNz
<- [phttp.1] 401

we got back an auth! what is it?
dXNlciU0MGxvY2FsaG9zdDpwYXNz == user%40localhost:pass

ok so we're back to where we started...

Joined: Dec 2002
Posts: 5,404
Hoopy frood
Offline
Hoopy frood
Joined: Dec 2002
Posts: 5,404
Thanks for confirming. As far as I can tell, the reason for this is that RFC3986 does not permit using an @ sign (or any of the other reserved characters that can delimit a URI) in either the username or the password. Both need to be separately percent-encoded by the scripter, and then separated by a ":" colon, before being used in the URI passed to $urlget(), in order for the URI to be parsed correctly.

In this case, it looks like mIRC needs to assume that the scripter always percent-encodes the username and password, and it should therefore always percent-decode the username and password after it has parsed the URI, before passing them to WinINET. That should resolve the issue. This change will be in the next beta.

The only possible backward-compatibility issue is if a scripter is already using username:password and one of these contains a combination of characters that looks percent-encoded. Hmm.

It may also be worth adding a percent encoding/decoding switch to the $encode()/$decode() identifiers in this case.

Note: RFC3986 indicates that using user:pass in the URI is deprecated, however since mIRC is parsing the URI independently of WinINET, it should continue to work in future.

Last edited by Khaled; 11/01/24 05:12 PM.

Link Copied to Clipboard