mIRC Home    About    Download    Register    News    Help

Page 2 of 2 < 1 2
Topic Options
#265202 - 16/03/19 09:44 AM Re: $urlget bugs / discussion [Re: SykO]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4445
Loc: London, UK
Originally Posted By: SykO
urs:pass seems to be the problem. I currently tested sending Authorization header as follows:
Code:
bset -t &header 1 Authorization: Basic $encode(usr:pass,m)
$urlget(http://localhost:port/,gf,&target,noop,&header)

and it works

Is anyone else experiencing an issue with using user:password in $urlget()? I have tried to reproduce SykO's issue but in my tests, $urlget() is correctly logging into password protected folders.

Top
#265203 - 16/03/19 12:35 PM Re: $urlget bugs / discussion [Re: Khaled]
Loki12583 Online
Hoopy frood

Registered: 22/01/04
Posts: 1268
I have tried user:pass even on a url which returns a redirect and still had no problem

Top
#265204 - 16/03/19 05:58 PM Re: $urlget bugs / discussion [Re: Loki12583]
SykO Offline
Ameglian cow

Registered: 17/07/14
Posts: 27
Might just be on my end, but I do find it weird that using the header directly works.

My OS:
Edition: Windows 10 Education
Version: 1803
OS build: 17134.648

Top
#265205 - 17/03/19 12:28 PM Re: $urlget bugs / discussion [Re: SykO]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4445
Loc: London, UK
Originally Posted By: SykO
Might just be on my end, but I do find it weird that using the header directly works.

My OS:
Edition: Windows 10 Education
Version: 1803
OS build: 17134.648

Hmm. If it isn't working for you, it will likely not work for others as well.

If you use SmartSniff to monitor http packets, can you see if mIRC is sending the Basic Authorization header? This will only be visible with http and not https. For https, you would need to install something like Fiddler (which installs a root certificate to enable https decryption).

Also, can you let me know the type of server you are logging into? Is it linux/windows/apache/cpanel/amazon/etc.? I might be able to set up a similar service to test it out.

Top
#265208 - 17/03/19 03:40 PM Re: $urlget bugs / discussion [Re: Khaled]
0nslaught Offline
Ameglian cow

Registered: 11/08/04
Posts: 27
Originally Posted By: Khaled
Quote:
Add a switch to allow redirection to be followed with depth, -dN with N = 0 for infinite redirection, or N > 0 for N redirection

Currently, $urlget() gives up after 10 redirects. It does not detect cyclical redirections. As far as I know, most browsers have a redirect limit of between 10 to 20 redirects. Instead of adding an option for this, I would rather make it behave in a standard way. I could increase the limit to 20 but 10 seems reasonable? I would not want to allow infinite redirects.


How about a switch to disable auto-redirects?

Top
#265388 - 13/04/19 01:01 AM Re: $urlget bugs / discussion [Re: Loki12583]
Loki12583 Online
Hoopy frood

Registered: 22/01/04
Posts: 1268
$urlget() does not process relative redirects properly. It seems to simply append the location to "scheme://domain/". It should instead construct an effective uri as described here:

https://tools.ietf.org/html/rfc7231#section-7.1.2
https://tools.ietf.org/html/rfc7230#section-5.5

Bug 1: $urlget() reverts to default port (80/443) when processing relative redirect from non-default port: http://localhost:8080/

Bug 2: $urlget() does not construct effective request uri. A request to "/relative/sub" with a redirect "./sub2" should create a new request for "/relative/sub2"

Bug 3: $urlget() adds an extra "/" with relative redirect (same as bug 2 really, simply appending instead of constructing the effective uri)

I have tested $urlget() against an actual nginx server using relative redirects and against a local mirc implementation (below). Once the local servers are listening Chrome dev tools can be used to cross reference behavior: http://localhost:8080/relative

Quote:
Request on 8080
> GET /relative HTTP/1.1
> Accept: */*
> Accept-Encoding: gzip, deflate
> User-Agent: mIRC
> Host: localhost:8080
> Connection: Keep-Alive
> Cache-Control: no-cache
< HTTP/1.1 302 Temporary Redirect
< Location: /relative/sub
< Connection: close
< Content-Length: 0
<
-
Request on 80
> GET //relative/sub HTTP/1.1
> Accept: */*
> Accept-Encoding: gzip, deflate
> User-Agent: mIRC
> Host: localhost
> Connection: Keep-Alive
> Cache-Control: no-cache
< HTTP/1.1 302 Temporary Redirect
< Location: ./sub2
< Connection: close
< Content-Length: 0
<
-
Request on 80
> GET /./sub2 HTTP/1.1
> Accept: */*
> Accept-Encoding: gzip, deflate
> User-Agent: mIRC
> Host: localhost
> Connection: Keep-Alive
> Cache-Control: no-cache
< HTTP/1.1 200 OK
< Connection: close
< Content-Length: 7
< failure
-
url      http://localhost:8080/relative
redirect http://localhost/./sub2
method   get
type     binvar
target   &target
alias    urlget.callback
id       1066
state    ok
size     7
resume   0
rcvd     7
time     296
reply    HTTP/1.1 200 OKConnection: closeContent-Length: 7
response failure


Code:
alias urlget.test {
  urlget.listen
  
  var %url = $iif($1,$1,http://localhost:8080/relative)
  var %id = $urlget(%url,gb,&target,urlget.callback,)
}

alias urlget.callback {
  var %id = $1
  
  echo -ag -
  echo -agi9 url      $urlget(%id).url
  echo -agi9 redirect $urlget(%id).redirect
  echo -agi9 method   $urlget(%id).method
  echo -agi9 type     $urlget(%id).type
  echo -agi9 target   $urlget(%id).target
  echo -agi9 alias    $urlget(%id).alias
  echo -agi9 id       $urlget(%id).id
  echo -agi9 state    $urlget(%id).state
  echo -agi9 size     $urlget(%id).size
  echo -agi9 resume   $urlget(%id).resume
  echo -agi9 rcvd     $urlget(%id).rcvd
  echo -agi9 time     $urlget(%id).time
  echo -agi9 reply    $urlget(%id).reply

  if ($urlget(%id).type == binvar) && ($bvar($urlget(%id).target,0)) {
    echo -agi9 response $bvar($urlget(%id).target,1-3000).text
  }
}

alias urlget.listen {
  if (!$sock(urlget.listen)) socklisten -d 127.0.0.1 urlget.listen 80
  if (!$sock(urlget.listen2)) socklisten -d 127.0.0.1 urlget.listen2 8080
}

on *:socklisten:urlget.listen*:{
  echo -ag -
  echo 4 -ag Request on $iif($sockname == urlget.listen,80,8080)
  var %sockname = urlget.client. $+ $ticks
  if ($sock(%sockname)) return

  sockaccept %sockname
}

on *:sockread:urlget.client.*:{
  var %header

  if (!$sock($sockname).mark) {
    sockread %header
    while (%header != $null) {
      if ($regex(%header,/GET (\S+)/)) {
        var %request = $regml(1)
      }
      echo 3 -ag > %header
      if (sub isin %header) %x = $true
      if ($regex(%header,Content-Length: (\d+))) {
        hadd -m $sockname content-length $regml(1)
      }
      sockread %header
    }
    if ($sockbr) sockmark $sockname $true
  }

  if ($sock($sockname).mark) && ($sock($sockname).rq) {
    sockread &read

    while ($sockbr) {
      hinc $sockname content-read $sockbr
      echo 6 -agi2 > $bvar(&read,1-3000).text

      sockread &read
    }
  }

  if ($hget($sockname,content-length) == 0) || ($v1 == $hget($sockname,content-read)) {
    var %redirect, %data

    if      (/relative/sub2 == %request)  %data = success
    else if (/relative/sub isin %request) %redirect = ./sub2
    else if (/relative isin %request)     %redirect = /relative/sub
    else %data = failure

    noop $socket.respond($sockname,%data,%redirect)
  }
}

alias -l sockwrite {
  echo 12 -ag < $3-
  sockwrite $1-
}

alias -l socket.respond {
  var %sockname = $$1, %data = $2, %redirect = $3
  if ($3) sockwrite -n %sockname HTTP/1.1 302 Temporary Redirect
  else    sockwrite -n %sockname HTTP/1.1 200 OK
  if ($3) sockwrite -n %sockname Location: %redirect
  sockwrite -n %sockname Connection: close
  sockwrite -n %sockname Content-Length: $len(%data)
  sockwrite -n %sockname $+($crlf,%data)
}

Top
#265409 - 16/04/19 04:24 PM Re: $urlget bugs / discussion [Re: Loki12583]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4445
Loc: London, UK
Quote:
$urlget() does not process relative redirects properly.

Thanks for testing these out.

Fixing these has turned out to be more complicated than I expected. I am trying to use standard windows APIs for URL parsing but their limitation means that I need to either write my own custom URL parsing routine, which is not advisable, or use a URL parsing library, of which many exist at differing levels of sophistication and size, ranging from tens of lines to thousands of lines of code. In addition, it's hard to know how well-tested these libraries are. Some of them implement recent security fixes.

As for relative redirects, again, I really should be using rfc-compatible, established, tested code instead of writing my own. I will need to look into this.

Top
#265427 - 19/04/19 06:44 PM Re: $urlget bugs / discussion [Re: Loki12583]
Khaled Offline


Planetary brain

Registered: 04/12/02
Posts: 4445
Loc: London, UK
Thanks for the test script. The latest beta implements several changes to $urlget() that should fix the issues mentioned in your post.

Top
Page 2 of 2 < 1 2