|
Joined: Oct 2003
Posts: 3,918
Hoopy frood
|
OP
Hoopy frood
Joined: Oct 2003
Posts: 3,918 |
Given the ubiquity of HTTP(S) out there today, especially the number of useful HTTP-only APIs, it would be great if mIRC had basic HTTP client abstractions to avoid the complexity (and limitations) of manually writing /sock code to send HTTP requests. The following is one of many proposals for how the commands might work. There might be existing scripts that could be used for reference, but here's a simplistic API: Synchronous identifier mode (easy-mode):
; A simplistic SYNCHRONOUS HTTP request function. Returns the
; status code and body, but does not provide access to headers.
; For complex HTTP requests, use the asynchronous function.
;
; Returns <status code> [response body]
$httpreq(<method>, <full URI>, [biohuN], [body], [&outbin],
[<Header: value>, ...])
Switches:
b: indicates that a body is provided.
i: indicates that the body token is a binvar.
o: indicates that the outbin binvar is provided to store output.
In this case, $httpreq returns ONLY the status code.
h: indicates that a list of "Header: value" parameters are provided.
uN: timeout the request after N seconds if there is no response.
(XXX: support decimals or make this milliseconds?)
Example:
//echo -a Response: $httpreq(GET, http://example.org/res?q=foo, h,
Authorization: my-api-token)
; echos: Response: 200 SomeDataHere
Asynchronous session-based evented mode:
; Opens/closes an HTTP session.
; No error if the handle is already open.
/httpopen <handle>
/httpclose <handle>
; Starts a request but doesn't send it.
; Error if /httpstart was executed without matching /httpsend.
/httpstart [-os] <handle> <method> <full URI>
-o: open the handle if not already open
-s: call /httpsend immediately (no body/headers)
; Adds a header to a request
; Error if no pending /httpstart transaction exists.
/httpheader [-b] <handle> <header> <value>
-b: the value parameter is a binvar.
; Adds a body to a request
; Error if no pending /httpstart transaction exists.
/httpbody [-b] <handle> <body|&binvar>
-b: the body parameter is a binvar.
; Sends the last built request on handle
; Error if no pending /httpstart transaction exists.
/httpsend <handle>
; Handles a response from a request on a session
on HTTPRESPONSE:<handle>:<uri matchtext>:
<comma separated matchtext statuscodes?>:/commands
The event would expose a few identifiers:
$httphandle - the handle of the session
$httpuri - the URI of the response (matches the URI of the
resource we called in /httpstart)
$httpstatus - the status code
$httpheader(N/Name) - the Nth header in the response or the
header named Name if text is provided.
$httpbody - the response body
Possibly also /httpread <%var|&binvar> which would copy the
body into a var/binvar.
Full example usage:
alias makereq {
httpstart -o mysession POST http://example.org/posts/create
httpheader mysession Authorization: my-api-token
httpbody mysession The body to send
httpsend mysession
}
; Handle initial success response
on *:HTTPRESPONSE:mysession:*/posts/create:200: {
; more correct to use 2* as status code match
echo -a Success! Creation data: $httpbody
; I can now make a follow-up request using the same session
; using any stored cookie data.
httpstart $httphandle GET $httpheader(Post-URI)
httpheader $httphandle Authorization: my-api-token
httpsend $httphandle
}
; Handle follow-up request
on *:HTTPRESPONSE:mysession:*:2*: {
echo -a Post body: $httpbody
}
; Handle failures
on *:HTTPRESPONSE:mysession:*:4*,5*: { ; handle 4xx 5xx responses
echo -a Error occurred when retrieving $httpuri
}
Note that the session based mode allows for important useful benefits, namely: 1. It can use keep-alive connections to efficiently make multiple requests. 2. It would follow redirects (3xx responses), which are non-trivial to handle manually. 3. It would (optionally?) keep track of Cookie data. This is a must for many "login-to-post" scraper scripts out there, which send down session IDs for follow up authentication. Currently, manually scripting socket code to interact with these kinds of websites is extremely complex. I could see this being turned off, but if this is optional it should be on by default. 4. It would transparently handle HTTPS connections using mIRC's already existing SSL libraries and certificate management experience. There might be better API choices here. It's possible to merge the session concept into the identifier mode too. Ultimately the important part is simplifying the overhead in crafting an HTTP request. The use case would be scripts making requests to APIs like Google, IMDb, GitHub, Dropbox, IFTT, etc., or even just their own scripted services to check for script updates and download them. A lot of these should be simple one-liners (or 3-4) but are much more complex due to the manual socket scripting that is currently required-- not to mention out of reach for typical users. HTTP abstractions would allow less advanced scripters to interact with simple HTTP APIs. Note that this doesn't really solve the remaining problem of processing the response data, but it's a good first step. I suppose if this were implemented, it would be great to have a follow-up set of abstractions for JSON processing, like, say a $json() identifier that could pull data out using some query language a la JSONPath or jq.
- argv[0] on EFnet #mIRC - "Life is a pointer to an integer without a cast"
|
|
|
|
Joined: Jul 2006
Posts: 4,222
Hoopy frood
|
Hoopy frood
Joined: Jul 2006
Posts: 4,222 |
Very good idea, I wanted to suggest that, can you remove the [code] tag which makes it unreadable though?
#mircscripting @ irc.swiftirc.net == the best mIRC help channel
|
|
|
|
Joined: Oct 2003
Posts: 3,918
Hoopy frood
|
OP
Hoopy frood
Joined: Oct 2003
Posts: 3,918 |
I wrapped long lines in the code blocks. Should be easier to follow now.
- argv[0] on EFnet #mIRC - "Life is a pointer to an integer without a cast"
|
|
|
|
Joined: Jul 2006
Posts: 4,222
Hoopy frood
|
Hoopy frood
Joined: Jul 2006
Posts: 4,222 |
Ok so, as far as the synchronous mode is concerned, I think it's pretty much perfect as it is, maybe an optional switch uN, to abort the request if it takes too long, in order not to lock mIRC ($httpreq() returning $timeout). For the asynchronous mode, you're are missing a parameter for the protocol itself (well, synchronous mode also suffer this problem), I don't think enforcing http 1.1 is a good idea (that could change to a greater version anyway), we should be able to specify 1.0. For the session, I rather think that by default, it should be disabled, that we should be able to enable it with a switch in /httpstart, sounds better imo, but can't really tell why. For the Json part, well something built-in would be better, but the COM script by Sreject (Froggie in here) is more than good enough ; Error if /httpstart was executed without matching /httpsend. I don't understand this, did you mean an error if /httpsend is used without /httpstart?
#mircscripting @ irc.swiftirc.net == the best mIRC help channel
|
|
|
|
Joined: Oct 2003
Posts: 3,918
Hoopy frood
|
OP
Hoopy frood
Joined: Oct 2003
Posts: 3,918 |
For the asynchronous mode, you're are missing a parameter for the protocol itself (well, synchronous mode also suffer this problem), I don't think enforcing http 1.1 is a good idea (that could change to a greater version anyway), we should be able to specify 1.0. Not sure I agree here. Enforcing 1.1 is a super great idea, because it's the current standard, and almost all user agents use it. It's a widely accepted default. I'd be surprised if you can find a single server that only supports 1.0 requests. I don't see any reason to ever need to send a 1.0 request in 2015. HTTP/2 is already being built out (your browser probably already supports it), HTTP/1.0 today would be like using HTML3 to build web pages. Forcing 1.0 would break connection keep-alives and plenty of other stuff. In general, there's no need for the client to control this. Remember, the whole point of this is that mIRC is abstracting the protocol. The user typically shouldn't care what protocol version is being used. The idea is that mIRC could eventually upgrade its internals to use HTTP/2 without users even knowing or caring, when it becomes the standard, assuming the protocol fundamentals remain compatible ( which they likely will). Of course, you could always add a switch for this later if there was a good reason for it, but I've never needed to drop down to 1.0 for any HTTP clients I've used... ever. I like the idea for a timeout in synchronous mode, I'll add that. I don't understand this, did you mean an error if /httpsend is used without /httpstart? This is poorly written-- I mean /httpstart should fail if there is a pending transaction, i.e., you cannot call /httpstart twice in a row on the same session. For the session, I rather think that by default, it should be disabled, that we should be able to enable it with a switch in /httpstart, sounds better imo, but can't really tell why. Are you suggesting that session handles would be optional and by default it would just use some "main session"? I suppose this could work for sending requests, but scripting the response events would be pretty much impossible without some kind of handle ID to associate your events with. You wouldn't want script Bar triggering for a request sent by script Foo because they were (ab)using the main session. With the -o switch, handles are pretty lightweight and easy enough to use.
- argv[0] on EFnet #mIRC - "Life is a pointer to an integer without a cast"
|
|
|
|
Joined: Jul 2006
Posts: 4,222
Hoopy frood
|
Hoopy frood
Joined: Jul 2006
Posts: 4,222 |
Well, I'm not saying we should force 1.0, but some servers will default to gzip encoding in the response on 1.1, while they wouldn't on 1.0, which makes thing complicated in mIRC scripting, at least I recall some tricky situation like this, I agree otherwise.
As far as session goes, I misunderstood what you meant with it, you didn't really describe it either (poorly written eh), I was talking about forwarding (keeping track) cookies from a request to another, and perhaps following redirection, which I think should be optional then. If a session means an handle, and it looks like it, then it's all good.
N in millisecond for uN is probably better, but it would be good enough as second
Last edited by Wims; 06/11/15 04:34 AM.
#mircscripting @ irc.swiftirc.net == the best mIRC help channel
|
|
|
|
Joined: Oct 2003
Posts: 3,918
Hoopy frood
|
OP
Hoopy frood
Joined: Oct 2003
Posts: 3,918 |
Well, I'm not saying we should force 1.0, but some servers will default to gzip encoding in the response on 1.1, while they wouldn't on 1.0, which makes thing complicated in mIRC scripting, at least I recall some tricky situation like this, I agree otherwise. Ah, well this is another reason HTTP abstractions shine-- a Transfer-Encoding of gzip should be handled transparently by the user-agent (mIRC), and should be invisible to the user. In other words, scripts should be able to parse gzip encoded responses without even needing to know that they are gzipped, because it should be unpacked by mIRC prior to getting to the user. This is what almost all HTTP abstraction libraries do in other languages, and likely what WinHTTP does (or can do) if that library were to be used to implement this. We would actually WANT mIRC to use gzip compression whenever possible, since it would improve network transfer speeds, so I would actually imagine that mIRC would have gzip on unless otherwise specified. I actually forgot to add this to benefits list. Incidentally you can disable gzip transfer on most servers, even with HTTP/1.1, by providing an Accept-Encoding header of 'identity' or empty string. Any server that handles HTTP/1.1 (read: virtually every server) will understand this. But again, a client shouldn't be doing this unless there is a good reason to force the user-agent to disable compression across the wire, since it doesn't represent the data that the user deals with. But yea, if you wanted to forcibly disable gzip, this is how you should be doing it, not by dropping down to HTTP/1.0. And to answer the session/handle question: yes, a handle refers to a handle to a session. Handle == session, pretty much. I might have alternated the terms a few times, and admittedly, I considered renaming "handle" to "session" midway through the proposal. It could be, if that simplifies things.
- argv[0] on EFnet #mIRC - "Life is a pointer to an integer without a cast"
|
|
|
|
Joined: Jul 2006
Posts: 4,222
Hoopy frood
|
Hoopy frood
Joined: Jul 2006
Posts: 4,222 |
Ah ok, I didn't think you were suggesting mIRC should handle the gzip encoding. Abstracting http doesn't necessarily mean the response is 'decoded' for you, it could be served as is and we would still call it abstracting, but yeah, handling gzip encoding for us would be awesome. You're right for the Accept-Encoding header, but I remembered using some broken http servers where using 1.0 was a workaround, but I guess that's irrelevant to the abstraction and it should use 1.1.
#mircscripting @ irc.swiftirc.net == the best mIRC help channel
|
|
|
|
Joined: Apr 2010
Posts: 969
Hoopy frood
|
Hoopy frood
Joined: Apr 2010
Posts: 969 |
As far as native libraries; I'd suggest WinINET over WinHTTP. Its a superset of WinHTTP but is scoped to the window user's profile, where as WinHTTP is not. WinHTTP is meant for services or similar that may need to run as a super user/admin With that said, both WinHTTP and WinINET provide full support of the HTTP/1.1 standard, with a few 'ease of use' abstractions such as for ssl(HTTPS) requests, auto (de)compression of gzip, PAK and various others, and local caching. -- As far as feed back: 1: Neither method of performing an HTTP request should use client-side caching unless its a subsequent 'session/keep-alive' request, or at the request of the scripter. I know WinHTTP/INET has a caching system, but cannot remember if its opt-in or opt-out. 2: Both methods should support specifying, at the least, a connection duration timeout. That is, if the connect remains open for longer than the specified duration it should be closed and an error raised. 3: If the abstraction includes redirect handling: - There should be a way to opt-out of such
- There should be a switch to set a limit on the number of redirects to follow with a way to reset the 'count'
4: Synchronous connections should be immediately terminated if the user inputs the ctrl+break key combination. 5: Synchronous requests should result in an error if the body would exceed a (smallish) buffer-size limit: - Users of slower internet speeds will experience varied freezing times for mIRC from large responses
- due to mIRC being a 32bit program, if the response is large enough its contents will, at the least, have to be wrote to a paging file thus resulting in slow access of the data.
Neither of which would make for a nice end-user experience. 6: Async HTTP requests should have events for: - The request (headers and body) has finished being sent to the server; raised prior to any 'response' events
- When the full server head is received - making statuscode and headers available until the 'session' is closed
- Progressively as the body is received - making the portion of the body received since the previous firing accessible as both string or into a bvar
Last edited by FroggieDaFrog; 06/11/15 03:34 PM.
|
|
|
|
Joined: Oct 2003
Posts: 3,918
Hoopy frood
|
OP
Hoopy frood
Joined: Oct 2003
Posts: 3,918 |
Agreed with all comments above. No client side caching (for a bare non-web-browser HTTP client you generally never want this), connection timeouts (see uN switch on sync mode, which can be on async mode), redirect opt-out or restriction (could add an rN/-rN switch for this), ctrl+break should kill a sync request, and yes, progressive events throughout the request cycle would be nice as well.
- argv[0] on EFnet #mIRC - "Life is a pointer to an integer without a cast"
|
|
|
|
|