mIRC Home    About    Download    Register    News    Help

Print Thread
Updating file via socket instead of overwrite. #117607 17/04/05 07:38 PM
Joined: Apr 2005
Posts: 11
TheCat5001 Offline OP
Pikka bird
OP Offline
Pikka bird
Joined: Apr 2005
Posts: 11
Well that's about as specific as I can make the topic title.
I'm not sure if it's possible, but I currently have a socket script that downloads a .txt file from a server, but everytime I want to update the file I have to rewrite it completely. As it has been becoming quite large lately (almost 800kB) the download is getting slower and slower.

Now I'd like to fix this before it actually starts to cause problems. Is there a possibility to check where the file on the server changed compared to the file on my hard drive and update it accordingly?

There are lines added, modified and deleted over the entire file every hour, it doesn't just add new lines to the end.


Well I hope this is kind of clear, if you read it a few times it should make sense. Now let me hope it's possible.


A shotgun is always the answer.
Re: Updating file via socket instead of overwrite. #117608 17/04/05 08:41 PM
Joined: Dec 2002
Posts: 1,922
O
Online Offline
Hoopy frood
Offline
Hoopy frood
O
Joined: Dec 2002
Posts: 1,922
Use the If-Modified-Since header. An example follows.

Code:
; Type /upd to begin.
 
alias upd {
  set %upd.site    www.example.com
  set %upd.request /
  set %upd.file    example.html
  set %upd.header  1
  sockclose upd
  sockopen upd %upd.site 80
  _info Connecting to %upd.site
}
 
alias -l _info echo -sc info * upd: $1-
alias -l _cleanup {
  sockclose upd
  unset %upd.*
}
 
On *:sockopen:upd:{
  if $sockerr {
    _info Error: unable to connect.
    _cleanup
    return
  }
  _info Requesting file %upd.request
  sockwrite -tn $sockname GET %upd.request HTTP/1.1
  sockwrite -tn $sockname Host: %upd.site
  sockwrite -tn $sockname Connection: close
  if $isfile(%upd.file) {
    sockwrite -tn $sockname If-Modified-Since: $&
      $gmt($file(%upd.file).mtime)
  }
  sockwrite -tn $sockname 
}
 
On *:sockread:upd:{
  if %upd.header {
    var %var
    sockread %var
    tokenize 32 %var
    if HTTP/* iswm $1 {
      if     $2 == 200 { 
        ; OK
        _info Receiving data...
        .remove %upd.file
      }
      elseif $2 == 304 {
        ; Not Modified
        _info File not modified.
      }
      else {
        _info Error: unrecognized reply: $1-
        _cleanup
      }
    }
    elseif !$1 {
      ; A blank line ends the header section
      set %upd.header 0
    }
  }
  else {
    _info Writing to disk: %upd.file
    sockread &var
    bwrite %upd.file -1 &var
  }
}
 
On *:sockclose:upd:{
  _info Done.
  _cleanup
}

Re: Updating file via socket instead of overwrite. #117609 18/04/05 04:33 PM
Joined: Apr 2005
Posts: 11
TheCat5001 Offline OP
Pikka bird
OP Offline
Pikka bird
Joined: Apr 2005
Posts: 11
Thanks, but wouldn't that just download the entire file again if it has changed since that certain time?

I know it changes every hour, but I'm searching a way so I only have to download the lines that were changed or added.

And find a way to delete the lines that don't exist in the file on the server anymore.


A shotgun is always the answer.
Re: Updating file via socket instead of overwrite. #117610 18/04/05 10:46 PM
Joined: Sep 2003
Posts: 4,230
D
DaveC Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
That is not possable, since you cant findout the modified lines untell you have the new file to compare to the old file, then you already have downloaded the new file, so why compare them. Downloading if the file is newer is the only method you have, unless the website was to develop a seperate updates only file, which i highly doubt would ever happen.

Re: Updating file via socket instead of overwrite. #117611 19/04/05 05:49 AM
Joined: Apr 2005
Posts: 4
T
tisyn Offline
Self-satisified door
Offline
Self-satisified door
T
Joined: Apr 2005
Posts: 4
If the file being downloaded is only updated at the end of the file then a resume (range) can be done instead of a full download.

Last edited by tisyn; 19/04/05 06:25 AM.
Re: Updating file via socket instead of overwrite. #117612 19/04/05 07:27 AM
Joined: Sep 2003
Posts: 4,230
D
DaveC Offline
Hoopy frood
Offline
Hoopy frood
D
Joined: Sep 2003
Posts: 4,230
Do you even read things before you reply to them?

Re: Updating file via socket instead of overwrite. #117613 19/04/05 11:29 AM
Joined: Feb 2004
Posts: 2,019
FiberOPtics Offline
Hoopy frood
Offline
Hoopy frood
Joined: Feb 2004
Posts: 2,019
Quote:
There are lines added, modified and deleted over the entire file every hour, it doesn't just add new lines to the end.


Gone.
Re: Updating file via socket instead of overwrite. #117614 19/04/05 04:09 PM
Joined: Apr 2005
Posts: 11
TheCat5001 Offline OP
Pikka bird
OP Offline
Pikka bird
Joined: Apr 2005
Posts: 11
Quote:
That is not possable, since you cant findout the modified lines untell you have the new file to compare to the old file, then you already have downloaded the new file, so why compare them.


That's what I thought, thanks for the help anyway.


A shotgun is always the answer.