Updating file via socket instead of overwrite.

Print Thread

Updating file via socket instead of overwrite. #117607 17/04/05 07:38 PM
Joined: Apr 2005 Posts: 11 Belgium TheCat5001 OP Pikka bird
OP TheCat5001 Pikka bird Joined: Apr 2005 Posts: 11 Belgium	Well that's about as specific as I can make the topic title. I'm not sure if it's possible, but I currently have a socket script that downloads a .txt file from a server, but everytime I want to update the file I have to rewrite it completely. As it has been becoming quite large lately (almost 800kB) the download is getting slower and slower. Now I'd like to fix this before it actually starts to cause problems. Is there a possibility to check where the file on the server changed compared to the file on my hard drive and update it accordingly? There are lines added, modified and deleted over the entire file every hour, it doesn't just add new lines to the end. Well I hope this is kind of clear, if you read it a few times it should make sense. Now let me hope it's possible. A shotgun is always the answer.

Re: Updating file via socket instead of overwrite.

#117608 17/04/05 08:41 PM

Joined: Dec 2002

Posts: 1,922

Online

Hoopy frood

Online

Hoopy frood

Joined: Dec 2002

Posts: 1,922

Use the If-Modified-Since header. An example follows.

Code:

; Type /upd to begin.
 
alias upd {
  set %upd.site    www.example.com
  set %upd.request /
  set %upd.file    example.html
  set %upd.header  1
  sockclose upd
  sockopen upd %upd.site 80
  _info Connecting to %upd.site
}
 
alias -l _info echo -sc info * upd: $1-
alias -l _cleanup {
  sockclose upd
  unset %upd.*
}
 
On *:sockopen:upd:{
  if $sockerr {
    _info Error: unable to connect.
    _cleanup
    return
  }
  _info Requesting file %upd.request
  sockwrite -tn $sockname GET %upd.request HTTP/1.1
  sockwrite -tn $sockname Host: %upd.site
  sockwrite -tn $sockname Connection: close
  if $isfile(%upd.file) {
    sockwrite -tn $sockname If-Modified-Since: $&amp;
      $gmt($file(%upd.file).mtime)
  }
  sockwrite -tn $sockname 
}
 
On *:sockread:upd:{
  if %upd.header {
    var %var
    sockread %var
    tokenize 32 %var
    if HTTP/* iswm $1 {
      if     $2 == 200 { 
        ; OK
        _info Receiving data...
        .remove %upd.file
      }
      elseif $2 == 304 {
        ; Not Modified
        _info File not modified.
      }
      else {
        _info Error: unrecognized reply: $1-
        _cleanup
      }
    }
    elseif !$1 {
      ; A blank line ends the header section
      set %upd.header 0
    }
  }
  else {
    _info Writing to disk: %upd.file
    sockread &amp;var
    bwrite %upd.file -1 &amp;var
  }
}
 
On *:sockclose:upd:{
  _info Done.
  _cleanup
}

Re: Updating file via socket instead of overwrite. #117609 18/04/05 04:33 PM
Joined: Apr 2005 Posts: 11 Belgium TheCat5001 OP Pikka bird
OP TheCat5001 Pikka bird Joined: Apr 2005 Posts: 11 Belgium	Thanks, but wouldn't that just download the entire file again if it has changed since that certain time? I know it changes every hour, but I'm searching a way so I only have to download the lines that were changed or added. And find a way to delete the lines that don't exist in the file on the server anymore. A shotgun is always the answer.

Re: Updating file via socket instead of overwrite. #117610 18/04/05 10:46 PM
Joined: Sep 2003 Posts: 4,230 D DaveC Hoopy frood
DaveC Hoopy frood D Joined: Sep 2003 Posts: 4,230	That is not possable, since you cant findout the modified lines untell you have the new file to compare to the old file, then you already have downloaded the new file, so why compare them. Downloading if the file is newer is the only method you have, unless the website was to develop a seperate updates only file, which i highly doubt would ever happen.

Re: Updating file via socket instead of overwrite. #117611 19/04/05 05:49 AM
Joined: Apr 2005 Posts: 4 T tisyn Self-satisified door
tisyn Self-satisified door T Joined: Apr 2005 Posts: 4	If the file being downloaded is only updated at the end of the file then a resume (range) can be done instead of a full download. Last edited by tisyn; 19/04/05 06:25 AM.

Re: Updating file via socket instead of overwrite. #117612 19/04/05 07:27 AM
Joined: Sep 2003 Posts: 4,230 D DaveC Hoopy frood
DaveC Hoopy frood D Joined: Sep 2003 Posts: 4,230	Do you even read things before you reply to them?

Re: Updating file via socket instead of overwrite. #117613 19/04/05 11:29 AM
Joined: Feb 2004 Posts: 2,019 Leuven, Belgium FiberOPtics Hoopy frood
FiberOPtics Hoopy frood Joined: Feb 2004 Posts: 2,019 Leuven, Belgium	Quote: There are lines added, modified and deleted over the entire file every hour, it doesn't just add new lines to the end. Gone.

Re: Updating file via socket instead of overwrite. #117614 19/04/05 04:09 PM
Joined: Apr 2005 Posts: 11 Belgium TheCat5001 OP Pikka bird
OP TheCat5001 Pikka bird Joined: Apr 2005 Posts: 11 Belgium	Quote: That is not possable, since you cant findout the modified lines untell you have the new file to compare to the old file, then you already have downloaded the new file, so why compare them. That's what I thought, thanks for the help anyway. A shotgun is always the answer.

Link Copied to Clipboard

Forums Scripts & Popups Updating file via socket instead of overwrite.