mIRC Home    About    Download    Register    News    Help

Print Thread
#241053 17/03/13 07:03 PM
Joined: Mar 2013
Posts: 1
R
RayGFX Offline OP
Mostly harmless
OP Offline
Mostly harmless
R
Joined: Mar 2013
Posts: 1
Does anyone know how to do an HTML scraper? I want it so that when a new topic is posted on a certain website, it sends a message to a channel.

I want it when a topic is created on this website, it messages a certain channel:

http://forums.zybez.net/forum/354-clan-fights/

Anyone wanna help?

RayGFX #241058 17/03/13 11:22 PM
Joined: Sep 2005
Posts: 2,881
H
Hoopy frood
Offline
Hoopy frood
H
Joined: Sep 2005
Posts: 2,881
Code:
alias -l hsh { return $qt($scriptdirforum_zybez.hsh) }
alias -l hsh_save { 
  if ($hget(forum_zybez)) { hsave forum_zybez $hsh }
}
alias -l hsh_load {
  if ($isfile($hsh)) {
    if (!$hget(forum_zybez)) { hmake forum_zybez 100 }
    hload forum_zybez $hsh
  }
}

alias check_new_threads {
  if (!$hget(forum_zybez)) { inc %forum_zybez_firstrun }
  sockopen forum_zybez forums.zybez.net 80
  sockmark forum_zybez /forum/354-clan-fights/
  if ($1 isnum 1-) { .timer 1 $1 check_new_threads $1 }
}
alias post_to_channel {
  var %threadid = $gettok($1-,1,9), %threadurl = $gettok($1-,2,9), %threadtitle = $gettok($1-,3,9)
  msg #yourchannel Thread ID: %threadid - URL: %threadurl - Title: %threadtitle
}

on *:sockopen:forum_zybez:{
  if ($sockerr) { echo -a * ERROR reading forum.zybez.net: $sock($sockname).wsmsg ( $+ $sock($sockname).wserr $+ ) }
  else {
    var %s = sockwrite -n $sockname
    %s GET $sock($sockname).mark HTTP/1.1
    %s Host: $sock($sockname).ip
    %s Accept: */*, *.*
    %s Connection: close
    %s
  }
}
on *:sockread:forum_zybez:{
  var %data
  sockread %data
  if ($regex(%data,/<a itemprop="url" id="(.+?)" href="(.+?)" title='(.+?)' class='topic_title' >/)) { 
    if (!$hget(forum_zybez,$regml(1))) { 
      hadd -m forum_zybez $regml(1) $+($regml(2),$chr(9),$regml(3)) 
      if (!%forum_zybez_firstrun) { post_to_channel $+($regml(1),$chr(9),$regml(2),$chr(9),$regml(3)) }
    }
  }
}
on *:sockclose:forum_zybez:{ unset %forum_zybez_firstrun }
on *:start:{ hsh_load }
on *:exit:{ hsh_save }


/check_new_threads - this will check only once.
/check_new_threads <N> - this will check every <N> seconds for new threads.

The first time this script runs it won't message the channel with any threads as you wouldn't want it to post like 30+ threads and flood you off.

I would strongly recommend you get the site owner's permission before you tell this to check every few minutes.

Change "#yourchannel" in the post_to_channel command to change the channel you want to post to.

Hope this helps smile


Link Copied to Clipboard