Spidering Hacks - O'Reilly Media

CHAPTER ONE


Hack #2 – Best Practices for You and Your Spider    
…

Be Liberal in What You Accept
… This is an inexact science, to put it mildly. …
…
Monitor your spider’s output on a regular basis to make sure it’s working as expected [Hack #31], make the appropriate adjustments as soon as possible to avoid losing ground with your data gathering, and design your spider to be as adaptive to site redesigns [Hack #32] as possible.
…
Don’t Reinvent the Wheel

• …
• Best Practices for You

If you must scrape HTML, do so sparingly. If the information you want is avail- able only embedded in an HTML page, try to find a “Text Only” or “Print this Page” variant; these usually have far less complicated HTML and a higher content-to-presentation markup quotient, and they don’t tend to change all that much (by comparison) during site redesigns.
Hack #4 – Registering Your Spider
By the way, you might think that your spider is minimal or low-key enough that nobody’s going to notice it. That’s probably not the case. In fact, sites like Webmaster World (http://www.webmasterworld.com) have entire forums devoted to identifying and discussing spiders. Don’t think that your spider is going to get ignored just because you’re not using a thousand online servers and spidering millions of pages a day.
Naming Your Spider
… There are web sites, like http://www.iplists.com, devoted to tracking IP addresses of legitimate spiders. …
Hack #5 – Preempting Discovery
No matter how gentle and polite your spider is, sooner or later you’re going to be noticed. Some webmaster’s going to see what your spider is up to, and they’re going to want some answers.
…
Hack #6 – Keeping Your Spider Out of Sticky Situations Hack
Bad Spider, No Biscuit!
… There is nothing stopping a disgruntled site from revising its TOS to deny a spider’s access, and then sending you a “cease and desist” letter. … Spidering another site’s content and reappropriating it into your own framed pages is bad. Don’t do it. …
Competitive IntelligenceSome sites complain because their competitors access and spider their data—data that’s publicly available to any browser—and use it in their com- petitive activities. You might agree with them and you might not, but the fact is that such scraping has been the object of legal action in the past. Bid- der’s Edge was sued by eBay (http://pub.bna.com/lw/21200.htm) for such a spider. …
Possible Consequences of Misbehaving Spiders
… But considering lawyer’s fees, the time it’ll take out of your life, and the monetary penalties that might be imposed on you, a lawsuit is bad enough, and it’s a good enough reason to make sure that your spiders are behaving and your intent is fair. …
CHAPTER TWO
Assembling a Toolbox
Hacks #8–32
…


Chapter 4 Gleaning Data from Databases
…
Hack #69 – Aggregating RSS and Posting Changes
-> meta feeds, aggregating feeds, …

jobs.perl.org needs a couple of changes -- let's start brain storming!

For me as a freelancer it's very clear:

• There must be separate feeds for freelance and salaried staff.
• There should be an opportunity of commenting on the job postings, e.g. if the original poster doesn't close the job, it makes sense to get that information from somebody else, maybe from somebody who was somehow involved. Yes, that cannot happen anonymously.
• …

What else?

Yes, I tried to contact Ask Bjørn Hansen at ask(AT)perl.org before I started this here, but to no success.

"Senior Software Engineer - Perl" / "Germany, Karlsruhe" / "Pay rate: 70,00 €/h" / CLOSED

"Closed", so the recruiter says.
What a pity, that comments on job postings on jobs.perl.org are not possible.

"Tour de Babel" by Steve Yegge

[...]
My whirlwind tour will cover C, C++, Lisp, Java, Perl, (all languages we use at Amazon), Ruby (which I just plain like), and Python, which is in there because — well, no sense getting ahead of ourselves, now.
[...]

"A Quick Tour of Ruby" by Steve Yegge

Very nice to read.

Ruby used to annoy me simply by existing. I first heard about Ruby years ago, in maybe 1997 or 1998, and folks said it was kind of like Perl, but "cleaner", whatever that meant. Ruby fans back then seemed like a tiny minority of rebels and fringe separatists.
Ruby irked me primarily because we already had Perl, which was working just fine thank you very much. And if for some strange reason you didn't like Perl, we had Python. If Perl fans were dog owners, and Python fans were cat owners, then Ruby fans seemed like ferret owners. They could go on and on about how much they adored their beady-eyed albino stretch-limo rats, and how cute they were, but we all knew they were just looking for attention. Nobody really wants a pet rat. (Ferret owners will correct me and say they're not rodents; they're more closely related to weasels and skunks. As if that helps.) Regardless, I didn't want to have anything to do with Ruby.
Last year, though, I was looking at a bunch of different languages in the hopes of finding one to replace Perl for small- to medium-sized tasks. One day my magic Perl dust had worn off rather suddenly, and I'd joined the growing ranks of people who were beginning to notice the emperor was a wee bit underdressed. But all the alternatives to Perl looked pretty bad themselves, and I started judging languages by how far I'd get into the reference manual before throwing it across the room.
I eventually picked up a Ruby book -- ...

Steve ...'s home page.

I personally keep loving both of them. I can afford that in the comp.lang.* area and in some others as well, but that doesn't concern my girl-friend, of course.

I actually came across Steve, when I searched for elisp.

iPhone apps, that I need sooner or later

• Telefonkarte (Calling Card): supplies you with support for all sorts of call-through telecom providers, even your FRITZ!Box at home can serve as one and is indeed supported by this app
• …

how to avoid to accidentally Quit Firefox?

Is there any config. variable?
There is a checkbox labeled "Warn me when closing multiple tabs". That does the job.

first steps in IRC with pidgin

1. "Add Account"  for each IRC server/user pair (e.g. irc.freenode.net), that you want to use, within pidgin with IRC as protocol
2. "Join a Chat" (below Buddies), select the right Account (i.e. one of your IRC protocol/server accounts), enter the Channel (including the '#'), leave the Password blank! here we are!

Did I mention recently, how much I love my pidgin?
I did all this with a (fink) pidgin on my MacBook running Snow Leopard (OS X), but I don't doubt, it will also run on my openSUSE Samsung notebook.

networks and logos

Where do you get to the personalised logos resp. badges of misc. networks:

• xing
• linkedin
• ...

To be continue …

how to structure my stuff on github.com

I need help. Does anybody want to instruct me, how to set up my stuff resp. repositories on github.com? My profile there is this. I created a RELAX-NG schema earlier today, and that's the first to upload.

• Filesystem Hierarchy Standard
• Linux packagement on sabi.co.uk
• ...

bulk upload of events in XML at XING

At this link XING tells us, that they accept XML files in a certain format for uploading events to their site. They supply us with some nice but rather informal documentation in PDF, but there is no schema. Now that made me a little curious today, I downloaded their template file, created a few variants of the sample entry, and running trang I created a RELAX-NG schema file (anybody interested?). trang seems to not recognise date-time values of tags, so that always needs a little manual post-processing.
O, yes, I proudly uploaded one such event that way. (Actually on the first attempt I got the year wrong, that was quite embarrassing.)

Erekat to Meridor: Without two-state solution by year's end, 'you will sweat'

http://www.haaretz.com/news/diplomacy-defense/erekat-to-meridor-without-two-state-solution-by-year-s-end-you-will-sweat-1.298382?localLinksEnabled=false

my approach to HTTP scripting, web harvesting, page scraping, and all that

• use LiveHTTPheaders in Firefox
• resp. ieHTTPHeaders in IE for extracting the relevant HTTP traffic;
• run my script scan_HTTP_log_file.pl on that and create a raw perl script (raw because e.g. it doesn't know, that it includes session IDs etc. as constant literals),
• that makes use of my JHwis toolkit around libcurl, it also includes a truely working cookie machine.
• on the Curl web-site you can also find formfind.pl, that helps a lot in finding all the "input" tags.
• now you have a raw perl script, that you can enrich will all necessary condition handling – there is all kinds of stuff in that code (like in any code generated from visiting a web-site), that you need to replace by dedicated handling, like session IDs and similar fields and URL ingredients; associate coordinate names with the resp. image names.
• on CPAN lives a module by the name of HTML::TableExtract for extracting HTML tables;
  I wrapped it up a little, so that I can make use of it on a command line.
• that command line utility supplies me with all necessary options for related tasks regarding navigating through HTML and all its tables, just what HTML::TableExtract actually does.
• I love writing "small" utilities in perl resp. ruby, that I wrap up bash resp. zsh scripts.

Years ago (I guess the situation is not so much different nowadays) libcurl was just so much more powerful than LWP, that I simply had to got for libcurl. Read "Using cURL to automate HTTP job" aka "The Art Of Scripting HTTP Requests Using Curl", that's IMHO the major evergreen in that area. libcurl does a lot more than just simple PUT, GET and that sort of thing.
All that makes my swiss army knife of web harvesting and page scraping.
Reality is seriously more challenging than text book examples, trust me!
Right, I could make all of that open source. I just recently started my open sourcerer career, after SzabGab had stayed in my place for a couple of days around LinuxTag.org/2010 at Berlin.

I should also mention Daniel Stenberg, the father of curl. IMO without his great work the art of HTTP scripting would not stand, where it stands today.

Right: last not least: no, I am not into dealing with AJAX and all that. For the last couple of years my approach has been: with the toolset I described above I can still manage "all" tasks without caring for AJAX. It does not matter enough.

page scraping with ruby: scrubyt

A rather interesting thread on the scrubyt Google Group.

visit Cristina Brizola's web spaces and buy something from her, maybe even a Frida Kahlo portrait copy!!!

Set up a flickr.com account for "mother of son#2".
She needs migrating from spaces.live.com to flickr.com rather sooner than later.

Do me a favour and buy some of her pictures!
She also does pretty nice Frida Kahlo portrait copies.

Yes, I know, I still have to create articles on her on en.wikipedia.org and de.wikipedia.org.

http://www.flickr.com/people/CrisBrizola

http://CristinaBrizola.spaces.live.com/

http://en.wikipedia.org/wiki/Cristina_Brizola

http://de.wikipedia.org/wiki/Cristina_Brizola

me on Google Buzz

I would love to have a few more followers there. Please!!!

shared synchronised editing with GNU Emacs

I continuously edit my ~/diary on my openSUSE Linux notebook and on my MacBook.
Emacs helps me there quite a lot. In case I changed something on that file on some other computer than the one I am currently sitting in front of, it warns me and also offers me to revert to the last version saved to disk.
I really love that.

Timer Utility - i use this on mac os x

Alarm clock, countdown timer, and stopwatch.
great stuff!!!
almost like the iPhone utility

MacMetronome - i use this on mac os x

Facebook does (not) import my Google Buzz public feed

First it didn't, what surprise ;-)

Update / 2010-06-23 17:00:
Works now.

I actually first tried that through "Import an External Blog", and you can only name there a single one. But they seem to reject that because of some details in the XML, that they don't like ;-)
But you can always use NetworkedBlogs for more letting other streams like blogs and whatever RSS feeds flow into "your" Facebook (wall).

Photo Booth on OS X

It was time to change the pics on Facebook, Skype etc. again.
Wondered how to take a few pics using the MacBook.
Finally found Photo Booth.
Move the utility's window to that corner of the screen, where you want your face to look at, otherwise your eyes look into the "wrong" direction. Sounds obvious, I know.

we remember the end of WW I and its armistice in Compiègne Forest

• article on en.wikipedia.org
• the armistice at the end of WW II has also taken place in Compiègne Forest
• today is still a bank holiday in Belgium

GoogleCL: Command-line tool for Google services

GoogleCL: Command-line tool for Google services

CGI::IDS - PerlIDS - Perl Website Intrusion Detection System (XSS, CSRF, SQLI, LFI etc.)

CGI::IDS - PerlIDS - Perl Website Intrusion Detection System (XSS, CSRF, SQLI, LFI etc.)

some old funny Python anti-Perl and Ruby anti-Python propaganda

Have you ever come across this this piece of Python anti-Perl propaganda? The article that referred to it is actually Ruby propaganda and dated 2004-07, so that anti-Perl propaganda is even older than that. The Ruby now found a followup. Look and enjoy!

By any chance: isn't there any gifted artist around, that the Perl community can make use of in order to create some Perl-minded stuff?

Update / 2010-08-05:
In response to my re-share here more sweet perl vs. ruby vs. python propaganda got created by Mark Keating:
http://mdk.per.ly/2010/06/25/propaganda/ :

debugging my .procmailrc

It's usually not really big fun, if you find a message like this in your procmail log file: "procmail: Missing action".

Well, I don't yet know of a procmail syntax checker, which is rather a pity, so the debugging is of a dynamic kind.

Just add "VERBOSE=on" in the beginning of your .procmailrc, enjoy the flood of "procmail: No match on ...,", and find your "procmail: Missing action" occasionally in the middle of it. You found the corrupted rule then. The rule in question might actually be a very rarely used one resp. one pretty at the end of your .procmailrc, so it may take a while resp. a couple of incoming mails, until it gets used and shows up.
Once you found and corrected the offender, I am quite sure, you will remove "VERBOSE=on" very quickly.

"Mock the Web Service" - O'Reilly Broadcast

Mock the Web Service - O'Reilly Broadcast

spoilt files from bittorrent

I recently saw "a friend" ;-)  who complained, that he had downloaded huge movie files from a bittrorrent, and when he tried to watch them, it turned out, that a special viewer is needed, and that that special viewer even makes a monthly subscription necessary. Well, under these circumstances downloading movies from bittorrents don't make sense.

FIFA worldcup on my Google Calendar

I just wanted to add the games to my calendar, now there are so many game events in my calendar, that I don't really see anything any more. And as there is one calender per team (and I only selected a couple of teams), it's not even easy to see the calendar wi/o FIFA worldcup. What a mess! ;-)

correcting red eyes on pictures

Looks like Picasa 3 for Windows seems to be the appropriate tool for doing that. Free and easy to use -- these reasons are good enough for me.
Update / 2010-06-22:
Alright, alright, I can simply use iPhoto on OS X for that. Silly me!
Did quite a few snapshots, uploaded them to the relevant places …
Life seems to be far easier with the right tools, i.e. on a Mac, esp. if it's running a Unix derivate …

2010 LinuxTag and Perl::Staff on Picasa

to proselytize

For those, that don't trust me (again), that there is no such word: http://en.wiktionary.org/wiki/proselytize . For some reason I prefer this word over the similar http://en.wiktionary.org/wiki/evangelize . Right, they don't actually show the some progress of the process (ROTFL!!!), but they go to the same direction.

@szabgab: I had to think of you and p-e-r-l, when I wrote this. (Using the dashes, so this article does not get picked up by the "respective" blog article grabber, as szabgab occasionally is soooo embarrassed over what I say.) In German we also have the phrase "Proselyten machen", which in this context sounds really, really funny – therefore ROTFL.

a new wonderful book on DocBook by Norman Walsh: "DocBook 5: The Definitive Guide"

Norman is an excellent writer, and it's good fun reading his books. If you are interested in DocBook, then get this book:

http://oreilly.com/catalog/9781449389604 = DocBook 5: The Definitive Guide - O'Reilly Media

I purchased the PDF recently from O'Reilly, and I just printed a single page for a friend, who looks like being my newest DocBook proselyte.