Saturday, October 30, 2010

Web crawler - Wikipedia, the free encyclopedia

Web crawler - Wikipedia, the free encyclopedia

Web scraping - Wikipedia, the free encyclopedia

Web scraping - Wikipedia, the free encyclopedia

screen-scraper.com

screen-scraper.com

Google group scrubyt is gone

Once in a while I am curious to see, what goes on in the scrubyt are.
I have a few Atom and RSS feed URLs stored in my feed reader (Firefox Sage), but I don't reach them any more. The group just does not exist any more:
Cannot find scrubyt
The group named scrubyt has been removed because it violated Google's Terms Of Service.
I assume, it got spammed to much, the moderators didn't care, and finally the resp. Google Groups surveillance instance had to go that step.

I still wonder, what the future of scrubyt is. Visit this previous link, even if only for the list of competing software (ruby only, I assume).

hpricot | RubyGems.org | a swift, liberal HTML parser with a fantastic library

hpricot | RubyGems.org | your community gem host

Nokogiri – an HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3 selectors… and much more

Nokogiri

No Javascript support.

web scraping afternoon

This wasn't meant to be yet another web scraping afternoon.

This afternoon started with me trying to recover a little from a hard time.
I had two probation days for a web-site testing job with Selenium, I am in the middle of a couple of recruitment processes, and I don't want to tell you about the real trouble.


  • I got intrigued to search oreilly.com for literature on Selenium and found a "Short Cut" document.
  • I found something.
  • I had a few looks over the chapter on "twill".
  • Before I really dived into the chapter on Selenium, I summed up, what I really liked and disliked about Selenium.
  • Of course, being able to use XPath is great.
  • With Selenium you somehow aren't aware at all, that there is Javascript being made use of on a web-site, but you just leave this to the browser engine, initially to Firefox and to the Selenium IDE.
  • I actually hate it, if your HTTP scripting depends on desktop computers running a browser and some remote control software to connect your server, where you "HTTP scripts" actually run, and the web browser(s), that you make use of.
  • I did a little superficial research on: perl/ruby + mechanize + xpath.
  • Yes, there is still scrubyt around, but isn't  that vaporware now itself?
  • Found perl's WWW::Scraper::TidyXML - "TidyXML and XPath support for Scraper". Not bad. But then it's from around 2003, and it seems to be vaporware. My e-mail to the author could not get delivered ("over quota"), so I guess, it's seriously no longer maintained.
  • WWW::Mechanize::Firefox seems to be nice, have a look at WWW::Mechanize::Firefox::Cookbook!
  • …

Friday, October 29, 2010

Scrappy: All Powerful Web Harvester, Spider, Scraper fully automated - search.cpan.org

Scrappy - search.cpan.org

EDI for Ruby (edi4r)

EDI for Ruby (edi4r)

Actually they refer to EDIFACT here.

You can use this software to output JSON, which you can process in any other software than.

WWW::Mechanize::Firefox - search.cpan.org

WWW::Mechanize::Firefox - search.cpan.org

Support for Javascript and XPath.

What about recording resp. capturing such a script?

perl, cpan: WWW::Scripter

WWW::Scripter - search.cpan.org

From the POD there:
DESCRIPTION 
This is a subclass of WWW::Mechanize that uses the W3C DOM and provides support for scripting.
No actual scripting engines are provided with WWW::Scripter, but are available as separate plugins. (See also the "SEE ALSO" section below.)

So it supports DOM, but no XPath expression yet.
And there is Javascript support through plugins.

An Introduction to Testing Web Applications with twill and Selenium - O'Reilly Media

An Introduction to Testing Web Applications with twill and Selenium - O'Reilly Media

To cheap not to own it – I thought a little, now I am reading it.

HSDD = hypoactive sexual desire disorder

A link to the abstract of the conference article / press release.

From that abstract:
CONCLUSION: Cerebral activation patterns in women with HSDD differs from those in women with normal sexual function and may reflect differences in how they interpret sexual stimuli.
In other words: Women with low libidos ‘have different brains’.

Have a good laugh!!!

Here is a lengthy discussion of the "miserable" approach in that article.

Thursday, October 28, 2010

Selenium+XPather: e.g. verifyTextPresent vs. verifyElementPresent

Selenium usually records string clicks and tests instead of true native language independent XPath expressions. But you can always find the right XPath expression yourself (resp. with the help of XPather, a Firefox extension), and make use of it in your selenium code.

Caveat: the XPath expression, that XPather tells you, needs yet another '/' in the beginning to be useful in your Selenium code.

Yes, these XPath expressions are lengthy, and you may think they are overspecifying your location in question, but then: when will that lengthy XPath expression ever fail? If your HTML programmer changes his code. And that's exactly, what you should insist of being informed of in the first place. Track your HTML programmer! If you don't, he will screw you w/o any mercy. You don't want to screw him, but you need to know the consequences of what he is doing. Actually not in every detail, but more details are better than no details at all.

We replaced verifyTextPresent with verifyElementPresent, and it worked "out of the box". We gained native language independence immediately.

Wall Street 2: Money Never Sleeps (2010)

Wall Street 2: Money Never Sleeps (2010)

My Wednesday (2010-10-27) night movie.

Very nice. Good entertainment.

Now I know, how to pronounce the surname "Schwartz" in English. That's one of the main characters in that movie.

Thanks to my movie night sponsor EN!!!!

Selenium: strftime, sprintf

I would like to see a strftime or an sprintf in Selenium.

Javascript has a printf, but only for files, not for strings.
(Maybe there is a way to regards strings as files, but my Javascript competence is not good enough for that.)

I found Javascript code, that implements sprintf.

(I only introduced "Selenium" in the title, because the Tweet created from a title w/o it looks stupid.)

misc. Selenium links


Selenium Training - The Automated Tester:

Un-enumerable thanks to DS, who provided me with these links above and much more!!!

Here are a few more links, that I found helpful:

Monday, October 25, 2010

Exodus 34:30 Moses' face was so radiant, that they couldn't bear it

29: When Moses came down from Mount Sinai with the two tablets of the Testimony in his hands, he was not aware that his face was radiant because he had spoken with the LORD.
30: When Aaron and all the Israelites saw Moses, his face was radiant, and they were afraid to come near him.

In German from Martin Buber's translation:
Es geschah,
als Mosche vom Berge Ssinai herabstieg
die zwei Tafeln der Vergegenwärtigung in Mosches Hand,
als er vom Berg herabstieg
– Mosche wußte aber nicht, daß von seinem Reden mit ihm die Haut seines Antlitzes strahlte –,
sah Aharon und alle Söhne Jissraels Mosche an:
da, die Haut seines Antlitzes strahlte,
und sie fürchteten sich, zu ihm zu treten.

Saturday Night Fever (1977)

Saturday Night Fever (1977)

Where were you, when they showed this movie? What was your life like?

yet another posting of mine on the FRITZ!Boxes

Somebody asked me a few questions regarding the FRITZ!Boxes, and I think, it makes sense to answer them here on this blog.

No, I haven't been digging into the FRITZ!Boxes "at levels well below the standard web UI".

There is no SSH server, that comes with the FRITZ!Boxes.There are private web-sites, where you can find a ready-made dropbear SSH server. I got mine that way. And a gkrellmd as well.

There should be a way, to create yourself a development system targeted towards the "busybox" on your particular FRITZ!Box, but there are various slightly different processors being employed. I wasn't successful there, when I tried. There is ip-phone-forum.de, where the FRITZ!Boxes get discussed a lot, but the preferred language there is German, although they would most certainly try to answer question, that people ask in English.But the threads are mostly in German. Be aware, that the main community for the FRITZ!Boxes is located in Germany and I guess also in Austria and in Switzlerland. You may also find a HOWTO, that explains how to set up a development environment.
But actually ... - see above!

The "international" 7390 "speaks" English, but I don't think, there a lot on frequent firmware updates for it. They only provide frequent firmware updates for their resp. battleship, and that is "certainly" not the *international* version of the 7390 but the "German" version -- but how do I actually know? I don't own an international 7390, I only brought one to Martinique and saw it working for like 2 weeks.

I assume the firewall on the FRITZ!Boxes is homegrown.

What tools and languages are available?
Well, basically they use the "busybox" software and what is built into that.
Nowadays there is also a "lua" interpreter provided. AVM seem to use that for CGI purposes.
perl etc. are regarded far to resource hungry.

Saturday, October 23, 2010

The Road (2009)

The Road (2009)

Post-apocalyptic …

It made me cry, it almost made me throw up. It's definitely not going to cheer you up.

This was my Saturday night movie.

I dislike Facebook apps pulling list of friends

I never let then do that, and I hope you (as my Facebook friend) don't either – if I notice you let them throw  their propaganda at me, I will have to remove you as my Facebook friend.
Why isn't there a Facebook flag to generally opt out of that kind of thing? I guess that's against there business model. How bad.

"Fear, uncertainty and doubt" - Wikipedia, the free encyclopedia

Fear, uncertainty and doubt - Wikipedia, the free encyclopedia

"Embrace, extend and extinguish" - Wikipedia, the free encyclopedia

Embrace, extend and extinguish - Wikipedia, the free encyclopedia

Apple: iChat vs. FaceTime (XMPP?)

In his recent keynote Steve Jobs announce FaceTime for OS X as something new. But didn't we all wonder, why they didn't just extend iChat to allow us to talk to somebody using FaceTime on the iPad or iPhone? I would rather prefer that.

Actually I would like to know, whether FaceTime and iChat are XMPP base or whether Apple prefers to invent proprietary protocols. Even “embrace and extend” would be better than that. Hmm, I don't really mean “embrace, extend and extinguish”, but that's what “embrace and extend” sadly redirects to on Wikipedia. I mean: does the world really need yet another messaging standard? There is SIP around, and it should actually get replaced by XMPP, but that's not really going to happen for the next 15 years, I assume. Google and Facebook employ XMPP for Google Talk resp. the Facebook chatting facility, and I like that.

Thursday, October 21, 2010

the coming App Store for Mac OS X: the next Apple jail?

First it sounds like the App Store will help the users to have a single point, where to find and update apps for OS X. But then: look what it means on iOS, i.e. the iPhone and the iPad: it means: apps will only find their way to your device through Apple's "placet", and finally through the App Store, and Apple will always earn money through 3rd parties' contributions.

Deutsches Historisches Museum Berlin: Exhibition: Hitler and the Germans. Nation and Crime

Deutsches Historisches Museum Berlin - Hitler and the Germans. Nation and Crime - Exhibition

Location: Deutsches Historisches Museum, "Ausstellungshalle von I. M. Pei".

Opening hours: 7 days a week 10:00 through 18:00, on Friday through 21:00.
2010-10-15 .. 2011-02-06.

I found the exhibition far too overcrowded. The captions made very bad use of 3 different fonts, occasionally making the text rather unreadable. I never saw any exhibition as badly made as this one.

Date format(s) - Wikipedia, the free encyclopedia

Date format(s) - Wikipedia, the free encyclopedia

I really love ISO 8601 as date format.