XMLTV tv_grab_be fix/workaround
Last week, tv_grab_be suddenly stopped working. This script updates my MythTV TV guide. Because the grabber malfunctioned, I missed a few episodes of my favorite shows.
When I ran the script from the commandline, I got the following:
kvanhertum@MythTV:/usr/share/perl5/XMLTV$ tv_grab_be
using config filename /home/kvanhertum/.xmltv/tv_grab_be.conf
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">
finding channels: could not get channels page http://www.teveblad.be/ndl/zender.asp, aborting
Google-ing a bit, I found this post at the XMLTV mailing list. It states that the site hosting the TV guide (http://www.teveblad.be) blocked the user agent used by XMLTV.
Digging a bit in the code, I found the line which sets the user agent used.
Change the following line in /usr/share/perl5/XMLTV/Get_nice.pm
# ORIGINAL $ua->agent("xmltv/$XMLTV::VERSION"); # CHANGE IT TO $ua->agent("IE/3.1"); # or anything similar
That’s it, tv_grab_be is now working again !
Thanks a lot!
I’m kinda worried, though. So these grabbers are getting the data without Teveblad’s consent?
Then why do they have those XMLs up, anyway?
I’m still having the same problem even after changing the user agent.
Did I miss something ? I’m using the windows xmltv version.
regards,
Gert
I only tried it on MythBuntu. You still keep getting the same error message ?
Thank you so much. I was really not willing to pay for a license for JXMLTV.
Hi
Same with me here, I’m also on windows, downloaded the newest xml version, and then found your blog. Unfortunately I still get the same error message…. Any clue?
Thanks,
Cakes
I’ll try it this weekend on Windows.
I’ll post my results.
Gr,
Kristof
I just had a quick look at the Windows version. The file you download from the XMLTV project page, is a precompiled binary. It includes the source files only for reference (they are not called by the executable). In order to make the modification work, you’ll have to rebuild the complete binary.
I think rebuilding the app is quite time consuming for me, but nevertheless I’ll have a look later on.
You could also install Perl for Windows (and all needed modules). That will work guaranteed !
Hey Kristof,
This is excellent news! Thanks ! I do have perl installed but I haven’t done more with it than just running a script. How do I rebuild an executable?
Cakes
Normally, you don’t compile Perl scripts into binaries. Perl gets interpretted/compiled at runtime when you execute the script.
You can however compile it into a binary, but this is usually not done (e.g. on Unix it’s just the source files, no binary). In order to run the tv_grab_be script (not the binary) on Windows, you need to have Perl for Windows installed, and all the necessary referenced modules. I’ll put that information only tomorrow, or the day after that.
Kristof
Tnx for the quick solution.
Has someone posted a bug on this yet?
I created a bug for this.
http://sourceforge.net/tracker2/?func=detail&aid=2175391&group_id=39046&atid=424135
Youri,
I don’t think this is a bug. Teveblad blocked XMLTV on purpose, as all other User Agents still get access to the site. This is why I believe this fix won’t enter the ‘mainline’ of future XMLTV versions.
I’m well aware that this is not a bug in the xmltv code. I feel however that the devs should be informed about this.
I also doubt they will implement a hotfix for this, but at least now they are aware of the error and it can be referenced by anyone on the tracker.
Ok, so we’ve received a response from the devs. Not quite wat i hoped for, but still…
I think we should consider on changing the script so it fetches the data from another source. Any suggestions??
Yep, that’s what I was afraid for … Does that mean they’ll completely remove the script from their package ?
Anyone volunteering to write a new grabber ?
It looks like they intend to.
It definitely got someone’s attention…
http://www.nabble.com/tv_grab_be-dutch-version-blocked-by-provider-td19968061.html
Hey,
I’ve found this alternative: http://mc2xml.110mb.com/
Works much faster than xmltv, less than 10 seconds. And as a bonus, it gives you 2 weeks worth of tv listings. That’s goodby xmltv for me !
Cakes,
thanks for posting that excellent link ! It really is way faster than XMLTV. Hooray for 2 weeks of tv listing ! Using Microsoft’s Media Center service to update a Linux media center, how nice is that
I’ve tried to use XMLTV for Windows from source, but this required installing nmake, c compilers etc to install the Term::ProgressBar module. This was too much work for me, so I abandoned the install.
Sorry Windows guys/girls, but even so I think the alternative Cakes just mentioned, is MUCH better.
Ugh, I like to think I’m logical about everything, but I did wish they would have sugar-coated their “cancellation” of our grabber.
*BAM* The grabber will be removed from the distribution.
Anyhoo, this makes me think more and more people are starting to use mythtv & co in Flanders
This grabber has been available for ages…
Cakes, this might sound a little gay but….
I LOVE YOU!!!!!!
OMG, this thing is faaaaast! First time I ran it, it only took 5sec to complete.
Thanks a million!
Interesting, they added a little suffix to the listings:
“Het gebruik van software om gegevens te extraheren uit deze website voor persoonlijk gebruik of voor weergave op een andere website (’screen scraping’) is verboden.”
Basically, in English, it says:
“The usage of software to extract data from this website for personal use or for display on other websites (screen scraping) is prohibited”
Mind the term “prohibited” (verboden) and NOT “illegal”, since there are no laws concerning screen scraping. (Correct me if I’m wrong, laws in different countries may vary)
There’s an interesting discussion here:
http://developers.slashdot.org/article.pl?sid=03/02/07/1849243&tid=156
Only problem is, the fix is now broken
Anyway, mc2xml is a better alternative …
@skerit
Interesting discussion at /. !
mc2xml FTW!!!
And it even fills in the correct frequencies!
Can anybody explain a little how to interface the mc2xml output to be used in my MythTV installation?
I’m a basic-to-middle-knowledged user of MythTV, but I still don’t understand the xmltv-import of program data…
Thanks
@dreamer:
Check the mc2xml website (http://mc2xml.110mb.com) for instructions on using it with MythTV
Hoi kristof, long time not seen. I’m switching to mc2xml too. The user agent fix didn’t last long.
I only made one mistake. I added zipcode 10000. Yep is all want wrong sundenly I had over 300 channels added.
So read the faq on the mc2xml website (http://mc2xml.110mb.com). Use your correct zipcode. The thing knows the right channels and even the frecencies. I did loose the icons, but I’ll fix that meself.
PS: how is your hospital patient tracking stuff going?
Teveblad is quite determined.
They even tried to block the IE/3.0 user agent. Changing it to IE/6.0 worked for a while, but now I believe they changed the location of their pages.
We could hack on a bit further, but it’s no use.
I just hate the way they handled this. I’m quite sure we could have reached an aggreeable arrangement. Instead they had to go down a dirty road, trying to convince people screen scraping is illegal (even if they did not use those words)
Anyway, moving on.
What I like about mc2xml is that it’s lightning fast!
But using a microsoft service makes me feel a bit uneasy. What is their stance on this, anyway? Is it a free-for-all service or could we expect this program to stop working in a few weeks, too?