Last.fm Real Time Stats Read Me

Table of content

This thing
Terms of use
Changing
Contact
Requirements
Installing
ToDo (know issues)
Done (to show my progress ;-)
Things to know/F.A.Q.
Undocumented stuff

Terms of use:

Last.fm Real Time Stats
Copyright (C) 2006, 2007 Paul Cornelissen
Contact: pcornelissen@gmail.com
Web: http://www.bit-byters.net

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Changing:

As this software is GPL'ed, you are free in using and modifying it, but I ask you to please contact me if you change, modify or copy from it and to be so kind as to include (a link to) the code so I can learn from it.

Contact:

You can contact me at:
pcornelissen -AT- gmail.com

You can find my blog where I write about this project at:
http://www.bit-byters.net/

Requirements

You don't really need a lot on your server to run this. Only PHP and MySQL are really required and these are supported by most hosting providers.

PHP
- Tested on PHP 5.2.3.
- You do need ALLOW_URL_FOPEN ON.
- You do need the GD-PNG module to draw images.
- Zlib is required for Gzipped logging. This can be turned of.
MySQL
Cron Tabs
This is a little more complicated.
Apache is the recommended HTTPD, but anything should suffice. It's recommend because I use a .htaccess file to secure the /inc/ dir in case of PHP parser failure.
Having a database management utility such as PhpMyAdmin simplifies things a lot.

Installing

Modify /rts/inc/config.php and add your database connection settings.
Copy the RTS folder to your web-page
Create the database structure using the SQL given in SQL.txt
Add a user to the mm2_users table
Add a crontab for recenttracker.php.
A 1 minute interval is recommended. Something like this:
* * * * * cd /home/www/rts/; /usr/bin/php /home/www/rts/recenttracker.php
Tip: If you set your E-mail address correctly, the crontab wil E-mail you if anything fails.
Listen to some music.

ToDo

			- Use getval() function where possible.
			- Install script.
			- Management page.
			- Make the statprediction.php HTML IE compatible and HTML 4.01 strict 
			  compliant.
			- Make a decent index.php (this is just a copy of a page from my web-site)
			- Use Mod_Rewrite
			- Fix Character set for statprediction.php
			- image.png doesn't show without context in IE(7)
			- full numeric usernames are not supported
			- Include mode, no head and tail, so page can be included in another page.
			- Storing settings in session vars
			- Move Grapher to RTS
			- Rewrite to enable SVG image generation

Done

			- Cache weekstats for last.fm in time
			- Week scrolling for statprecdiction.php
			2.0.2
			- You can now turn off logging and gzip compression in config.php
			- htmlentities() in statprediction.php
			- Bug: sorting between image and statprediction differs
			- Bug: Recent tracks seem not to update when a script is called.
			- Use XML data source.
			- Fix inefficiency with disconnecting before and reconnecting after 
			  recenttracker.php is called.
			- Fix low compression rates on logs.
			- Implement page counter
			- Bug: Very long artist or tracks names failed to insert. Cut off at 255.
			- Make this document HTML
			- Position are now the same for artists and track with the same playcount for
			  image.php and statprediction.php
			2.0.1
			- bug: Color variable names are incorrect in image.php
			- Improve Documentation massively! (actually with 2.0.0...)
			- A footer include for e.g. Google analytics.
			- Parsetimes are now rounded
			- Make scripts week-parameter accept absolute week#. >1 is absolute, format 
			  WWYY, 2 digit year, 2 digit week.
			2.0.0
			- Log to gzipped files.
			- Configurable colours for image output.
			- Put most config options in a single file
			- Use more named constants
			- A whole lot of stuff before I started documenting

Things to know/F.A.Q.

Limits on database size:
There are some limitations on the size of the datasets. I've deliberately chosen relatively small data types to keep tables size low. You can just increase this if you want.
Things you might consider:
- track id is of the type smallint unsigned autoincrement, this means a maximum of 65535 track titles. You're most likely to out-grow this one first.
- artist id is of the type smallint unsigned autoincrement, this means a maximum of 65535 artists.
- user id is of type tinyint unsigned autoincrement, this means a maximum of 255 users. (autoincrement starts at 1)
Gap on Sundays
A last.fm stats collection week, the period in which stats are collected is every week starting at Sunday 12:00 GMT, this is to make it happen on Sunday in all the different time zones. RTS behaves in the same way for collecting data, only when displaying the stats it switches at Monday 0:00 server time. In most server configurations this will implicate Monday 0:00 GMT. That's 12 hours later. This is done because Last.fm usually doesn't update your week stats within 12 hours and because there isn't any substantial amount of data to generate stats with anyway.
What load will this put on my server?
Short: I believe any normal server should easily be able to cope.
Longer: This off course depends on the number of users. You need to regularly run recenttracker.php and this script then fetches new recent plays. Also it's called every time stats are displayed, but then it only fetches the affected user if not updated in the last 30sec. ($updatefloodinterval in config.php) I currently track about 30 users and on a busy day this results in about 4,5K updates. That means 4,5K request to the last.fm web services. That's on average about 150 requests per user per day.
image.php outputs a PNG image. This uses some memory on the server side. If I remember correct, imagecreatetruecolor() allocates about 9byte's per pixel. So 160*200*9=281KB. Nothing impressive. I consider data traffic low as well. Due to the excellent compression by PNG of the very simple images generated I manage to have an average file size of 8.4KB per image with 40759 requests served in august 2007.
Database loads might be a bit higher. Not for image.php, since it only needs 2/3 queries (username/id, play count and the top list). Statprediction.php has a bit higher load since it also does a query for every artist in the top list to fetch the top tracks by that artist. On my server parse times are still very acceptable in any case, staying well below 1sec for anything I've seen, tested and tried. You can lower the maximum list length in config.php, default is 500.
If you don't have crontabs
There are several things you can do. You cannot just not use them, because it's crucial to regularly fetch the recent plays. Last.fm only provides a feed for the last 10, so if you scrobble more tracks before the script updates, it won't see them. What you can do:
- Bribe your server-admin. (e.g. send a nice email)
- Let somebody else create a crontab that requests your recenttracker.php
- If you have a high volume web-site that you can rely upon to regularly call recenttracker.php you can use it. This does include using image.php as a forum signature or avatar, because, remember, every time stats are displayed they are also updated.
- Use a different computer to regularly initiate an update. Provided that your computer is probably running when you're scrobbling music. Under windows you might want to have a look a the scheduler service, under *NIX use crontab.
  DOH!!
RTS missed some tracks!
This does happen. Even if you have perfectly good crontabs and 100% availability on your side. Why you ask? This is why:
- Last.fm often only has intermittent service, this will cause tracks to be backlogged and processed as one big batch when it comes back up and then the feed only provides info on the last 10.
- This also applies when your client, or your users client for that matter, backlogs client side. E.g. after a fight with your ISP, in a train with bad GPRS and HSDPA reception, when you have a linksys router or probably as well when you iScrobble your iPod with more than 10 tracks.
This is why I've called it a prediction. To explicitly not pretend to be exact, although it's very well possible that it is.
I can't use dynamic images on <insert site>!
I've had this problem on serveral occasions with serveral people. That's why I'm planning on using mod_rewrite. For the time being: Most scripts aren't that smart, just add '&.jpg' or whatever and it'll vallidate in most cases. The generating script will not notice the argument you've just added, since it has a bad syntax and it'll be ignored.
So if you can't use dynamic images, use:
http://yourdomain.tld/rts/image.php?id=1&.jpg
instead of:
http://yourdomain.tld/rts/image.php?id=1
My host has ALLOW_URL_FOPEN OFF
This is required. Without this the scripts can't load file from outside the domain it's run on and thus it can not load the recent track list. The options in this case are to either convince your provider to turn it on or if your provider supports cURL you can edit recenttracker.php. There is a comment stating what you should change in the file. Also it makes perfect sense for your provider to turn it back on. Many scripts rely on this function. Server- side machine reading of feeds is a very common practice. This script relies on it, but a lot of CMS- and Blog software also extensively use it.

Undocumented stuff

There is!

Created by Paul Cornelissen (pcmadman) The Bit-Byters Network
Visit pcmadman's last.fm profile or contact him at pcornelissen -at- gmail.com

Created by Paul Cornelissen (pcmadman)
Visit pcmadman's last.fm profile or contact him at pcornelissen -at- gmail.com

MENU

Last.fm Real Time Stats Read Me

Table of content

Terms of use:

Changing:

Contact:

Requirements

Installing

ToDo

Done

Things to know/F.A.Q.

Limits on database size:

Gap on Sundays

What load will this put on my server?

If you don't have crontabs

RTS missed some tracks!

I can't use dynamic images on <insert site>!

My host has ALLOW_URL_FOPEN OFF

Undocumented stuff