Notes on Hadoop on Amazon Elastic MapReduce

I am on a project which requires Hadoop for crunching millions of documents. Being new to Hadoop, I was faced with a learning curve.  Here are a couple of notes on the experience.

Hadoop Streaming

Working with streaming jobs on Hadoop is straightforward enough, except when it comes to controlling how Hadoop treats the input files.

The input files I needed to process were large (100's MB). Hadoop insisted on chunking those to smaller size but unfortunately the default splitting wasn't compatible with my file type.  I ended up fooling Hadoop in accepting bigger minimum split size:

-D mapred.min.split.size=10737418240

Hadoop with Custom JAR

Although one has full control of Hadoop when using custom JAR jobs, I must say I had quite a rocky ride getting my stuff to work.

First off, I needed a a way to package the dependent JAR librairies inside the main JAR.  I was forced to write a custom ANT build file to do this.

The trick is to get the dependent jars in the directory ./lib for Hadoop EMR (I don't know about the standard Hadoop, sorry) to be able to include those in the Java classpath.

Second, I wanted to use Amazon S3 as both input and output for files.  The trick there is to grab FileSystem objects based on the URI scheme of the files.

    Path opath=new Path(outputPath);

    FileSystem ofs=opath.getFileSystem(conf);

Forget a single of those and the whole job crashes and you end up paying Amazon for nothing... $100's of dollars down the drain ironing up this one.

Lego Style Software Design

My company recently got the chance to work on a very interesting project (which unfortunately I can't divulge at the moment).  What made the project fun to work on, aside from the product level challenges, was the fact that we were given much more architectural flexibility than usual.

Amongst the work emerged the following:

Python Functional Tools

I got the chance to come by a pretty cool project recently: Python Moka.  It consists of functional programming friendly implementations of standard Python dictionary and list classes.  Then it struck me:  wouldn't be nice to have Erlang-ish pattern matching functionality to Python?

So I crafted a small Python package scratching an itch I have had for way too long:  function dispatching based on pattern matching.  For those interested, here are the relevant links:

• Project on Github
• Main page on Systemical

Feedback welcome :)

EC2 architecture notes

I've updated the home page of Systemical: you'll find a bunch of useful links to documents:

• An EC2 architecture
• Distributed Leader Election Protocol
• Data-Flow based EC2 architecture

Enjoy :)

Amazon AWS tools - jldaws

Today I am open-sourcing yet another project.  It consists in a collection of Linux scripts related to Amazon Web Services (AWS).

The project's home page can be found here whilst the code repository is there.

My life in the Cloud

I am pretty much 100% in the Cloud now. Here's how I do it.

Services

• Email:  Google GMAIL (with "Tasks")
• Calendar:  Google Calendar
• Documents:  Google Docs (with a sprinkle of DropBox, going away soon enough)
• Music: combination of Rdio and Grooveshark
• Movies: Netflix  (on my Android devices, Apple TV and MacBook Pro)
• Code: Github
• Project Issue Tracker:  Lighthouse

What I gain:

• Mobility
• Availability
• Persistence
• Version Control

I don't need to worry about loosing / forgetting my laptop for work:  I just grab a spare one at work. Hence, business continuity is greatly improved over the traditional Microsoft "stack".

So, on my Linux/OSX PCs, I only need the following installed software:

• Google Chrome  (the auto-sync feature for bookmarks, extensions etc. is way cool :)
• Eclipse (with PyDev, Google Plugin, Egit)
• git

When it comes time to setting-up a complete working environment, I can turnaround one in under 1/2 hour! 

Security

The first reaction I get when I expose my workflow:  is this Cloud stuff secure?  To which I respond with a combination of the following:

• I would rather trust Google and their army of sysadmins than me when it comes to external security threats
• I never write down passwords nor credit card numbers in documents. Period.
• Doing business comes down to trust - being part of society requires trust
• My data is more secure located in the Cloud than on my laptop / desktop : those are too easy to lift

Availability

Another common reaction is about availability. I don't know about you guys, but Internet Access is pretty available nowadays and certainly Google is. For those of you worried about this dimension, consider:

• I have access through my Android mobile devices (carrier #1)
• I have access through my 3G dongle (carrier #2)
• I have access through my Cable Modem (carrier #3)

Yes I subscribe to 3 different carriers - I don't remember the last time I lost connectivity and was out of commission, business wise.

Software based Message Switch

Media-Keys over Web-Socket

My newest open-source project is a Media-Keys over Web-Socket application.  The project's details can be found here.

Application
The main use-case I wanted to address with this component is the ability to control music playback of web-sites such as GrooveShark and Rdio through the keyboard media-keys.

Usage
A Google Chrome extension will follow shortly.

Using Google Sites as Content Management System

I recently had to build a new web site from ground-up.  One thing I didn't want to do is go off installing Wordpress / Drupal etc. on a virtual machine.  As much as I love technology, I can't be bothered anymore with administrative tasks such as backups.
I decided to give Google Sites another go but I quickly was confronted with the limitations of this platform. That's when I decided to couple to great Google products:  Sites and AppEngine.

View the presentation

A Service Registration Web-Service API

It's been a while since I have last blogged... been very busy with my company Data-Tester Inc.

I have the pleasure to announce the immediate availability of another Web-Service hosted on Systemical : ipreg.  This service allows, for example, mobile devices (such as Android) to "discover" available LAN services).
Why is this useful at all you might ask?  For the curious:  it boils down to limitations related to multicast support in mobile devices.  For more details, read the details here.

iTunes playcount updater script

I am pretty busy these days with my new job ( I am heading a start-up in the telecommunications market ) but I took a couple of hours to craft a couple of scripts.

The first script (lastfm_gettracks) downloads all of a user's song track information from Last.fm. It can be found on Pypi here.  More information can be perused here.

The second script takes as input the file generated by the former script and updates the iTunes database on an OSX machine (sorry, I don't do windozes). Documentation for this script can be found here whilst package details can be found on Pypi here.

Logo contest

Help me choose a logo for my new company.  The logo contest on 99designs has ended and it is now time to vote!

VOTE HERE

Thanks!

Chrome Framed extension

I have just published my latest Google Chrome extension:  Chrome-Framed.  The extension allows opening a tab with 2 columns 'frame'.   This functionality enables a user 2 pages side-by-side in the same chrome window.

Enjoy!

Musicbrainz and Last.fm proxies updated

Just a quick note to let users of musicbrainz-proxy-dbus & lastfm-proxy-dbus applications that new versions are available on my PPA.  Notable changes include:

- Tray icon support: the main application window can now be closed and a tray icon can control them
- Help button linking to a web accessible page

Web Services available on Systemical.com

I've decided to open access to some web services I built.  These services are hosted on Google AppEngine and the documentation can be found here.

• Google Moderator: an RSS feed for a "series"
• Stackoverflow: an RSS feed for a user's statistics on http://stackoverflow.com/
• Github: an RSS feed for a user's statistics on http://github.com/

I will be opening access to more services so stay tuned!

Last.fm Proxy with DBus interface

I just realized I haven't made a public announcement regarding an application that constitutes a critical piece of my Rhythmbox arsenal:  " lastfm-proxy-dbus ".

This Linux Gnome application serves as "proxy" to a Last.fm user's scrobbling history. The data collected through this proxy serves to populate the "playcount" field of Rhythmbox's database.

More information can be found here.

Rhythmbox Synclastfm plugin

Today I announce the availability of a brand new version (2.02) of my Rhythmbox plugin "Synclastfm" (see previous post).

This version packs the much anticipated "batch update" feature i.e. the plugin will walk the user's history on Last.fm in order to update Rhythmbox tracks 'playcount' field.

More information can be found here.  Have fun and don't forget to provide feedback!

Musicbrainz Proxy service over DBus

I have just finished another building block on my quest to taming my music library: a Linux based application serving as 'proxy' (with caching of course) to Musicbrainz webservice.  The service is exposed through DBus.

Why?
Correlating music tracks across disparate applications / services can be daunting. For example, suppose one  scrobbles his/her tracks to Last.fm. Furthermore, let's say one wishes to keep his/her favorite Music Player in sync with his/her Last.fm profile.  This goal appears, on the surface, to be an easy one but as it turns out, support for finding & cataloging music tracks coherently across various applications / services remains elusive at the moment.

Also worth noting:  access to Musicbrainz's webservice is monitored.  Webservice API access is limited to 1call/sec: if many applications make uncoordinated access to the service, the user's machine might get blacklisted.  As I intend to make the track mbid a central piece of information for correlating my music tracks, I needed to address this issue.

Solution
With this application, it will be easier to integrate the various other building blocks of my music management arsenal. See the other project SyncLastfm that will benefit from this addition.

Project
The project's home page can be found here.  The associated Ohloh page can be found there.

Update:  the application will be installed under Applications / Other.
Update2: more information is accessible here.

Some Google Gadgets

I am in the process of building up my company website based on Google Apps, I need some Google Gadgets to embed but as I am unsure about the majority of the gadgets listed in the directory, I have created  some of my own.

• Skype Account Information
• Ohloh Profile Flair
• Stackoverflow Profile Flair
• Twitter Profile Flair with latest tweets

Links to the relevant gadgets are provided on my projects page.  I would rather not duplicate them here as I am most likely to forget to update them whereas on the project page they'll get my attention.

Last.fm Web Service desktop interface through DBus

Not too long ago, I started my quest to Music Library Faiyland (my daughter would love this quote). On this journey, I have completed a couple of intermediate steps ( Rhythmbox plugin - Syncing with Last.fm and Rhythmbox plugin for Last.fm desktop client ) and now I am happy to announce another milestone: a DBus accessible interface to Last.fm's Web Service API.

API
The current DBus interface is rudimentary but can be easily extended ( I have abstracted Last.fm's API to a manageable degree ). I will certainly be expanding the methods in the near future based on my needs and "customer feedback" ;-) Below is a non-exhaustive list of the API:

• Account related:
• get/set Username
• getAuthUrl : retrieve a URL for authenticating the service against a user
• setApiKey , setSecretKey
• Track related
• track.addTags
• track.getTags
• track.removeTag

I have only implemented these methods because they get me what I need at the moment: the possibility to "rate" my music tracks through a hack using the tags to hold the rating for a track. E.g. a 5 stars track would get "R5" as tag.

Message Based Design Technique
In this project, I have further experimented with a design technique based on "Message Passing". I have made some notes on the subject here.

Project
The home page of this project can be found on my Github repository.  A Debian package can be found on my Launchpad PPA.