Showing posts with label Programming. Show all posts
Showing posts with label Programming. Show all posts

Saturday, September 8, 2012

Notes on Hadoop on Amazon Elastic MapReduce

I am on a project which requires Hadoop for crunching millions of documents. Being new to Hadoop, I was faced with a learning curve.  Here are a couple of notes on the experience.

Hadoop Streaming

Working with streaming jobs on Hadoop is straightforward enough, except when it comes to controlling how Hadoop treats the input files.

The input files I needed to process were large (100's MB). Hadoop insisted on chunking those to smaller size but unfortunately the default splitting wasn't compatible with my file type.  I ended up fooling Hadoop in accepting bigger minimum split size:

-D mapred.min.split.size=10737418240

Hadoop with Custom JAR

Although one has full control of Hadoop when using custom JAR jobs, I must say I had quite a rocky ride getting my stuff to work.

First off, I needed a a way to package the dependent JAR librairies inside the main JAR.  I was forced to write a custom ANT build file to do this.

  

   
    
      
      
    
      
  


The trick is to get the dependent jars in the directory ./lib for Hadoop EMR (I don't know about the standard Hadoop, sorry) to be able to include those in the Java classpath.

Second, I wanted to use Amazon S3 as both input and output for files.  The trick there is to grab FileSystem objects based on the URI scheme of the files.

    Path opath=new Path(outputPath);
    FileSystem ofs=opath.getFileSystem(conf);

Forget a single of those and the whole job crashes and you end up paying Amazon for nothing... $100's of dollars down the drain ironing up this one.

Sunday, February 5, 2012

Python Functional Tools

I got the chance to come by a pretty cool project recently: Python Moka.  It consists of functional programming friendly implementations of standard Python dictionary and list classes.  Then it struck me:  wouldn't be nice to have Erlang-ish pattern matching functionality to Python?

So I crafted a small Python package scratching an itch I have had for way too long:  function dispatching based on pattern matching.  For those interested, here are the relevant links:



Feedback welcome :)

Thursday, February 2, 2012

EC2 architecture notes

I've updated the home page of Systemical: you'll find a bunch of useful links to documents:


Enjoy :)

Tuesday, June 15, 2010

Rhythmbox Synclastfm plugin

Today I announce the availability of a brand new version (2.02) of my Rhythmbox plugin "Synclastfm" (see previous post).

This version packs the much anticipated "batch update" feature i.e. the plugin will walk the user's history on Last.fm in order to update Rhythmbox tracks 'playcount' field.

More information can be found here.  Have fun and don't forget to provide feedback!

Monday, February 8, 2010

Last.fm Web Service desktop interface through DBus

Not too long ago, I started my quest to Music Library Faiyland (my daughter would love this quote). On this journey, I have completed a couple of intermediate steps ( Rhythmbox plugin - Syncing with Last.fm and Rhythmbox plugin for Last.fm desktop client ) and now I am happy to announce another milestone: a DBus accessible interface to Last.fm's Web Service API.

API
The current DBus interface is rudimentary but can be easily extended ( I have abstracted Last.fm's API to a manageable degree ). I will certainly be expanding the methods in the near future based on my needs and "customer feedback" ;-) Below is a non-exhaustive list of the API:

  • Account related:
    • get/set Username
    • getAuthUrl : retrieve a URL for authenticating the service against a user
    • setApiKey , setSecretKey
  • Track related
    • track.addTags
    • track.getTags
    • track.removeTag

I have only implemented these methods because they get me what I need at the moment: the possibility to "rate" my music tracks through a hack using the tags to hold the rating for a track. E.g. a 5 stars track would get "R5" as tag.

Message Based Design Technique
In this project, I have further experimented with a design technique based on "Message Passing". I have made some notes on the subject here.

Project
The home page of this project can be found on my Github repository.  A Debian package can be found on my Launchpad PPA.

Thursday, October 8, 2009

Erlang DBus Client Interface Library

When working on Linux, it is hard to ignore DBus for communicating with applications around the Desktop. Since I love Erlang so much, I wanted the capability to interface with DBus through Erlang scripts and/or applications hence I crafted an interface library.


The project in question is available at Erlang-DBus on GoogleCode. Note that the Erlang Port API library is also required: EPAPI.

Thursday, September 24, 2009

Mindmap of my projects

Here is a mindmap of my projects. I intend to keep this map up-to-date.

Monday, August 10, 2009

Daemons in Erlang

I've written Linux daemons in Python & C/C++ before and I must say that enough documentation is available online to make this experience as smooth as I believe it can be. Granted there are some interesting cases to take care of (e.g. zombies) but overall the procedure is fairly straightforward. Now my new found love in Erlang brings me once again to writing daemons.

First attempt
For my first go, I inspired myself with the procedure I had used with both Python & C/C++:
  • Use /var/run as a registry to hold the running state of the daemon
  • For starting the daemon, check /var/run for a running PID & if found, assume it is that of the daemon and abort
  • For stopping the daemon, check /var/run for a running PID & if not found, assume no daemon instance is running and thus spawn one
It turned out that to work OK but it seemed utterly complex and inflexible given that Erlang provides such excellent communication capabilities. On my next project, I had to try something else.
My second attempt
This time around, I relied on Erlang's intrinsic distributed programming capabilities to help manage the life-cycle of the daemon.
  1. A Python script serves as top-level manager ( it could very well be written in bash or Perl but I happen to like Python a lot and given its quasi ubiquity on Linux based distros, it is a safe operational choice ). The said script exports the two familiar management commands: start and stop.
  2. Start procedure: the manager script spawns a controller escript (Erlang Script) with for command-line parameter "status" . The return code ( exported through Erlang's erlang:halt/1 function ) is inspected by the manager script to determine if a daemon is already running. If not, the manager script spawns (with the option -detached) the actual Erlang based daemon.
  3. Stop procedure: the manager script spawns a controller escript with for command-line parameter "stop". The said controller script uses Erlang's rpc module to query a potential running daemon for its process PID (the Erlang emulator system PID (retrievable through os:getpid/0) i.e. not the PID of the daemon running process in the Erlang emulator). If the said RPC call succeed, the controller script issues a system command "kill -9 PID".
The communication between the controller script and the actual daemon is done through RPC and requires that both ends register with EPMD (Erlang's Port Mapper Daemon). I used the "short naming" convention to achieve this (the -sname option for erl).

Example of controller script written in escript:



#!/usr/bin/env escript
%% -*- erlang -*-
%%! -sname etrx_control
%%
%% @author Jean-Lou Dupont
%%

code_ok() -> 0.
code_error() -> 1.
code_daemon_found() -> 2.
code_daemon_not_found() -> 3.
code_lib_not_found() -> 4.
code_node_not_found() -> 5.


err_lib() -> "erlang-transmission not found".
err_daemon() -> "daemon not found".
err_node() -> "transmission node not found".

msg_pid() -> "daemon found, pid: ".
msg_usage() -> "usage: etrx_control [-q] [status|stop]".
msg_kill() -> "stop command sent".


main(["-q", "stop"]) -> run(quiet, stop);
main(["-q", "status"]) -> run(quiet, status);
main(["stop"]) -> run(verbose, stop);
main(["status"]) -> run(verbose, status);

main([]) ->
msg(verbose, code_ok(), msg_usage()),
halt(code_ok());

main([_Cmd]) ->
msg(verbose, code_ok(), msg_usage()),
halt(code_error()).


run(Feedback, stop) ->

add_cwd(),

case getstatus() of
daemon_not_found ->
msg(Feedback, code_daemon_not_found(), err_daemon());

{pid, Pid} ->
os:cmd("kill -9 "++Pid),
msg(Feedback, code_ok(), msg_kill());

{error, lib_not_found} ->
msg(Feedback, code_lib_not_found(), err_lib());

{error, node_not_found} ->
msg(Feedback, code_node_not_found(), err_node())
end;


run(Feedback, status) ->

%%for development
add_cwd(),

case getstatus() of
daemon_not_found ->
msg(Feedback, code_daemon_found(), err_daemon());

{pid, Pid} ->
msg(Feedback, code_daemon_found(), msg_pid(), Pid);

{error, lib_not_found} ->
msg(Feedback, code_lib_not_found(), err_lib());

{error, node_not_found} ->
msg(Feedback, code_node_not_found(), err_node())

end.


add_cwd() ->
{ok,Cwd}=file:get_cwd(),
Cp=Cwd++"/ebin",
code:add_pathsa([Cp]).


getstatus() ->
try
Status=rpc(status),
case Status of
rpcerror ->
daemon_not_found;

{pid, Pid} ->
{pid, Pid}
end
catch
error:undef ->
{error, lib_not_found};

_X:_Y ->
{error, node_not_found}
end.


msg(Feedback, Code, Msg) ->
case Feedback of
verbose ->
io:format("etrx_control: ~s~n", [Msg]);
_ ->
ok
end,
halt(Code).

msg(Feedback, Code, Msg1, Msg2) ->
msg(Feedback, Code, Msg1++Msg2).


%%%%%%%%%%%%%%
%% RPC related
%%%%%%%%%%%%%%

rpc(Command) ->
case dorpc(Command) of
rpcerror ->
daemon_not_found;

Response ->
Response
end.



dorpc(Message) ->
Node=tools:make_node(transmission),

case rpc:call(Node, transmission_daemon, api, [Message], 2000) of
{badrpc, _Reason} ->
rpcerror;

Other ->
Other
end.

Tuesday, June 30, 2009

Erlang Syntax Highlighter


/**
* Code Syntax Highlighter.
* Version 1.5.2
* Copyright (C) 2004-2008 Alex Gorbatchev
* http://www.dreamprojections.com/syntaxhighlighter/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, version 3 of the License.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see http://www.gnu.org/licenses/.
*/

/*
* Erlang syntax contributed by Jean-Lou Dupont http://www.jldupont.com/
* file: shBrushErlang.js
*/
dp.sh.Brushes.Erlang = function()
{
// According to: http://erlang.org/doc/reference_manual/introduction.html#1.5
var keywords = 'after and andalso band begin bnot bor bsl bsr bxor '+
'case catch cond div end fun if let not of or orelse '+
'query receive rem try when xor'+
// additional
' module export import define';

this.regexList = [
{ regex: new RegExp("[A-Z][A-Za-z0-9_]+", 'g'), css: 'vars' },
{ regex: new RegExp("\\%.+", 'gm'), css: 'comment' },
{ regex: new RegExp("\\?[A-Za-z0-9_]+", 'g'), css: 'preprocessor' },
{ regex: new RegExp("[a-z0-9_]+:[a-z0-9_]+", 'g'), css: 'mod_func' },

{ regex: new RegExp('"(?!")(?:\\.|\\\\\\"|[^\\""\\n\\r])*"', 'gm'), css: 'string' },
{ regex: new RegExp("'(?!')(?:\\.|(\\\\\\')|[^\\''\\n\\r])*'", 'gm'), css: 'string' },

{ regex: new RegExp(this.GetKeywords(keywords), 'gm'), css: 'keyword' },
];

this.CssClass = 'dp-erl';
this.Style = '.dp-erl .vars { color: rgb(184,134,11); }' +
'.dp-erl .mod_func { color: #CC00FF; }';
};

dp.sh.Brushes.Erlang.prototype = new dp.sh.Highlighter();
dp.sh.Brushes.Erlang.Aliases = ['erl', 'erlang'];


Test:


%% Author: Jean-Lou Dupont
%% Created: 2009-06-19
%% Description: Phidget InterfaceKit driver
%%
%% MESSAGES GENERATED:
%% ===================
%% {phidgeterror, {{Serial, Code, String}, date(), time(), now() }}
%% {device, {{Serial,Type,State, Version,Name,Label}, date(), time(), now() }}
%% {din, {{Serial, Index, Value}, date(), time(), now() }}
%% {dout, {{Serial, Index, Value}, date(), time(), now() }}
%% {sout, {{Serial, Index, Value}, date(), time(), now() }}
%%
%%
%% SUBSCRIPTIONS:
%% ==============
-module(ifk).

%%
%% MACROS
%%
-define(DRV_IFK_DEBUG, "pem_drv_ifk_debug").
-define(DRV_IFK, "pem_drv_ifk").

-define(SUBS, [phidgetdevice]).

%%
%% Exported Functions
%%
-export([
start_link/0,
start_link/1,
stop/0
]).

-export([
loop/0,
loop_handler/1,
handle_phidgetdevice/2,
filter_device/4,
handle_ifk/3,
handle_active/2,
handle_active/4,
handle_inactive/2,
handle_inactive/4,
ifk_drv/2,
send_to_reflector/1,
handle_crashed_driver/1,
clean_driver/1
]).

%% =============
%% API Functions
%% =============
start_link() ->
start_link([]).

start_link(Args) ->

{debug, Debug}=base:kfind(debug, Args,false),
DrvPath = base:pole(Debug, true, false, ?DRV_IFK_DEBUG, ?DRV_IFK),
LD = [{driver_path, DrvPath}],
NArgs = lists:append(Args, LD),

base:ilog(?MODULE, "start_link: Args[~p]~n",[NArgs]),

Pid = spawn_link(?MODULE, loop, []),
register( ?MODULE, Pid ),

?MODULE ! {args, NArgs},
{ok, Pid}.

stop() ->
?MODULE ! stop.

%% =====================================================
%% MAIN LOOP
%% =====================================================
loop() ->
receive

%% Send the 'ready' signal
{args, Args} ->
put(args, Args),
{driver_path, DrvPath} = base:kfind(driver_path, Args),
put(driver_path, DrvPath),
switch:subscribe(?MODULE, ?SUBS);

%% Send the 'ready' signal
{switch, subscribed} ->
%%base:ilog(?MODULE, "subscribed~n",[]),
switch:publish(?MODULE, ready, self());


stop ->
base:ilog(?MODULE, "exiting~n", []),
exit(ok);


%%verify that it is an "InterfaceKit" device
{_From, phidgetdevice, {M, Ts}} ->
%%base:ilog(?MODULE,"received 'phidgetdevice'~n", []),
handle_phidgetdevice(M, Ts);

{driver, Serial, Port, Pid} ->
base:ilog(?MODULE,"received driver info, Serial[~p] Port[~p] Pid[~p]~n", [Serial, Port, Pid]),
put({port, Serial}, Port),
put({pid, Serial}, Pid),
put({serial, Port}, Serial);

{crashed, Port} ->
clean_driver(Port);

%%don't know what todo
Other ->
base:ilog(?MODULE,"received unknown msg: [~p]~n", [Other])

end,
?MODULE:loop().

clean_driver(undefined) ->
base:ilog(?MODULE, "clean_driver: received undefined", []);

clean_driver(Port) ->
Serial = get({serial, Port}),
base:ilog(?MODULE, "clean_driver: Serial[~p] Port[~p]~n", [Serial, Port]),
erase({port, Serial}),
erase({pid, Serial}),
erase({serial, Port}).




%% =====================
%% HANDLER
%% =====================
handle_phidgetdevice(Msg, Ts) ->
%error_logger:info_msg("~p: handle_phidgetdevice, Msg[~p]~n", [?MODULE, Msg]),
{Serial, Type, State} = Msg,
filter_device(Serial, Type, State, Ts).

% we just want the InterfaceKit devices!
filter_device(Serial, Type, State, Ts) ->
case Type of
"PhidgetInterfaceKit" ->
handle_ifk(Serial, State,Ts);

_ ->
notifk
end.

%% Spawn 1 driver for each InterfaceKit device in "active" state
%% and get rid of detached device(s)
handle_ifk(Serial, inactive, Ts) ->
handle_inactive(Serial, Ts),
ok;

handle_ifk(Serial, active, Ts) ->
%error_logger:info_msg("~p: handle_ifk: Serial[~p] active~n", [?MODULE, Serial]),
handle_active(Serial, Ts),
ok;

handle_ifk(Serial, State, _) ->
base:elog(?MODULE, "handle_ifk: Serial[~p] INVALID STATE[~p]~n", [Serial, State]),
ok.


%% Open the driver if not already done
handle_active(Serial, Ts) ->
Port = get({port, Serial}),
Pid = get({pid, Serial}),
handle_active(Serial, Port, Pid, Ts).

handle_active(Serial, _, undefined, Ts) ->
handle_active(Serial, undefined, invalid, Ts);

% not sure this one is required
handle_active(Serial, undefined, undefined, Ts) ->
handle_active(Serial, undefined, invalid, Ts);


% Not active... yet
handle_active(Serial, undefined, invalid, _Ts) ->
DriverPath = get(driver_path),
%%error_logger:info_msg("~p: handle_active: DriverPath[~p]~n", [?MODULE, DriverPath]),
Pid = spawn(?MODULE, ifk_drv, [DriverPath, Serial]),
base:ilog(?MODULE, "handle_active: Serial[~p] Pid[~p]~n", [Serial, Pid]),
ok;


% Is it really active?
handle_active(Serial, _Port, Pid, Ts) ->
Active = is_process_alive(Pid),
case Active of
true ->
ok;
false ->
% clean-up required!
erase({pid, Serial}),
erase({port, Serial}),
handle_active(Serial, undefined, undefined, Ts)
end.


%% Close the driver if still active
handle_inactive(Serial, Ts) ->
Port = get({port, Serial}),
Pid = get({pid, Serial}),
handle_inactive(Serial, Port, Pid, Ts),
ok.

%not even defined it seems... nothing much to do
handle_inactive(_Serial, undefined, _, _) ->
ok;

handle_inactive(_Serial, _, undefined, _) ->
ok;

handle_inactive(Serial, Port, Pid, _Ts) ->
Active = is_process_alive(Pid),
case Active of
true ->
erlang:port_close(Port),
erlang:exit(Pid, kill),
erase({port, Serial}),
erase({pid, Serial}),
ok;
false ->
ok
end,
ok.

ifk_drv(ExtPrg, Serial) ->
%%error_logger:info_msg("~p:ifk_drv: Serial[~p] Pid[~p]~n",[?MODULE, Serial, self()]),
process_flag(trap_exit, true),
Param = erlang:integer_to_list(Serial),
Port = open_port({spawn, ExtPrg++" "++Param}, [{packet, 2}, binary, exit_status]),
error_logger:info_msg("~p: ifk_drv: Serial[~p] Port[~p] Pid[~p]~n",[?MODULE, Serial, Port, self()]),
put({port, Serial}, Port),
put({serial, Port}, Serial),

% signal back some useful mapping
?MODULE ! {driver, Serial, Port, self()},
loop_handler(Port).


%% =====================
%% IFK DRIVER loop
%% =====================


loop_handler(Port) ->
receive

{Port, {exit_status, _}} ->
base:elog(?MODULE, "loop_handler: an ifk driver exited/could not load~n", []),
handle_crashed_driver(Port),
exit(crashed);


{Port, {data, Data}} ->
Decoded = binary_to_term(Data),
%%base:ilog(?MODULE, "loop_handler: decoded msg[~p]~n", [Decoded]),
send_to_reflector(Decoded);

Msg ->
base:ilog(?MODULE,"loop_handler: msg[~p]~n", [Msg])
end,
loop_handler(Port).


send_to_reflector(Decoded) ->
{Msgtype, Msg} = Decoded,
M = {Msg, {date(), time(), now()}},
switch:publish(?MODULE, Msgtype, M).


%% =======================
%% CRASHED DRIVER RECOVERY
%% =======================


handle_crashed_driver(Port) ->
error_logger:warning_msg("~p: handle_crashed_driver: Port[~p] Pid[~p]~n", [?MODULE, Port, self()]),
?MODULE ! {crashed, Port}.

Tuesday, June 9, 2009

EPAPI: Erlang Port driver API library

Some time ago, I decided to turn to Erlang for my pet project (more and this in a later post). Erlang is a very nice language, it features:
  • asynchronous messaging
  • soft real-time capabilities
  • built-in multi-threading (with O(1) scheduler)
  • fault-tolerance & high availability functions (OTP release)
  • much more...
My project requires interfacing to Phidgets devices (USB connected sensors & other interface) and no Erlang client libraries exist for those so I decided to roll my own (I'll share my library at a later time).

The first step in interfacing with the Phidgets from Erlang was to devise a Port Driver library which I am making available here. For those who wish to get a taste of the API, jump directly to the documentation.

Have fun and don't hesitate to comment/contribute!

Sunday, May 31, 2009

Chrome extension for Diigo

UPDATED: Chrome's Extension subsystem keeps evolving. The latest changes break my extension and so I'll wait until Chrome is more stable before attempting to reissue this one.

I love the speed and sleeknest of Google Chrome but one thing that disappointed me was the lack of extensions. Now the wait is over: there is support for extensions in the developper release of Chrome.

Diigo's bookmarklet is nice but having to press the bookmarklet button each time I visit a site isn't much fun so I decided to package an extension for it: the extension is available here.

For those of you who do not know Diigo, one feature that distinguishes it from the other social bookmarking sites (such as Delicious) is its highlighting feature.

Thursday, February 19, 2009

First contact with Microsoft Azure

I got my LiveID invitation code 2days ago so I decided to invest 15minutes evaluating Azure. I am already pretty familiar with Google AppEngine having developed my own Web site on it but since I am striving to be "technology agnostic", I needed to investigate Microsoft Azure PaaS.


Revision Control
My first shock was to learn that SVN integration does not exist with Microsoft Visual Studio. I wonder how Microsoft developers handle this critical activity. This process must be addressed but I do not know how, being a Microsoft .NET newbie.

Deployment
It appears as though the VS environment automates the creation of the CS file ( akin to a Java WAR file it seems ). But the kicker: one must go through the web portal to upload the file to the Azure fabric... a tedious process to say the least compared to the "one click" AppEngine deployment made possible through some simple scripting in Eclipse.

I'll surely follow the "Hello World" tutorial and hopefully get a better feel for Azure in the coming weeks.

Monday, January 12, 2009

My Python library: command-line backup tools mainly

Through my daytime work, I use online tools a lot ( e.g. Gliffy, MindMeister )  so one could say I am confident about conducting work online.  Sometimes though I feel I need a safety net and this is why I like to use my moonlight time to calm my qualms about the possibility of loosing my work.


To this effect, I have created a python library with several command-line tools ( cross-platform i.e. Windows and Linux, tested on Ubuntu ) to perform, amongst other tasks, backup of my online documents.  The library documentation page is available here.

Thursday, December 18, 2008

Gliffy diagrams backup tool

Delicious  is a useful tool. I use it in an unconventional way sometimes: I bookmark some resources with tags which serve as indicators for my backup tools. As an example, when I tag my Gliffy diagram pages with the my-diagrams tag, my backup tool grabs the corresponding diagrams and makes local copies on my server.

The backup tool I am referring to is readily available here.  I configure the backup tool to run periodically through cron jobs on my Linux server.

Friday, October 31, 2008

MySQL load balancing



1) The traditional way of handling a two-tiered WEB architecture:


2) The software load-balancing approach:

3) The hardware load-balancing way:



With a versatile (i.e. scriptable) hardware load-balancer (e.g. F5's BIG-IP), one could scale the traditional architecture (see 1) without having to change the application. In this situation, one is trading software development effort against the cost of additional equipment.


Tuesday, October 28, 2008

Green Software: From Garbage Collection (GC) to Resource Recycling

I confess: I am a recycling freak.  When it comes to better our environment, I am all ears. 


I would like to offer a suggestion to the software community: in the green spirit, why couldn't we use the term Resource Recycling instead of referring to Garbage Collection?  That would help (albeit just a tad) setting the green reflex throughout the younger generations.

Wednesday, October 15, 2008

Auto-properties for Subversion (Subclipse)

Too many times I had to dig the web to find how the Subclipse Eclipse plugin could be configured for auto-properties so I decided I would keep a record here.


1) Create a directory where the config file will be stored. On my development machine, I usually put it somewhere like worspace/configs/subclipse/config
2) Create a file named config in the directory in question:


3) Finally, go to Windows->Preferences->Team->SVN and the Configuration Location to Use Directory and browse to the aforementioned directory created in step 1.



Wednesday, September 3, 2008

Add to Google Reader bookmarklet

Wow!  Google Chrome is out and of course I couldn't wait to try it out... that was until I discovered that automatic RSS/ATOM feed detection wasn't part of the deal.  I wrote a quick bookmarklet for addressing this situation: just drag-and-drop the following bookmarklet to your bookmarks (works on Chrome, Firefox and IE): AddToGoogleReader

The code is hosted on Google AppEngine so performance shouldn't be an issue. Have fun!

Thursday, July 17, 2008

Design Pattern in PHP: Object Pool Pattern

Still working on my AOP framework for PHP, I stumble upon an interesting problem: when there are many files that require weaving, at some point memory usage was spiking very high (north of 256M) . For environments where memory is a scarce resource, I needed object recycling.

Here is what I came up with: