Mindmap of my projects

Here is a mindmap of my projects. I intend to keep this map up-to-date.

ejabberd: changing the Erlang cookie for inter-node communications

I was recently playing around with ejabberd crafting myself a module in Erlang. I require the module to communicate with an external node through the RPC facility. Of course, Erlang mandates having the same cookie value throughout a node domain. To make matters worse, ejabberd does not appear to allow configuring the said cookie.

A solution to this problem: copy the desired .erlang.cookie file to /var/lib/erjabberd with the proper permissions (user: ejabberd, group: ejabberd, user-read-only).

Let me find the bug tracker for ejabberd now...

Daemons in Erlang

I've written Linux daemons in Python & C/C++ before and I must say that enough documentation is available online to make this experience as smooth as I believe it can be. Granted there are some interesting cases to take care of (e.g. zombies) but overall the procedure is fairly straightforward. Now my new found love in Erlang brings me once again to writing daemons.

First attempt

For my first go, I inspired myself with the procedure I had used with both Python & C/C++:

• Use /var/run as a registry to hold the running state of the daemon
• For starting the daemon, check /var/run for a running PID & if found, assume it is that of the daemon and abort
• For stopping the daemon, check /var/run for a running PID & if not found, assume no daemon instance is running and thus spawn one

It turned out that to work OK but it seemed utterly complex and inflexible given that Erlang provides such excellent communication capabilities. On my next project, I had to try something else.

My second attempt

This time around, I relied on Erlang's intrinsic distributed programming capabilities to help manage the life-cycle of the daemon.

1. A Python script serves as top-level manager ( it could very well be written in bash or Perl but I happen to like Python a lot and given its quasi ubiquity on Linux based distros, it is a safe operational choice ). The said script exports the two familiar management commands: start and stop.
2. Start procedure: the manager script spawns a controller escript (Erlang Script) with for command-line parameter "status" . The return code ( exported through Erlang's erlang:halt/1 function ) is inspected by the manager script to determine if a daemon is already running. If not, the manager script spawns (with the option -detached) the actual Erlang based daemon.
3. Stop procedure: the manager script spawns a controller escript with for command-line parameter "stop". The said controller script uses Erlang's rpc module to query a potential running daemon for its process PID (the Erlang emulator system PID (retrievable through os:getpid/0) i.e. not the PID of the daemon running process in the Erlang emulator). If the said RPC call succeed, the controller script issues a system command "kill -9 PID".

The communication between the controller script and the actual daemon is done through RPC and requires that both ends register with EPMD (Erlang's Port Mapper Daemon). I used the "short naming" convention to achieve this (the -sname option for erl).

Example of controller script written in escript:

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -sname etrx_control
%%
%% @author Jean-Lou Dupont
%%

code_ok()               -> 0.
code_error()            -> 1.
code_daemon_found()     -> 2.
code_daemon_not_found() -> 3.
code_lib_not_found()    -> 4.
code_node_not_found()   -> 5.


err_lib()    -> "erlang-transmission not found".
err_daemon() -> "daemon not found".
err_node()   -> "transmission node not found".

msg_pid()    -> "daemon found, pid: ".
msg_usage()  -> "usage: etrx_control [-q] [status|stop]".
msg_kill()   -> "stop command sent".


main(["-q", "stop"]) ->  run(quiet,   stop);
main(["-q", "status"]) -> run(quiet,   status);
main(["stop"]) ->   run(verbose, stop);
main(["status"]) ->   run(verbose, status);
 
main([]) ->
 msg(verbose, code_ok(), msg_usage()),
 halt(code_ok());
 
main([_Cmd]) ->
 msg(verbose, code_ok(), msg_usage()),
 halt(code_error()).


run(Feedback, stop) ->

 add_cwd(),
 
 case getstatus() of
  daemon_not_found ->
   msg(Feedback, code_daemon_not_found(), err_daemon());
     
  {pid, Pid} ->
   os:cmd("kill -9 "++Pid),
   msg(Feedback, code_ok(), msg_kill());
   
  {error, lib_not_found} ->
   msg(Feedback, code_lib_not_found(), err_lib());
   
  {error, node_not_found} ->
   msg(Feedback, code_node_not_found(), err_node())  
 end;  
 

run(Feedback, status) ->

 %%for development
 add_cwd(),
 
 case getstatus() of
  daemon_not_found ->
   msg(Feedback, code_daemon_found(), err_daemon());
     
  {pid, Pid} ->
   msg(Feedback, code_daemon_found(), msg_pid(), Pid);
   
  {error, lib_not_found} ->
   msg(Feedback, code_lib_not_found(), err_lib());
   
  {error, node_not_found} ->
   msg(Feedback, code_node_not_found(), err_node())  
 
 end.
 

add_cwd() ->
 {ok,Cwd}=file:get_cwd(),
 Cp=Cwd++"/ebin",
 code:add_pathsa([Cp]).


getstatus() ->
 try
  Status=rpc(status),
  case Status of
   rpcerror ->
    daemon_not_found;
    
   {pid, Pid} ->
    {pid, Pid} 
  end
 catch
  error:undef ->
   {error, lib_not_found};
  
  _X:_Y ->
   {error, node_not_found}
 end.


msg(Feedback, Code, Msg) ->
 case Feedback of 
  verbose ->
   io:format("etrx_control: ~s~n", [Msg]);
  _ ->
   ok
 end, 
 halt(Code).
 
msg(Feedback, Code, Msg1, Msg2) ->
 msg(Feedback, Code, Msg1++Msg2).


%%%%%%%%%%%%%% 
%% RPC related
%%%%%%%%%%%%%%
 
rpc(Command) ->
 case dorpc(Command) of
  rpcerror ->
   daemon_not_found;
 
  Response ->
   Response
 end.
 


dorpc(Message) ->
 Node=tools:make_node(transmission),
 
 case rpc:call(Node, transmission_daemon, api, [Message], 2000) of
  {badrpc, _Reason} ->
   rpcerror;
  
  Other ->
   Other
 end.

Social toolbox

I am extensively online for both my personal & professional lives. Over the years, I've come to integrate a number of tools for keeping organized whilst socializing. My main requirements are:

• Capacity to organize feed items (e.g. categorize)
• Capacity to publish (share) selected feed items
• Capacity to subscribe to feeds
• Capacity to view feed items in real-time (as much as possible)
• Capacity to view feed items in one window

Following is a diagram of my current toolbox that meets these requirements:

Sites without an RSS/ATOM feed

Once in a while I encounter sites without an RSS/ATOM feed (mainly on "local sites" e.g. town news) . I use Yahoo! Pipes to build a feed in such cases.

What is Diigo?

Diigo is a cool service: it is a social bookmarking site sporting the added features of "text highlighting" and sticky notes.

What tools would you suggest I add to my arsenal?

Erlang Syntax Highlighter

/**
 * Code Syntax Highlighter.
 * Version 1.5.2
 * Copyright (C) 2004-2008 Alex Gorbatchev
 * http://www.dreamprojections.com/syntaxhighlighter/
 *
 *  This program is free software: you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation, version 3 of the License.
 *
 *  This program is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *  GNU General Public License for more details.
 *
 *  You should have received a copy of the GNU General Public License
 *  along with this program.  If not, see http://www.gnu.org/licenses/.
 */

/*   
 * Erlang syntax contributed by Jean-Lou Dupont  http://www.jldupont.com/  
 * file: shBrushErlang.js
 */
dp.sh.Brushes.Erlang = function()
{
 // According to: http://erlang.org/doc/reference_manual/introduction.html#1.5
    var keywords = 'after and andalso band begin bnot bor bsl bsr bxor '+
        'case catch cond div end fun if let not of or orelse '+
        'query receive rem try when xor'+
        // additional
        ' module export import define';

    this.regexList = [
        { regex: new RegExp("[A-Z][A-Za-z0-9_]+",      'g'), css: 'vars' },
        { regex: new RegExp("\\%.+",                  'gm'), css: 'comment' },
        { regex: new RegExp("\\?[A-Za-z0-9_]+",        'g'), css: 'preprocessor' },
        { regex: new RegExp("[a-z0-9_]+:[a-z0-9_]+",   'g'), css: 'mod_func' },
        
        { regex: new RegExp('"(?!")(?:\\.|\\\\\\"|[^\\""\\n\\r])*"', 'gm'), css: 'string' },
        { regex: new RegExp("'(?!')(?:\\.|(\\\\\\')|[^\\''\\n\\r])*'", 'gm'), css: 'string' },

        { regex: new RegExp(this.GetKeywords(keywords), 'gm'), css: 'keyword' },
        ];

    this.CssClass = 'dp-erl';
    this.Style =    '.dp-erl .vars     { color: rgb(184,134,11); }' +
                    '.dp-erl .mod_func { color: #CC00FF; }';
};

dp.sh.Brushes.Erlang.prototype  = new dp.sh.Highlighter();
dp.sh.Brushes.Erlang.Aliases    = ['erl', 'erlang'];

Test:

%% Author: Jean-Lou Dupont
%% Created: 2009-06-19
%% Description: Phidget InterfaceKit driver
%%
%% MESSAGES GENERATED:
%% ===================
%% {phidgeterror, {{Serial, Code, String},                  date(), time(), now() }}
%% {device,       {{Serial,Type,State, Version,Name,Label}, date(), time(), now() }}
%% {din,          {{Serial, Index, Value},                  date(), time(), now() }}
%% {dout,         {{Serial, Index, Value},                  date(), time(), now() }}
%% {sout,         {{Serial, Index, Value},                  date(), time(), now() }}
%%
%%
%% SUBSCRIPTIONS:
%% ==============
-module(ifk).

%%
%% MACROS
%%
-define(DRV_IFK_DEBUG, "pem_drv_ifk_debug").
-define(DRV_IFK,       "pem_drv_ifk").

-define(SUBS,          [phidgetdevice]).

%%
%% Exported Functions
%%
-export([
  start_link/0,   
  start_link/1,  
  stop/0
        ]).

-export([
   loop/0,
   loop_handler/1,
   handle_phidgetdevice/2,
   filter_device/4,
   handle_ifk/3,
   handle_active/2,
   handle_active/4,
   handle_inactive/2,
   handle_inactive/4,
   ifk_drv/2,
   send_to_reflector/1,
   handle_crashed_driver/1,
   clean_driver/1
   ]).

%% =============
%% API Functions
%% =============
start_link() ->
 start_link([]).

start_link(Args) ->
  
 {debug, Debug}=base:kfind(debug, Args,false),
 DrvPath = base:pole(Debug, true, false, ?DRV_IFK_DEBUG, ?DRV_IFK),   
 LD = [{driver_path, DrvPath}],
 NArgs = lists:append(Args, LD),
 
 base:ilog(?MODULE, "start_link: Args[~p]~n",[NArgs]),

 Pid = spawn_link(?MODULE, loop, []),
 register( ?MODULE, Pid ),
 
 ?MODULE ! {args, NArgs},
 {ok, Pid}.

stop() ->
 ?MODULE ! stop.

%% =====================================================
%% MAIN LOOP
%% =====================================================
loop() ->
 receive
  
  %% Send the 'ready' signal
  {args, Args} ->
   put(args, Args),
   {driver_path, DrvPath} = base:kfind(driver_path, Args),
   put(driver_path, DrvPath),
   switch:subscribe(?MODULE, ?SUBS);
  
  %% Send the 'ready' signal
  {switch, subscribed} ->
   %%base:ilog(?MODULE, "subscribed~n",[]),
   switch:publish(?MODULE, ready, self());
  
  
  stop ->
   base:ilog(?MODULE, "exiting~n", []),
   exit(ok);

  
  %%verify that it is an "InterfaceKit" device
  {_From, phidgetdevice, {M, Ts}} ->
   %%base:ilog(?MODULE,"received 'phidgetdevice'~n", []),
   handle_phidgetdevice(M, Ts);
  
  {driver, Serial, Port, Pid} ->
   base:ilog(?MODULE,"received driver info, Serial[~p] Port[~p] Pid[~p]~n", [Serial, Port, Pid]),
   put({port,  Serial}, Port),
   put({pid,   Serial}, Pid),
   put({serial, Port},  Serial);
  
  {crashed, Port} ->
   clean_driver(Port);
  
  %%don't know what todo
  Other ->
   base:ilog(?MODULE,"received unknown msg: [~p]~n", [Other])
 
 end,
 ?MODULE:loop().

clean_driver(undefined) ->
 base:ilog(?MODULE, "clean_driver: received undefined", []);

clean_driver(Port) ->
 Serial = get({serial, Port}),
 base:ilog(?MODULE, "clean_driver: Serial[~p] Port[~p]~n", [Serial, Port]),
 erase({port, Serial}),
 erase({pid, Serial}),
 erase({serial, Port}).




%% =====================
%% HANDLER
%% =====================
handle_phidgetdevice(Msg, Ts) ->
 %error_logger:info_msg("~p: handle_phidgetdevice, Msg[~p]~n", [?MODULE, Msg]),
 {Serial, Type, State} = Msg,
 filter_device(Serial, Type, State, Ts).

% we just want the InterfaceKit devices!
filter_device(Serial, Type, State, Ts) ->
 case Type of
  "PhidgetInterfaceKit" ->
   handle_ifk(Serial, State,Ts);
 
  _ ->
   notifk
 end.

%% Spawn 1 driver for each InterfaceKit device in "active" state
%%  and get rid of detached device(s)
handle_ifk(Serial, inactive, Ts) ->
 handle_inactive(Serial, Ts),
 ok;

handle_ifk(Serial, active, Ts) ->
 %error_logger:info_msg("~p: handle_ifk: Serial[~p] active~n", [?MODULE, Serial]),
 handle_active(Serial, Ts),
 ok;

handle_ifk(Serial, State, _) ->
 base:elog(?MODULE, "handle_ifk: Serial[~p] INVALID STATE[~p]~n", [Serial, State]),
 ok.


%% Open the driver if not already done
handle_active(Serial, Ts) ->
 Port = get({port, Serial}),
 Pid  = get({pid,  Serial}),
 handle_active(Serial, Port, Pid, Ts).

handle_active(Serial, _, undefined, Ts) ->
 handle_active(Serial, undefined, invalid, Ts);

% not sure this one is required
handle_active(Serial, undefined, undefined, Ts) ->
 handle_active(Serial, undefined, invalid, Ts);


% Not active... yet
handle_active(Serial, undefined, invalid, _Ts) ->
 DriverPath = get(driver_path),
 %%error_logger:info_msg("~p: handle_active: DriverPath[~p]~n", [?MODULE, DriverPath]),
 Pid = spawn(?MODULE, ifk_drv, [DriverPath, Serial]),
 base:ilog(?MODULE, "handle_active: Serial[~p] Pid[~p]~n", [Serial, Pid]), 
 ok;


% Is it really active?
handle_active(Serial, _Port, Pid, Ts) ->
 Active = is_process_alive(Pid),
 case Active of
  true ->
   ok;
  false ->
   % clean-up required!
   erase({pid, Serial}),
   erase({port, Serial}),
   handle_active(Serial, undefined, undefined, Ts)
 end.


%% Close the driver if still active
handle_inactive(Serial, Ts) ->
 Port = get({port, Serial}),
 Pid  = get({pid, Serial}),
 handle_inactive(Serial, Port, Pid, Ts),
 ok.

%not even defined it seems... nothing much to do
handle_inactive(_Serial, undefined, _, _) ->
 ok;

handle_inactive(_Serial, _, undefined, _) ->
 ok;

handle_inactive(Serial, Port, Pid, _Ts) ->
 Active = is_process_alive(Pid),
 case Active of
  true ->
   erlang:port_close(Port),
   erlang:exit(Pid, kill),
   erase({port, Serial}),
   erase({pid,  Serial}),
   ok;
  false ->
   ok
 end,
 ok.

ifk_drv(ExtPrg, Serial) ->
 %%error_logger:info_msg("~p:ifk_drv: Serial[~p] Pid[~p]~n",[?MODULE, Serial, self()]),
    process_flag(trap_exit, true),
 Param = erlang:integer_to_list(Serial),
 Port = open_port({spawn, ExtPrg++" "++Param}, [{packet, 2}, binary, exit_status]),
 error_logger:info_msg("~p: ifk_drv: Serial[~p] Port[~p] Pid[~p]~n",[?MODULE, Serial, Port, self()]),
 put({port,   Serial}, Port),
 put({serial, Port},   Serial),
 
 % signal back some useful mapping
 ?MODULE ! {driver, Serial, Port, self()},
 loop_handler(Port).


%% =====================
%% IFK DRIVER loop
%% =====================


loop_handler(Port) ->
 receive
   
  {Port, {exit_status, _}} ->
   base:elog(?MODULE, "loop_handler: an ifk driver exited/could not load~n", []),
   handle_crashed_driver(Port),
   exit(crashed);
   
   
  {Port, {data, Data}} ->
   Decoded = binary_to_term(Data),
   %%base:ilog(?MODULE, "loop_handler: decoded msg[~p]~n", [Decoded]),
   send_to_reflector(Decoded);
  
  Msg ->
   base:ilog(?MODULE,"loop_handler: msg[~p]~n", [Msg])
 end,
 loop_handler(Port).


send_to_reflector(Decoded) ->
 {Msgtype, Msg} = Decoded,
 M = {Msg, {date(), time(), now()}},
 switch:publish(?MODULE, Msgtype, M).


%% =======================
%% CRASHED DRIVER RECOVERY
%% =======================


handle_crashed_driver(Port) ->
 error_logger:warning_msg("~p: handle_crashed_driver: Port[~p] Pid[~p]~n", [?MODULE, Port, self()]),
 ?MODULE ! {crashed, Port}.