network management in an heterogeneous environment

I love titles with plenty of buzz-words :)

After a bit of google-ing I finally found the right combination of tools that fit my needs. Luckily they are all packaged for debian and well documented. Auto-configure your network is an old topic. There are plenty of articles and howtos. Moreover the gnome NetworkManager already solves many problems alone. But since I don't use NetworkManager I tried to find a solution that works well with wicd.

The "right" combination for me so far is to use guessnet and ifplugd to handle my wired connection and use Wicd for the wireless network. In theory, you can go for a fully automated solution using guessnet and wpa_supplicant, but I find much better to hide all the complexity of wpa_supplicant and just use a simple - and unobtrusive - UI to handle my wireless.

First thing is to convince Wicd to leave alone my wired connection. This is pretty easy since Wicd wont touch your wired interface if it finds an entry in /etc/network/interfaces for it. Moreover to selecting always switch to a wired connection when available will make Wicd to leave alone you routing tables even if there is a wireless connection available that is marked as automatically connect to this network.

Second you need to install guessnet and ifplugd

aptititude install guessnet ifplugd

I've noticed that the default settings for ifplugd work pretty well except for the -q option that sometimes lives my interface configured even if I've been disconnected for quite a while. Using ifplugd default, I re-obtain an address every time the cable is plugged-in, maybe causing a small delay in some occasion, but I prefer waiting that de-configuring the interface by hand.

Guessnet needs a tiny bit of work. The debian package has already nice defaults that you can copy directly in your /etc/network/interfaces.

# Use guessnet
mapping eth0
        script guessnet-ifupdown
        map default: dhcp
        map timeout: 3
        map verbose: true

iface interface inet manual
        test missing-cable
        pre-up echo No link present.
        pre-up false

# By default, perform DHCP
iface dhcp inet dhcp
These will make sure you avoid long timeouts and you leave the interface configured even if no cable is connected.

Now you have to add one entry per network that you want to configure:

iface home inet dhcp
        test1-peer address 192.168.0.30 mac 88:43:e1:xx:xx:xx
For example to use dhcp on your home network (and of course run any if-up / if-down script as you like). Or :
iface work inet static
        address  192.168.0.1
        netmask 255.255.255.224
        gateway  192.168.0.234
        dns-search buuu.org
        dns-nameservers  192.168.0.30
        test peer address 192.168.0.1 mac 00:1e:xx:xx:xx
to set a static address at work. Guessnet is nice and well integrated with ifupdown. Very flexible and documented. I had however the impression that from time to time the documentation I found on the net was not in sync with the latest release...

I've also noticed that a component of guessnet is not in the package anymore. Guessnet-scan is a small program that will output a stanza to be used with ifupdown based on values sniffed on the wire. This is a kinda of extreme / hackish way to get connected but i reckon a nice tool to have. Hopefully it will make it back in the package sometimes soon...

ps: first post on debian planet !

Average: 1.4 (14 votes)

quote of the day

" I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant" — Alan Greenspan

Average: 1 (8 votes)

FLOSS crisis management software / community

The second workshop I attended at the open word forum was about the involvement of the free and open source community in natural disasters. Quite frankly I didn't know anything about these efforts and I was amazed by the level of commitment of this community.

The first speaker introduced sahana that is a FOSS disaster management system. The Wikipedia has a page about its story and development. Mark Prutsalis introduced the role of free software and crowd-sourcing in recent years highlighting its strengths and flexibility compared to government agencies. In particular sahana and the people from crisis common played a big role in the first days of the relief efforts during the Haiti earthquake early this year.

Another important player is the ushahidi community. The build a number of tools that are increasingly used to connect poeple during natural disasters, monitor political elections, diseases, fires, etc. It's clear that build awareness among people and empower them with these kind of tools can really give back to the community a voice that was long lost in the global village. And there are many projects that go in this direction.

On really important components of these technologies is openstreetmap. Differently from other commercial map providers, and thanks to free licenses and open standards, openstreetmap allowed to visualize and organize the over-growing information that can be collected through crowd sourcing. In this context open standard plays a really important role giving the possibility to exchange information and to easily mashup new different sources in some useful way.

Another nice effort is the to build tools is lead by the Crisis Common community. Last week they organized a Crisis Camp in Paris. Unfortunately I could not attend, but I really applause the effort and the spirit of such gatherings.

Last link I want to give is about NGO in a Box. Despite not directly related to crisis management and crowd sourcing, this kind of tools are certainly a terrific help to the humanitarian aid community and it definitely worth mention them.

Average: 1.1 (62 votes)

My own private info

I've just attended the workshop Datastream: My own private info at the Open World Forum in paris. A very sensitive topic nowadays and the speakers around the table raised a number of interesting points.

Sunil Abraham, Policy Director, Center for Internet and Society (India), started his contribution pointing out the privacy is very much a related to the local culture and history of a country. In India for example, the expectation of privacy is dramatically different from western countries. A very interesting example is the amount of information that is already "encoded" in the name of the person. Being India a country based on a cast system, the name of the person, not only gives away the social status and religion of a person, but also his sex and location. Very common question during every day conversation are also related to salary, spending and wealth. Questions that are somehow taboo in western countries. This lead to very different perception and expectation of privacy that is not clearly reconcilable with western practice and policies. This also allows the Indian government to establish policies that that from a western point of view are completely unacceptable.

A different angle is proposed by the sociologist Dominique Cardon working for Orange. He points out the important difference between government surveillance versus collateral surveillance, as the stoking from people in their circle as parents, neighborhoods, etc. The large majority of people when confronted with questions about privacy often show great concerns and fears about the big brother spying on them, However, he points out, there is a clear cut between these concerns and the quantity of information that each individual then puts on the web. The problem being the distinction that unconsciously people make between data they want to share with their social network and the world as a whole (composed of governments and unknown individuals). Facebook and other social network greatly emphasize the idea of a network of friends giving people the false idea that the data they share is truly private, or restrict to a small circle of friends, when reality shows that these tools are often exploited by other individual or entities to dig information and make profit on personal data.

Somebody in the audience framed the same problem as and identity issue in the digital worlds. As he put it, people starts to develop split digital personalities (i.e. a personal facebook account and a work facebook account) in order to defend themselves from snooping and surveillance. There is clearly a need to reconcile these personalities by technological means creating privacy contexts instead of fostering the creation of completely separated and antithetic digital personae that can have repercussions on the way poeple behave in the real world.

Djordje Djokic (European rights and privacy protection on the Internet), gave a quite broad overview of the political issues related to privacy. Despite privacy being a fundamental right upon which many other rights are based, from a legal prospective it is impossible to define privacy. This is due to the fact that national policies are local while digital privacy is a global problem. If we put together this with the cultural difference among states, in the short term it will be very difficult to safe guard citizens against privacy speculators. He also made an interesting points about privacy safe heavens that can attract activists and agencies due to better legal protections.

I point I made is about the future. It seems that the entire debate was focus about the state actual state of affairs. The FLOSS community has debated for long time now about privacy and technological solutions. Enlightening talks like Eben Moglen many times this year gave rise to interesting projects like the freedom box project . The Diaspora project that will hopefully take off the ground sometimes soon, promises to offer and distributed and decentralized alternative to facebook. Status Net and Identica are also two very interesting platforms built on free and open source software that I hope will take over, or at least pave the path for commercial alternatives.

One of the biggest challenge of course is about education. People don't understand the pitfall of many so called "free" services like facebook or twitter. These companies effectively make money on your willingness to give away information about you, your friends and your life. The large majority of the community is not aware of these problems. This makes it very difficult to privacy advocates to push policies changes because of the lack of interest with the general public. Politicians in particular do not really grasp these problems. Now we even start to see regional and national politician embracing privacy-less social medias making it difficult to for the public to move away from them at the price of being excluded from the democratic life of their country.

National education certainly do not have yet in their curricula topics such as privacy and new medias. Kids often learn from their peers and are enticed by the rich offer of these companies. This state of affair allows facebook or google founders to declare that people today do not have anymore an expectation of privacy. I personally strongly disagree with this position and I hope these will change in the future with privacy aware social media, maybe decentralized, but certainly built in a way to let individuals to retain complete ownership and control over their digital life.

The path to shift the actual tendency is certainly steep. A first step, from a technological point of view, is to create something stable and sound. But the second step, to get weight among todays' big players is to create new and exciting services. Selling an alternative to somebody that does not understand the problem of privacy is already difficult enough (and the story about desktops on linux should already have show why this does not work in this monopolistic world). The strong selling point should be about new services, exciting new way to interact and seamlessly integration with nowadays platforms.

Average: 1.5 (14 votes)

connect django and rfoo

This evening I spent 30 minutes to try out rconsole in the package rfoo . It's a simple environment to inspect and modify the namespace of a running script.

If you are on debian, you need to install two packages :

sudo aptitude install cython python-dev

Then download the source code. If you want to try it out without installing you have to compile it with the --inplace option :

python setup.py build_ext --inplace

Now you're ready to go. Add in your views.py file the following code:

from rfoo.utils import rconsole
rconsole.spawn_server()

In a console type python scripts/rconsole. Keep in mind that you have to adjust your import search path in order to use the rconsole script without installing the library.

Then you can now directly call all methods in your views from the console. For example, imagine you have a search view, then you can call it with :

>>> from django.http import HttpRequest
>>> request = HttpRequest()
>>> search(request,"debian")
<django.http.HttpResponse object at 0x2bc7490>
>>> search(request,"debian").content
'<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml">\n  <head>\n    
....
>>>

I've to say that using rconsole for debugging it not very useful. pdb or winpdb are much more powerful and versatile. It was worth a try anyway...

Update

After getting in touch with the author of rconsole, I think it is important to put in context this post. I've tried rconsole with django in mind. On one hand, I was looking for a debugger that I could use in an early development stage of a project. In this context, I think a bloking debugger can do a much better job then rconsole to help the programmer to inspect variables and insert break points. rconsole is a non-blocking debugger and it is not the right tool.

On the other hand, rconsole can be of great help when debugging a live application when you don't have the luxury to stop your server. In this regard rconsole is very lightweight and unobtrusive, and I think it can be of great help.

I had the impression I've been a bit unfair in my judgment...

Average: 1.1 (69 votes)

cut and pasting in gnome / iceweasel

I've always wondered why iceweasel won't paste my selection into the search area from a gnome terminal. It turned out that to put something in the gnome clipboard from a gnome terminal is not enough to select the piece of text with your mouse, but you need to explicitly do "Ctrl + shift + v" . Then in iceweasel you can use "shift insert" . Kind of brain-dead for me. So if i want to search a word I've on a terminal I've to :

  • select the word with the mouse
  • ctrl + insert OR Ctrl + shift + v to store the word in the clipboard
  • alt + tab to get focus on iceweasel
  • ctrl + k to focus on the search form
  • shift + insert OR Ctrl + v to copy
  • return

wowwwwww !

This nice post explain the root of the problem related to the gnome terminal. This gives a bit of general background.

If you just want to search something, you can try the gnome applet gspot. It's nice, works (I tried only 5 minutes), but doesn't allow you to configure common short-cuts. You are always obliged to use the mouse. googlizer is another applet that has the same problem are gspot, and it limited to google only (while gspot has many search engines built-in).

bah, for the moment I'm a bit stuck ...

Update

A very nice solution :

google() {
    search=""
    #echo "$1"
    for term in $*; do
        search="$search%20$term"
    done
    xdg-open "https://www.google.com/search?q=$search"
}

shell power !

Average: 1.1 (59 votes)

kprintf and failwith

This is just a quicky to start off the day. I often write fatal error message using a combination of Printf.eprintf ;; exit 1 ;; failwith ;; assert false ;; etc ... For example to throw a fatal exception with a message I would write the overly verbose

failwith (Printf.sprintf "this is a fatal error in module %s" modulename)
I see these idioms everywhere and I find them a bit ugly...

If we use Printf.kprintf we can write the statement above in a bit more compact way as:

let fatal fmt = Printf.kprintf failwith fmt ;

Moreover, we add a label to the function fatal and instantiate it once in every module we can get a localize error message for free. Something like :

let fatal label fmt =
  let l = Printf.sprintf "Fatal error in module %s: " label in
  Printf.kprintf (fun s -> failwith (l^s)) fmt
;;
val fatal : string -> ('a, unit, string, 'b) format4 -> 'a = <fun>
# let local_error = fatal "module" ;;
val local_error : ('_a, unit, string, '_b) format4 -> '_a = <fun>
# local "aaaa %d %d" 1 1;;
Exception: Failure "Fatal error in module module: aaaa 1 1".
# local_error "message %d %d" 1 1;;
Exception: Failure "Fatal error in module module: message 1 1".

It would be awesome to have a localized version in the source code as with assert , but I don't think this is possible to do in a generic way. Something like : Exception: Failure "Fatal error in module module (line 144, 63): message 1 1".

I guess this can be done with camlp4. We can catch the line and column like :

let (line,col) =
        try assert false
        with Assert_failure ("", line, col) -> (line,col)
and then feed this info in fatal . Maybe I'll get 5 mins to write this macro. This cannot be done statically as the line reported of assert false will always be the same ...

Average: 1.1 (64 votes)

python itertools and groupby

who said that ignorance is a bliss didn't try python :) This is the assignment : you have a list of dictionaries with a field date and you want to group all these dictionaries in a map date -> list of dictionaries with this date.

The first solution that came to my mind was something ugly like :

def group_by_date(qs):
    by_date = {}
    for r in qs :
        l = by_date.get(r['date'],[])
        l.append(r)
        by_date[r['date']] = l
    return by_date

for example :

In [36]: group_by_date([{'date' : 1},{'date' : 2}])
Out[36]: {1: [{'date': 1}], 2: [{'date': 2}]}

6 lines of python !!! unacceptable. It hurt my eyes and it is not easy to read. The good people on the #python irc channel adviced me to check the collections.defaultdict and this is actually pretty neat. Now I can write something like

from collections import defaultdict
def group_by_date(qs):
    by_date = defaultdict(list)
    for r in qs :
        by_date[r['date']].append(r)
    return by_date

In [47]: group_by_date([{'date' : 1},{'date' : 2}])
Out[47]: defaultdict(<type 'list'>, {1: [{'date': 1}], 2: [{'date': 2}]})

Nice, but still ... and we can do better ! itertools.groupby on the rescue :

from itertools import groupby
qs = [{'date' : 1},{'date' : 2}]
[(name, list(group)) for name, group in itertools.groupby(qs, lambda p:p['date'])]

Out[77]: [(1, [{'date': 1}]), (2, [{'date': 2}])]

Ah ! Nirvana :) I've to admit the most readable solution is using defaultdict, but this solution using groupby is a wonderful power-tool. If you understand list comprehension, this is a very natural solution to the problem.

Average: 1.7 (18 votes)

important things in life

Linux for the masses ... so what is important for the average user ? Well. I've realized I managed to use only Linux since '94 because I'm the antithesis of average user. I've asked a friend what is important. She told me

  • I want to listen my music
  • I want to download and organize my photos
  • I want to sync my data with my iphone (sic!)
  • I want to browse the net
  • I want from time to time to write a document and be able to print it at work.

Easy ? not really :( and I'm a bit disgruntled about this. Let's go in order. Here I'm using ubuntu 10.04.

Music

I've started with rhythembox. It's the default music player in ubuntu. I've been using it for years. While configuring it on a new collection of music, the very first bug I've encountered is the bug #537272 . No way to get out of it of this loop. And if you use rhythembox on a collection that comes straight from windows media player, you might imagine the frustration as wma is the default encoding. To solve this problem I converted overnight all the wma files to ogg with ffmepeg. This solves the import problem, but it should be easier...

So far, so good, but now I want to sync these songs with his iphone. Well. rhythembox does not allow you to do that, or maybe it does, but certainly not out-of-the box and in a way that I can explain to joy user. The iphone is there in the menu, but there is no way to copy music on it (it an iphone with the version 4.0 of the firmware fresh from an apple store). I've tried to look everywhere. It seems a very common problem and syncing indeed works with older version of the firmware, but no luck with the latest "updates".

On top of that there are various shortcoming of the interface :

  • searching (it seems that rhythembox is not able to search at the same time in the song title, filename, meta data and provide a list of songs that match your request)
  • jump to the current song (I've to install a plugin for it even )
  • lyrics (right click, property, lyric .dahhhhh a bit less intuitive way ? other player do this better)
  • well not pretty (I've a friend that choose a model of a laptop just because it has a white keyboard... so you can imaginable that for some people these things are important)

So I tried Exile . It's a more usable player. The interface is more "human" with many features that make joy-user to feel home . But It crashes far too often and it does not even see the iphone.

Then I tried Listen music player that is a pretty nice player. Written in python and pretty stable. But Alas, not iphone. But much better then ryhtmbox as a player and I settled on this one for the moment.

I've tried also Gtkpod but there it seems it only works with older versions of the firmware.

Last I tried Amarok (on a gnome system, but I hope that the dependency system is able to install it in any case). I hoped this would work as everybody says on the net that is the most mature player out there... For me it failed to import the music library and out of frustration I gave up trying the iphone support...

Photos

I've been using f-spot for years. It gets better and better, but it seems that it's default behavior of copying and reorganizing the photo collections by date is completely against the mind setting of most of the people. Why is this the default ? Let people live their life as they want ? More you try to bend them toward a different schema, more you risk to loose them in the process. F-spot has also a number of shortcomings, sometimes it crashes, but all in all it pretty usable. And it is also able to read and index photos from the iphone. Yeeee :) Something that bluffed me is the time it takes to export photos to a folder. Copying 1000 photos to a usb pen took 2 hours !!!!!!! while copying photo directly from nautilus takes only 30 mins ... This was not cool.

F-spot is also terribly slow with its slide show. It does not allow to rotate photos during the slide show and sometimes it stalls up 5 to 5 seconds to redraw a new photo...

I've tried also showtell, the upcoming default ubuntu photo manager... It failed to import from f-spot large part of my photos on the first run. I kinda of gave up, but I've high hopes...

Other

I've to say that we often tend to focus on negative aspects. On ubuntu the chrome broweser runs very well delivering a more then acceptable user experience. Openoffice also works nicely both with old word documents and to edit new document. Thumb up here. Nothing to say.

Conclusion

When I started writing this article I had a very negative picture. I've to say that on a second thought things are not that bad after all. The main problem are not surprisingly proprietary formats. From wma files to the f***ing iphone. They are all playing against FOSS applications and the have the high hand all the time. We can try to catch up, but it's difficult. And trying to explain this to the average user is difficult. They says that simple things in life should be simple. And while this is true in general, they fail to understand is that what is simple for windows / apple user is extremely difficult in the FOSS world because of the lack of open standards and commercial practice that force this state of affairs. I use an openmoko as day phone and I never had any of these problems :)
Average: 1 (54 votes)

command line parsing in python

if a good API is meant to let you write code more concisely, I would say the argparse achieves this goal perfectly. For example, with getopt you usually write something like :

try :
    opts, cmdline = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
except getopt.GetoptError, err:
    print str(err)
    usage()
    sys.exit(1)
output = None
verbose = False
for o, a in opts:
    if o == "-v":
        verbose = True
    elif o in ("-h", "--help"):
        usage()
        sys.exit()
    elif o in ("-o", "--output"):
        output = a
    else:
        assert False, "unhandled option"
if not cmdline :
    usage()
    sys.exit(1)

This is just to add two command line options, help and verbose and without taking care of positional arguments... Now, the magic of argparse:

    parser = argparse.ArgumentParser(description='description of you program')
    parser.add_argument('-v', '--verbose')
    parser.add_argument('timestamp', type=int, nargs=1, help="a unix timestamp")
    parser.add_argument('inputfile', type=str, nargs=1, help="input file")
    args = parser.parse_args()
    parser.print_help()

tadaaaaa .

usage: import.py [-h] [-v VERBOSE] timestamp inputfile

description of you program

positional arguments:
  timestamp         a unix timestamp
  inputfile             input file

optional arguments:
  -h, --help            show this help message and exit
  -v VERBOSE, --verbose VERBOSE

And this has all the bells and whistle you want. Like checking if you pass both positional arguments, if the file exists, if the timestamp is really and integer file, etc. Very nice indeed !

Average: 1.3 (16 votes)
Syndicate content