Tag: freesoftware

Bostonian and Cantabrigian Debianistas should waste no time in celebrating Etch’s release. Join a group of us celebrating the release tonight (April 8) at Grendel’s Den in Harvard Square.

Things you should know:

• We’ll meet up at 21:00.
• Directions are online.
• Food is either half-price or $1 with a drink.

Call me if you’re lost in the neighborhood or have questions. I hope to see a few of you there. New faces are, of course, welcome.

Last November I wrote about an iPod liberation party we held in Cambridge. As I mentioned then, it was a huge success. Mika just got around to polishing up a short documentary video she made from footage at the event and a few interviews afterward. The documentary is called, What’s Wrong With My iPod? and it acts as an introduction to DRM on iPods and what we can do about it.

The video introduces the concept of DRM, the dangers it brings, and describes the role it plays on the iPod. The second half is about iRony, our iPod liberation party, and RockBox.

Please pass the video links around to your friends — especially those who might not be up to speed on DRM. If it inspires you, think about running your own iPod Liberation Party (instructions here).

You can view it on blip.tv (only MP4 until their converter catches up), on YouTube, or you can just download the OGG theora file (also in high quality).

I forgot to mention the coolest thing about the last Ubucon in my summary.

I heard that my friend Sean Moss-Pultz (the person who started the OpenMoko project) would be in New York the week of Ubucon so I managed to contact him and get him to drop by on his way to the airport so we could have lunch.

To my surprise, he had one of the early versions of the free-phone for me (I wasn’t expecting one for several weeks). I had brought my OLPC XO so, for the first time ever (as far as we know) we managed to get the two coolest, and most important, technology platform projects in the world together.

While the XO and OpenMoko share a commitment to freedom, the similarities between the projects are, in fact, also skin deep. If we all work hard, we can look forward to a future that is free. Apparently, it’s also white with bright trim.

Apologies for the pictures taken from my current inferior, both ethically and technically, mobile phone.

I’m coming down to New York this Friday for the second Ubuncon. Ubucon is a small(ish) user organized and oriented Ubuntu "unconference." Apparently, that means that it’s not very organized — which adds flexibility and is considered a good thing!

Both Ubucons to date have been held at Google offices. This one will be held in the Google office in New York City. I’ll be giving at least one talk. In all likelihood, I will be giving a talk about participation in the Ubuntu community and another more technical crash introductory course in building Ubuntu packages. Finally, I’ve had Pearson’s ship a dozen or so copies of the Official Ubuntu Book which I’ll be signing and handing out.

If you want to go, you should check out the schedule, and the info pages in the wiki and maybe even RSVP. You will need a LP/wiki account to do so.

See you in New York! Please contact me if you want to get together while I’m in New York.

I’m been perplexed by the recent fracas around the possibility of Ubuntu shipping non-free drivers by default as part of the feisty release goal to bring the bling. The issue has been discussed by many people, most recently and eloquently in a blog post by Scott James Remnant.

I have never been 100% comfortable with our (Ubuntu’s) decision to ship proprietary drivers. The Ubuntu philosophy document — which Mark Shuttleworth and I drafted originally — says quite unambiguously:

Every computer user should have the freedom to run, copy, distribute, study, share, change and improve their software for any purpose, without paying licensing fees.

I believe this deeply. Consequently, I feel that binary drivers in Ubuntu are, and always have been, a failure to live up to our principles. If our philosophy document is still representative, most people in our community should be a little uncomfortable with them — even if they think, as most of us do, that shipping them is the correct thing to do.

Early on Ubuntu decided that binary firmware and hardware drivers would be distributed and quasi-supported in restricted. While not completely comfortable with the move, I could understand and defend it for three reasons:

1. The drivers were easily removable and not enabled by default unless necessary to achieve functioning hardware.
2. As part of this process, we promised to work with vendors to accelerate the freeing/open-sourcing of their drivers.
3. Perhaps most importantly, I couldn’t imagine not using them if it were the choice between non-free drivers or inoperable hardware.

The proposal to include proprietary nVidia driver in place of the existing free nVidia driver is important because it changes, or eliminates, the first and third justifications. Since the first issue has been discussed in depth, it seem appropriate to focus on the third and final point. The point is perhaps best introduced with an observation:

While I have met many people who research and buy hardware to avoid the need for non-free drivers, something I do myself, I have never met anyone who voluntarily chooses inoperable hardware on a computer they use over working hardware through a non-free driver.

While not airtight, this observation is precisely why the old policy of using non-free drivers only when no free drivers exist works so well. By effectively synonymizing use of the non-free driver with use of the hardware, the crisis created by non-free drivers is rendered paradoxical: anybody using a piece of hardware that only works with non-free drivers is, by definition, either using non-free drivers or not using the hardware. The only way to not use non-free drivers in situations where no free drivers exist is to not use hardware that requires them. If you use different hardware, you will not need to use non-free drivers. This important justification is completely inapplicable to the proposal to include non-free drivers in place of free (but not perfect) alternatives.

Second, while shipping non-free drivers is wrong because we lose our freedom and violate our philosophy, the desire for basic hardware support is universal and important enough that it’s defensible. One reason that the "binary by default for bling" situation alarms me is that I don’t believe that the desire for desktop bling is as universal, or as strong, as the desire for working hardware.

An analogy might be stealing food to feed your family and stealing a CD from the record store. Both are wrong because stealing is wrong, but I have a lot more sympathy for the hungry thief in the first example because CDs are not as necessary as food and because many people would rather go without a new CD than steal. We can ask: "Which is worse: selling your soul for a decent job or selling your soul for a PS3?" In an absolute sense, neither is worse; but it’s easier to support the former than the latter. I’m very impressed with Beryl — but I have trouble empathizing strongly with the need for a spinning desktop cube.

New users may be impressed and attracted to an Ubuntu desktop powered by proprietary drivers. They might also be impressed by proprietary applications. However, it is self-defeating to attract new users to our free platform and principles by compromising our core values. Many users already make the decision to use non-free video drivers and Ubuntu makes it very easy to do so once they have made that choice. Installing binary drivers by default is choosing for the large number of users who will stick with any default. It may be a conservative stance but as long as we believe that a non-insignificant number of users would choose to live without bling before compromising Ubuntu’s philosophy and goals, we should make the difficult decision to side with freedom by default.

When discussing this issue, many people say, it’s not my decision or it isn’t my call. Regardless of whether anyone agrees with me and regardless of who the final decision is up to — and it’s not clear to me now — I believe that the Ubuntu community has the power to inform and shape the development of their distribution. As the Ubuntu community, we can make our position heard. The best place to do that right now seems to be the Accelerated X specification comments page. One can also email the the Technical Board (for technical comments) and the Community Council (for principle or philosophy related issues).

Over the years, I’ve published some of the articles I am most proud of on Advogato. As you can imagine, I was sad to see demise of Advogato announced early September and thrilled to see that by the end of the month, Steve Rainwater had stepped up to save the site.

For a belated welcome of Advogato back from the edge of the grave, I’ve published a short essay about the effectiveness of an analogy of software and writing on the site. In it, I introduce the analogy and argue that it is useful in advocating free software and programming education and in justifying the disproportionately large amount of programming related projects in the free software world. Here’s the intro:

Advocates of free and open source software, myself included, like to talk about the "democratizing" effect of free software. Others, especially non-programmers, are quick to point out that the only technical people can take advantage of half of the enumerated freedoms in FOSS. The freedoms to modify and collaborate mean little if you don’t know to program. Over time, I have come to the conclusion that the only good solution to this problem — and one that I was initially quite opposed to — is to teach everyone to program. In considering this position, the processes of reading and writing provide a useful analogy when considering the processes of using and creating computer software. Additionally, the analogy provides an powerful justification for the fact that FOSS programmers produce a disproportionately large amount of software that is primarily of interest to FOSS programmers.

Please check it out on Advogato and feel free to leave a comment on my blog if you don’t have an Advogato account.

I was very happy to hear that the Masters of the Universe (MOTU) loved my suggestion and named their nascent governing board Council Grayskull.

I apologize for boring the vast majority of my readers who do not use PyBlosxom but I’m trying to at least mention each "released" projects here once for the sake of completeness — even if it’s only interesting to a small number of them.

I posted a note a few months ago mentioning a stronger CAPTCHA I had created to deal with blog spam. A few friends pointed out that they were having excellent luck with Akismet which filters using a model similar to some of those old collaborative spam filters.

Building off earlier work, I build an Akismet plugin for PyBlosxom. I’ve gone through a couple of iterations on the PyBlosxom lists and the results have been great. If you use don’t use PyBlosxom, there are plugins for WordPress (where the system originated) and I imagine for other blogging systems as well. Its an effective and attractive alternative to a CAPTCHA.

You can grab my plugin on my PyBlosxom hacks page

About a 6 weeks ago, I uploaded a great piece of software into Debian. The package is libwww-mediawiki-client-perl but you can also get it by installing the package mvs. The software seems to be a couple years old — but it’s new to me and I’m been thrilled with it so I thought I would share.

The package is a library but its usefulness to me centers around a cute little command-line wrapper to the library called mvs. mvs essentially provides a simple CVS-like interface to an instance of Mediawiki that facilitates offline editing of Mediawiki pages and much more.

Here’s a quick little walk-through that I wrote for the README.Debian file that might give someone a pretty good description of how the software works:

Step 1: make a directory to store pages from this Mediawiki instance:

 mkdir en-wp cd en-wp 

Step 2: log in to the host with your username/password:

 mvs login -d wikipedia.org -l en -u 'Benjamin Mako Hill' -p password 

Step 3: download a page that you want to edit by adding ".wiki" to the end:

 mvs update Granrojo.wiki 

Step 4: edit the file to make changes:

 vim Granrojo.wiki 

Step 5: preview your changes:

 mvs preview Granrojo.wiki 

Step 6: commit your changes into the wiki:

 mvs commit --minor yes -m 'made spelling fix' Granrojo.wiki 

And that’s all there is to it! It brings a whole series of things that were almost prohibitively difficult through the web interface into reach (e.g., diffing two different pages to see if they’ve diverged) and its changed the way that I interact with Wikipedia in some exciting ways.

Thanks to Mark Jaroski for writing and maintaining the software!

Since my weblog catastrophe a few months ago, I’ve been spending a little more time getting my PyBlosxom setup on copyrighteous into shape.

Initially, my efforts were focused on a series of patches to PyBlosxom and to several important plugins. As I became a little more familiar with the code base, I realized that in an evening or a little bit of downtime, I can pretty easily create my own plugins. I’ve already managed to accumulate a few of them.

I’ve created a page on my website to hosts these and have put what I’ve got publishable so far up there. I’ve got two more that I’m still in the process of vetting on the PyBlosxom mailing list but that I’ll post there (and announce on this blog) soon.

The first plugin that I’ve published there is a very simple plugin I wrote to display ads (I am experimenting with Google ads but it should work with other ad providers or even for non-ads) but only to those who get to my old blog entries by searching for them.

My thinking was inspired by an old post by my friend Evan where he described setting up Google advertisements in a way that would not affect his core readership but might allow him to use blogging to make his activism financially sustainable. While I suspect (and hope!) that most my own readership’s Internet experience is mediated by Adblock and Filterset.G, the plugin will spare even those that don’t while showing ads to those I don’t know yet.

I’ve been spending more time that I want to in the last week talking about should or should not be on Planet Debian. In particular, there is some disagreement on what should happen to the blogs of people who resign from the project.

In the interest of my own sanity, here is the text I just posted on the wiki page that talks about Planet Debian:

Planet Debian is for any active and directly involved participant in the Debian development community. Inclusion in Planet should reflect a relationship that already exists — it is not meant to create one. Inclusion is not restricted to people who are currently Debian developers nor are ex-developers necessarily barred from inclusion.

Defining activity and direct involvement is tough and there are many ways of participating in Debian (e.g., packaging, translations, administration, etc). I am happy to consider all of these types of involvement for the purposes of inclusion in Planet. In terms of deciding what is enough activity, I am daunted by the fact that there has been academic research on the difficulties of deciding what an active member in Debian is. As a result I will continue to rely primarily on what blog owners themselves feel unless the evidence available to me points to the contrary.

Planet participants should include feeds that provide stable permanent URLs so that they do not flood planet repeatedly and should include content in English only. An inability to do either of these will be grounds for exclusion from Planet until this can be fixed.

In the past, I’ve made incorrect assumptions about whether people did or did not want to be included in Planet. I’ve apologized to those involved. In the future, I will ask people that appear inactive to me or that have resigned from the project if they feel that they still qualify under the terms above. If I approach you about inclusion in Planet, please don’t be offended or assume that I, or the project, doesn’t want you on Planet. There are feeds in Planet Debian that annoy me but I have yet to remove a feed from Planet that I dislike.

Over the past few months, I’ve dealt with something of a blog spam nightmare on Copyrighteous: my blog running the PyBlosxom weblog software.

Wiser and less stubborn individuals might given up on either PyBlosxom or the ability to receive public comments. However, I find PyBlosxom unique in its flexibility and great ReStructured Text support and am always frustrated with others’ blogs that don’t accept comments. At the end of the day, I couldn’t bring myself to part with either.

Historically, my blogspam protection has been to use a simple weak CAPTCHA and to have my blog software email me each time a comment is successfully submitted so that I can (with a built-in macro in mutt) delete each spam comment that slips in. This has worked well for the last couple years.

This summer, Mika pointed out that my blog was full of Chinese link spam that I had not noticed or been notified about. Around the same time, I realized that my website had been dealt a massive spam penalty by Google and was basically not showing up in any search results.

I have spent a significant amount of time over the last month repairing the damage and working to prevent it from reoccuring. I’m documenting this process here in the hopes that it might save other time and energy.

Upon reflection, the situation could have been prevented the in three relatively easy ways — all of which I have now implemented.

• Had the PyBlosxom comment.py plugin’s mail function been working properly, I would have known that I was being spammed.
• Had PyBlosxom been configured to only make blog entries (and not comments) indexable by search engines, Google and others never would have seen the link spam.
• Had I installed a stronger CAPTCHA, I might have blocked the spam from being submitted in the first place (although at the expense of the participation in comments by visually impaired users).

A month or so, hours of work, and a Google reinclusion request later, my website is beginning to show up in search entries again. Hopefully this message will help save others from a similar fate.

Fixing PyBlosxom’s Comment Notification

The most critical problem was a bug in PyBlosxom’s contributed comment.py plugin and its comment notification system. In short, the email based comment notification system failed silently if the body of the email — which included the full text of the comment — included any non-ASCII UTF-8 encoded text.

I’ve filed a bug against PyBlosxom and included a patch that fixes this issue. However, since this is a rather critical problem and because PyBlosxom releases tend to be few and far between, it might be worth patching your system now. My patch is against version 1.3 but can easily be modified and applied to version 1.2.

Hiding Comments from Search Engines

The major reason that the successful spam became a problem was that it triggered Google’s abuse detector, resulted in a spam penalty, and made all of the (non-spam) material on my website more difficult for others to find. A simple way to prevent this is to hide all comments from the search engines.

I’ve done this by creating a new PyBlosxom flavor that shows comments (and allows them to be input) which is not indexed by search Engines and to remove comments altogether from the default indexable flavors.

To do this, I removed all of the comment-* templates from html flavor and created a new flavor called comment.flav that included the comment templates. I also had to make the comment submit action point to the new flavor and to change the "Comments: N" link to point instead to .comment flavor rather than the .html. The rest of the template is simply symbolic links to the the HTML template.

The next step is to ensure that the comment flavor is not indexed by search engines. I found two ways of doing this and did both. The first was to add a "no index" meta tag to the header of each .comment page. It looked like this:

 <meta name="robots" content="noindex, nofollow"> 

This is necessary because the robots.txt standard, the normal way to tell search engines not to index a page, does not support wildcards.

Luckily, Google (and others I imagine) do support an extension to Robots.txt that allows you to use wildcards. To take advantage of this, I created a robots.txt for mako.cc that blocks indexing all of the comment flavor. The following robot.txt did the trick for me:

 User-agent: Googlebot Disallow: /copyrighteous/*.comment$ 

An Improved CAPTCHA

Ultimately, the best solution would be to keep the spam from showing up on the blog at all.

The only decent PyBlosxom CAPTCHA is the "nospam" plugin by Steven Armstrong. It is a simple image-based CAPTCHA and I was running it when I was spammed. It uses PIL but generates purely number-based strings and does some minimum obfuscation. Basically, spambots were able to break the CAPTCHA and toward the end, I was receiving thousands of pieces of a blog spam a day.

I’ve incorporated the PIL image generation code from Mediawiki’s ConfirmEdit/FancyCaptcha extension into nospam.py with this patch which I have also sent to Steven Armstrong — nospam.py’s original author. It’s much stronger.

Apologies, of course, go to all of my vision impaired users. Image-based CAPTCHAs really are evil. In this situation though with many thousands of attempts of a day, the alternative is that I will turn off comments altogether — the standard (and poor) lesser of two evils argument.

Ultimately, I will write a python implementation of a new strong text-based CAPTCHA I’ve invented that uses commonsense knowledge and pulls off some cool data acquisition in the process. I presented this project at the Wikimania Hacking Days and at a Media Lab open house for AAAI 2006 where I got universally positive and useful feedback. CAPTCHA inventor and recent genius-grantee Luis von Ahn seemed to like the idea too. I’ll write more about this on another day though.

I’ve upgraded Planet (the software that runs Planet Debian) to version 2.0. It’s been a while since I touched the planet software so many issues that had annoyed users of planet should now be remedied by this upgrade. I think people will very happy with the upgrade.

In the process, I really screwed up planet for the moment. New posts from the last day or so may not be showing up. Other people may appear to have flooded planet when in fact they’ve done nothing at all. This is not their fault but I’ve had a hard time going through and picking up the pieces left by the upgrade. Please just bear with me.

If you notice a little bit of funkiness in Planet Debian in the next day or so, please wait a day or two (or a post or two) before contacting me to debug the problem. Thanks for your patience everyone.

If you have questions, don’t hesitate to get in contact with me.