Posts Tagged ‘open source’

Mailman Archives Issue Resolved

I have been battling a weird archives issue with our GNU Mailman mailing list server.  We have some lists that archive properly when e-mail is sent to them.  We have other lists where the e-mail is delivered but does not show up in the archives.  We also have lists where e-mail sent to them disappears and is never heard from again.  I have been hassling with this permissions issue literally for years now.  I picked the baton up again today and decided to try to bring this one home.  First I started in the mailman error logs:

/var/log/mailman/error

Jul 26 12:25:43 2010 (2755) Uncaught runner exception: [Errno 13] Permission denied: ‘/var/lib/mailman/archives/private/listname/index.html’Jul 26 12:25:43 2010 (2755) Traceback (most recent call last):
File “/usr/lib/mailman/Mailman/Queue/Runner.py”, line 112, in _oneloop
self._onefile(msg, msgdata)
File “/usr/lib/mailman/Mailman/Queue/Runner.py”, line 170, in _onefile
keepqueued = self._dispose(mlist, msg, msgdata)
File “/usr/lib/mailman/Mailman/Queue/ArchRunner.py”, line 73, in _dispose
mlist.ArchiveMail(msg)
File “/usr/lib/mailman/Mailman/Archiver/Archiver.py”, line 217, in ArchiveMail
h.close()
File “/usr/lib/mailman/Mailman/Archiver/pipermail.py”, line 324, in close
self.write_TOC()
File “/usr/lib/mailman/Mailman/Archiver/HyperArch.py”, line 1094, in write_TOC
toc = open(os.path.join(self.basedir, ‘index.html’), ‘w’)
IOError: [Errno 13] Permission denied: ‘/var/lib/mailman/archives/private/listname/index.html’
Jul 26 12:25:43 2010 (2755) SHUNTING: 1280155615.876646+a19c8dce602a83897d29592d36d618fc80195ec7
I didn’t remember seeing this particular error message before (of course I didn’t write down the old ones) so I started over again with fresh eyes.  After googling for a long time I ran across this nugget:
>>I ran several times check_perms -f and it says all is fixed.
>
>
> check_perms is lying (actually, there are many files, as opposed to
> directories, that check_perms doesn’t check). The above file and all
> files in /var/lib/mailman/archives/private/ excluding those in
> /var/lib/mailman/archives/private/*/database/ need to be group
> writable.
>
> Once you fix these permissions, you could run bin/unshunt to add the
> shunted messages to the archive, but there is an issue in that the
> messages have been successfully added to
> /var/lib/mailman/archives/private/mylist.mbox/mylist.mbox, and
> unshunting will add them again.
>
> Rather than trying to fix archive permissions, I suggest you verify
> that /var/lib/mailman/archives/private/mylist.mbox/mylist.mbox
> contains all the lists posts from inception to date, and mayby verify
> there are no stray “From ” lines in message bodies with bin/cleanarch,
> and then stop Mailman and rebuild the archive with
>
>  bin/arch –wipe listname
>
> and then start Mailman. This way, the pipermail archive will be
> completely rebuilt with correct permissions.
>
> This is one reason why I always recommend when moving lists to just
> move the LISTNAME.mbox/LISTNAME.mbox file and build the archive on the
> new machine with bin/arch.
>
> Note if you do this, remove the shunted messages from qfiles/shunt/ so
> they don’t accidently get unshunted in the future.
What?  The check_perms utility doesn’t fix all of the needed permissions in order for mailman to run properly?  Why won’t it complain when the daemon starts up then?  Why won’t it at least say that some of the permissions have been correct but not all of them? I’ve been running this mailman installation for several years now and I (and all my buddies who also run mailman) have always held up check_perms as the gold standard for making sure that mailman works properly.  I wish I had known about this a few weeks ago when I was moving hundreds of gigabytes of files from one server to another.  The bit about moving lists would have saved me (and my end users) a lot of time.

Mailing List Blues

The Conference mailing list server is down again. I’ve been monitoring disk utilization on the list server for awhile now in an attempt to keep the server up until after the building move. Once I realized that we were going to run out of space again I decided to take the server down preemptively.

GNU Mailman

We have a long running history with this software.  Sometime around 2003 I was tasked with setting up a mailing list solution for the Conference.  Several of our local churches had also expressed interest so I had to find something cheap, scalable and fast.  GNU Mailman was the perfect choice.  It’s free and open source software, you can continue throwing lists at it and it supports lists of all sizes.

The list server is my oldest Linux installation.  I was a lowly Windows admin at the time so my good friend Alan Swartz helped me with the original Red Hat installation.  My how times have changed.  Back then I had a brief list of commands to create new lists, reboot the server and perform a few basic administrative tasks.  I wish I had kept a copy of that original handwritten list but alas, it is lost to the sands of time.  This software has proven robust over the years as it has moved across several physical computers and 3-4 different Linux distributions.

Victims of our own success.

It would seem that too much of a good thing always lead to problems.  The mailing list server maintains an archive of all of the e-mail that is sent over each of the mailing lists.  These files grow over time as new messages are sent.  Over time disk space can become a problem.  It took us several years of constant usage to amass a corpus of around 80 gigabytes (GB).  Mailman must have changed how it stores e-mail because over the course of a year or so we shot up to around 280 GB.  Maybe people realized that you can send attachments to the lists?  Once things get back to normal I plan to dig into why these list archives are growing so quickly.

Everyone loves a good history lesson but why is the server down today?  The simple answer is that the hard drive is full (again).  Once it fills up the mailman daemon stops responding.  Since I am out of the office it could take me a good while to discover that the system is down.  That’s why I decided to go ahead and replace it.

On June 17th the system went down due to a full hard drive.  With the building move coming up soon I decided to try temporarily remove the larger archives from the internal mailing lists.  This would free up enough hard disk space to keep the server running (hopefully) until well after the building move when I could properly schedule an outage.  I’ve been monitoring the disk utilization since then, moving archives as I can.  Unfortunately, I’ve moved all of the larger ones and was forced to move forward with plans to switch the drive.

Last night I pulled the 320 GB drive and replaced it with a 1.5 Terabyte (TB) drive.  It takes awhile to copy the archives back to the new drive however.  Overnight 60 GB of the 280 GB data store copied.  I expect progress all day and will bring the system back online as soon as I can.  Hopefully this will buy us a good bit of time before I have to permanently retire some of the archives.

Update: Friday, July 9th 2010 @ 12:17 PM
The list server is back up!  We have plenty of available disk space now.  I’m hoping that this one will last us a good while.  I still need to research what is eating so much disk space but moving forward we should be in good shape!

Is Sugar The Laptop Or The Operating System?

Nicholas Negroponte is at it again, giving an interview in Singapore and discussing the major failings of the OLPC project.  I was struck by one thing that he said:

Putting a crank-shaft on the XO laptop was a mistake, but the biggest mistake was not having Sugar run as an application “on a vanilla Linux laptop”, said OLPC founder and chairman Nicholas Negroponte.

“Sugar should have been an application [residing] on a normal operating system,” he told ZDNet Asia in an interview. “But what we did…was we had Sugar do the power management, we had Sugar do the wireless management–it became sort of an omelet. The Bios talked directly with Sugar, so Sugar became a bit of a mess.”

After spending several years working in IT as a career I have learned that there is at times a disconnect between the words of management and the actual inner workings of a product.  This looked funny to me so I wondered what the actual people working behind the scenes thought of this.  Turns out Sugar wasn’t as bad as advertised:

Here’s the problem: through a somewhat regrettable set of naming decisions, the name “Sugar” came to represent two entirely different things. It was the name for the new learning-oriented graphical interface that OLPC was building, but it was also the name for the entire XO operating system, one tiny part of which was Sugar the GUI, and the rest of which was mostly Fedora Linux.

Nicholas, evidently, still remains blissfully unaware of any of this. As is plain to see from his own words, what he considers to be the biggest mistake of the project has nothing to do with Sugar the GUI, and everything to do with the gross, hairy, complicated systems development work that OLPC was doing to support the XO’s special hardware features. And to be clear, I mean “short bus special”, not “shiny unicorn special”.

Let me explain something to you. For most of OLPC’s existence, we had about two guys working on Sugar the UI. They were GUI developers, with GNOME backgrounds. They were not at all the same people doing systems development work to support our hardware. No resources were taken away from systems development to do Sugar. If Sugar hadn’t happened at all, we would have still had to do all the systems work to get Linux working on the XO, and it would have still taken just as long. So if you’re looking for things to blame, Sugar is not the droid you are looking for.

In truth, the XO ships a pretty shitty operating system, and this fact has very little to do with Sugar the GUI. It has a lot to do with the choice of incompetent hardware vendors that provided half-assedly built, unsupported and unsupportable components with broken closed-source firmware blobs that OLPC could neither examine nor fix.

So we wound up with a keyboard whose keys get stuck. A dual-mode touchpad, capacitive and resistive, where one mode doesn’t work at all, and the other makes the cursor spontaneously jump around and sometimes shuts off the touchpad altogether, prompting OLPC kernel developers to beg for saner hardware in the next round. We had board engineering issues that made power management practically impossible. We had a custom display controller chip that was incomplete in some regards, and completely broken in others. We had an embedded controller that blocks keyboard events and stops machine suspend, and to which we — after a long battle — received the source, under strict NDA, only to find a jungle of nested if statements, twelve levels deep, and no code history. (The company that wrote the code doesn’t use version control, see. They put dates into code comments when they make changes, and the developers mail each other zip files with new versions.) And we had a wireless chip that is so far beyond fucked, it’s just about funny.

(Each of those words is a different link. Click them all, I dare you.)

Thinking back, there’s a hardware incident I remember particularly fondly: one of our vendors sent us a kernel driver patch which enhanced support for their component in our machine. They chose to implement the enhancement by setting up a hole which allowed any unprivileged user to take over the kernel, prompting our kernel guy to send a private e-mail to the OLPC tech team demanding that, in the future, we avoid buying hardware from companies whose programmers are, direct quote, “crack-smoking hobos”.

In the end, Nicholas’ bit of interview nonsense just doesn’t pass the smell test. Customers aren’t stupid. There’s close to a million XOs out there; if Sugar was OLPC’s biggest mistake, Windows on the XO would be selling like hotcakes. Let me remind you, then, that the number of Windows-based XOs that OLPC has sold is exactly zero.

So next time you hear Nicholas break out the egg metaphors and wave his hands about the Sugar that doomed it all, shrug and smile. Hell, If I were a meaner person, I’d ask Nicholas why it is that Windows — you know, the Windows from Microsoft, mercifully unstained with the mistake of Sugar — can’t even shut down an XO without throwing up a blue screen of death.

I honestly don’t know what to say to this.  It’s a shame that the top down management style of the OLPC project nearly killed it.  I remember sitting around with my IT buddies excited about the future of Sugar and the XO laptop.  To be honest, most of us have moved on to something else.  What a shame…

Microsoft Not So Altruistic After All

I recently wrote that Microsoft suddenly released some code to the open source community.  Turns out their motives weren’t as pure as first thought:

Sometimes, some things are just too good to be true. Earlier this week, Microsoft made a relatively stunning announcement that it would contribute some 20000 lines of code to the Linux kernel, licensed under the GPL. Microsoft isn’t particularly fond of either Linux or the GPL, so this was pretty big news. As it turns out, the code drop was brought on by… A GPL violation.

This story begins when Stephen Hemminger, principal engineer with open-source network vendor Vyatta and Linux contributor, finds out that a network driver in Microsoft’s Hyper-V uses open source components licensed under the GPL. These components were statically linked to closed-source binaries, which the GPL does not allow.

Consequently, Hemminger contacted Linux Driver Project head Greg Kroah-Hartman, who works for Novell. Commendably, Hemminger wanted the case to be worked out without fireworks and massive media attention. “Since Novell has a (too) close association with Microsoft, my expectation was that Greg could prod the right people to get the issue resolved,” he writes on his blog.

When the code drop was announced Monday, nor Microsoft, nor Kroah-Hartman spoke of the violation. To confirm the story, About-Microsoft blogger Mary-Jo Foley contacted Kroah-Hartman, and he confirmed that Hemminger is indeed correct: the code drop seems to have been brought on by a GPL violation.

A “break from the ordinary” and a “significant milestone”? None of that – just a silently handled case, with an overdose of marketing spin, to prevent a major embarrassment for Microsoft.

I expect Microsoft to be evil and work to destroy the open source software movement.  This isn’t much of a surprise.  It is a bit of a disappointing end to this story though…

Microsoft Stuns Linux Community

This is absolutely stunning:

In an historic move, Microsoft Monday submitted driver source code for inclusion in the Linux kernel under a GPLv2 license.
The code consists of four drivers that are part of a technology called Linux Device Driver for Virtualization. The drivers, once added to the Linux kernel, will provide the hooks for any distribution of Linux to run on Windows Server 2008 and its Hyper-V hypervisor technology. Microsoft will provide ongoing maintenance of the code.

Linux backers hailed the submission as validation of the Linux development model and the Linux GPLv2 licensing.

Microsoft said the move will foster more open source on Windows and help the vendor offer a consistent set of virtualization, management and administrative tools to support mixed virtualized infrastructure.

“Obviously we are tickled about it,” said Jim Zemlin, executive director of the Linux Foundation. “Hell has frozen over, the seas have parted,” he said with a chuckle.

I am surprised by this move.  The Linux community would be well served to review this code very carefully.  Are their motives completely noble?

While observers hail Microsoft’s Linux kernel code submission as good for the industry and a substantial step forward, the move isn’t pure altruism.

The drivers will make it easier to support Linux guest operating systems in their emerging cloud infrastructure, and it will guarantee Windows is a part of every enterprise conversation around virtual Linux servers.

And virtualization integration baked into the Linux kernel appears to provide Microsoft with a heavy stick with which to beat up VMware.

“Why should Microsoft let a religious distaste for Linux get in the way of making a lot of money on Windows Server 2008 being the hypervisor under all those Linux servers?” asked Jeffrey Hammond, an analyst with Forrester Research. “Microsoft’s desire to take money away from VMware and other alternatives has outweighed its distaste for embracing Linux and the GPL. That is a sign of the opportunity they see here.”

Microsoft, however, won’t have an exclusive on virtualization drivers in Linux. VMware has certified kernel mode para-virtualization drivers but administrators have to install them separately because they are not part of the mainline Linux kernel.
“Microsoft is taking a short cut,” said Chris Wolf, an analyst with the Burton Group.”This is a big deal. When you get in the mainline Linux kernel it is a competitive advantage for Microsoft.”

I thought so…

Sugar On A Stick

Sugar on a stick!

Sugar on a stick!

The MIT Technology Review has published an article detailing the latest efforts on behalf of the Sugar Labs project.  The newest version of the Sugar operating system is designed to install on a USB thumb drive or CDROM.  If it is installed on a thumb drive then user files can be written to it and saved in between sessions.

The underlying operating system is built on Fedora 11, which was just recently released.  It represents a significant improvement from earlier versions of Sugar.  The educational software has greatly improved along with overall stability.

This is an effort that I have been following with great interest since it was part of the One Laptop Per Child (OLPC) project.  After OLPC decided to stop trying to use Linux in their revolutionary laptop, the main developers of the operating system left to pursue the project further.  They have been busily improving the operating system and porting it to different architectures.

The best part is that a $200 computer can now be stored on a $5 USB key:

The open-source education software developed for the “$100 laptop” can now be loaded onto a $5 USB stick to run aging PCs and Macs with a new interface and custom educational software.

“What we are doing is taking a bunch of old machines that barely run Windows 2000, and turning them into something interesting and useful for essentially zero cost,” says Walter Bender, former president of the One Laptop per Child (OLPC) project. “It becomes a whole new computer running off the USB key; we can breathe new life into millions of decrepit old machines.”

It will be interesting to see if this incrementalist approach will be more successful long term than the original vision.  Deploying massive amounts of cheap laptops to third world countries has not produced any outstanding successes yet.  Time will tell how this works out.  For now though, I’m going to try it out with my kids.  I’m interested to see how they do with it.

You can find full details on the project in the Sugar Labs wiki.

Digital Socialism

The latest issue of Wired Magazine features an article called The New Socialism: Global Collectivist Society Is Coming Online.  The author, Kevin Kelly, lays out an interesting case for a new form of socialism.  Today’s social networks (Facebook, Twitter, MySpace, Wikipedia, etc…) are all examples of this behavior.  After reading the article however, I must admit to some trouble with the language.  We start with this:

Bill Gates once derided open source advocates with the worst epithet a capitalist can muster. These folks, he said, were a “new modern-day sort of communists,” a malevolent force bent on destroying the monopolistic incentive that helps support the American dream. Gates was wrong: Open source zealots are more likely to be libertarians than commie pinkos. Yet there is some truth to his allegation. The frantic global rush to connect everyone to everyone, all the time, is quietly giving rise to a revised version of socialism.

Already in the first paragraph we see several terms that are charged with meaning.  No matter how we use them, words have their own definition.  If a writer chooses to use a word loaded with negative connotations it will carry that meaning no matter how hard it is explained away.  Starting off the article by comparing free and open source software to communism is a tough way to start.  The author does tries to explain it away but the words still ring heavily at the beginning.

The author does at least admit to the overly charged nature of his words:

I recognize that the word socialism is bound to make many readers twitch. It carries tremendous cultural baggage, as do the related terms communal, communitarian, and collective. I use socialism because technically it is the best word to indicate a range of technologies that rely for their power on social interactions. Broadly, collective action is what Web sites and Net-connected apps generate when they harness input from the global audience. Of course, there’s rhetorical danger in lumping so many types of organization under such an inflammatory heading. But there are no unsoiled terms available, so we might as well redeem this one.

I disagree.  There are plenty of unsoiled terms we may use.  Why not adopt some of the language of the Creative Commons movement?  Can we not refer to open source software as a Town Commons?  How about using the term freedom?  I’ll let the elites bounce around their ideas but we have to come up with something.  We must move away from Microsoft FUD (fear, uncertainty and doubt) and politically charged terms when trying to describe the open source movement.  The average American uses terms such as socialism, communism and fascism interchangeably, even though they are very different.  They simply don’t know the difference.  When the word socialism is read the majority of Americans immediately associate it as bad.

Socialism is a legal framework.  The government imposes a system on the people and everyone is compelled (either willingly or unwillingly) to join and function within the system.  This is not the case with social networks.  No one is compelled to create or edit a Wikipedia article.  No one has to upload their entire lives to FacebookTwitter is voluntary.  While there is greater power in collaborative social networks, all are free to leave.  This is surely not the case with socialism.

Using this terminology makes sense to people like me, but to the rest of the general public it really does hurt the overall movement.  Socialism is a system that while similar, is not a very good comparison.  You should read the article.  I am interested in what you have to say about it.  I do however, regret this article, and hope that it doesn’t set the movement back too far.

Digital Textbooks

Gov. Arnold Schwarzenegger is pushing for California to adopt digital textbooks in favor if printed ones.  This is an effort to help reign in an out of control budget deficit.  This makes perfect sense.  We expect our children to cart around half their body weight using textbooks that are often out of date before they are sent to the printers.  Textbooks are traditionally distributed on a six year cycle!  Money quote:

“So just think about the last six years, all the things that happened,” Schwarzenegger said Monday. “For instance, the Iraq war, the country’s first African-American president … all of this you wouldn’t have in those textbooks.”

A six year cycle?  An entire class of students could pass through high school and not study these recent events!  With the technology we have today, this is simply inexcusable.  We must do something.  Why not use open source textbooks?  From the article:

While not all open-source books are free, they usually have more lenient copyright licenses than do print textbooks – or digital books provided by mainstream publishers. Educators can download and distribute them at will without facing additional costs. Typically, the cost of producing the text is offset by foundations or private donations.

The open source license makes perfect sense.  This frees up students and teachers to be able to copy and use the information contained within the textbooks, free from the restrictions of traditional copyright.  This important change would break the strict control on knowledge, granting the freedom to innovate, adapt and stay current with the latest information.

I was discussing issues of copyright just last week at our annual conference. Ten years ago when I was in college (ok, I’m dating myself here) almost no one had any computer equipment in the classroom.  We carried around our textbooks and used ink pens and notebooks to take notes.  Now when I visit a college campus everyone has laptops, iPods, iPhones, Blackberries etc…  The times have changed.  The educational opportunities are endless.  I asked the question what education would look like in another ten years?  The answer we came up with was that textbooks would be electronic.  The lines will have blurred between print and digital such that a student would have one device that does everything.  Perhaps I was wrong.  It looks like ten years from now is going to happen a lot earlier than I thought.

Follow

Get every new post delivered to your Inbox.

Join 498 other followers