Rosetta Code Blog

I’ve been averaging 70-90 hours of work per week for a few weeks now, and a lot of work on Rosetta Code has had to be put off.  So here’s a TODO list of accumulated things that need to done on Rosetta Code, and have either been in the works for a long time or have been planned. (And by “planned”, I mean that some of these things have been ideas that just won’t go away.)

At the top of the list; These things are either urgent or are already in the “pipeline”:

• Mod Alias needs to be set up on RC, as mod_rewrite’s ‘+’ handling became broken in the switch form mod_php to fcgi, and we’ve back in the bad old days of C++ pointing to C.  At least there’s something we can do about it now…
• ImplSearchBot needs to be fixed.  It’s editing almost 400 pages every four hours, when it only needs to be editing between one and ten, on average.
• ImplSearchBot’s Subversion repository (where it keeps the JSON caches of category contents) needs to be opened up for general consumption.
• ImplSearchBot’s Subversion repository needs to be abused to generate RSS feeds containing interesting events per language.
• There are some bugs in the way Rosetta Code’s syntax highlighting deals with leading whitespace.  Details are in the relevant Village Pump page.  There also appears to be a bug breaking Unicode support with at least some languages when dealing with the string “møøse”.  Not sure why this would be.
• Need to finish RC promo video.  Looking for suitable audio to sync.
• Find out what causes Recent Changes RSS feed to spit out batches of duplicate items a couple times a week.
• Rewrite the Rosetta theme from scratch.

Things that I want to start on:

• Frame generator task, sourcing from our resident graphics library tasks and sinking to mencoder/FFmpeg/DirectShow/VirtualDub/AviSynth.
• Get Rosetta Code content published via POD services to help pay for services (and thus potentially increasing capacity).
• See if it would be possible to set up an XMPP server with “chat rooms” for every Rostta Code category, and a resident bot that mentions updates to pages in each category.
• Investigate OpenID auth.  Not sure whether or not it’s worthwhile; Serving as an auth source means crypto and greater attention to security. (I know it’s not *required*, but I have my standards as a webmaster…)
• Get the Village Pump a bot to rotate out old, stale pump pages and float active ones to the top of the list. (reddit/digg style, I suppose, but without up/downvoting.)
• Get thumbnails working on the site again via background process.
• Word square generator task
• More fractal tasks
• Markov chain tasks
• More concurrency tasks
• More automata tasks. (Conway’s Game of Life is an example of these.)
• Tasks involving signal processing. (High-pass, low-pass, band-pass,  histograms, Fast Fourier Transform, etc.)
• More graphics and/or UI tasks (2D and 3D), using cross-platform APIs like SDL and OpenGL.  Of course,  I wouldn’t reject DirectDraw, Direct3D, GDI, WPF, Direct2D, Gtk, Qt, Tk, wxWidgets, or even straight Xlib, if people will provide code.

I’d like to see a bit of a shift away from theoretical tasks to practical tasks, and move from explicitly contrasting languages to identifying where a language’s abilities can be taken advantage of for things that programmers often need to do.

If anyone wants to give these a try (especially creating more tasks, creating RC promo material, or anything that requires a bot), go ahead, give it a shot!  I haven’t had a whole lot of time of late.

(Yes, I know there are a lot of links; It comes from having a lot of proper nouns and other interesting concepts…)

I figured out what killed the server Saturday–It was ImageMagick.  The ALGOL 68 Dragon Curve animated GIF is fairly large.  Someone went to visit the GIF’s page on RC, and MediaWiki ran ‘convert’ to create a thumbnail.

MediaWiki ran (approximately; the paths and filenames have been changed to protect the server, and because I tested it locally) ran was:

convert -background white -size 781 ALGOL_68_Dragon_curve_animated.gif -coalesce -thumbnail '781x599!' -depth 8 out.gif

That command, with that data, takes several seconds to run on my Phenom 9650 desktop at home, and my machine has a darn sight more CPU available to programs than anything running within RC’s VPS slice.  As a result, when MediaWiki ran convert, it, via mod_php, via Apache spent several seconds trying to generate a thumbnail.  It would have eventually finished, except that whoever was looking at the page got bored, and refreshed.  Several times.  When I discovered that the server was having issues, the server load average was up around 16.  RC’s slice usually hovers between 0.00 and 0.15.

Three things have been implemented to fix this problem.  First, I’ve switched from mod_php to FastCGI, with the assistance of some of the folks in #mediawiki on FreeNode.  As a result, we get back an HTTP 500 ISE when commands run by the server take longer than expected to complete. (For whatever reason, mod_php was simply hanging.)  Second, I’ve turned off thumbnails.  None of the images on the site are large enough to make them worthwhile, for the time being.  I’ll look into backgrounding convert processes in a way that doesn’t take down the site, but that’s going to be low-priority for now. The third bit came as part of my debugging.  All external commands run by MediaWiki will now be logged, to leave some sort of trace for when this kind of problem happens again.

Sorry about the downtime. I’m not entirely certain what the cause was, but the fix has been to switch from mod_php to fcgi, and correct a few caching settings in MediaWiki. My best guess is that a high-traffic site linked to (or embedded) the Algol 68 Dragon Curve animated GIF thumbnail, which was apparently causing several hung instances of ImageMagick’s convert tool. I’ll know more when I have time to look at the logs and analytics data tonight.