slyfox

For programming, blog post tutorial series are very common and helpful. Having kindly individuals document the design and construction of systems using neat technology is exceedingly valuable for the programming community. The most thorough and handy one that I have found in the last few days has been the SpawnLink Introduction to Erlang’s Open Telecom Platform. Erlang is a major interest of mine, and I have been trying to learn OTP off and on for a while. Previous blog tutorials have taught me everything from Rails tricks to the concepts of neural networks.

I started playing with the otp branch on my erlang_ann (Artificial Neural Network) with the intent of learning by doing and possibly also writing up my experiences into a tutorial. Then I started thinking about whether git itself could be used for making the guides. It seems like appropriately fine grained commits with detailed commit messages could serve as an interested medium for telling stories about code. The power of this approach is that the instructions and rationale are written as the code evolves, with the commentary always corresponding to a full tree of code. This would allow easier experimentation at all steps through the tutorial. I don’t know how well this will work, but I’m trying it out on erlang_ann.

Now the only issue that remains is my chronic lack of overlap between time and energy.

The upcoming election should be pretty interesting. I’m just not quite sure how it will end. At various points in this year, I felt certain that the election had been decided one way or another, but it has stayed dynamic. It promises to hold a few more surprises, in any case.

I have mixed feelings about this election. I, being an internet geek, supported Ron Paul in the primaries. I’m pretty moderate, with the tendency to come off as flamboyantly libertarian. Neither of the candidates truly grab me, but McCain-Palin has been scaring me greatly as of late. I’m pessimistic about the prospects for real, positive change under Barack Obama, and I definitely have my qualms about his various social programs. Overall, his charisma and his openness to diplomacy and hearing outside opinions are what lean me in his general direction. I feel that he has the chops to be a perfectly fine president. At the very least, I doubt he would be catastrophic.

My support for Obama might be relatively unenthusiastic, but my opposition to McCain-Palin is vehement. I view McCain as basically a chump who would represent a lighter, friendlier third term for Bush. He is undoubtedly better than our boy George, but he isn’t actually good. There might have been a time when he actually was a maverick, but those days are long gone now (2-4 years I would say). Sarah Palin on the other hand is a living embodiment of all my worst dreams. She represents nearly everything that I don’t want at any political position above editor of a high school newspaper. Suffice it to say that I get all kinds of warm fuzzies from having leaders who put at least a couple million years between us and the dinosaurs.

In the state election, my ticket becomes more split. For the 5th district House race, I am a strong supporter of Virgil Goode. I disagree with him on many issues, but I feel that he has a good grasp on what’s good for the district. His stances on economic issues also bring a smile to my face. I wish more Representatives would agree with his fiscal restraint. For the Virginia senate race, I am voting for Mark Warner. He is a fantastic politician, and I wish I there were more Radical Centrists that I could give my support.

This election promises to be close and probably critical. I was on the fence for a long time about the president, but given the alternative, I believe that Obama deserves the full force of our support. Hopefully, everything will turn out for the best

After setting up my blog with hobix installed on my Dreamhost account yesterday, I realized that I wasn’t dreadfully fond of that situation either. Ideally I should be able to write content in the most convenient manner possible and have it somehow get to the interwebs. I’m much happier using my local install of Vi and having full control of all the libs that bake my blog into a delicious website ready for dissemination. I am also happy when my data can survive me randomly overwriting critical things.

Given these general desires, I set out to modify hobix to run on my laptop and then, using git and github, get the compiled website to my webserver. Hobix seemed to operate in a very simple manner than would make it easy to modify, so I forked it on github and got to work. I quickly added in a hook and configuration option to commit and push to the remote repo specified by blahg. I then hooked that up to a repo on github. When changes are pushed to github, a post-receive message is sent to a PHP file on my webserver that then performs a git pull. All the plumbing is working now.

While I was hacking this together, I ran into some issues that required going deeper into the code of Hobix. I ended up completely deleting the lockfile capabilities and also changing some of the path handling code that screwed up the templates when generated on Windows. While I was going through this, I realized just how much code was there. It seems to be an extremely fully featured and complicated piece of blogging software. I’m not entirely sure how I feel about that. I’m getting somewhat burnt out on blog features. I kind of feel that I should be able to compile my blog using make like god intended. If I get some more free time, I will probably end up trying my hand at a Rake based blog build system.

I really like my current setup and I think Hobix is quite the tool. I will continue using Hobix while I think of ways that might make me happier about my blog system.

Typo angered me for the final time by not working for the last month. I am now trying this Hobix tomfoolery. I will be gradually migrating old postings over to my new blag.

Processing and Ruby-Processing are really awesome programs for visualizing things and making pretty doodads. Ruby-Processing is great because it uses the JRuby Java bridge to expose all of Processing’s immense power to normal Ruby code. Since I use JRuby at work for data processing anyway, Processing seemed like a natural fit for being braindead easy for drawing.

When I went to hack it into my current project I was sorely disappointed to discover that Java was squawking about SecurityExceptions and signer mismatches when run from jruby. First, how is code-signing a first-class language function? Second, this appears to be a known issue. I really hate Java.

Out of frustration, I decided to hack at it and try to recreate the jar so it would be unsigned and happy. Doing this seemed to work as I can now use Processing wherever I want in my application. If you want to use Ruby-Processing in your JRuby app, download core.jar and ruby-processing and place them wherever your other lib files live.

I created this jar by unzipping the normal ruby-processing core.jar, removing all metadata, and rebuilding it like

jar cf core.jar processing/core/*.class

This seems to work and is generating a color bar for me now.

I started work Monday. It’s fun to work on an Air Force Base, because your commute gets all sorts of fun things like F-15s and F-22s making their landing approach right over where you are driving. My team is pretty cool so we should be able to get stuff done. When I arrived we hadn’t gotten our data yet, so I experimented with the different directions in which my chair could swivel.
The other guys on my team know the admin password and have to do IT grunt work. They have also picked up various data processing tasks from other teams. Because of this, I’m the one currently tasked with working on the actual ASCENDS project.
We got the data Wednesday and were pretty shocked by what we saw. The data formats don’t seem to correspond in any sort of logical manner. I can’t find the sync-points between altimeter/GPS readings and the CO_2 instrument data. Judging from my skimming of the IDL code that was on the DVD with the data, it can’t sync them either. There are magic numbers somewhere that we just don’t know. The fallback in the face of ignorance is, of course, plotting what we can and making it pretty in Google Earth.
To this end, I’m using gigantron. It’s weird how helpful it is for these sorts of tasks. The ascends_viz project on github shows the current work. With the exception of a few hitches (JRuby JDBCSqlite3 driver mostly), it has been pretty smooth for data exploration. My teammates also seem to like what they have seen of gigantron.
Work is also demonstrating how annoying Java is. My team lead has been warring with it for days. CLASSPATH is an evil signature of Satan, and JNI and the hdf4 jars are among the signs of the apocalypse. Observing how well Java complicates and obfuscates nearly everything makes me wonder if humanity has committed some form of mortal sin that has caused God to hate us. Write-once run-anywhere might be the cruelest joke ever. Most of the C/C++ that I write is more cross-platform. Java just makes it difficult enough to even get things working on one platform that you can’t be dicked into testing it on others and therefore must assume that it works as advertised. I hate Java.
Most of the JRuby issues we have had have been Java issues (or Ubuntu trying to use gcj without telling us). JRuby is a really cool implementation. The one thing that I hadn’t noticed when I was setting up the gigantron database stuff was the Work-In-Progress status of the JDBCSqlite3 driver. It seems to work, but it errors out in odd ways when using migration methods. I believe that MySQL will behave much better for our project, and I will try it and benchmark that tomorrow.
All in all, it is coming together. Code is being written and I am enjoying myself. Cool technical details or fun KML will be posted when I can.

This summer I am again working for NASA DEVELOP to create visualizations of carbon flumes over the US to demonstrate some of the possibilities with the proposed ASCENDS mission. Basically, my team will be taking a large amount of random data and a bunch of scientific models and trying to squeeze impressive visuals out of it. We don’t really know exactly how things will work out, but there will be a lot of data processing and exploration. Our team will also probably be tasked with helping other teams process data for their projects.

In my time with NASA DEVELOP, I have done a lot of data processing scripting. Most of the time these tasks are pretty straight-forward and simple, but things become pains in our asses when we have to go in later and refine and build on these tasks. Generally these scripts are just hacks and not meant to grow. A little bit of organization and encapsulation would go a long way in keeping everything sane.

Of course, I’m lazy, so making an organized attempt at data processing with directory structures and buttons not labeled “Shock the Monkey¹” for a simple script that will take at most 10 minutes to complete and hand test is not something that is going to happen.
Over the last year, I have spent an inordinate amount of time being some form of a Rails fanboy (I also spent some time stalking Alan Kay and preparing a hippie van to follow a Phish tour), so I had seen the benefits of arcane magick in programming. Thinking about my upcoming woes with the maintenance of poorly thought out DP scripts, I realized that I could have my organization — without having to give up time that could be spent making references to episodes of Family Guy. Clearly, code generation has my back in this situation.

In this spirit, I spent yesterday hacking away with RubiGen to create Gigantron: Processor of Data. It is currently a simple set of generators that create a directory and file structure to allow for organized and tested DP projects.

I was inspired by a blog post I found on reddit about Organized Bioinformatics Experiments to ape those techniques for my data processing. Gigantron imports data into SQLite (or any other db you feel like) and accesses it through DataMapper models. The actual logic and transformations that we write for the big bucks go in Rake tasks that operate on these models.

Backing data into an RDBMS and then getting at it through the DataMapper ORM adds some initial leg work and bloats some simple tasks, but it also encourages better treatment of input sources and couples your code and its data much less. Data formats and requirements change and having to untangle business logic from uncommented shortcut, good-enough parse jobs are a way to ruin a perfectly good day. The other advantage to this method is that if, for whatever reason (performance), the data must stay in whatever godawful format it was shipped in (damn you HDF), the model abstraction can still make it appear like any other datasource that comes out of our DB. It can be our little secret.

I’m not entirely convinced that Rake is the right golden hammer for this job, but it does make a compelling case for itself. I need to actually try it out in the real world to see if it will work with me rather than against me in production. I think it is the Right Way™, but I need usage and feedback to know for sure.

That’s the other thing. I have written only enough of a framework for it to be something that I would give a try in production. I don’t know what works or doesn’t, but I have a system that I think will save me some time and can be evolved into something that totally doesn’t suck. It is the usage of it that will allow me to find areas of improvement and evolve Gigantron into a true processor of data/destroyer of cities. Hopefully a few gigabytes of real satellite and atmospheric data will help clarify my ideas with it.

If this looks like something that could help in some way for your work, I would be tickled pink to have other people using it. I am very interested in adding anything that could help productivity as that is the true concern here.

Check it out:

• Gigantron RubyForge
• Gigantron Github

¹ I listen to Peter Gabriel while hacking sometimes, sue me.

It happens to us all. We go to Wikipedia with the intention of finding some specific information on some specific topic. Several hours later we realize that we are reading about Sexual Abuse on Pitcairn Island or something equally unrelated to High-Energy Particle Physics or whatever the initial topic was. Oftentimes the articles in between are forgotten and only revealed when one transforms into the smarmy smart ass at the next game of Trivial Pursuit. Much like waking up in a bathtub full of ice, missing a kidney, this loss of time and memory raises unsettling questions about recent events.

A rather old XKCD confirmed that I am not the only person experiencing this most curious malady.

I’m tired of the contents of the “3 hours of fascinated clicking” time block being unknown. I think I am reasonably sound of mind and the connections that get me from point A to point Z on the Wiki would make some sense in context. I might be wrong, but I want to find out with evidence.
I’ve hacked together a simple hacktempt at graphing this solution. Basically I have an extremely simple greasemonkey script that runs on en.wikipedia.org and captures the current page and the referer. It then runs some AJAX that tells a local mongrel hackjob to update the database of connections. The local mongrel server also has an HTTPHandler (localhost:9999/show) that uses Graphviz to render a fresh hot png to be delivered to the web-browser. This handler also takes a query string with start and end to set the date range of interest.

The code is uglier than Fergie on a rainy day, but it works and I find the results to be pretty fascinating.

The code is available on Github

If it amuses you or you have any suggestions let me know.

For those living in rural regions, the Wal-Mart serves as a center of commerce and culture. Much like the village greens of the old world, Wal-Mart provides a meeting place for the like-minded pillars of community. As one browses through camouflage clothing that serves equally well for hunting as it does for gaining social acceptance at the area schools or peruses the selection of NASCAR memorabilia, one can meet and greet all the movers and shakers of the hamlet. Discount sales of Big Mouth Billy Bass wall mountings and John Wayne posters are guaranteed to bring together the finest ladies and gentlemen. The spring time places this retail location as the center of the thriving debutante culture of the region. Suitors and suitees swarm the aisles for some of that old fashioned summer loving. Verily, this shop among shops provides for all the needs of the community in which it resides.

Wal-mart stocks everything needed to maintain a proper and culturally fulfilled existence. One can experience the diversity of the world’s cuisine in the comfort of their own home through a thousand varieties of Hamburger Helper, microwaveable burritos, and the finest DiGiorno Pizza. After an enjoyable meal the likes of which could not be replicated in any larger, morally bankrupt community, one can retire to a tastefully decorated sitting room that exudes the patriotism that can only come from having an eagle statuette with a Thomas Kinkade 9/11 with rippling American flag chiaroscuro screen printed on the wings. The soothing melodies of prima uomos like Scott Stapp of Creed and Austin Winkler of Hinder can delicately waft through one.s leisure time. The impeccable selection and wide range of goods offered by Wal-Mart is unmatched by other retailers. One can tell the quality of the establishment from the fact that it sells warm PBR, Dale Earnhardt shower curtains, and all material goods in between.

While the commercial utility of the institution cannot be understated, it serves an important — perhaps a more important even — role as the center of society. As the place to see and be seen, one sees the arbiters of fashion and trends carefully lurking around the self checkout lanes, checking people out. It is truly a place for every age. From the children stuffed five to a cart being pushed around by consummate professionals in the field of tanning, to the hormonal teenage males in their pickup trucks desperately clinging on to any shred of female affection sent their way, and the adults in their slightly better pickup trucks, Wal-mart contains representation from every race, creed, color, national origin, and societal level that the average product of our fine educational programs could imagine. To further enhance the socialization, one must intermingle with all these groups at a snail’s pace. This is the simple by-product of the legions of ambling socialites who must exert all concentration and effort on merely clearing the sides of the aisles without beaching themselves. Their single minded pursuit of inefficient locomotion is adequately offset by the depth and breadth of their conversations — which are conveniently related at volume levels that are amicable to the prospect of idle eavesdropping. The conversations cover the range of happenings at both Tunstall AND Dan River. The Wal-mart elite certainly understands the importance of being true to one’s school, even if it has not been one’s school for the better part of three decades. Without exception, these conversations are exceptionally interesting and relevant to the affairs of a grown citizen. Whether it is the smell of deer urine being used in a last ditch effort to woo the opposite sex in the fall or the feeling of the pursuit of someplace that is open after 8p.m., Wal-mart exudes the ephemeral nostalgia of a better era. Once one has experienced the Wal-mart scene in all of its infinitesimal1 glory, one cannot be satiated by lesser means.

One cannot underestimate the importance of this integral institution to this community or many others. It exists in a manner that enriches and extends the opportunities present in this delightful slice of society. It protects and shelters the surrounding citizens from unwanted experiences in the broadening of culture and knowledge. It insulates the people from the higher quality of other retailers. It ensures the status quo. It creates such a level of comfort, that referencing such a place without a well-positioned definite article elicits cries of blasphemy and heresy. The Wal-mart is the natural progression2 of this nation.s retail organizations. Wal-mart has the buying power to stock more useless stuff cheaper. The only way Wal-mart could become better is if it offered an in house payday loan office! In one-thousand years, after the moral and societal depravities that we hypocritically denounce cause the collapse of this golden era, anthropologists (and other tweed clad, effete, out-of-touch, elitist, liberal, intellectual types) will look back on the machinations of corporations like Wal-mart and finally understand it for the good that it was.