Archive for June, 2007|Monthly archive page
Most of you weather enthusiasts have probably already heard about this, but since there are a fair number of you that come by here that aren’t completely absorbed in the latest weather news, I thought I’d bring it to your attention.
In short, QuikSCAT is the only instrument we have to measure surface winds over the ocean . It makes images that look like this one of Hurricane Katrina. And this one of Hurricane Frances:
Due to various technical problems, it’s ability to continue transmitting data back to Earth is very uncertain. Bill Proenza, the new Director of the National Hurricane Center, has come under heavy criticism from superiors at the NOAA for speaking about the vulnerable condition of the satellite (source).
What does that mean for you and me? Larger evacuation zones (since storm intensity would be much less certain) and a measurable decline in the accuracy of forecasting tropical cyclones—some say by as much as 16%. Oh, and it may take until 2016 to replace it.
I’m not going to go on a crusade against the politics behind it; that isn’t my purpose. And I also don’t want to contribute to any doomsday media frenzy. But as someone that’s spent his entire life benefiting from this instrument I can’t help but wonder what the practical effects of this could be for those living in tropics. After all, it runs completely counter to what we hear every year: that the accuracy of our forecasting system is increasing.
Can you imagine Dr. Steve Lyons saying “well, last year we would have had a better idea, but as far as this storm is concerned we really can’t be sure”?
Better gas up early!
About halfway through the previous post, Stormpulse gained a partner. Brad was my Manager at the company I worked for in Chicago. I still remember the coffee break when he asked me what sorts of things I was in to, and I replied very casually “oh, weather.” I don’t think it made much an impression on him at the time, but over the course of late 2005 and early 2006, Brad would continue to hear about my little ‘windy project’ (as my father has called it).
We worked together for a year before working together on Stormpulse, and our projects together went well. In fact, as a two-person team working on the cutting-edge aspects of our previous employer’s website, we often felt like a startup in incubation, brainstorming for hours and forging ahead even when others didn’t grasp the essence of what we were doing.
In “The 18 Mistakes That Kill Startups“, Paul Graham identifies the #1 cause of startup death as ‘Single Founder':
“Even if you could do all the work yourself, you need colleagues to brainstorm with, to talk you out of stupid decisions, and to cheer you up when things go wrong.”
For us, brainstorming can be a little punchy, but never personal. Talking each other out of stupid decisions is mostly about being preemptive in identifying our direction so that we don’t go in circles (no time for that) and also raising the bar on one another based on our strengths. Stormpulse.com probably wouldn’t exist today without these checks-and-balances, because without a founder, I know there are times when I would have quit. Not because it was too hard, but because the path has been so long.
I left the company first, in August 2006. Brad and I had been working on the site together for a few months, and when I decided to make the jump from Chicago to Florida (for personal, not professional reasons), the opportunity presented itself to make Stormpulse my full-time job. So we did the logical thing and hired me, thus enabling me to leave behind my cubicle and about the best corporate work environment a person can imagine, for an upstairs den, my laptop, a few pens and a lot of empty source files.
It was time to find out if we could execute. My role was chief trailblazer, his chief architect. For the next year Brad would have his nose to the grindstone making the project into a real enterprise application, handling detail after detail on matters of performance, reliability, security, and installation. If Brad weren’t on the team, Stormpulse.com would be written in Python as a TurboGears application, still running on a single server (likely FreeBSD), riddled with security holes, and frequently timing out due to inadequate database performance.  Instead, today Stormpulse is an Amazon EC2 + S3 production with tight control and a real migration path. And that’s just the beginning.
The subtle but critical truth is, if I didn’t have a co-founder, I wouldn’t know what I was missing. Even more importantly, I would be missing what I’ve found to be the real x-factor: the motivational glue that says “I’m not going to let it die, because this is bigger than I am.” 
 The front-end would be different too. Brad has a knack for picking something about the interface that needs the smallest visible tweak in the universe. The worst part is that I usually balk at the suggestion initially, but, when I finally try it, he ends up being right (after about 10 iterations). I suppose I can live with that.
 Paul Graham calls this one of the most powerful forces in human nature. I’m inclined to agree.
“The theory of the Long Tail is that our culture and economy is increasingly shifting away from a focus on a relatively small number of “hits” (mainstream products and markets) at the head of the demand curve and toward a huge number of niches in the tail. As the costs of production and distribution fall, especially online, there is now less need to lump products and consumers into one-size-fits-all containers. In an era without the constraints of physical shelf space and other bottlenecks of distribution, narrowly-target goods and services can be as economically attractive as mainstream fare” (emphasis added)
His speech resonated with me insofar as it underscored my reason for avoiding the Google Maps API when developing Stormpulse. Simply put, today’s mapping API’s don’t jive with the long tail.
The way to appeal to the long tail is not simply to provide manifold opportunities for input (user-generated content). Google is doing this with their My Maps feature, as are many mashup that depend on Google’s API.
Instead, the key to jiving with the long tail is providing a meaningful context for user-generated content. To many this probably implies the social network. Indeed, that’s a part. But what about display? I’m sure it goes without saying, but a little red balloon can only represent so many things well (carnivals, whoopie cushion factories, et al.). In fact, at this point I am downright numb to that ubiquitous icon of all things geospatial.
Before you say that I am only being negative, I will say this: many Google Maps mashups are impressive (this one plotting Chicago gangs even left such a strong impression that I vividly dreamt about it). And I’m sure that these thoughts of mine aren’t very inspiring or original. So, allow me to quote Chris Anderson one more time, this time regarding a recent review of the long tail theory written by Bear Sterns:
[…] in a world of infinite choice, content is only as valuable as your ability to find it. They call that “context and aggregation”, and it’s what both Google [Search] and your favorite blogger do when they filter the web according to a narrow lens, be it your expressed search term or their own sensibility. (read the post)
So, mashup engineers: are you displaying your data through a narrow lens? Are you being sufficiently narrow (and vertical)? Again, one-size-fits-all doesn’t do this. To me, Google Maps is a very normal lens, and many of the decisions about what to display and not to display have been made for you. This isn’t all good. For example, I couldn’t *not* call out the Cape Verde islands at a hurricane-tracking website and still feel that I’d made a site truly devoted to tropical cyclones. And yet this kind of control is missing from the one-size-fits-all solution.
And what is your visitor’s sensibility? I take this to mean their entire sense for the universe your data lives in. Is it really filled with balloons and green arrows, yellow streets and . . . well, you get the idea. Or is it something else?
Google Maps opened the floodgates for mapping street-level phenomena, but the real long tail solution will be a platform for creating niche mapping applications, not a generic mapping application with a finite ability to customize.
The previous post left off with Stormpulse having been cobbled together from a very large text file and a few lines of PHP for creating XML. It was shortly after this that the project went in to its slowest period of development. From late 2004 to early 2005, I was working full-time as a software developer at a medium-sized private company in Chicago, the hurricane season had ended, and I still didn’t have any idea that it would ever see the light of day (most of my solo projects don’t).
But then, through a friend, I was introduced to a somewhat uncommon programming language called Python. I had heard of it before through a co-worker, but, as is so often the case when a programmer encounters a new programming language, aversion set in, and I rationalized that I didn’t need that silly-sounding language. I was what Paul Graham would call a true Blub programmer (from his essay Beating the Averages):
“[W]hen our hypothetical Blub programmer looks […] up the power continuum, he doesn’t realize he’s looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.”
But then I met another friend who by chance got me to look at Python (literally, over his shoulder), and with it TurboGears, a Python web framework—and that changed everything. I still remember the euphoria of shortly thereafter writing a search program in under 300 lines that could return to me all of the storms that had ever passed by a given city. This was made possible by tapping into a GPS database I had assembled out of text files I had, of course, also found out on the web.
Now the data was interactive, responsive—fun! I remember keying in arguments to the search function and gleefully announcing to anyone that would listen: “Look! All the storms that have ever passed through West Palm Beach! Isn’t that so cool?!” Given that I was the only one with an emotional connection to that factoid, it’s not surprising that no one else thought it was quite as cool. But no matter; the real epiphany was that up until that point I thought that kind of information was something that only the real weather folks could have. But now it was mine, and for what seemed like very little effort (less than I expected).
I started using my morning breaks at work to sketch tiny database diagrams (I had multiple tables now, after all!) and to scheme over the next feature—”Similar Storms”. I wanted to see all of the storms that had ever taken a path similar to Hurricane Charley, for example. Eventually this was solved with the help of a friend, and I found this dataset to be even cooler than the last (and so did he).
But then I became a father (a continuing joy :-)), and life went on, and the project died down again. Then the 2005 hurricane season happened, and a close friend of mine reminded me repeatedly how I should have already had Stormpulse out there. That I had missed the golden opportunity. *sigh*
Determined not to let all of the effort go to waste, I spent the rest of 2005 and the beginning of 2006 coding the mapping window in my spare time. It wasn’t easy. Not only was it my first ActionScript project, but I also decided in my bountiful wisdom that I was going to do the Earth in 3D. Really. I wanted Google Earth (or better yet, NASA World Wind) in a web browser.  And who doesn’t? Unfortunately, I eventually had to gut much of my work. But at least by that time I was so sick of converting coordinates and accounting for the true shape of the Earth (it’s not really a perfect sphere you know) that I was able to embrace a 2D implementation quickly and wholeheartedly. 
The site had become more than a hobby; it was hard work. But around this time I also began to believe that I could actually make something better than other hurricane-tracking websites.
 It was NASA World Wind that inspired me to have real-time cloud cover for Stormpulse. Before seeing it in World Wind I didn’t know it could be done (i.e. that the data was freely available).
 Since I still get this question, I guess it’s worth repeating: the mapping window for Stormpulse isn’t driven by Google Maps or any other API. I’ll save the long answer to ‘why?’ for another post, but in short, for me it’s about total control.
The amount of weather information being collected and circulated on the Internet these days boggles my mind. Much of this collection is done by the government’s National Oceanic and Atmospheric Administration (NOAA), parent of the National Hurricane Center (NHC), the source drawn upon my most of our website’s features. Since the NOAA is a government agency, all of that information is immediately in the public domain. This is the upshot, and this also makes us happy because it made it possible to start a weather website with relatively little initial capital. 
But there’s also the downside (or, as I would later discover, the opportunity). Because the NOAA is a government agency and the government has been interested in weather for a very long time (largely to support the operation of the military), much of the data that gets shipped around the Internet gets packaged in arcane, Mainframe-era formats (example), and the graphical displays are usually rather dated:
Before the summer of 2004 I never really thought about either of these facts: that weather data is abundant and free, and that weather data is often trapped inside formats that make it illegible to all but the most die-hard enthusiasts. It was back in 2004 that I discovered the first fact: that I could simply go right out on the web and get storm track data back to 1851 with a relatively simple Google search. With a little bit of parsing (at that time with PHP) I had myself a searchable, sortable database. 
And to me, the track data contained more than just numbers. The numbers told stories (888 millibars!) , and names like Andrew, Mitch, and Floyd brought back memories of tracking storms during my childhood and young adult years in South Florida . With all due respect, none of the sites I was visiting at the time really seemed to make the data sing the same tune of anticipation and excitement that surrounds the live event of a tropical cyclone forming and churning across the Atlantic.
Of course, the goal for me at the time wasn’t to make the data sing. The goal was just to have a live feed of data on the storms as they developed. It was a challenge for me as a novice programmer, and there was just something satisfying about seeing the updates come out from the NHC and then minutes later being able to refresh on my little computer and see . . . new data!
It would be a year or two before the second fact—that weather information is often locked in painfully-confusing formats, would finally dawn on me as an opportunity; in the meantime, I had lots of data to scrape, parse, and mine for the fun of it, and Stormpulse , a few little scripts on top of a couple of database tables, had been born.
 While the thought that the core of our data is entirely free once scared me, I now embrace it. After all, I heard that Amazon.com got started off a free list of ISBN numbers, but who’s going to replace them now?
 To be honest, I was too young to remember the millibars, but seeing the data brought back some vivid memories: that my Dad’s co-worker was in the Yucatan when it hit, and that we still have the VHS tape of the satellite image progressions shown on the evening news. I can still see the intense red and orange surrounding the eye as it passed through the Caribbean, and the word ‘Gilbert’ scrawled on the label.
 You’re right, of course; neither Mitch nor Floyd hit South Florida. But I had a good friend from Honduras whose family and friends were in Mitch’s path of devastation, and Floyd . . . well, can we understate the importance of the Bermuda high?
We are very happy to announce that Stormpulse has received another set of upgrades and enhancements. The main thrust was to make the site feel more current, whether the Atlantic Basin is active or quiet. In summary, we’ve released:
- Cloud cover of the Earth that’s self-updating and current. No more staring at Barry weeks later. The cloud cover you see should always be less than six hours old.
- Satellite image browsing. This one took up a lot of time, but we finally have a real, navigable interface into our satellite imagery. On the home page, you’ll see the latest images for all regions: Caribbean, Florida, Gulf of Mexico, Puerto Rico, Texas, Eastern U.S., Mid-Atlantic U.S., Northeast U.S., West Atlantic, Northwest Atlantic, North Atlantic, Central Atlantic, East Atlantic. I’m happy to say that the URL’s are fairly clean as well–for example, http://www.stormpulse.com/satellite/florida will show you the latest imagery for Florida, http://www.stormpulse.com/satellite/gulf-of-mexico will show the latest imagery for the Gulf of Mexico. Currently we are only grabbing images from the GOES-12 and UK Met. satellites, but we will be adding links to the high-resolution Aqua and Terra MODIS satellites in the future.
- Option to turn off cloud cover. There’s now a toggle at the top-right of the map window for turning clouds (o) On or ( ) Off. If there’s a green dot on the ‘On’ option, they’re on, and . . . well, you get the idea.
- Circles showing the extent of storm force winds. Known as ‘wind radii’ in weather lingo, these are concentric circles drawn on top of the storm’s path to illustrate how far tropical depression, tropical storm, and hurricane-force winds extend from the storm’s center. These help to answer the question of if and when the storm will be upon you (tropical cyclones are not small-scale events!). This data is available for all presently-developing storms and storms dating back to 2004. Keep an eye out, because every storm has a unique wind radii ‘signature’, and as we all know, winds can vary dramatically in each ‘quadrant’ of the storm. For example, take a look at Rita‘s symmetry compared to Wilma‘s later stages.
- Tropical Weather Outlook. We’re now displaying the latest Tropical Weather Outlook in the top-left corner of the home page. The Tropical Weather Outlook summarizes the National Hurricane Center‘s thoughts on if and when we could see new storm development in the tropics.
- Added new hurricane-tracking buoys. We’ve added Buoy 42059 and Buoy 41043 to our database. Floating in the eastern Caribbean Sea and north of Puerto Rico, respectively, these buoys are in prime positions for monitoring tropical cyclone activity. View a map of NOAA’s buoy expansion project.
- Tweaked the display of our buoy data so that the main buoys along the path from the Cape Verde islands to the Yucatan Peninsula and Gulf of Mexico are visible when you’re zoomed farther out.
That’s all for now. Let us know what you think. As always, thanks for staying tuned; I do believe the best is yet to come . . .