Dark Sky Journal

Announcing Forecast

2013-03-26T00:00:00-07:00

We’re extremely excited to announce the launch of Forecast, a new global weather service.

Check it out the announcement on our Forecast blog.

We make Dark Sky — a brand new kind of weather app. Dark Sky predicts precisely when it will rain or snow at your exact location and has the most beautiful radar visualizations you’ve ever seen.

It’s Crazy How Disruptive We Are

2013-02-05T00:00:00-08:00

Our friends at Fast Company just posted this video they made where Adam and I say words and only come off as moderately strange.

Open Source

2012-11-20T00:00:00-08:00

At Dark Sky, we believe strongly in open source. Without some of the wonderful libraries out there, both the Dark Sky app and the Dark Sky API would have been utterly impossible to write. (Unless, maybe, you’re some kind of super-human cross-disciplinary programming savant.) Unfortunately, some of the tools we needed, we couldn’t find in the open-source toolbox. So we wrote them, and today we’re officially releasing some of them into the wild.

The Dark Sky API is written entirely (well, almost entirely) in Node.JS. We picked it from the beginning since it’s so darned fast and lightweight, we can power the entire API off of a few teensy virtual machines. All of the modules we’re releasing today were developed for the API, and so they’re all Node.JS modules. If you want to use them, they’re all handily available in NPM for your convenience.

cache-helpers: A set of abstracted, commonly-used caching patterns. The Dark Sky API is quite fast, and most of the reason for that is lots and lots of caching.
pngparse: a pure-JS means for parsing PNG images into a pixel array. We’ve mentioned before that Dark Sky stores gridded data as images: this module is how the API reads those images.
sphere-knn: an acceleration data structure for making k-nearest-neighbor queries on a sphere. Since most weather data comes from ground-based radar stations, looking up what stations are near a requested location is a common operation for us!
string-hash: A teensy little module for hashing strings to integers.
tz-lookup: Quickly lookup the timezone of a given latitude/longitude pair. We originally made use of the tzwhere module, but found it far too slow for production use. tz-lookup is a tad less precise, but an order of magnitude faster.

All of these modules are well-tested and have been in production use for a few months, so any kinks should be worked out by now. And, of course, we love to see forks and pull requests, so feel free to contribute to the modules if they don’t meet your needs!

We plan to release more of our modules in the future, but we’re working really hard on international support and some secret projects, so you’ll just have to wait. (It’ll be worth it, promise!)

Jason LaPorte for Dark Sky

2012-09-19T00:00:00-07:00

INT. SCENE - DARK SKY OFFICE
SHOT - BLURRED FRAME FOCUSES TO REVEAL JAY

                        JASON LAPORTE
              Hi, I'm Jason LaPorte of the Dark Sky 
              Company.

              I come from the future: only an hour, 
              but sometimes that's all you need. You 
              see, in the future, everybody knows 
              when it's going to rain; and now, 
              thanks to the advanced technology of 
              the Dark Sky App, you can, too.

              So join us, and stay dry.

                      (Jay smiles)

[Throw credits in white text over Jay who is just standing 
there. He continues to smile and eventually says “Thank you.” 
and then he fades out, leaving just the white credits on black. 
Text fades out then Jay fades back in.]

                        JASON LAPORTE
              Oh, in an hour the economy still sucks.

App Store “Optimization”

2012-08-23T00:00:00-07:00

App Store Optimization: The act of fiddling with two arbitrary strings of text in order to rank higher in searches in the Apple App Store.

Here is a history of Dark Sky’s rankings, in yellow, for three search terms. Can you spot when we “engaged in search optimization”?

Search for “Weather”

Search for “Forecast”

Search for “Radar”

A few observations:

It appears that the only things that matter are the list of app keywords and the app title (which happen to be the two fields that a developer cannot modify once an app has been published). The description doesn’t seem to effect search results.
Changing our title from the nice and simple “Dark Sky” to the ungainly “Dark Sky - Weather Radar, Hyperlocal Forecasts, and Storm Alerts” had a big effect. Look at those juicy keywords!
You’ve got to play games with the keywords field. Inexplicably, you need to include pluralized words as separate keywords (e.g., both “forecast” and “forecasts”). And be sure to remove spaces in between your commas so you can cram in an extra word or two!
Being in the Top 10 Sales / Grossing lists don’t matter. Topping the chart in your category doesn’t matter. Being Fireballed doesn’t matter. Consistently improving your app doesn’t matter. What matters is adding the right words to your app’s title field.
The good news is this: Turns out you can have a successful app without appearing in any search results. Dark Sky has done remarkably well. Although I suspect this is bad news for Apple... it indicates search is so broken that it’s not the primary means for finding new apps by many (most?) iOS users.1

Whenever I search the App Store, it’s almost always to find a specific app that I’ve already heard about elsewhere. I’ve given up using it for exploration.

Notifications

2012-08-15T00:00:00-07:00

It seems there’s been a lot of confusion regarding notifications, so we wanted to take a moment to clear some of it up regarding two facets of the feature: battery life, and privacy.

Battery Life

iPhones and iPads have two methods of gathering your location: the GPS, which is power-hungry but accurate to within meters, and cell-tower triangulation, which is low-power but somewhat less accurate.1 Apple makes both available to developers, allowing them to decide if they really need the added accuracy or not. As it turns out, most developers just take the most accurate method they can even when it’s not necessary, which has led to Location Services having a bad reputation for draining the battery.

When you’re using the Dark Sky app itself, we use the GPS: the accuracy is important to the app, and we don’t use it for long enough to hurt your battery. When you enable notifications, though, we need to keep a connection to Location Services indefinitely; on the upside, we don’t need as much accuracy, so we use cell-tower triangulation instead. As a result, enabling notifications in Dark Sky will not drain your battery.

Unfortunately, we’ve received a number of negative reviews on the App Store claiming the opposite. But don’t you believe them!

Privacy

Enabling notifications will have Location Services keep us up-to-date with your location, so we can provide accurate notifications even if you move. We understand that all location-based services are based on trust, and several of our users have expressed concern about trusting us with their location. Since trust is based on honesty, we want to tell you exactly what information we collect, and what we do with it.

If you enable notifications, Dark Sky stores four pieces of information:

A unique code, specific to Dark Sky, identifying your device. When we want to notify you, we give Apple that code and the notification message. Only Apple has the ability to associate these codes with your device: neither we nor anybody else can identify you or your device from this code.
Your current location. We use this to keep an eye on the weather where you are, and send you a notification if it’s going to rain soon.
Your location at the last time we checked the weather for you. If you’ve moved far enough away, we know that our previous forecast might not be accurate anymore, and we’ll check the weather again, just to be safe.
Your notification settings (including threshold and do-not-disturb times). We have to store these in order to make use of them on the server.

We store nothing else about you, and all of the above data is transmitted securely (via SSL). Finally, as mentioned in our privacy policy (available from the Dark Sky settings panel), we will never sell, rent, or share any of your personal data with anyone.

If you have any questions about us or our privacy policy, please don’t hesitate to send us an email. But at the end of the day, if you don’t feel that we’re worthy of your trust, simply do not enable notifications.

Well, technically, there’s a third method: the device can also work out where it is by triangulating nearby Wi-Fi networks (which is how users with only a Wi-Fi model iPad or iPod Touch can be located). It's accurate and low-power, but often doesn't work in rural areas. ↩

Version 2.0

2012-08-14T00:00:00-07:00

In the life of a software product, each major version upgrade is considered a major milestone in that product’s life. Today, Dark Sky 2.0 is available in the App Store, and as a celebration of it’s first “birthday,” we thought it would be nice to take a brief look at the app’s past, present, and future.

Where We’ve Been

Last November, we started a Kickstarter project to bring an idea of ours to life. 1,200 wonderful people believed in that idea as much as we did, securing enough funding for us to begin the hard work of turning Dark Sky the prototype into Dark Sky the app.

In programming, there’s an old adage that “the first 90% of a project is fun, the second 90% wears you down, and the third 90% makes a great product.” We found that to hold from the very beginning, and really began to buckle down into that second 90% of the work in February, even going so far as to hire another guy to help reign in all the bugs and polish the interface. We slipped our planned March release date, but did manage to release Dark Sky 1.0 a month later.

Of course, Dark Sky isn’t just a product, it’s a service: it needs servers to run and has bugs that need fixing. So, unlike some Kickstarter projects that have a definitive start and end date, our little project had to keep on chugging, which meant we needed to sell copies of the app to keep the lights on and feed our families. Sales began slowly: getting the word out is a difficult process, the app had a few large bugs, and several early reviewers had perhaps unrealistic expectations of how well one can predict the weather (even though we do a lot better than the weatherman). For a while we were worried about being able to keep things running, but sales slowly picked up through a combination of advertising and positive press from bloggers and journalists. We now have over 35,000 users!

Where We Are Today

Version 2.0 is a free upgrade now available in the App Store. In addition to bugfixes and minor improvements, we’ve included two new major features: push notifications and a national radar view.

Push Notifications are a feature that our users have been requesting since we first launched last Spring. By enabling notifications within the app, we will tell you whenever it’s going to rain in the next ten or fifteen minutes, so you’ll never get caught in the rain—even if you forget to check the app. We’ve actually been working on this feature since the beginning, but it’s been very tricky to implement: we’re not quite 100% confident that we’ve gotten it perfect. For that reason, we’re currently considering the system to be experimental: if you find that notifications aren’t behaving exactly the way you’d expect, we’d love to hear from you. As always, we strive to improve the app with every release.

Expect notifications to be available in the developer API sometime in the near future.

The national radar view is a new radar mode that you can access with the touch of a button from the standard radar view. This view is similar to Dark Sky’s existing radar view, but shows the entire continental US at once, so that you can get a sense for the “bigger picture” of what’s happening in the country right now. It has exactly the same smooth, beautiful radar tracking as you’ve come to expect from Dark Sky, on a different scale. And in this mode, you can scroll back in time over twelve hours.

Where We’re Going

Since launching Dark Sky, we’ve gotten hundreds of requests to expand our coverage, globally if possible. We’re thrilled about the excitement that Dark Sky has generated, and now that version 2.0 is out of the way, we’re starting to work on international support in earnest. As we’ve told the many people who have asked us, international radar data is fragmented directly along national lines and generally less accessible as it is in the US: each country controls its own radar technology, has its own licensing fees, and has its own export laws. As a result, we’re limited to tackling the problem one country at a time, based on the number of requests we get for that country. If you’re interested in having Dark Sky where you live, be sure to let us know.

Additionally, with our new national view and international support on the horizon, our current interface is going to get cluttered quickly. We plan to completely revise our radar view in order to provide as elegant and beautiful a user interface for viewing as much of the world’s weather as possible.

Thanks to our 1,200 Kickstarter backers and our 35,000 customers, the future looks bright for Dark Sky. Every time we hear that we saved someone from a rainstorm, it brings a smile to our faces. We’re excited about where we’ve come from, and where we’re going. Dark Sky has just had it’s first “birthday.” Here’s hoping for many more.

How Dark Sky Calculates Temperature

2012-05-31T00:00:00-07:00

When we set out to build Dark Sky, our plan was to focus exclusively on precipitation for the next hour. It’s what we do differently, and it’s what we do well: we wanted to leave the longer-term prediction to other apps. Our users have asked us repeatedly for more, though: it’s frustrating to have to use multiple apps just to get a single picture of the day.1

Really, we find it flattering that our users want to spend more time in Dark Sky. So, you may have noticed that version 1.2 (from last week) included the current temperature, and Dark Sky 1.3 (out today!) includes 24-hour forecasts of both precipitation and temperature. We’ve talked previously about how Dark Sky finds the precipitation down to your exact location; we thought we’d shed a little light on how we do the same for the temperature.

In addition to its approximately 150 radar stations, the National Weather Service also operates about 1500 ground stations that take more traditional readings, including temperature, dew point, wind speed, and so on; additionally, they provide hour-by-hour forecasts of this data over the next day. All of this information is publicly available just like the radar data is. It’s really easy for us to know the temperature at those specific places around the country simply by downloading it from the government’s servers and looking up the temperature at the desired time. As an example, here are the ground stations and their temperatures for upstate New York:

But what about everywhere in between those stations? Well, temperature is a very low-resolution phenomenon: it doesn’t change very much even over moderate distances. One approach (and, in fact, the first one we tried) is to simply find the nearest station to a given location and use its temperature. This works pretty well most of the time, but starts to fall apart if the nearest station is on top of a mountain, or far away: we could be off by several degrees. A couple degrees doesn’t sound like much, but trust me: we get complaints over even minor issues!

A better solution (and the one we now use2) is to average the temperatures of nearby stations; but which stations do we pick? The technique that we settled on is one that has been successfully applied to similar scenarios in computer graphics, called triangulation:3 we just “connect the dots” between stations until we’ve covered the entire country in triangles; then, to figure out which stations to average, we find which triangle the desired location is in, and take a weighted average of the three stations that form that triangle.4

That’s really all there is to it! For sake of comparison, below is the “nearest station” version side by side with the “averaged” version. You can see how much more smoothly temperatures blend over distances:

We really like using Apple’s iPhone weather widget alongside Dark Sky, but we found out later that it isn’t available on the iPad—and since one of our developers only uses an iPad, we knew we had to do something about it! ↩
We find that doing these kinds of predictions on our servers, rather than in the app itself, is preferable because we can continually improve accuracy without having to send out an app update. ↩
The exact method we use is called Delaunay triangulation: it tries to form triangles that are as close to equilateral as possible. This ensures that the stations that form the vertices of the triangle are as close (on average) to any point within the triangle as possible. Our JavaScript implementation of Delaunay triangulation is open source. ↩
The fastest way to check whether a point is within a triangle is to find that point’s barycentric coordinates; this has the beneficial side effect of also giving us the weighting coefficients for each vertex for free. ↩

Kickstarting an iOS App, Part 2: Physical Rewards

2012-05-28T00:00:00-07:00

Dark Sky for iPhone and iPad was made possible through Kickstarter. In this series, we’re talking about what we wished we knew when we started the project: previously, we looked at how we got our backers their digital rewards; in this second part of the series, we’ll be talking about the process of getting our backers their physical rewards.

As mentioned in the previous article, all three of us at Dark Sky came from the world of web development, and if creating and releasing an iOS app was on the fringes of our realm of experience, creating and sending out physical rewards was completely off the map. The two physical rewards provided to our backers were silk-screened T-shirts and silk-screened umbrellas; getting these rewards to our backers was by far the most arduous part of the Kickstarter campaign, even considering the positively wonderful support given us by Design It Together, the local print studio that actually produced the rewards for us.

We got a late start on designing the T-shirts, because our goal for them was so straightforward: the Dark Sky icon in medium-gray on a dark gray background. We handed off our design to the print studio and expected things would go pretty quickly… and then we discovered that “asphalt” and “charcoal” are two different colors, and that the medium-gray ink shows up differently under varying lighting conditions, and that women’s shirts print darker because of their denser weave, and so on; furthermore, each iteration involved several days of waiting simply due to logistics. In total, the process we expected to take a month ended up taking two-and-half, through no specific fault of any party.

After we had the rewards, we still had to ship them out to each of our backers. It seems that places like Amazon have spoiled us into thinking that shipping is cheap; in our experience, the cost of shipping is always much higher than you expect. We looked at every reputable carrier we could find, and in the end found that the United States Postal Service had the best domestic and international rates by far (via flat-rate priority mail boxes), but even then it cost us as much to ship the rewards as it did to have them made (and, in the case of international orders, up to three or four times as much).

USPS has a website from which you can order large quantities of flat-rate boxes—for free!—and have them delivered right to you (though, in our case, the boxes were held at the local post office since they believed a residential address ordering 250 flat-rate boxes was an error). There is a time investment involved in folding the boxes, folding the shirts, printing the shipping labels (which can also be done online) and affixing shipping labels to each; but this part of the process actually went faster than we expected and didn’t factor significantly into the overall reward fulfillment time. After that, we only needed to wait on the postman to pick everything up.

(In our experience, while the post office sapped lots of our time and was generally unhelpful, we found that the two or three mail carriers that we interacted with were wonderful human beings, going out of their way to help us make the mailing process as easy as possible. If you ever find yours off-duty, buy them a drink!)

While we personally didn’t actively have to invest much time and effort into producing the physical rewards, there was a lot of waiting and back-and-forth involved in both getting them made and mailing them; if we were to do this all again, we would get started on it immediately upon the project’s successful backing and try to do as many parts of it in parallel as possible. We are also quite certain that, given the enormous cost of shipping, there has to be a better way; we’d do more research the next time around, and would encourage others to do so as well.

Dark Sky for Developers

2012-05-22T00:00:00-07:00

People see Dark Sky and they see an app, but that’s only half the story. Not even half.1 What you hold in your hand when you use our app is a mere visualization of an entirely unique weather prediction system. This system has been developed by the two smartest people I’ve ever met2 and is every bit as impressive as the app itself.

Utilizing the US government’s doppler radar, and applying whiz-bang science,3 this Dark Sky Net — for lack of a better term — provides us with minute-by-minute predictions for exact locations and human-readable descriptions of what’s about to happen. We developed this system to be incredibly easy to communicate with over the Internet, resulting in a very powerful API. And when you make something powerful you want to share it with the world.

Ladies and gentleman, I give you Dark Sky for Developers…

We know that these kind of forecasts must have applications in any number of markets / human endeavors. In fact, we are actively pursuing a number of licensing deals. But why limit Dark Sky to only the projects we can think of? So we’re offering free unlimited trial access to our API for anyone who wants to know the [near] future.

To facilitate this, we’ve built you a simple dashboard where you can monitor your API usage, manage your secret key, and read the documentation. Just don’t go and use our API in anything that is open to the public or commercial without talking to us first. When you reach that stage just send us an email and we can figure out how best to work together.

Now please, go make something!4

47%, to be precise. ↩
I can say that, they can’t. ↩
Neural nets, optical flow, cloud computing, object tracking, and Node.js — to drop a few buzzwords in no apparent order. ↩
Namely, an Android app that does short-term forecasting. We’re a bit busy at the moment. Thanks! ↩

Kickstarting an iOS App, Part 1: Digital Rewards

2012-05-08T00:00:00-07:00

Kickstarter has become very big news on the Internet, and for good reason: the Internet is all about novel ideas, and Kickstarter has become one of the most visable means of funding those ideas that would otherwise never get off the ground. Our project, Dark Sky, was one of those ideas; and thanks to our (beloved, wonderful, attractive) Kickstarter backers, we were able to meet our funding goal and build our iOS app.

Of course, the story didn’t end when we met our goal, and we learned quite a bit along the way. Since so many other folks are looking to Kickstarter as a way to fund their own projects, we wanted to contribute some of the things we wished we knew when we started. In this first part of a series, we’ll be talking about the process of getting digital rewards out into the world.

All three of us came from the world of web development, and moving to app development is a bit of a change for us; the most obvious difference is that instead of being able to roll out changes and bugfixes whenever we want, we’re constrained to a release cycle—but that’s a story for a different day. The other major difference, and the one we’d like to talk about today, is in distribution: on the web, everything you make is immediately public to the world; on iOS, though, there was a process involved in getting our Kickstarter backers their rewards. We never really thought about this ahead of time, thinking—we assume, like many others—that we would just give each of our backers promotional codes. Unfortunately, that avenue just isn’t viable: Apple gives us 50 codes for each version of the app, and releasing an update takes (at least!) a week to go through approval. In order to get enough promo codes to give one to each of our backers, we would have to update the app once a week for six months—and that doesn’t even include using any codes to promote the app! Obviously, promo codes are simply infeasible for rewards.

The solution we settled on was Apple’s gifting mechanism: essentially, buying each of our backers their copy of the app. This isn’t quite as arduous as it sounds: everything is done right from within iTunes, and you are allowed to provide Apple a comma-separated list of email addresses to gift to, though there is a limit on the number of characters that can be submitted at once; we were able to send about fifty copies of the app in each batch. A couple hours later, each of our Kickstarter backers had an email with their copy of Dark Sky.

Well, almost. The process wasn’t all rainbows and unicorns; we encountered more than a couple problems with it.

Apple places a limit on the number of gift purchases that can be made in a certain period of time. We aren’t sure what the exact parameters of this limit are, but after having made some 500 purchases of the app in an hour we were given an error telling us that we couldn’t make any further purchases for a while. After waiting an hour, we were able to finish making our purchases without issue.
Strangely enough, our credit card company regards making a thousand purchases of the same product in a short period of time as illegitimate behavior, and as a result placed a hold on our credit card until we called them and straightened everything out.
Apple takes a 30% cut of all App Store purchases, and while we were expecting overhead costs for the physical rewards we offered, we hadn’t counted on having to spend an additional $2,000 (that’s fully 5%!) of our gross Kickstarter funding on digital rewards. Obviously, these costs can be minimized by setting the initial app price very low and raising it as soon as we’d finished gifting it to our backers, but this seemed to us dishonest, and a poor way to begin our business relationship with Apple.
After purchasing a gift for a particular email address, Apple sends that email address an email from “iTunes Store <do_not_reply@itunes.com>” with the subject “— sent you an iTunes Gift;” a large percentage of our backers never received these emails, whether due to them giving us incorrect email addresses, or losing the email in their spam folders. In fact, even backers who use Apple Mail would often end up with the email being marked as spam. We ended up spending quite a lot of time working with backers who never received their apps individually over email; in some cases, the lost emails were found in spam folders, but in other cases we needed to use promo codes or even buy additional copies of the app in order to resolve the situation, further adding to our overhead.
When someone is gifted a copy of the app (or given a promo code), it seems that they are not able to leave a review in the App Store. While we can appreciate the reasoning behind this, it is unfortunate, since many apps live or die by their reviews and an initial burst of positive reviews would help to encourage sales, especially for a “premium” app like ours.

All in all, the process wasn’t especially painful and we’d go this route again, but we’d budget our time and money much more carefully: it’s difficult to spend several hours daily on responding to support emails when there are crucial bugs to fix in your brand new app, and it’s difficult to spend a few thousand dollars when you’re trying to get your fledgling company to take off.

How Dark Sky Works

2011-11-07T00:00:00-08:00

A lot of people have been wondering how Dark Sky works. In this post I’ll try to shed some light on what goes on under the hood.

Getting the data

All our radar data comes from the National Oceanic and Atmospheric Administration. NOAA operates a network of over 140 radar stations spread across the United States and its territories, and one of the advantages of being a US citizen is that you get access to this data for free. They’re mandated by law to make their information public, but they go above and beyond by providing real-time access via the web and FTP.

We download the raw radar data in binary format. But if you’re looking to get started playing around with weather radar, a better place to start is perhaps RIDGE, which boils the raw data into color-coded images that are easier to work with.

Cleaning it up

Weather radar is noisy. There’s lots of ground clutter, bug and bird migrations (!), and other artifacts that can be confused for precipitation. Here’s an example:

So we’ll need to clean this up. How do we do it? Well, the noise mostly consists of low-intensity data -- the light blue areas. One technique (which this gentleman writes more about), is to simply remove all low-intensity data. It clears away the noise, and leaves most of the actual precipitation data.

However, for our purposes, “most” isn’t good enough. By removing low-intensity data indiscriminately, we’re removing valuable data from the leading and trailing edges of the storms. This data is crucially important if we’re going to predict when it will start and stop raining. So we’ll have to be a little more clever.

After staring at enough radar images, it becomes pretty clear which signals are real precipitation data, and which are noise. Not only does noise tend to be low-intensity, but it has a recognizable “texture” which is distinct from actual storms. It’s easy to see with the eyes, but hard to explicitly quantify or define. So how do we write a program to remove it? This type of I-know-it-when-I-see-it, but ill-defined situation is ripe for some machine learnin’.

In this case, we opted for neural nets. Using the Fast Artificial Neural Network C-library, we trained a neural net with thousands of “blobs” of data hand-separated into two categories: noise, and not-noise.

The training process takes a while to run but the end result is a small, blazing fast program that accurately identifies somewhere between 90% and 95% of the noise, with very few false positives.

(To take care of the remainder, we spent an embarrassingly large amount of time and effort writing heuristic clean-up code by a tedious process of trial and error. But we’re keeping that part secret ;-)

The end result looks something like this:

Extracting storm velocity

Now that we have the radar data and we’ve cleaned it up, we come to the real meat of the app: velocity extraction.

But first, let me talk a little about the philosophy of our approach...

Weather is chaotic. It’s the quintessential non-linear dynamical system. Complicated fluid dynamics combined with a spinning globe, uneven terrain, and energy pumped into the system from the sun makes the weather extremely hard to predict with any accuracy. Meteorologists have spent a century coming up with increasingly sophisticated models and they still can’t reliably talk about the weather -- even generally -- much more than a week in advance.

So how can we justify our claim of being able to predict precipitation down to the very minute?

Well, here’s the thing: while the weather becomes chaotic and unpredictable at large timescales (hours to days), its behavior becomes increasingly linear at smaller and smaller timescales. To see this in action, take a look at cumulus clouds drifting across the sky. They tend to move in relatively straight lines:

We’ve found that precipitation bands are even more coherent and behave approximately linearly over the course of minutes, and in many cases up to an hour or more. So how do we go about quantifying this linear motion? Math, that’s how!

We use various computer vision algorithms to compare multiple radar image frames and create a map of velocity. Specifically, we use OpenCV, an open-source computer vision library, which comes with a number of optical flow and object tracking algorithms (for a great introduction to the topic and to see some sample code, check out this tutorial by David Stavens at the Stanford AI Labs).

The end result looks something like this:

Actually, we don’t draw little arrows. Instead, we encode the velocity in a more useful form: colors. We create a 3-channel image where the red channel represents velocity in the x-direction, the blue channel represents velocity in the y-direction, and the green channel represents change in storm intensity:

(If you’re so inclined, open the image in Photoshop and check out the color channels individually)

Prediction and Interpolation

So, now that we’ve extracted storm velocity, it’s time to use it to predict the future!

The reason we encode velocity data as an image is so we can pass it off to the GPU on the iPhone and iPad. Both the storm prediction and the smooth animations are calculated on the device itself, rather than the server, and all the magic happens directly on the GPU.

I’m going to be a little vague here, since this part constitutes our “secret sauce”, and is the component of our system that is in most active development. So instead, here’s a pretty video of the app in action:

Monitoring error

A prediction is worthless unless it is not only accurate, but reliably accurate. A large amount of our effort is focused on measuring the error rate of Dark Sky predictions. Some storms are more coherent and stable than others, so how far into the future we can project varies over time and at different geographical locations.

Whenever we process a new radar image, we go back to previous images and project them forward, creating a map of what we think the storm will look like in the present. We then compare this with the latest radar image to see how close we got.

We are constantly doing this check in real time, for every radar station. This lets us monitor our accuracy, and helps us quantify how effective future improvements are.

In conclusion...

So Dark Sky consists of a number of different moving parts that all need to fit together to create accurate predictions and pretty animations. This post glosses over many of the details, but I hope it helps get across the essential process and our approach to short-term weather forecasting.

It’s very much a numerical & statistical approach, rather than a meteorological one. Since we’ve launched the Kickstarter project, we’ve had several people (many of them meteorologists) criticize it because it eschews physical modeling of atmospheric fluid dynamics. And it’s true that our system will never be able to predict six hours in the future, let alone a day from now. But by focusing on the immediate future, and leaving everything beyond to the meteorologists, we can create a highly accurate app that is useful in a surprisingly wide range of applications.

Dark Sky Saved My Wedding Day

2011-11-01T00:00:00-07:00

I got married this past June. We decided to have the wedding outdoors, on the banks of the beautiful Connecticut River. Fearing rain, we got a big ol' tent to house the reception, but we really wanted the ceremony outside overlooking the river and were hoping we could keep the rain at bay with happy thoughts and magical incantations.

Unfortunately, I woke up the day of the wedding to find grey skies… Grey skies that gradually gave way to light rain… Rain that they forecasted would last all day.

Every time the rain would die down, I’d get hopeful, only to have that hope dashed with the precipitation’s return. As the time of the ceremony approached, we had to make a decision whether to keep everything under the tent, or go ahead as planned. Everyone told us not to chance it and just have the ceremony under the tent. Not so fast! Enter Dark Sky!

I’d been working on Dark Sky for some time, and it’s been a prototype for more than a year. More importantly, I made sure the servers were running for the Big Day. So I grabbed my iPhone, opened up the app, and took a gander at the chart.

Despite the ominous clouds, Dark Sky reported that the rain would hold off for just under an hour… enough time to have the ceremony. So we rounded up a posse of groomsmen and had the entire area set up in 20 minutes. The ceremony went off without a hitch and we were back under the tent by the time the rain returned.

Dark Sky saved my wedding day!