Big data, Landsat and earth science

Big Data Helps Scientists Dig Deeper: Supercomputing + 40 Years of Landsat Images = A New Era for Earth Science

Very nice overview of how the Landsat archive has jump started new avenues in earth science, ecoinformatics, and public communication and participation.

In 2008, the U.S. Geological Survey took 3.6 million images acquired by Landsat satellites and made them free and openly available on the Internet. Dating back to 1972, the images are detailed enough to show the impact of human decisions on the land, and they provide the longest continuous view of Earth’s landscape from space.

Since January 1, 2000, more than 4.3 million scenes have been captured by Landsat satellites and made available to the public. Landsat 8 was launched in February 2013, significantly boosting the rate to as many as 1,400 images each day. (Graph by Joshua Stevens, using data collected from the U.S. Geological Survey acquisitions archives.)

http://earthobservatory.nasa.gov/Features/LandsatBigData/

California Water Use Map

In response to Gov. Jerry Brown's announcement yesterday, calling all California residents to reduce water use by 25%, the folks at the New York Times put togther a nice interactive map. The map shows residential water use in California in gallons per day.

Take a look here!

Mapping the Berkeley Boom: Social Media and Mapping Help Unravel a Mystery

Last night we heard the Berkeley Boom again.  We’ve been hearing this thunderous boom quite frequently in the last month here in Berkeley, but this one sounded bigger than most.  Car alarms went off on the street.  The dog jumped.  “What IS that?” I wondered aloud.  With a quick search on the internet I found that that the Berkeley Boom is a phenomena whose Twitter reports are being actively mapped.  While Berkeley police and residents still have no idea what the mystery boom is, through the combined powers of social media and mapping we are gathering an understanding of where it is happening.  As Berkeley residents continue reporting the boom (#BerkeleyBoom), perhaps we’ll get to the bottom of this, the newest of Berkeley’s many mysteries. 

For more on the Berkeley Boom see the Berkeleyside article: http://www.berkeleyside.com/2015/03/31/the-unsolved-mystery-of-the-berkeley-boom/

Map from Berkeleyside Article:

The drought indeed hits home: Berkeley water less than its usual quality

The hills and lawns might look green still, but the drought has hit the east bay hard. The sparkling, clean, tasty water we usually have delivered through our taps via  the Mokelumne River Basin in the Sierra Nevada. Get out Britas!

From our favorite and fastest source for local news Berkeleyside: 

The drinking water for 1 million customers of East Bay Municipal Utilities District had an “off” odor and taste over the weekend and, while EBMUD is fixing the issue, customers might have to get used to it. The culprit? The drought.

EBMUD usually draws the drinking water for the majority of its customers from the bottom of Pardee Reservoir, about 100 miles east of Berkeley, according to Abby Figueroa, a spokeswoman for EBMUD. But on Thursday, the water district started taking water from the top portion of the reservoir. The water there is warmer and contains some algae, so even though it was treated before gushing into pipes in Berkeley, Oakland and elsewhere, there was a peculiar smell.

Route from the Mokelumne River Basin in the Sierra Nevada, to the East BayAccordingly there was a run on Brita filters at all local hardware/houseware stores.

New water restrictions for California announced.

Satellites can be vulnerable to solar storms

I don't use ocean color data, but found this report of interest nonetheless. From the HICO website. HICO is the Hyperspectral Imager for the Coastal Ocean.

HICO Operations Ended. March 20, 2015

In September 2014 during an X-class solar storm, HICO’s computer took a severe radiation hit, from which it never recovered.  Over the past several months, engineers at NRL and NASA have attempted to restart the computer and have conducted numerous tests to find alternative pathways to communicate with it.  None of these attempts have been successful.  So it is with great sadness that we bid a fond farewell to HICO.

Yet we rejoice that HICO performed splendidly for five years, despite being built in only 18 months from non space-hardened, commercial-off-the-shelf parts for a bargain price.  Having met all its Navy goals in the first year, HICO was granted a two-year operations extension from the Office of Naval Research and then NASA stepped in to sponsor this ISS-based sensor, extending HICO’s operations another two years.  All told, HICO operated for 5 years, during which it collected approximately 10,000 hyperspectral scenes of the earth.

Most of the HICO scenes taken over sites worldwide are available now, and will remain accessible to researchers through two websites:  http://oceancolor.gsfc.nasa.gov/ and http://hico.coas.oregonstate.edu.  HICO will live on through research conducted by scientists using HICO data, especially studies exploring the complexities of the world’s coastal oceans.

Data science for the 21st century: building a new team of researchers

Berkeley is one out of eight new awards from the National Science Foundation's recently launched NSF Research Traineeship (NRT) program. These programs develop innovative approaches to graduate training used across these projects include industry internships, international experiences, citizen science engagement, interdisciplinary team projects, and training in communication with the media, policy makers, and general public.

Our program at UC Berkeley is called Data Science for the 21st centur: DS421.  Three Grand Challenges motivate our program:

1. Data: data acquisition, assimilation, and analysis, and the resulting challenges and opportunities for the research community and society at large. The data revolution is a potentially disruptive advance that challenges the norms and traditions of scientific research. Data science is an opportunity, entailing a revolution in training and a reorientation of research priorities. Open science— open access to datasets, literature, scripted workflows and the like—is a fundamental transformation that integrates scientific publication with the underlying data, analysis, and reasoning, using metadata and machine-readable research products to facilitate a semantic web of knowledge. These practices will make our research reproducible and transparent, documenting the evidentiary basis for scientific conclusions and their implications for policy.
2. System dynamics: coupled human-natural systems and their responses to rapid environmental change. Social-ecological systems display a complex array of ecological and social processes interconnected across broad spatial, temporal, and socio-political scales. Our current approach to understanding ecological and economic systems is dominated by partial equilibrium models that are poorly suited to the dynamics of rapidly changing systems. Important research avenues include: characterizing the dynamics and feedbacks among and within systems to better plan for cross-scale and nonlinear uncertainties; identifying the proximity of tipping points or other critical transitions; understanding how the spatial structure of interactions affects system dynamics; and detecting and attributing responses to environmental and climatic drivers. Real-time data analytics combined with long-term monitoring and forecasting are critical tools to address to these challenges.
3. Action: evidence-based proposals in public policy, natural resource management, and environmental design to mitigate the impacts of rapid environmental change, and enhance societal resilience and sustainability. Effective decision-making depends on networks of diverse stakeholders, with rapid feedback between individuals and groups to evaluate the impact, efficiency, equity, and efficacy of policy and management actions. This third component is at the core of a practical data science ethic critical for translating science to societal benefit, and makes use of our partnerships with academic, private, governmental, and non-governmental organizations.

Cutting across these challenges, all students, and especially those engaged in interdisciplinary research,
need excellent communication skills and the ability to adjust content and style to reach their audiences. Welcome to the new cohort!

Mapsense talk at BIDS for your viewing pleasure

Here is Erez Cohen's excellent talk from the BIDS feed: http://bids.berkeley.edu/resources/videos/big-data-mapping-modern-tools-geographic-analysis-and-visualization

Title: Big Data Mapping: Modern Tools for Geographic Analysis and Visualization

Speaker: Erez Cohen, Co-Founder and CEO of Mapsense

We'll discuss how smart spatial indexes can be used for performant search and filtering for generating interactive and dynamic maps in the browser over massive datasets. We'll go over vector maps, quadtree indices, geographic simplification, density sampling, and real-time ingestion. We'll use example datasets featuring real-time maps of tweets, California condors, and crimes in San Francisco. 

The BIDS Data Science Lecture Series is co-hosted by BIDS and the Data, Science, and Inference Seminar. 

About the Speaker

Erez is co-founder and CEO at Mapsense, which is builds software for the analysis and visualization of massive spatial datasets. Previously Erez was an engineer at Palantir Technologies, where he worked with credit derivatives and mortgage portfolio datasets. Erez holds a BS/MS from UC Berkeley's Industrial Engineer and Operations Research Department. He was a PhD candidate in the same department at Columbia University.

print 'Hello World (from FOSS4G NA 2015)'

FOSS4G NA 2015 is going on this week in the Bay Area, and so far, it has been a great conference.

Monday had a great line-up of tutorials (including mine on PySAL and Rasterio), and yesterday was full of inspiring talks.  Highlights of my day: PostGIS Feature Frenzy, a new geoprocessing Python package called PyGeoprocessing, just released last Thurs(!) from our colleagues down at Stanford who work on the Natural Capital Project, and a very interesting talk about AppGeo's history and future of integrating open source geospatial solutions into their business applications. 

The talk by Michael Terner from AppGeo echoed my own ideas about tool development (one that is also shared by many others including ESRI) that open source, closed source and commercial ventures are not mutually exclusive and can often be leveraged in one project to maximize the benefits that each brings. No one tool will satisfy all needs.

In fact, at the end of my talk yesterday on Spatial Data Analysis in Python, someone had a great comment related to this: "Everytime I start a project, I always wonder if this is going to be the one where I stay in Python all the way through..."  He encouraged me to be honest about that reality and also about how Python is not always the easiest or best option.

Similarly, in his talk about the history and future of PostGIS features, Paul Ramsey from CartoDB also reflected on how PostGIS is really great for geoprocessing because it leverages the benefits of database functionality (SQL, spatial querying, indexing) but that it is not so strong at spatial data analysis that requires mathematical operations like interpolation, spatial auto-correleation, etc. He ended by saying that he is interested in expanding those capabilities but the reality is that there are so many other tools that already do that.  PostGIS may never be as good at mathematical functions as those other options, and why should we expect one tool to be great at everything?  I completely agree.

10-year anniversary for the GIF

I'm musing, contemplating and writing on the decade 2005-2015, as this is the GIF's 10-year anniversary. What a decade it was. Here I'll post and add to some of the key events that helped transform mapping (and the GIF) in the last 10 years.

Key background events

• 1996. Mapquest launched.
• 1997. Skynet becomes self-aware.
• May 2000. Selective Availabilility on GPS turned off, leading the way for GPS in smartphones.
• The Scan Line Corrector (SLC) on the Landsat 7 ETM+ instrument failed May 31, 2003.
• 2004. Open Street Map founded.
• March 2004. Yahoo! maps launched, first slippy maps (click and drag to pan and zoom the map).
• 2004. NASA releases WorldWind.
• October 2004. Google acquires Where 2 allowing AJAX map tiling to a desktop client.
• October 2004. Google acquires Keyhole.

What made 2005 such a crazy year

• Google Maps launches in February, and goes mobile in April.
• The first mashup: Paul Rademacher's Housingmaps.org. His original post on Craigslist asking for feedback: https://forums.craigslist.org/?ID=26638141
• Google Maps API launches in June.
• NASA's Blue Marble Next Generation released.
• Google Earth launches in June.
• Hurricane Katrina hits in August. Simple webmaps for the disaster proliferate, and ESRI and GE get on the scene.
• Kellylab's first blog post in September.
• GIF launches and hosts our first GIS Day in November with Michael Jones, formerly of Keyhole.
• The back-up solar array drive on Landsat 5 began failing and was not able to provide the power needed to charge the batteries. November 26.

Where we are in 2015

We've gone through a number of transitions in the world of mapping:

• Data have transitioned from being siloed, and found in clearinghouses to being open and provided through APIs.
• We’ve moved from desktop computing to cloud computing.
• Webmaps have transitioned from using proprietary stacks to networks with multiple open and proprietary options.
• We’ve moved from imagery gathered monthly or seasonally to daily; footprints are smaller, and our focus has shifted from local focus to global coverage.
• Our planimetric 2D view is changing with lidar and radar sensors.
• Visualization has moved from static cartography or simple animations to dynamic interactive visualization.
• Finally, mapped content is no longer anonymous or regulated, but highly personal and narrative.

Key GIF milestones:

• 2005 GIIF (Geospatial Imaging and Informatics Facility) launches
• 2006 OakMapper changes from ArcIMS to Google Earth API
• 2008 GIIF becomes GIF
• 2008 OakMapper 2.0 launches
• 2008 SNAMP website launches
• 2011 Cal-Adapt goes live
• 2013 EcoEngine/HOLOS goes live
• 2014 LandCarbon launches
• 2014 GIF and Cal-Adapt go to the White House
• 2014 vtm.berkeley.edu goes live, built from the HOLOS API
• 2015 Spatial Data Science bootcamp in May

Onwards and upwards!

A brief history of the digital map

Nice consise history of mapping from his lecture "The Ubiquitous Digital Map" by Gary Gale, Director of Global Ccommunity Programs, HERE.

http://www.vicchi.org/2013/02/19/the-ubiquitous-digital-map-abridged/