Spatial Data Science Bootcamp 2016!

Last week we held another bootcamp on Spatial Data Science. We had three packed days learning about the concepts, tools and workflow associated with spatial databases, analysis and visualizations. Our goal was not to teach a specific suite of tools but rather to teach participants how to develop and refine repeatable and testable workflows for spatial data using common standard programming practices.

2016 Bootcamp participants

On Day 1 we focused on setting up a collaborative virtual data environment through virtual machines, spatial databases (PostgreSQL/PostGIS) with multi-user editing and versioning (GeoGig). We also talked about open data and open standards, and moderndata formats and tools (GeoJSON, GDAL).  On Day 2 we focused on open analytical tools for spatial data. We focused on Python (i.e. PySAL, NumPy, PyCharm, iPython Notebook), and R tools.  Day 3 was dedicated to the web stack, and visualization via ESRI Online, CartoDB, and Leaflet. Web mapping is great, and as OpenGeo.org says: “Internet maps appear magical: portals into infinitely large, infinitely deep pools of data. But they aren't magical, they are built of a few standard pieces of technology, and the pieces can be re-arranged and sourced from different places.…Anyone can build an internet map."

All-in-all it was a great time spent with a collection of very interesting mapping professionals from around the country. Thanks to everyone!

Speaking of Data Viz...

 Check out this cool Google #MapsHack... here!

 Google maps+Color= Awesome Art

California Water Use Map

In response to Gov. Jerry Brown's announcement yesterday, calling all California residents to reduce water use by 25%, the folks at the New York Times put togther a nice interactive map. The map shows residential water use in California in gallons per day.

Take a look here!

Mapsense talk at BIDS for your viewing pleasure

Here is Erez Cohen's excellent talk from the BIDS feed: http://bids.berkeley.edu/resources/videos/big-data-mapping-modern-tools-geographic-analysis-and-visualization

Title: Big Data Mapping: Modern Tools for Geographic Analysis and Visualization

Speaker: Erez Cohen, Co-Founder and CEO of Mapsense

We'll discuss how smart spatial indexes can be used for performant search and filtering for generating interactive and dynamic maps in the browser over massive datasets. We'll go over vector maps, quadtree indices, geographic simplification, density sampling, and real-time ingestion. We'll use example datasets featuring real-time maps of tweets, California condors, and crimes in San Francisco. 

The BIDS Data Science Lecture Series is co-hosted by BIDS and the Data, Science, and Inference Seminar. 

About the Speaker

Erez is co-founder and CEO at Mapsense, which is builds software for the analysis and visualization of massive spatial datasets. Previously Erez was an engineer at Palantir Technologies, where he worked with credit derivatives and mortgage portfolio datasets. Erez holds a BS/MS from UC Berkeley's Industrial Engineer and Operations Research Department. He was a PhD candidate in the same department at Columbia University.

print 'Hello World (from FOSS4G NA 2015)'

FOSS4G NA 2015 is going on this week in the Bay Area, and so far, it has been a great conference.

Monday had a great line-up of tutorials (including mine on PySAL and Rasterio), and yesterday was full of inspiring talks.  Highlights of my day: PostGIS Feature Frenzy, a new geoprocessing Python package called PyGeoprocessing, just released last Thurs(!) from our colleagues down at Stanford who work on the Natural Capital Project, and a very interesting talk about AppGeo's history and future of integrating open source geospatial solutions into their business applications. 

The talk by Michael Terner from AppGeo echoed my own ideas about tool development (one that is also shared by many others including ESRI) that open source, closed source and commercial ventures are not mutually exclusive and can often be leveraged in one project to maximize the benefits that each brings. No one tool will satisfy all needs.

In fact, at the end of my talk yesterday on Spatial Data Analysis in Python, someone had a great comment related to this: "Everytime I start a project, I always wonder if this is going to be the one where I stay in Python all the way through..."  He encouraged me to be honest about that reality and also about how Python is not always the easiest or best option.

Similarly, in his talk about the history and future of PostGIS features, Paul Ramsey from CartoDB also reflected on how PostGIS is really great for geoprocessing because it leverages the benefits of database functionality (SQL, spatial querying, indexing) but that it is not so strong at spatial data analysis that requires mathematical operations like interpolation, spatial auto-correleation, etc. He ended by saying that he is interested in expanding those capabilities but the reality is that there are so many other tools that already do that.  PostGIS may never be as good at mathematical functions as those other options, and why should we expect one tool to be great at everything?  I completely agree.

Turf: Advanced geospatial analysis for browsers and node

Mapbox introduced a new javascript library for spatial analysis called Turf.js. On the "Guides to getting started" page, it claims that Turf "helps you analyze, aggregate, and transform data in order to visualize it in new ways and answer advanced questions about it."

Turf provides you with functions like calculating buffers and areas. Other common functions include aggregation, measurement, transformation, data filtering, interpolation, joining features, and classification. The detailed explanations of these functions can be found on this page.

Turf seems like a cool tool to try out if you want to provide spatial analysis functions on your webGIS applications.

Wow! new D3 panel visualization

Mike Bostock's visualizations are glorious. This gives me lots of ideas for visualizing temporal variability across space. I can't embed this here, but I recommend you check it out.

http://bost.ocks.org/mike/drought/

They say: We published a more serious graphic today on drought’s effect on crops, but this was a fun animation we made to sanity-check parsing drought data. NOAA publishes monthly values for the Palmer Drought Severity Index going all the way back to 1895! Dark purple represents extreme drought, while dark green represents extreme moisture. In effect, this is a crazy electric version of Haeyoun Park and Kevin Quealy’s graphic, Drought’s Footprint.

Web mapping of high res imagery helps conservation

One of our collaborators on the Sonoma Vegetation Mapping Project has sent work on how web mapping and high resolution imagery has helped them do their job well. These are specific comments, but might be more generally applicable to other mapping and conservation arenas.

1. Communicating with partnering agencies.
   
   • In the past year this included both large wetland restoration projects and the transfer of ownership of several thousands of acres to new stewards.
2. Articulating to potential donors the context and resources of significant properties that became available for purchase.
   
   • There are properties that have been identified as high priority conservation areas for decades and require quick action or the opportunity to protect would pass.
3. Internal communication to our own staff.
   
   • We have been involved in the protection of over 75 properties, over 47,000 acres. At this time we own 18 properties (~6500 acres) and 41 conservation easements (~7000  acres). At this scale high quality aerial imagery is essential to the size of land we steward and effective broad understanding. The way it is served as a seamless mosaic means it is available to extremely experienced and intelligent people who find the process of searching and joining orthorectified imagery by the flight path and row cumbersome or inefficient.
4. Researching properties of interest.
   
   • Besides our own internal prioritization of parcels to protect, I understand that we receive a request a week for our organization’s attention towards some property in Sonoma. Orienting ourselves to the place always includes a map with the property boundary using the most recent and/or high quality imagery for the parcel of interest and its neighbors.  This is such a regular part of our process that we created a ArcGIS Server based toolset that streamlines this research task and cartography. The imagery service we consume as the basemap for all these maps is now the 2011 imagery service.  This imagery is of high enough resolution that we can count on it for both regional and parcel scale inspection to support our decisions to apply our resources.
5. Orienting participants to site.
   
   • Our On the Land Program uses the imagery in their introduction maps to help visitors on guided hikes quickly orient to the place they are visiting and start folding their experience and sense of place into their visit.
6. Complementing grant applications.
   
   • Grants are an important part of the funding for major projects we undertake. High quality imagery facilitates our ability to orient the grant reviewer and visually support the argument we are making which is that our efforts will be effective and worthy of funds that are in short supply.
7. Knowing what the resources on a property are is an essential part of thoughtfully managing them.
   
   • In one example we used the aerial imagery (only a year old at the time) as a base map for botanists to classify the vegetation communities. These botanists are not experts in GIS, but by using paper maps with high resolution prints in the field they were easily able to delineate what they were observing on the ground on features interpreted in the photo.  We then scanned and confidently registered their hand annotations to the same imagery, allowing staff to digitized the polygons that represent the habitat observed. These vegetation observations are shared with Sonoma County and its efforts to map all the vegetation of Sonoma County.
8. Conservation easement monitoring makes extensive use of aerial imagery.
   
   • In some cases we catch violations of our easements that are difficult to view on the ground, for example unpermitted buildings by neighbors on the lands we protect, illegal agriculture or other encroachment. It is often used to orient new and old staff to a large property before walking their and planning for work projects that might be part of prescribed management.
9. The imagery helps reinforce our efforts to communicate the challenge to preserve essential connectivity in the developed and undeveloped areas of Sonoma County.
   
   • In the Sonoma Valley there is a wildlife corridor of great interest to us as conservation priority. Aerial imagery has been an important part of discussing large land holdings such as the Sonoma Developmental Center, existing conserved land by Sonoma Land Trust and others, and the uses of the valley for housing and agriculture.
10. Celebration of the landscape cannot be forgotten.
    
    • We often pair this high quality aerial imagery with artful nature photography. The message of the parts and their relation to the whole are succinctly and poetically made. This is essential feedback to members and donors who need to see the numbers of acres protected with their support and have the heartfelt sense of success.

We look forward to the continued use of this data and the effective way it is shared.
 
We hope that future imagery and other raster or elevation data can be served as well as this, it would benefit many engaged in science and conservation.

Thanks to Joseph Kinyon, GIS Manager, Sonoma Land Trust

Mapping Ebola Over Time

HealthMap is an automated electronic information system that monitors data from electronic media sources (e.g. social media, government websites, physician social networks) in order to visualize and foster an understanding of infectious disease outbreaks around the world.  The system is credited with recognizing the current Ebola epidemic in West Africa nine days before the World Health Organization was able to do so (see: http://www.politico.com/story/2014/08/healthmap-ebola-outbreak-109881.html?hp=l8).  Here you can access their visualization of the spread of Ebola across West Africa, and later into Europe and the United States: http://healthmap.org/ebola/#timeline.

Flight visualization

This is quite lovely. You've seen it before, but this is a nice version.

http://www.theguardian.com/world/video/2014/jul/03/transatlantic-flight-paths-of-2-524-planes-on-single-day-video-animation