Spatial Data Science Bootcamp 2016!

Last week we held another bootcamp on Spatial Data Science. We had three packed days learning about the concepts, tools and workflow associated with spatial databases, analysis and visualizations. Our goal was not to teach a specific suite of tools but rather to teach participants how to develop and refine repeatable and testable workflows for spatial data using common standard programming practices.

2016 Bootcamp participants

On Day 1 we focused on setting up a collaborative virtual data environment through virtual machines, spatial databases (PostgreSQL/PostGIS) with multi-user editing and versioning (GeoGig). We also talked about open data and open standards, and moderndata formats and tools (GeoJSON, GDAL).  On Day 2 we focused on open analytical tools for spatial data. We focused on Python (i.e. PySAL, NumPy, PyCharm, iPython Notebook), and R tools.  Day 3 was dedicated to the web stack, and visualization via ESRI Online, CartoDB, and Leaflet. Web mapping is great, and as OpenGeo.org says: “Internet maps appear magical: portals into infinitely large, infinitely deep pools of data. But they aren't magical, they are built of a few standard pieces of technology, and the pieces can be re-arranged and sourced from different places.…Anyone can build an internet map."

All-in-all it was a great time spent with a collection of very interesting mapping professionals from around the country. Thanks to everyone!

ESRI @ GIF Open GeoDev Hacker Lab

We had a great day today exploring ESRI open tools in the GIF. ESRI is interested in incorporating more open tools into the GIS workflow. According to www.esri.com/software/open, this means working with:

1. Open Standards: OGC, etc. 
2. Open Data formats: supporting open data standards, geojson, etc. 
3. Open Systems: open APIs, etc. 

We had a full class of 30 participants, and two great ESRI instructors (leaders? evangelists?) John Garvois and Allan Laframboise, and we worked through a range of great online mapping (data, design, analysis, and 3D) examples in the morning, and focused on using ESRI Leaflet API in the afternoon. Here are some of the key resources out there.

• Main ESRI Open Information: http://www.esri.com/software/open
• Slide deck from today: http://slides.com/alaframboise/geodev-hackerlabs#/
• Afternoon example using Leaflet: http://esri.github.io/esri-leaflet/
• ESRI's developer toolkits: https://developers.arcgis.com/en/, including
• ESRI's javascript API: https://developers.arcgis.com/javascript/beta/

Great Stuff! Thanks Allan and John

Spatial Data Science Bootcamp March 2016

Register now for the March 2016 Spatial Data Science Bootcamp at UC Berkeley!

We live in a world where the importance and availability of spatial data are ever increasing. Today’s marketplace needs trained spatial data analysts who can:

• compile disparate data from multiple sources;
• use easily available and open technology for robust data analysis, sharing, and publication;
• apply core spatial analysis methods;
• and utilize visualization tools to communicate with project managers, the public, and other stakeholders.

To help meet this demand, International and Executive Programs (IEP) and the Geospatial Innovation Facility (GIF) are hosting a 3-day intensive Bootcamp on Spatial Data Science on March 23-25, 2016 at UC Berkeley.

With this Spatial Data Science Bootcamp for professionals, you will learn how to integrate modern Spatial Data Science techniques into your workflow through hands-on exercises that leverage today's latest open source and cloud/web-based technologies. We look forward to seeing you here!

To apply and for more information, please visit the Spatial Data Science Bootcamp website.

Limited space available. Application due on February 19th, 2016.

Spatial Data Science @ Berkeley May 2015

Bootcamp participants outside historic Mulford HallOur bootcamp on Spatial Data Science has concluded. We had three packed days learning about the concepts, tools and workflow associated with spatial databases, analysis and visualizations. 

Our goal was not to teach a specific suite of tools but rather to teach participants how to develop and refine repeatable and testable workflows for spatial data using common standard programming practices.

On Day 1 we focused on setting up a collaborative virtual data environment through virtual machines, spatial databases (PostgreSQL/PostGIS) with multi-user editing and versioning (GeoGig). We also talked about open data and open standards, and modern data formats and tools (GeoJSON, GDAL).

Analyzing spatial data is the best part! On Day 2 we focused on open analytical tools for spatial data. We focused on one particular class of spatial data analysis: pattern analysis, and used Python (i.e. PySAL, NumPy, PyCharm, iPython Notebook), and R Studio (i.e. raster, sp, maptools, rgdal, shiny) to look at spatial autocorrelation and spatial regression. 

Wait, visualizing spatial data is the best part! Day 3 was dedicated to the web stack, and visualization. We started with web mapping (web stack, HTML/CSS, JavaScript, Leaflet), and then focused on web-based visualizations (D3).  Web mapping is great, and as OpenGeo.org says: “Internet maps appear magical: portals into infinitely large, infinitely deep pools of data. But they aren't magical, they are built of a few standard pieces of technology, and the pieces can be re-arranged and sourced from different places.…Anyone can build an internet map."

All-in-all it was a great time spent with a collection of very interesting mapping professionals from around the country (and Haiti!). Thanks to everyone!

Mapsense talk at BIDS for your viewing pleasure

Here is Erez Cohen's excellent talk from the BIDS feed: http://bids.berkeley.edu/resources/videos/big-data-mapping-modern-tools-geographic-analysis-and-visualization

Title: Big Data Mapping: Modern Tools for Geographic Analysis and Visualization

Speaker: Erez Cohen, Co-Founder and CEO of Mapsense

We'll discuss how smart spatial indexes can be used for performant search and filtering for generating interactive and dynamic maps in the browser over massive datasets. We'll go over vector maps, quadtree indices, geographic simplification, density sampling, and real-time ingestion. We'll use example datasets featuring real-time maps of tweets, California condors, and crimes in San Francisco. 

The BIDS Data Science Lecture Series is co-hosted by BIDS and the Data, Science, and Inference Seminar. 

About the Speaker

Erez is co-founder and CEO at Mapsense, which is builds software for the analysis and visualization of massive spatial datasets. Previously Erez was an engineer at Palantir Technologies, where he worked with credit derivatives and mortgage portfolio datasets. Erez holds a BS/MS from UC Berkeley's Industrial Engineer and Operations Research Department. He was a PhD candidate in the same department at Columbia University.

print 'Hello World (from FOSS4G NA 2015)'

FOSS4G NA 2015 is going on this week in the Bay Area, and so far, it has been a great conference.

Monday had a great line-up of tutorials (including mine on PySAL and Rasterio), and yesterday was full of inspiring talks.  Highlights of my day: PostGIS Feature Frenzy, a new geoprocessing Python package called PyGeoprocessing, just released last Thurs(!) from our colleagues down at Stanford who work on the Natural Capital Project, and a very interesting talk about AppGeo's history and future of integrating open source geospatial solutions into their business applications. 

The talk by Michael Terner from AppGeo echoed my own ideas about tool development (one that is also shared by many others including ESRI) that open source, closed source and commercial ventures are not mutually exclusive and can often be leveraged in one project to maximize the benefits that each brings. No one tool will satisfy all needs.

In fact, at the end of my talk yesterday on Spatial Data Analysis in Python, someone had a great comment related to this: "Everytime I start a project, I always wonder if this is going to be the one where I stay in Python all the way through..."  He encouraged me to be honest about that reality and also about how Python is not always the easiest or best option.

Similarly, in his talk about the history and future of PostGIS features, Paul Ramsey from CartoDB also reflected on how PostGIS is really great for geoprocessing because it leverages the benefits of database functionality (SQL, spatial querying, indexing) but that it is not so strong at spatial data analysis that requires mathematical operations like interpolation, spatial auto-correleation, etc. He ended by saying that he is interested in expanding those capabilities but the reality is that there are so many other tools that already do that.  PostGIS may never be as good at mathematical functions as those other options, and why should we expect one tool to be great at everything?  I completely agree.

Questions about the Spatial Data Science Bootcamp? Read on!

In May, the GIF will be hosting a 3-day bootcamp on Spatial Data Science.

What is the significance of Spatial Data Science?

We live in a world where the importance and availability of spatial data is ever increasing, and the value of Spatial Data Science: big data tools, geospatial analytics, and visualization is on the rise. There are many new and distributed tools available to the geospatial professional, and the ability to efficiently evaluate and integrate the wide array of options is a critical skill for the 21^st century marketplace.  Spatial Data Science offers a modern workflow that includes the integration of data from multiple sources and scales; with open-source and web-based technology for robust data analysis and publication; with core spatial concepts and application of spatial analysis methods; and allows for the collaborations of people – companies, scientists, policy-makers, and the public.

Why come to the GIF to learn about it?

The Geospatial Innovation Facility (GIF) at UC Berkeley is the premier research and educational facility in the Bay Area that focuses on a broad vision of Spatial Data Science. The GIF has a decade-long history of successful GIS and remote sensing research projects. The GIF has also trained many students, researchers, and community members in geospatial techniques and applications through our popular workshop series and private consultation. With more recent advances in web-based mapping capabilities, the GIF has been at the forefront of complex web-based spatial data informatics (web-based data sharing and visualization), such as the Cal-Adapt  tool, which provides a wealth of data and information about California’s changing climate. Participants will get the benefit of our decade-long focus on Spatial Data Science: collaborative project development, rigorous spatial analysis methods, successful interaction with clients, and delivery of results to project managers, the public, and other stakeholders.

What are the key elements of the Bootcamp?

This Bootcamp is designed to familiarize participants with some of the major advances in geospatial technology today: big data wrangling, open-source tools, and web-based mapping and visualization. You will learn how and when to implement a wide range of modern tools that are currently in use and under development by leading Bay Area mapping and geospatial companies, as well as explore a set of repeatable and testable workflows for spatial data using common standard programming practices. Finally, you will learn other technical options that you can call upon in your day-to-day workflows. This 3-day intensive training will jump start your geospatial analysis and give you the basic tools you need to start using open source and web-based tools for your own spatial data projects.  

Interested in integrating open source and web-based solutions into your GIS toolkit? Come join us at our May 2015 Bootcamp: Spatial Data Science for Professionals. Applications due: 3/16/2015. Sign up here!

Geolunch Seminar Talk: Spatial Data Analysis with Python

Thanks to all who attended today!  Here is a link (modified after FOSS4G NA 2015) to the slides, and below are more links for some of the free resources that were highlighted.

Code Academy (programming tutorials): http://www.codecademy.com/
Coursera (full courses): https://www.coursera.org/courses?query=python
Python wiki pages: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
https://docs.python.org/2/tutorial/
Python at Berkeley (DLab): http://python.berkeley.edu/learning_resources.html
Python Books and Training: http://pythonbooks.revolunet.com/
http://www.learnpython.org/
ArcPy tutorials from ESRI: http://training.esri.com/gateway/index.cfm?fa=catalog.
webCourseDetail&courseid=2520
http://training.esri.com/gateway/index.cfm?fa=catalog.webCourseDetail&courseid=2523

List of spatial Python packages: https://github.com/SpatialPython/spatial_python/blob/master/packages.md

If you are a graduate student, check out the graduate student-led workshops for Spring 2015: http://goo.gl/forms/rUceY1I67n