Using OpenStreetMap data Nick Whitelegg Southampton Solent University, RM424

Using OpenStreetMap data
Nick Whitelegg
Southampton Solent University, RM424
[email protected]
What is OpenStreetMap?
OpenStreetMap[1] is a set of mapping data
covering the whole world
Liberally licenced (CC-by-SA)
Anyone can contribute
Rendered maps available, but the data is the
important thing!
See http://www.openstreetmap.org
Why OpenStreetMap?
Existing mapping data was not free (as in beer
or speech)
Licencing fees necessary to use maps online
Generally only the rendered maps were
available, not the underlying data
Data available to business, but at very high cost
This restricted how you can use the data
It's the data!
OSM has a default “slippy” map but access to
the raw data is what makes OSM
Developers can use this data in their own
applications
OSM has a web service API to allow access to
the raw data
Raw data also downloadable as XML files
Contributing
Survey roads, footpaths and POIs with a GPS
device
Use freely-available data sources (Ordnance
Survey out-of-copyright/OpenData [2], Bing
aerial imagery)
Use an OSM editor (JOSM [3], Potlatch [4]) to
draw highways on top of GPS traces or imagery
Overview of in-the-field
mapping procedure
Survey using a GPS device (e.g. smartphone,
Garmin GPS unit)
Load GPS track and waypoints into an editor
(web or standalone)
Download existing OSM data into the editor
Draw new roads, paths, points of interest in
OSM editor using your GPS track/waypoints
Upload changes to OSM
Editing the data
JOSM (josm.openstreetmap.de)
Standalone editor, written in Java
Powerful and flexible
Communicates with API over HTTP
Potlatch
(http://wiki.openstreetmap.org/wiki/Potlatch_2)
Online Flash Editor
Easy to use
iD (ideditor.com)
HTML5/JavaScript editor, in early development
Other sources of OSM data
In addition to direct surveys, the following
sources can be used to add data to OSM:
Aerial imagery (Bing; special agreement)
Out-of-copyright historical mapping (e.g. OS maps
from 50+ years ago, i.e. 1962 or before)
Ordnance Survey OpenData or other liberallylicenced official data sources
The editors can load in these data sources as a
backdrop to the OSM data
The OpenStreetMap API
API to allow developers to make use of OSM
data
REST-compliant web service providing data in
OSM's own XML format
Allows creation, retrieval and modification of
map data
Available at
http://www.openstreetmap.org/api/0.6/(operation)
Examples of operations on following slides
Nodes and Ways
OSM data consists of nodes and ways
A node represents a point feature
e.g. a point of interest, or a constituent point of a
line feature such as a road
A way represents a line or polygon feature
e.g. a road, a stream, a wood, a lake
Ways consist of two or more nodes
OpenStreetMap API Examples
Retreive map data in a given bounding box
(OSM XML format)
/map?bbox=-1.5,50.9,-1.4,51.0
Retrieve a given way (here, way with ID 223)
/way/223 (GET request, retrieves OSM XML)
Modify a way (here, way with ID 223)
/way/223 (PUT request, send OSM XML to URL)
OpenStreetMap XML format
<osm version="0.6">
<node lat="50.9" lon="-1.4" id="12345">
<tag k="place" v="city"/>
<tag k="name" v="Southampton"/>
</node>
<way id="223">
<nd id="1234" />
<nd id="1235" />
<tag k="highway" v="motorway" />
<tag k="ref" v="M27" />
</way>
...
OpenStreetMap XML Explanation
The <node> tag represents a node
It has an ID, latitude and longitude attributes and a
series of free-form tags describing the point
The <way> tag represents a way
A way is made up of constituent nodes
The XML consists of a series of node references,
referenced by node ID (the <nd> tag) together with
free-form tags describing the way
OSM tagging
Properties of map objects in OSM are described
by a series of tags
Each tag has a key and value
The key is the general class of property, the
value is the actual value for that property
Examples of tags for linear
ways
highway: the class of road (motorway, primary,
secondary etc)
maxspeed: maximum speed in km/h
ref: the road reference number
oneway: is it a one way road? (boolean)
Examples of tags for nodes
natural
If it's a natural feature, indicates the type (hill
summit, etc)
man_made
If it's a man made feature, indicates the type (water
tower, mast etc)
amenity
If it's an amenity (e.g. a pub, restaurant, recycling
centre etc) indicates the type
OSM way tagging example
A main road could be described by the following
tags:
highway=primary
Indicates it's a “primary” class road (A road in UK)
name=High Street
ref=A286
The road number
maxspeed=64
Maximum speed in km/h
OSM node tagging examples
Hill / Mountain:
natural=peak; name=Snowdon
Pub:
amenity=pub; name=The Crown
Place:
place=city; name=Southampton
Communications mast:
man_made=mast
Examples of tags for areas
Areas (polygons) are treated as ways (which
just so happen to be closed)
e.g. landuse for managed land
landuse=forest or landuse=industrial or
landuse=residential
or natural for natural land cover
natural=wood or natural=water or natural=heath
OSM “planet” files
Requesting large amounts of data from API can
place undue load on the OSM server
API is really designed for adding and editing
data, not retrieving
So OSM also makes its data available as
downloadable “planet” files in OSM XML format,
containing OSM data for the entire planet
Whole planet too large for most to use directly,
so country or regional extracts available
e.g. Geofabrik (http://download.geofabrik.de)
Extracting OSM data with
osmosis
Even with a country extract, chances are you
are only interested in some of the data
For example, a walkers' map might only need
data of interest to walkers
The osmosis tool
(wiki.openstreetmap.org/wiki/Osmosis) allows
you to extract only the data you need
osmosis
Osmosis allows you to (amongst other things):
Extract data by bounding box or polygon
Extract all nodes or ways containing a given key, or
key/value pair
Write OSM data to a MySQL database
Read OSM data from a database and write to an
OSM data file
osmosis
Examples of osmosis commands:
osmosis --rx file=uk.osm -–bounding-box top=52
bottom=50 left=-2 right=1 –-wx
file=southern_england.osm
This reads in uk.osm, extracts a bounding box defined by 50N, 52N, 2W and
1E, saving the results in southern_england.osm
osmosis –-rx file=uk.osm -–tf reject-ways –-tf
accept-nodes place=* --wx file=ukplaces.osm
This reads in uk.osm, rejects all ways and accepts only nodes with a place tag,
writing the result to ukplaces.osm
General principles of
osmosis
Has a set of commands specified as commandline options, e.g. --rx (read XML), --tf (tag filter)
Commands typically filter the input stream
So the following command:
osmosis –-rx file=uk.osm –-tf reject-ways –-tf acceptnodes place=* --tf accept-nodes amenity=* --wx
file=output.osm
will only give nodes with both a place tag and an
amenity tag
If we want nodes with either a place tag or an
amenity tag, we must use named pipes and stream
merging, see the osmosis page for details
Overpass API
An alternative to downloading the OSM data as
a flat file
The Overpass API is a third-party live web API
which allows OSM data (a local copy of the
OSM database) to be queried and returned as
XML
Uses an XML-based query language
See http://wiki.openstreetmap.org/wiki/Overpass_API
Setting up an OSM database
You may wish to setup your own copy of the
OSM database
e.g. for rendering your own maps or developing
a web service of some kind
Typically, a PostgreSQL / PostGIS database is
used for this
OSM data is imported into two tables
planet_osm_point and planet_osm_line
Setting up PostgreSQL /
PostGIS
PostgreSQL and its GIS-specific extensions,
PostGIS, normally available as packages for
Linux distributions
Installation fairly straightforward on current
Ubuntu or Debian distributions
Detailed installation instructions available at:
wiki.openstreetmap.org/index.php/Mapnik/PostGIS
Importing OSM data
Having set up the PostGIS database, OSM data
must be imported into it
A custom tool, osm2pgsql, is used for this
This can be downloaded from OSM's github
repository
Osm2pgsql allows us to control which features
are imported (by tag), minimising the size of the
database
See the osm2pgsql wiki page:
http://wiki.openstreetmap.org.index.php/Osm2pgsql
Map rendering
OK, you've set up a database containing OSM
data, but how are you going to generate your
own maps?
Two broad approaches:
Server side rendering (e.g. Mapnik)
Client side rendering (e.g kothic-js)
Mapnik: server-side
rendering
Mapnik (mapnik.org) is a C++ library (with
Python wrappers) for rendering high-quality
maps from geographical data
Mapnik is able to read data from (amongst other
data sources) PostGIS databases and uses an
XML style file to control the look-and-feel of the
final map
Developers create programs in C++ or Python
to render data using Mapnik from the PostGIS
database
Helper tools for Mapnik
Mapnik is powerful but can be tricky to setup
and use
So tools have been developed to ease creation
of Mapnik maps
One example is TileMill (mapbox.com/tilemill)
An easily-installable package which allows you to
design and render maps locally
Uses a locally-running webserver
Creating web maps
OK, we've got the maps rendered now, but
what about creating a Google-style “slippy
map”?
JavaScript mapping libraries are used here
Leaflet: www.leafletjs.com
OpenLayers: www.openlayers.org
You need to ensure you generate tiles in the
“Google XYZ” format
Brief intro to Google XYZ
The tiling system used by most web mapping
Map tiles (projected in “Spherical Mercator”)
described by an x,y,z coordinate scheme
The x and y coordinates represent the tile's
position in the world (x=0 is west; y=0 is north)
The z coordinate represents the zoom level
Google XYZ zoom levels
Zoom level 0 contains the whole world on one
256x256 pixel tile
Zoom level 1 zooms in by a factor of two, so the
whole world is contained by four 256x256 pixel
tiles
… and so on for zoom levels 2, 3, etc
See www.freemap.org.uk/course/ewt/webmapping.xhtml
Google XYZ zoom levels
Zoom level 0 : whole world as 256x256 tile
Google XYZ zoom levels
Zoom level 1: 4 tiles (2 rows, 2 cols) for world
Client-side rendering
Server-side rendering can be computationally
expensive, and requires large storage space
(cache) once rendered
Therefore not always the optimal solution
Client-side, in-browser HTML5 canvas
rendering possible with kothic-js (kothic.org/js)
kothic-js
JavaScript library developed as a port of the
Python renderer kothic
Renders GeoJSON data client-side using
HTML5 Canvas
GeoJSON is a standard JSON-based format for
representing geographical data, see geojson.org
Server-side scripts must be written to generate
the GeoJSON from a database using the
“Google” XYZ tile system
A Python implementation is available, produced by
the kothic developers
Other uses of OSM data
Location-based smartphone apps
Apps can include vector maps
Apps can find nearby points of interest downloaded
from OSM
Routing software
OSM data can be converted to binary formats
optimised for routing
Example site: Freemap
Freemap (www.free-map.org.uk) is an example
of a third-party site making use of OSM data
OSM site geared for UK walkers, showing
rights-of-way
OSM data for parts of the UK loaded into
PostGIS database with osm2pgsql
Server-side scripts generate GeoJSON from the
database
kothic-js is used client-side to render the data
OS LandForm-PANORAMA data[5] used for
contours
Example mobile app:
OpenTrail
OpenTrail (www.free-map.org.uk/downloads/android/opentrail.apk)
is Freemap's Android app
It shows vector maps generated from OSM data
using the Mapsforge library
(code.google.com/p/mapsforge)
It also allows users to find nearby POIs
These are delivered from Freemap's web service,
which is also used to generate the GeoJSON for
Freemap itself
Novel uses of OSM data:
example
Augmented reality is an interesting concept in
mobile development
The general idea is that the real world (e.g. camera
feed) is overlaid with virtual objects (e.g. POIs)
Hikar (see https://github.com/nickw1/Freemap/tree/master/java/hikar/)
is a “researchware” app which overlays OSM
data (provided by Freemap's API) on the
phone's camera feed
… and in doing so potentially offers a novel
approach to navigation for walkers
References
[1] OpenStreetMap, 2011 "OpenStreetMap" (online) Available: www.openstreetmap.org, last accessed 6/6/11
[2] Ordnance Survey, 2011 "OS OpenData" (Online) Available http://www.ordnancesurvey.co.uk/oswebsite/products/os-opendata.html,
last accessed 6/6/11
[3] OpenStreetMap, 2011 "JOSM" (online) Available: josm.openstreetmap.de, last accessed 6/6/11
[4] OpenStreetMap, 2011 "Potlatch 2" (online) Available: wiki.openstreetmap.org/wiki/Potlatch_2, last accessed 6/6/11
[5] Ordnance Survey, 2011 "Land-Form PANORAMA - small-scale height data of Great Britain",(online) Available:
http://www.ordnancesurvey.co.uk/oswebsite/products/land-form-panorama/index.html, last accessed 6/6/11
Thanks for listening
Any questions?