Ckanext-Spatial

Background and strategic fit

One important feature of HEF Data Catalog should be the ability to do a spatial search. This extension provides not only a spatial search but also to expand the metadata fields to add the spatial boundary to a resource.

• Extending metadata adding the resource boundary:
  • Manually: selecting one or multiple locations/boundaries using the map widget. The program will automatically convert it into valid GeoJSION geometries. Most importantly it can be drawn in a simple and complex geometrical shape.

  • Manually: typing a valid GeoJSION geometry

• Representing the spatial boundary for resources with the spatial field.
• Searching and filtering resources based on their locations and through the map widget

Installations and Requirements

To install ckanext-spatial on a production site:

Ubuntu 16.4 (PostgresSQL 9.3 and PostGIS2.1)

1. Install PostGIS:

   sudo apt-get install postgresql-9.3-postgis-2.1

2. Run the following commands. The first one will create the necessary tables and functions in the database, and the second will populate the spatial reference table:

   sudo -u postgres psql -d ckan_default -f /usr/share/postgresql/9.3/contrib/postgis-2.1/postgis.sql
   sudo -u postgres psql -d ckan_default -f /usr/share/postgresql/9.3/contrib/postgis-2.1/spatial_ref_sys.sql

3. Change the owner of spatial tables to the CKAN user to avoid errors later on:

   sudo -u postgres psql -d ckan_default -c 'ALTER VIEW geometry_columns OWNER TO ckan_default;'
   sudo -u postgres psql -d ckan_default -c 'ALTER TABLE spatial_ref_sys OWNER TO ckan_default;'

4. Execute the following command to see if PostGIS was properly installed:

   sudo -u postgres psql -d ckan_default -c "SELECT postgis_full_version()"

   You should get something like:

                                                            postgis_full_version
   ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
    POSTGIS="2.1.2 r12389" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER
   (1 row)

5. Install some other packages needed by the extension dependencies:

   sudo apt-get install python-dev libxml2-dev libxslt1-dev libgeos-c1

Install the extension

1. Activate your CKAN virtual environment, for example:

   ENV /usr/lib/ckan/default/bin/activate

2. Install the ckanext-spatial Python package into your virtual environment:

   pip install -e git+https://github.com/ckan/ckanext-spatial#egg=ckanext-spatial

3. Install the rest of Python modules required by the extension:
   

   pip install -r /usr/lib/ckan/default/src/ckanext-spatial/pip-requirements.txt

4. Restart CKAN. For example, if you've deployed CKAN with Apache on Ubuntu:

   sudo service apache2 reload

Configuration options

ckan.plugins = spatial_metadata spatial_query

#   When enabling the spatial metadata, you can define the projection in which extents are stored in the database with the following option. Use the EPSG code as an integer (e.g 4326, 4258, 27700, etc). It defaults to 4326:
#   will try to load "spatialx_schema.json" and "spatialxy_schema.json"
#   as dataset schemas
ckan.spatial.srid = 32633

# To define which backend to use for the spatial search use the following configuration option (see Choosing a backend for the spatial search):
ckanext.spatial.search_backend = solr-spatial-field

# To setup the backgroun map this configuration has been set:
#source: https://lists.okfn.org/pipermail/ckan-dev/2016-July/021484.html
ckanext.spatial.common_map.type = mapbox
ckanext.spatial.common_map.mapbox.map_id = satellite-streets-v11
ckanext.spatial.common_map.mapbox.access_token = pk.ey...

# The extension provides a snippet to add a map widget to the search form, which allows filtering results by an area of interest.
# To add the map widget to the sidebar of the search page, add the following block to the dataset search page template (myproj/ckanext/myproj/templates/package/search.html). If your custom theme is simply extending the CKAN default theme, you will need to add {% ckan_extends %} to the start of your custom search.html, then continue with this:
-------------------------------------------------------------------------------------
{% block secondary_content %}
{% snippet "spatial/snippets/spatial_query.html" %}
{% endblock %}
-------------------------------------------------------------------------------------

# By default the map widget will show the whole world. If you want to set up a different default extent, you can pass an extra default_extent to the snippet, either with a pair of coordinates like this:
---------------------------------------------------------------------------------------------------------------
{% snippet "spatial/snippets/spatial_query.html", default_extent="[[15.62, -139.21], [64.92, -61.87]]" %}
---------------------------------------------------------------------------------------------------------------

# or with a GeoJSON object describing a bounding box (note the escaped quotes):
---------------------------------------------------------------------------------------------------------------------
{% snippet "spatial/snippets/spatial_query.html", default_extent="{ \"type\":
\"Polygon\", \"coordinates\": [[[74.89, 29.39],[74.89, 38.45], [60.50, 38.45], [60.50, 29.39], [74.89, 29.39]]]}" %}
---------------------------------------------------------------------------------------------------------------------

# Using the snippets provided, if datasets contain a spatial extra like the one described in the previous section, a map will be shown on the dataset details page.
# To add a map to the sidebar, add the following block to the dataset page template (eg ckanext-spatial/ckanext/spatial/templates/package/read_base.html):
---------------------------------------------------------------------------------------------------------------------
{% block secondary_content %}
  {{ super() }}

  {% set dataset_extent = h.get_pkg_dict_extra(c.pkg_dict, 'spatial', '') %}
  {% if dataset_extent %}
    {% snippet "spatial/snippets/dataset_map_sidebar.html", extent=dataset_extent %}
  {% endif %}

{% endblock %}
---------------------------------------------------------------------------------------------------------------------

# For adding the map to the main body, add this to the main dataset page template (eg ckanext-myproj/ckanext/myproj/templates/package/read.html):
---------------------------------------------------------------------------------------------------------------------
{% block primary_content_inner %}

  {{ super() }}

  {% set dataset_extent = h.get_pkg_dict_extra(c.pkg_dict, 'spatial', '') %}
  {% if dataset_extent %}
    {% snippet "spatial/snippets/dataset_map.html", extent=dataset_extent %}
  {% endif %}

{% endblock %}
---------------------------------------------------------------------------------------------------------------------

Choosing a backend for the spatial search

solr

solr-spatial-field

postgis

<fields>
    <!-- ... -->
    <field name="bbox_area" type="float" indexed="true" stored="true" />
    <field name="maxx" type="float" indexed="true" stored="true" />
    <field name="maxy" type="float" indexed="true" stored="true" />
    <field name="minx" type="float" indexed="true" stored="true" />
    <field name="miny" type="float" indexed="true" stored="true" />
</fields>

<types>
    <!-- ... -->
    <fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
        spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
        autoIndex="true"
        distErrPct="0.025"
        maxDistErr="0.000009"
        distanceUnits="degrees" />
</types>
<fields>
    <!-- ... -->
    <field name="spatial_geom"  type="location_rpt" indexed="true" stored="true" multiValued="true" />
</fields>

User interaction and design

• Extending metadata adding the resource boundary:

  •  Manually (typing): as an additional field, resource owner needs to add the field "spatial" and give a valid GeoJSION geometry

    example Quelle erweitern

    {
      "type":"Polygon",
      "coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]
    }
    OR
    {
      "type": "Point",
      "coordinates": [-3.145,53.078]
    }

  • Manually (selecting): there already exists an additional field "spatial" and a map widget. Resource owner only needs to select one or multiple boundaries. The program will automatically convert it into valid GeoJSION geometries.

• Representing the spatial boundary
  • Opening up a resource metadata with the spatial information, there exists one widget on the bottom of the resource page showing the exact location of that resource on a map.
• Searching and filtering resources based on their locations
  • Users who want to look for specific resources from a special location can search it through the map widget by choosing a bounding box directly on the map.

Questions and Answers

Below is a list of Q&A from user sides:

Further steps

• Add a spatial widget in the metadata form by which a user can only draw the boundaries and the program automatically convert it into the correct format (instead of typing the geometry).
• Improve the geometry drawing.
• Upgrade Solr (at the moment v. 3.4.) to solr 8 and change the backend for the spatial search