# Welcome to LinkedGeoData: Providing OpenStreetMap data as RDF LinkedGeoData (LGD) is an effort to add a spatial dimension to the Web of Data / Semantic Web. LinkedGeoData uses the information collected by the OpenStreetMap project and makes it available as an RDF knowledge base according to the Linked Data principles. It interlinks this data with other knowledge bases in the Linking Open Data initiative. The project web site can be found [here](http://linkedgeodata.org). If you are running [Ubuntu](http://www.ubuntu.com) then this repository contains everything you need to transform OpenStreetMap data to RDF yourself. For other systems please consider contributing adaptions of the existing scripts. ## Quick Start The following commands should get you started with a running Monaco dataset: ```bash git clone https://github.com/GeoKnow/LinkedGeoData.git cd LinkedGeoData make clean make docker-compose up # Quirk: Sometimes the nominatim container startup fails # indicated by an error message that only reverse-only search is available # In that case restart the container: docker-compose restart lgd-nominatim-web ``` Services will run under these ports: * Nominatim: http://localhost:8012/nominatim * Sparqlify: http://localhost:8013/sparql * Ontop: http://localhost:8014/ and http://localhost:8014/sparql * Pubby: http://localhost:8021/ ### Important Notes * The default settings are in [env.dist](env.dist). * If the file `.env`. does not exist then the `make` invocation will also create it from `env.dist`. * The `.env` file by default contains the setting `PROJECT_NAME=linkedgeodata`. Make sure that the project name matches the lower-case spelling of the name of the folder containing the git repo. * Most configuration changes, such as port and database settings, take effect when restarting the containers. * Many data and config files are stored in volumes whose naming is `${parent-directory}_${service-name}-vol`. For example `linkedgeodata_lgd-osmosis-sync-vol`. You can check existing volumes with `docker volume ls`. * Before starting the containers the sources for the initial data and incremental updates can be configured. These settings should not be changed after starting the containers. * SQL scripts and RDB2RDF Mappings are located in the [sql](sql) and [sml](sml) folders, respectively. The build process bundles these up as a debian package that gets installed in the docker container on docker build. Therefore, changes to these resources require the debian package to be updated. ## Changing the Dataset The `.env` includes the properties which can be ajusted as needed. ``` OSM_DATA_BASE_URL=http://download.geofabrik.de/europe/monaco-latest.osm.pbf OSM_DATA_SYNC_URL=http://download.geofabrik.de/europe/monaco-updates/ ``` In addition, the `files` folder will be mounted as a shared volume to the data loading containers under `/files` so that custom datasets can be loaded as follows: ``` # Copy or download data to the files folder cp mydata.osm.pbf /files ``` ``` OSM_DATA_BASE_URL=file:///files/mydata.osm.pbf # Leaving the following property unset by commenting it out disables sync: #OSM_DATA_SYNC_URL= ``` ## Recent Updates * Please refer to the [Update Section](http://linkedgeodata.org/docs/updates) ## How It Works The architecture shown in the image below. The docker setup is located in the [linkedgeodata-docker](linkedgeodata-docker) folder. ![LGD Dockerized Architecture Overview](docs/assets/images/lgd-architecture-2021-01-18.png) * This project first uses Osmosis to initialize a 'raw' OpenStreetMap postgis database (using simple schema) from a `.osm.pdf` file. * Then, this database is extended with additional tables containing RDF mapping - and interlinking - information. Also, helper views are provided for simplifying access to the integrated information. * Further, a nominatim setup (based on osm2pgqsl) is performed to further enrich the initial database osm data. * A set of RDB2RDF mappings is provided that enables running SPARQL queries over the postgis database. The SPARQL-2-SQL rewriting engine we use is Sparqlify. * Dumps are generated by simply running preconfigured SPARQL queries. ## Project layout This project requires Make and Apache Maven to build (Java 11+ required). Maven builds are not dockerized. Therefore any built artifacts will be cached in the local repository as usual. ### Primary resources They include the SQL files and mappings that enable rewriting SPARQL queries to SQL ones over an OpenStreetMap (and Nominatim) database. Furthermore, bash scripts are avaiable for helping setting up an LGD database. The mavenization and dockerization re-package these resources. The following folders contain resources that are copied when building the `lgd-tools-resources` jar bundle * [sql](sql) SQL scripts for extending on OSM simple schema database * [sml](sml) The mapping files in SML format. Convert to e.g. r2rml using the mapping converter. The scripts in the bin folder become part of the debian packaging of the cli tools `lgd-tools-pkg-deb-cli` * [bin](bin) Bash scripts for setting LinkedGeoData up (Ubuntu) ### Thirdparty resources There is a pom.xml that creates a jar file from a snapshot of the nominatim git. Especially for development this is much faster than repeated clones of the full git repo. ### Contributions Welcome The docker-based architecture is aimed at making it easy to contribute new or alternative components that can sit side-by-side with the core of the system - which is the a virtual knowledge graph view over an OSM database. Please open issues for discussion. Examples include but are not limited to: * A more modern Linked Data Interface that displays GeoSPARQL geometries out-of-the-box * ~~Nicer SPARQL interface (YASGUI)~~ ✓ * Another data virtualization engine in order to ease comparision with the already integrated ones * Updates to the existing OSM-RDF mappings, proposals about how this system could be improved. * Proposals for a better IRI schema. For example, the 'triplify' in the IRIs is archaic. Migration to the pattern used by [Sophox](https://wiki.openstreetmap.org/wiki/Sophox) seems very worthwhile. Because of the virtual knowledge graph approach there should be no problem to use the legacy approach in parallel. * General proposals for architecture improvements (e.g. config options in the docker setup to improve modularity) Dockerfiles for services such as a Linked Data or SPARQL interfaces should be designed to allow configuration of the target SPARQL endpoint(s), ideally via the docker environment. ## License The content of this project are licensed under the [GPL v3 License](https://github.com/GeoKnow/LinkedGeoData/blob/master/LICENSE).