@prefix n4c: <https://nfdi4culture.de/id/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix schema: <http://schema.org/> .
@prefix fabio: <http://purl.org/spar/fabio/> .
@prefix nfdicore: <https://nfdi.fiz-karlsruhe.de/ontology/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix void: <http://rdfs.org/ns/void#> .
n4c:E3247
  rdfs:label "An automated Wikibase deployment pipeline for research data management."@en ;
  rdfs:seeAlso <https://nfdi4culture.de/resource/E3247.rdf>, <https://nfdi4culture.de/resource/E3247.ttl>, <https://nfdi4culture.de/resource/E3247.nt>, <https://nfdi4culture.de/resource/E3247.json> ;
  a schema:NewsArticle, fabio:NewsItem ;
  schema:image <https://nfdi4culture.de/fileadmin/user_upload/news_events/Deployment_pipeline_post-image2.png> ;
  schema:name " An automated Wikibase deployment pipeline for research data management" ;
  schema:headline " An automated Wikibase deployment pipeline for research data management" ;
  nfdicore:NFDI_0001008 "https://nfdi4culture.de/news/details/an-automated-wikibase-deployment-pipeline-for-research-data-management"^^xsd:anyURI ;
  schema:datePublished "2022-07-04"^^xsd:date ;
  schema:text """
As part of our work in the context of NFDI4Culture (see https://nfdi4culture.de/), a project aimed at setting out standards, guidelines and practical implementations of infrastructure for research data management, the Open Science Lab (see https://www.tib.eu/de/forschung-entwicklung/forschungsgruppen-und-labs/open-science) have been engaging with the construction of a reliable and configurable deployment process for Wikibase (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy). The requirements themselves have grown organically based on research needs in the context of facilitating LOD management and the annotation of complex media objects (see https://zenodo.org/record/5628847), and includes significant modification from the example Wikibase installation (see https://github.com/wmde/wikibase-release-pipeline/tree/main/example) configuration provided by WMDE (see https://wikimedia.de/). Our method of deployment has also matured from a collection of terminal commands to a fully-automated Ansible (see https://www.ansible.com/) playbook.
Our entire deployment process is now orchestrated by Ansible. This is a tool intended to allow developers to deploy systems in a highly predictable manner, and is attractive as it requires that nothing is pre-installed on target servers. It is also extremely scalable: if a playbook can write a deployment to a single instance, it can write to a hundred.
The time-saving aspect of Ansible has also been crucial, as the deployment playbook now contains a number of discrete stages, from installing prerequisite tools, building custom containers and performing post-installation configuration. We have included full NGINX (see https://www.nginx.com/) configuration, including automatically requesting SSL certificates (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/a13426bab4599457dd25cf262abd1d6e639a80b8/ansible/roles/nginx/tasks/main.yml#L16) as, although we no longer use this role in our own instances, this has been retained here as likely useful for other users who wish to deploy a proper HTTPS setup.
Within the playbook, it is worth breaking down what each of these roles are doing:
 	 	User (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/user/tasks/main.yml)
The process begins by creating a non-root user account on the server and migrating the required SSH keys to allow direct login to the new account.
 	Nginx (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/nginx/tasks/main.yml)
NGINX (see https://www.nginx.com/) is installed, certificates and configuration generated and the system is restarted ready for use. In the interests of pursuing idempotency (see https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html#term-Idempotency), certificates are generated only if they do not exist. Users should be aware while undertaking deployment testing that there are rate limits (see https://letsencrypt.org/docs/rate-limits/) imposed by LetsEncrypt which are easily reached.
 	Docker (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/docker/tasks/main.yml)
This stage installs Docker (see https://www.docker.com/) and Docker Compose (see https://docs.docker.com/compose/), which are required for the various containerised components of the installation. Containerisation is the process of wrapping an application within a generalised and predictable environment, which avoids the historical issues with applications behaving differently when installed on different systems.
 	Custom Query UI (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/customqueryui/tasks/main.yml)
Inspired by the excellent work undertaken by Rhizome (see https://rhizome.org/) customising their own query interface (see https://github.com/rhizomedotorg/artbase-query-gui), we have forked an instance with further modifications (see https://gitlab.com/nfdi4culture/ta1-data-enrichment/mvp-query-gui). These include changes to stylistic elements as well as adding custom prefixes for query building. Currently, this is deployed by submitting templated configuration files and building the Docker image directly as part of the playbook, but we will shortly publish this as a versioned container with configuration applied post-build.
 	Wikibase (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/tasks/main.yml)
Wikibase itself is then constructed. This is achieved by writing the necessary templates and scripts (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/tree/main/ansible/roles/wikibase/templates) to the target server and starting all Docker containers from a single Docker Compose configuration file (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/docker-compose.yml.j2). Much of this Docker configuration is unchanged from the default (see https://github.com/wmde/wikibase-release-pipeline/blob/main/example/docker-compose.yml), with a few key alterations to allow for HTTPS support (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/docker-compose.yml.j2#L145). The Wikibase configuration file (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2) has been modified to include extensions for hosting local media (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L19), requiring account sign in and approval prior to making edits (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L47), email server configuration to allow Wikibase to issue notifications (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L26), and a custom namespace for "Description" pages (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L54). The latter serves the common need in research data management scenarios to work with long-form natural language texts, which requires storage separate to the structured data statements. There are also the typical style updates: logo (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L7), icon (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L10) and Wikibase name (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/wikibase/templates/LocalSettings.php.j2#L4).
 	ConfirmAccount (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/confirmaccount/tasks/main.yml)
Although ConfirmAccount has been requested by the Wikibase configuration, the extension itself (see https://www.mediawiki.org/wiki/Extension:ConfirmAccount) must also be installed. This role downloads the appropriate release (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/confirmaccount/tasks/main.yml#L2), unarchives and places it in the required location (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/confirmaccount/tasks/main.yml#L6). The role also runs the required maintenance PHP script (see https://www.mediawiki.org/wiki/Manual:Maintenance.php) to update the Wikibase SQL database.
 	Prefixes (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/prefixes/tasks/main.yml)
Some prefix updates (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/prefixes/templates/prefixes.conf.j2) must be submitted to the query service configuration to allow for the customisation deployed as part of the built query user interface.
 	Reconcile (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/reconcile/tasks/main.yml)
The final stage of the deployment is to enable a reconciliation endpoint for writing data from OpenRefine (see https://openrefine.org/) to our Wikibase instance. Prior to building the application itself, it is necessary to add mandatory edit tags for multiple versions of OpenRefine, which is achieved via a small Python script (see https://gitlab.com/nfdi4culture/ta5-knowledge-graph/wikibase-deploy/-/blob/main/ansible/roles/reconcile/templates/openrefine-tag.py.j2). After this, the service configuration is written with instance specific information, and the service is launched.
As with other TIB projects, each Wikibase deployment comprises discrete instances for production, testing and development work, written to distinct Debian (see https://www.debian.org/) servers. The advantage of deploying via the same Ansible playbook to each of these servers is that testing and development work can take place in environments identical to the production instance. Concerning data synchronisation, this is performed using a SQL database snapshot process and is enacted periodically from production to test instances.
Maintenance of the Wikibase deployment is controlled by a separate playbook (see https://gitlab.com/nfdi4culture/ta1-data-enrichment/wikibase-maintenance/-/blob/main/ansible/playbook.yml), which manages the automated snapshotting process to ensure that data will be retained in the case of catastrophic server failure, or to facilitate regular database overwriting operations. A planned extension of this playbook will be the incorporation of system monitoring tasks to ensure that the services are responding as expected. This playbook will also be handed over to a scheduling system like Apache Airflow (see https://airflow.apache.org/) to manage regular operation without human interaction.
The use of Ansible is also useful beyond deployment, as system upgrades to Mediawiki, Wikibase and other containerised components can be consistently directed to all relevant servers. In this way, maintenance remains a sustainable task even when the number of running instances is scaled.
The process described above has allowed us to deploy instances of Wikibase which are specially customised to meet the needs of research data management use-cases with minimal effort and with confidence that the resulting services will be stable and ready to use. Future efforts will focus on additional strategies for managing multiple deployments, adding improved system monitoring and the incorporation of further tools and configuration changes as driven by project requirements.
&nbsp;
Author: Paul Duchesne, Open Science Lab, TIB
First published: July 6th, 2022
Contact Information:
Questions or suggestions? Simply contact the NFDI4Culture Helpdesk (see https://nfdi4culture.de/helpdesk.html) via a general request.""" ;
  schema:keywords n4c:E4386 ;
  schema:contributor [
    a schema:Role ;
    schema:contributor n4c:E2849 ;
    schema:sameAs n4c:E3282 ;
    schema:roleName "Author"
  ], [
    a schema:Role ;
    schema:contributor n4c:E1825 ;
    schema:sameAs n4c:E3283 ;
    schema:roleName "Contributor"
  ] ;
  nfdicore:NFDI_0000116 n4c:E1825 ;
  nfdicore:NFDI_0001027 n4c:E2849 ;
  schema:about n4c:E3759 ;
  schema:potentialAction [
    a schema:CreateAction ;
    schema:actionStatus "CompletedActionStatus" ;
    schema:agent n4c:E1825 ;
    schema:name "Task Area 5: Overarching technical, ethical and legal activities" ;
    schema:result n4c:E3247
  ] .
<https://nfdi4culture.de/resource/E3247.rdf>
  a void:Dataset ;
  void:feature <http://www.w3.org/ns/formats/RDF_XML> .
<https://nfdi4culture.de/resource/E3247.ttl>
  a void:Dataset ;
  void:feature <http://www.w3.org/ns/formats/Turtle> .
<https://nfdi4culture.de/resource/E3247.nt>
  a void:Dataset ;
  void:feature <http://www.w3.org/ns/formats/N-Triples> .
<https://nfdi4culture.de/resource/E3247.json>
  a void:Dataset ;
  void:feature <http://www.w3.org/ns/formats/JSON-LD> .
n4c:E4386 schema:name "News Item" .
n4c:E2849 schema:name "Paul Duchesne" .
n4c:E1825 schema:name "Task Area 5: Overarching technical, ethical and legal activities" .
n4c:E3759 schema:name "Wikibase4Research" .
