Difference between revisions of "ART-DECOR building and testing"
 
(21 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
This page will document how to build and test ART-DECOR in a automated process.
 
This page will document how to build and test ART-DECOR in a automated process.
  
−
==Building ART-DECOR==
+
=Building ART-DECOR=
−
The following steps are taken during an ART-DECOR build
+
The following steps are taken during an ART-DECOR build:
−
===eXist-db===
+
* install eXist-db
−
====Committing code====
+
* Create new ART-DECOR packages for installation in eXist-db database
−
Code is committed in a software repository and 2 logs are created:
+
* Create new vue build
−
* Development: [[http://art-decor.org/ADAR-dev/svnlog.html]]
+
* Restoring data
−
* Stable: [[http://art-decor.org/ADAR/svnlog.html]]
+
 
−
====Building XAR packages code====
+
=eXist-db=
 +
==Install eXist-db from installer==
 +
The part will install the eXist-db database.
 +
A Unix-like OS is assumed, but the main part of the work is done inside eXist-db.
 +
 
 +
=== Notes on the installation ===
 +
* Make sure JAVA is installed
 +
* Mac and Linux can use the .jar installer: java -jar INSTALLER_FILE.jar
 +
* Please note ART-DECOR uses a specific eXist-db installer which contains configuration for ART-DECOR. Details about the configuration can be found at: https://bitbucket.org/art-decor/art-decor-backend/src/exist-db-5/exist5/
 +
* Start the database: sudo INSTALLATION_FOLDER/bin/startup.sh or use wrapper scripts for linux (see below)
 +
* See also the installation process for Linux for older versions: [[ART-DECOR_installation_on_linux]]
 +
* See also: https://exist-db.org/exist/apps/doc/basic-installation
 +
 
 +
=== Install eXist-db on linux ===
 +
Installation procedure for Linux, tested on CentOS.
 +
 
 +
* Make sure JAVA is installed
 +
* Setup the directory structure
 +
<syntaxhighlight lang="bash">
 +
sudo mkdir /opt/art-decor-linux;
 +
cd /opt/art-decor-linux
 +
sudo mkdir tooling
 +
sudo mkdir tooling/backups
 +
sudo mkdir tooling/packages
 +
sudo mkdir tooling/packages_archives
 +
sudo mkdir tooling/scripts_archives
 +
sudo chown -R root:wheel /opt/art-decor-linux/
 +
cd /opt/art-decor-linux/tooling/packages
 +
</syntaxhighlight>
 +
 
 +
* Run the eXist-db installer
 +
<syntaxhighlight lang="bash">
 +
sudo su -l root
 +
# stop previous version of eXist on your system
 +
systemctl stop exist-db.service
 +
cd /opt/art-decor-linux/tooling/packages
 +
# Set the version:
 +
VERSION=5.3.0
 +
VERSIONSHORT=530
 +
# download the ART-DECOR eXist-db installer
 +
wget https://decor.test-nictiz.nl/pub/exist_art_decor_${VERSION}/exist-installer-${VERSION}_artdecor.jar --no-check-certificate
 +
# or curl https://decor.test-nictiz.nl/pub/exist_art_decor_${VERSION}/exist-installer-${VERSION}_artdecor.jar -o exist-installer-${VERSION}_artdecor.jar
 +
 
 +
# start the eXist-db installer:
 +
java -jar exist-installer-${VERSION}_artdecor.jar -console
 +
# output/input during the installer:
 +
# installation path contains VERSIONSHORT_CURRENTDATE
 +
Select the installation path:  [/usr/local/exist_atp_520_20200130]
 +
 
 +
# Use default choice for installation packages (these are eXist-db packages, not ART-DECOR)
 +
Select Installation Packages -> 0
 +
 
 +
# Usually not sensible to create shortcuts on a headless system:
 +
    ----------------------------------
 +
    Create shortcuts in the Start Menu
 +
    ----------------------------------
 +
    Enter Y for Yes, N for No:
 +
    N
 +
    ------------------------------------------
 +
    Create additional shortcuts on the desktop
 +
    ------------------------------------------
 +
    Enter Y for Yes, N for No:
 +
    N
 +
</syntaxhighlight>
 +
 
 +
* create backup of empty eXist-db
 +
This is handy for when you need to re-deploy the database. Another option for re-deploy is just running the installer again (then this step can be skipped)
 +
<syntaxhighlight lang="bash">
 +
mkdir -p /opt/art-decor-linux/tooling/backups
 +
cd /opt/art-decor-linux/tooling/backups
 +
tar zcvf exist_without_artdecor_${VERSIONSHORT}_$(date '+%Y%m%d%H%M').tgz /usr/local/exist_atp_${VERSIONSHORT}_*
 +
</syntaxhighlight>
 +
 
 +
* Create symlink
 +
<syntaxhighlight lang="bash">
 +
cd /usr/local
 +
ln -s  exist_atp_${VERSIONSHORT}_* exist_atp_${VERSIONSHORT}
 +
</syntaxhighlight>
 +
 
 +
* adduser existdb
 +
<syntaxhighlight lang="bash">
 +
adduser --system --shell /sbin/nologin --comment "eXist-db Service Account" existdb
 +
chown -h existdb:existdb /usr/local/exist_atp_${VERSIONSHORT}
 +
chown -R existdb:existdb /usr/local/exist_atp_${VERSIONSHORT}_20*
 +
</syntaxhighlight>
 +
 
 +
* check that there is no other eXist-db currently running
 +
<syntaxhighlight lang="bash">
 +
systemctl status exist-db-${VERSIONSHORT}.service
 +
ps aux |grep -i exist
 +
netstat -anp|grep :8080
 +
netstat -anp|grep :8
 +
</syntaxhighlight>
 +
 
 +
* create a wrapper script for systemd
 +
More info: https://exist-db.org/exist/apps/doc/advanced-installation#headless
 +
<syntaxhighlight lang="bash">
 +
location_symlink=/usr/local/exist_atp_${VERSIONSHORT}
 +
FILE=/etc/systemd/system/exist-db-${VERSIONSHORT}.service
 +
cat > ${FILE} << _EOF_
 +
[Unit]
 +
Description=eXist-db ${VERSIONSHORT} Server
 +
Documentation=
 +
After=syslog.target
 +
 
 +
[Service]
 +
Type=simple
 +
User=existdb
 +
Group=existdb
 +
ExecStart=${location_symlink}/bin/startup.sh
 +
 
 +
[Install]
 +
WantedBy=multi-user.target
 +
_EOF_
 +
</syntaxhighlight>
 +
 
 +
* configure systemd and set permissions
 +
<syntaxhighlight lang="bash">
 +
cd /etc/systemd/system
 +
chown existdb exist-db-${VERSIONSHORT}.service
 +
chgrp existdb exist-db-${VERSIONSHORT}.service
 +
echo eXist-db can be started/stopped/status with:
 +
echo sudo systemctl status exist-db-${VERSIONSHORT}.service
 +
echo sudo systemctl start exist-db-${VERSIONSHORT}.service
 +
exit
 +
 
 +
# as normal user, start eXist-db
 +
sudo systemctl status exist-db-${VERSIONSHORT}.service
 +
sudo systemctl start exist-db-${VERSIONSHORT}.service
 +
 
 +
# If you have replaced the version of exist-db on this server:
 +
sudo systemctl daemon-reload
 +
</syntaxhighlight>
 +
 
 +
* Note that for development environments, the systemd wrapper could be skipped and instead:
 +
Start eXist by navigating in a command window to the newly installed folder, bin directory and run:
 +
<syntaxhighlight lang="bash">
 +
cd /usr/local/exist_atp_${VERSIONSHORT}/bin; sudo nohup ./startup.sh &
 +
</syntaxhighlight>
 +
(no privileges required, eXist runs on a non-protected port, 8877).
 +
 
 +
* check status and logging files
 +
<syntaxhighlight lang="bash">
 +
sudo ps aux |grep -i exist
 +
sudo netstat -anp |grep :8877
 +
 
 +
sudo updatedb
 +
sudo locate exist.log
 +
sudo tail -f /usr/local/exist_atp_${VERSIONSHORT}/logs/*.log
 +
# or just tail: exist.log
 +
</syntaxhighlight>
 +
 
 +
=== eXist-db ART-DECOR package installation ===
 +
In this step we will install the ART-DECOR packages into eXist-db.
 +
 
 +
Perform the following steps:
 +
* If needed: Create SSH-tunnel to the server. localhost:8877 to server:8877
 +
* Check the eXist-db logs while you are installing packages below
 +
* Navigate in a browser to http://localhost:8877
 +
* Access the package manager:
 +
http://localhost:8877/exist/apps/dashboard/admin#/launcher
 +
http://localhost:8877/exist/apps/dashboard/admin#/packagemanager
 +
* Login as the admin user
 +
* Start the Package Manager and update all packages that are marked as updateable (from the eXist-db repository). Expected are: Dashboard, eXide, shared*
 +
 
 +
* Change the repository configuration to the ART-DECOR repository
 +
Browse to eXide: http://localhost:8877/exist/apps/eXide/index.html
 +
* Open this file: by clicking on '''directory''' and clicking to this location:
 +
/db/apps/packageservice/configuration.xml
 +
The default configuration allows you to install/update eXist-db packages from exist-db.org .xar package repository. Note that eXist-db can only handle one repository.
 +
* Change the repository stanza so that it points to the ART-DECOR .xar package repository (and comment the exist-db.org line out):
 +
<syntaxhighlight lang="xml">
 +
<settings>
 +
    <repositories>
 +
      <!--  <repository active="true" default="true">http://exist-db.org/exist/apps/public-repo</repository> -->
 +
        <repository active="true" default="true">http://decor.nictiz.nl/apps/public-repo-dev5</repository>
 +
</syntaxhighlight>
 +
* File, Save
 +
 
 +
* Access the package manager:
 +
http://localhost:8877/exist/apps/dashboard/admin#/packagemanager
 +
* Install ART-DECOR packages from the ART-DECOR repository with the package manager
 +
install, in the given order:
 +
# Advanced Requirements Tooling
 +
# DECOR core files
 +
# DECOR services
 +
# ART-DECOR System Services
 +
# Terminology Applications
 +
# DECOR example files
 +
# ART-DECOR API
 +
# OID Registry tools
 +
 
 +
 
 +
* API documentation:
 +
See:
 +
http://localhost:8877/exist/apps/api/api.html
 +
 
 +
* (optional) create users, demo users with which you can test ART-DECOR:
 +
Author: decor, editor, issues, terminology
 +
Project admin: decor, decor-admin, editor, issues, terminology
 +
 
 +
* create backup of eXist-db which has ART-DECOR installed 
 +
<syntaxhighlight lang="bash">
 +
VERSIONSHORT=(version you are running)
 +
sudo systemctl stop exist-db-${VERSIONSHORT}.service
 +
sudo systemctl status exist-db-${VERSIONSHORT}.service
 +
mkdir -p /opt/art-decor-linux/tooling/backups
 +
cd /opt/art-decor-linux/tooling/backups
 +
sudo tar zcvf exist_installed_artdecor_${VERSIONSHORT}_$(date '+%Y%m%d%H%M').tgz /usr/local/exist_atp_${VERSIONSHORT}_*
 +
sudo systemctl start exist-db-${VERSIONSHORT}.service
 +
</syntaxhighlight>
 +
 
 +
=== eXist-db troubleshooting ===
 +
A couple pointers if issues occur:
 +
* For problems with tokens/logging in. Make sure '''eXist-db EXPath Cryptographic library''' and '''JWT''' are installed and updated to the latest version. JWT relies on Crypto, so update Crypto first. Note that you may have to stop/start eXist-db after performing these upgrades.
 +
 
 +
==eXist-db ART-DECOR code update==
 +
===Commit code===
 +
Code is committed in a software repository, see [[ART_developers_manual#ART-DECOR_Next_generation_repository]]
 +
===Build .xar packages===
 
During this step code from the code repository is converted into .xar packages.
 
During this step code from the code repository is converted into .xar packages.
−
+
 
 
Documentation on how to run the script for updating the repository can be found here: TODO
 
Documentation on how to run the script for updating the repository can be found here: TODO
  
−
The result repositories are viewable here: [[Download#Repository]]
+
The result (development) repository is viewable here: http://decor.nictiz.nl/apps/public-repo-dev5/retrieve.html
  
−
====Installing .xars into a ART-DECOR instance====
+
Note: two other routes to get your own sourcecode into a running eXist-db database:
−
There is a docker script that will deploy a running instance of ART-DECOR with the latest .xar packages installed.  
+
* First locally, create a .xar package by running '''ant xar''' and install that package into the database: at localhost:8877, through the package manager. Or:
−
Documentation on how to run the script for installing ART-DECOR can be found here: TODO
+
* connect to eXist-db and save your code into the database. For instance at localhost:8877, through eXide. Please note that during creation of the .xar package any variables such as @project.title@ in expath-pkg.xml.tmpl, but maybe also in api.json are replaced by the values that are set in build.xml. Therefore the literal code in the git repository is different than the code created in the .xar package (which will be installed into eXist-db).
  
−
===Orbeon===
+
===Installing .xars into a ART-DECOR instance===
−
====Creating a Orbeon war file====
+
Documentation can be found here: [[ART-DECOR_building_and_testing#eXist-db_ART-DECOR_package_installation]]
−
Steps to create the art-decor.war package are documented here: [[ART_developers_manual#Creating_a_new_art-decor.war_package]]
 
  
−
====Installing a Orbeon war file into a ART-DECOR instance ====
 
−
Documentation on how to run the script for installing ART-DECOR can be found here: TODO
 
 
===Restoring data===
 
===Restoring data===
 
Test instances or production instances of ART-DECOR might need data restored into their database.
 
Test instances or production instances of ART-DECOR might need data restored into their database.
 +
For example:
 +
* Make sure you have backup from NEW installation (so you can rollback)
 +
* Copy data from old installation. Note that this works for version 5 -> version 5 eXist-db, but might not work between other major eXist-db versions.
 +
<syntaxhighlight lang="bash">
 +
OLD=/usr/local/exist_atp_511
 +
NEW=/usr/local/exist_atp_530
 +
sudo rm -Rf $NEW/data
 +
sudo cp -r --preserve=all $OLD/data $NEW
 +
</syntaxhighlight>
 +
 +
{{:ART_developers_vue}}
  
 
==Testing ART-DECOR==
 
==Testing ART-DECOR==
 
Tests that can be run against a ART-DECOR instance are documented at:
 
Tests that can be run against a ART-DECOR instance are documented at:
 
[[ART_developers_manual#ART-DECOR_testscenarios]]
 
[[ART_developers_manual#ART-DECOR_testscenarios]]

Latest revision as of 10:17, 28 July 2021

Tools.svg This article or section is in the middle of an expansion or major restructuring and is not yet ready for use. You are welcome to assist in its construction by editing it as well.

This page will document how to build and test ART-DECOR in a automated process.

Building ART-DECOR

The following steps are taken during an ART-DECOR build:

  • install eXist-db
  • Create new ART-DECOR packages for installation in eXist-db database
  • Create new vue build
  • Restoring data

eXist-db

Install eXist-db from installer

The part will install the eXist-db database. A Unix-like OS is assumed, but the main part of the work is done inside eXist-db.

Notes on the installation

Install eXist-db on linux

Installation procedure for Linux, tested on CentOS.

  • Make sure JAVA is installed
  • Setup the directory structure
sudo mkdir /opt/art-decor-linux;
cd /opt/art-decor-linux
sudo mkdir tooling
sudo mkdir tooling/backups
sudo mkdir tooling/packages
sudo mkdir tooling/packages_archives
sudo mkdir tooling/scripts_archives
sudo chown -R root:wheel /opt/art-decor-linux/
cd /opt/art-decor-linux/tooling/packages
  • Run the eXist-db installer
sudo su -l root
# stop previous version of eXist on your system
systemctl stop exist-db.service
cd /opt/art-decor-linux/tooling/packages
# Set the version:
VERSION=5.3.0
VERSIONSHORT=530
# download the ART-DECOR eXist-db installer
wget https://decor.test-nictiz.nl/pub/exist_art_decor_${VERSION}/exist-installer-${VERSION}_artdecor.jar --no-check-certificate
# or curl https://decor.test-nictiz.nl/pub/exist_art_decor_${VERSION}/exist-installer-${VERSION}_artdecor.jar -o exist-installer-${VERSION}_artdecor.jar

# start the eXist-db installer:
java -jar exist-installer-${VERSION}_artdecor.jar -console
# output/input during the installer:
# installation path contains VERSIONSHORT_CURRENTDATE
Select the installation path:  [/usr/local/exist_atp_520_20200130]

# Use default choice for installation packages (these are eXist-db packages, not ART-DECOR)
Select Installation Packages -> 0

# Usually not sensible to create shortcuts on a headless system:
    ----------------------------------
    Create shortcuts in the Start Menu
    ----------------------------------
    Enter Y for Yes, N for No:
    N
    ------------------------------------------
    Create additional shortcuts on the desktop
    ------------------------------------------
    Enter Y for Yes, N for No:
    N
  • create backup of empty eXist-db

This is handy for when you need to re-deploy the database. Another option for re-deploy is just running the installer again (then this step can be skipped) 

mkdir -p /opt/art-decor-linux/tooling/backups
cd /opt/art-decor-linux/tooling/backups
tar zcvf exist_without_artdecor_${VERSIONSHORT}_$(date '+%Y%m%d%H%M').tgz /usr/local/exist_atp_${VERSIONSHORT}_*
  • Create symlink
cd /usr/local
ln -s  exist_atp_${VERSIONSHORT}_* exist_atp_${VERSIONSHORT}
  • adduser existdb
adduser --system --shell /sbin/nologin --comment "eXist-db Service Account" existdb
chown -h existdb:existdb /usr/local/exist_atp_${VERSIONSHORT}
chown -R existdb:existdb /usr/local/exist_atp_${VERSIONSHORT}_20*
  • check that there is no other eXist-db currently running
systemctl status exist-db-${VERSIONSHORT}.service
ps aux |grep -i exist
netstat -anp|grep :8080
netstat -anp|grep :8
  • create a wrapper script for systemd

More info: https://exist-db.org/exist/apps/doc/advanced-installation#headless 

location_symlink=/usr/local/exist_atp_${VERSIONSHORT}
FILE=/etc/systemd/system/exist-db-${VERSIONSHORT}.service
cat > ${FILE} << _EOF_
[Unit]
Description=eXist-db ${VERSIONSHORT} Server
Documentation=
After=syslog.target

[Service]
Type=simple
User=existdb
Group=existdb
ExecStart=${location_symlink}/bin/startup.sh

[Install]
WantedBy=multi-user.target
_EOF_
  • configure systemd and set permissions
cd /etc/systemd/system
chown existdb exist-db-${VERSIONSHORT}.service
chgrp existdb exist-db-${VERSIONSHORT}.service
echo eXist-db can be started/stopped/status with:
echo sudo systemctl status exist-db-${VERSIONSHORT}.service
echo sudo systemctl start exist-db-${VERSIONSHORT}.service
exit

# as normal user, start eXist-db
sudo systemctl status exist-db-${VERSIONSHORT}.service
sudo systemctl start exist-db-${VERSIONSHORT}.service

# If you have replaced the version of exist-db on this server:
sudo systemctl daemon-reload
  • Note that for development environments, the systemd wrapper could be skipped and instead:

Start eXist by navigating in a command window to the newly installed folder, bin directory and run: 

cd /usr/local/exist_atp_${VERSIONSHORT}/bin; sudo nohup ./startup.sh &

(no privileges required, eXist runs on a non-protected port, 8877).

  • check status and logging files
sudo ps aux |grep -i exist
sudo netstat -anp |grep :8877

sudo updatedb
sudo locate exist.log
sudo tail -f /usr/local/exist_atp_${VERSIONSHORT}/logs/*.log
# or just tail: exist.log

eXist-db ART-DECOR package installation

In this step we will install the ART-DECOR packages into eXist-db.

Perform the following steps:

  • If needed: Create SSH-tunnel to the server. localhost:8877 to server:8877
  • Check the eXist-db logs while you are installing packages below
  • Navigate in a browser to http://localhost:8877
  • Access the package manager:

http://localhost:8877/exist/apps/dashboard/admin#/launcher http://localhost:8877/exist/apps/dashboard/admin#/packagemanager

  • Login as the admin user
  • Start the Package Manager and update all packages that are marked as updateable (from the eXist-db repository). Expected are: Dashboard, eXide, shared*
  • Change the repository configuration to the ART-DECOR repository

Browse to eXide: http://localhost:8877/exist/apps/eXide/index.html

  • Open this file: by clicking on directory and clicking to this location:

/db/apps/packageservice/configuration.xml The default configuration allows you to install/update eXist-db packages from exist-db.org .xar package repository. Note that eXist-db can only handle one repository.

  • Change the repository stanza so that it points to the ART-DECOR .xar package repository (and comment the exist-db.org line out):
<settings>
    <repositories>
      <!--  <repository active="true" default="true">http://exist-db.org/exist/apps/public-repo</repository> -->
        <repository active="true" default="true">http://decor.nictiz.nl/apps/public-repo-dev5</repository>
  • File, Save
  • Access the package manager:

http://localhost:8877/exist/apps/dashboard/admin#/packagemanager

  • Install ART-DECOR packages from the ART-DECOR repository with the package manager

install, in the given order:

  1. Advanced Requirements Tooling
  2. DECOR core files
  3. DECOR services
  4. ART-DECOR System Services
  5. Terminology Applications
  6. DECOR example files
  7. ART-DECOR API
  8. OID Registry tools


  • API documentation:

See: http://localhost:8877/exist/apps/api/api.html

  • (optional) create users, demo users with which you can test ART-DECOR:

Author: decor, editor, issues, terminology Project admin: decor, decor-admin, editor, issues, terminology

  • create backup of eXist-db which has ART-DECOR installed
VERSIONSHORT=(version you are running)
sudo systemctl stop exist-db-${VERSIONSHORT}.service
sudo systemctl status exist-db-${VERSIONSHORT}.service
mkdir -p /opt/art-decor-linux/tooling/backups
cd /opt/art-decor-linux/tooling/backups
sudo tar zcvf exist_installed_artdecor_${VERSIONSHORT}_$(date '+%Y%m%d%H%M').tgz /usr/local/exist_atp_${VERSIONSHORT}_*
sudo systemctl start exist-db-${VERSIONSHORT}.service

eXist-db troubleshooting

A couple pointers if issues occur:

  • For problems with tokens/logging in. Make sure eXist-db EXPath Cryptographic library and JWT are installed and updated to the latest version. JWT relies on Crypto, so update Crypto first. Note that you may have to stop/start eXist-db after performing these upgrades.

eXist-db ART-DECOR code update

Commit code

Code is committed in a software repository, see ART_developers_manual#ART-DECOR_Next_generation_repository

Build .xar packages

During this step code from the code repository is converted into .xar packages.

Documentation on how to run the script for updating the repository can be found here: TODO

The result (development) repository is viewable here: http://decor.nictiz.nl/apps/public-repo-dev5/retrieve.html

Note: two other routes to get your own sourcecode into a running eXist-db database:

  • First locally, create a .xar package by running ant xar and install that package into the database: at localhost:8877, through the package manager. Or:
  • connect to eXist-db and save your code into the database. For instance at localhost:8877, through eXide. Please note that during creation of the .xar package any variables such as @project.title@ in expath-pkg.xml.tmpl, but maybe also in api.json are replaced by the values that are set in build.xml. Therefore the literal code in the git repository is different than the code created in the .xar package (which will be installed into eXist-db).

Installing .xars into a ART-DECOR instance

Documentation can be found here: ART-DECOR_building_and_testing#eXist-db_ART-DECOR_package_installation

Restoring data

Test instances or production instances of ART-DECOR might need data restored into their database. For example:

  • Make sure you have backup from NEW installation (so you can rollback)
  • Copy data from old installation. Note that this works for version 5 -> version 5 eXist-db, but might not work between other major eXist-db versions.
OLD=/usr/local/exist_atp_511
NEW=/usr/local/exist_atp_530
sudo rm -Rf $NEW/data
sudo cp -r --preserve=all $OLD/data $NEW

Vue

Vue background information

More info on vue: https://vuetifyjs.com/en/getting-started/quick-start/

Vue installation for user

For users that want to install their own ART-DECOR instance:

Vue installation for developers

Vue debugging

https://vuejs.org/v2/cookbook/debugging-in-vscode.html

Chrome

  • Chrome DevTools: what requests are being sent to the back-end. Shows queries/responses.
  • extension: Vue Devtools and other debugging information.

Testing ART-DECOR

Tests that can be run against a ART-DECOR instance are documented at: ART_developers_manual#ART-DECOR_testscenarios

Retrieved from "https://wiki.art-decor.org/index.php?title=ART-DECOR_building_and_testing&oldid=8076"