Upgrade exist-db - Revision history

Ahenket: /* Modify the backup data before importing it */

2017-07-27T15:45:55Z

‎Modify the backup data before importing it

Ahenket: /* Modify the backup data before importing it */

2017-07-27T15:44:46Z

‎Modify the backup data before importing it

Ahenket: /* Modify the backup data before importing it */

2017-07-27T15:44:30Z

‎Modify the backup data before importing it

Ahenket: /* Modify the backup data before importing it */

2017-07-27T15:28:46Z

‎Modify the backup data before importing it

Mligtvoet: /* Modify the backup data before importing it */

2016-04-01T07:06:56Z

‎Modify the backup data before importing it

Mligtvoet: initial setup

2015-10-31T22:11:28Z

initial setup

New page

Since ART 1.6.0 we recommend updating the eXist-db database to version 2.2.
This section documents the actions that are needed to upgrade from eXist-db 2.1 to eXist-db 2.2.
If you installed eXist-db from eXist-db-setup-2.1-rev0000.jar, you are running the outdated eXist-db 2.1 and should consider updating to eXist-db 2.2.

As an alternative to the export/import for all data and users listed below:
If you are running with a couple of users, no SNOMED CT data and very little project data, it might be easier to create a full database export (see below), install eXist-db 2.2 as explained here: [[ART-DECOR_installation_on_linux|ART-DECOR installation on linux]], recreate the users as explained in the [[ART_maintenance_manual#Create_users|Maintenance manual/create users]], import your projects into /db/apps/decor/data (use dashboard/Collections/upload resources) and run the [[Fix_permissions_xquery|fix permissions xquery]].

The procedure the export/import all data and users is listed here.

Please note that the full terminology package '''SNOMED CT data''' is not supplied from our repository, due to licensing issues.
We do supply an empty stub package '''Terminology Data - SNOMED CT''' version 14.1.1, but this essentially contains no SNOMED CT, but just serves as an example on how to host this content.
Please skip the steps regarding SNOMED if you do not have the full terminology package '''SNOMED CT data''' installed.

==On the old eXist-db 2.1 instance==
Perform the actions in this section on the old eXist-db 2.1 version to create a full backup and export user information.

===Create a full database export===
Create a full database export.
See [[Create_database_backup|Create database backup]].

===Exporting usergroup information, user passwords and meta-information===
When exporting/importing a full exist-db database dump, user groups are not correctly handled.
For that reason we first export user group information.
To get all user groups, (hashed) user passwords and meta-information for each user:
[[Get_account_data_xquery|Get account data xquery]]

===Stop eXist-db===
To stop the server perform the following actions:
* Log in with SSH
* Stop the old exist-db server
<syntaxhighlight lang="bash">
sudo service exist_atp stop
</syntaxhighlight>

==Install the new eXist-db instance==
Perform the actions in this section to install eXist-db 2.2.

Start with the steps explained here:
[[ART-DECOR_installation_on_linux|ART-DECOR installation on linux]]
and stop at: '''eXist-db logging''', so you don't install ART-DECOR software & data because we first will import the data from the eXist-db 2.1 backup.

===Restore from the eXist-db 2.1 full backup===
===Modify the backup data before importing it===
eXist-db 2.2 comes with preinstalled applications: '''dashboard''', '''eXide''' and '''shared-resources'''.
We will remove these applications from the eXist-db 2.1 backup, to not overwrite the 2.2 versions.
Also we will remove the data from the package '''SNOMED CT data''' because restoring this data does not work properly (so we will deinstall/install it later on).
At the end of these steps we have a directory with unzipped database data with some parts deleted.

* Locate the eXist-db 2.1 full backup on the old server (of from the old eXist-db 2.1 instance):
/usr/local/exist_atp/webapp/WEB-INF/data/export/full-<datetime>.zip
* Log into the new server with SSH
* Copy the full backup to a temporary location and modify it:
<syntaxhighlight lang="bash" enclose="div">
# copy the full backup to your home directory in a directory like 20151031_exist_data
cp /usr/local/exist_atp/webapp/WEB-INF/data/export/full20151030-0323.zip ~/$(date '+%Y%m%d')_exist_data
# chown the zipped data
sudo chown $USER:$USER full20151030-0935.zip
# unzip the backup
unzip full*
# change directory
cd ./db
# remove dashboard, eXide and shared-resources applications
# first edit the contents file which contains information on the exported data:
vi __contents__.xml
# remove the following lines:
<subcollection name="dashboard" filename="dashboard"/>
<subcollection name="eXide" filename="eXide"/>
<subcollection name="shared-resources" filename="shared-resources"/>
# exist vi and save the file:
<ESC>:wq
# remove the data folders for the dashboard, eXide and shared-resources applications
rm eXide/ -Rf
rm dashboard/ -Rf
rm shared-resources/ -Rf
# remove the data from the package SNOMED CT data, skip this step if you do not have this package installed
# change directory
cd apps/terminology-data/snomed-data
# first edit the contents file which contains information on the exported data:
vi __contents__.xml
# Change these lines:
<subcollection name="meta" filename="meta"/>
<subcollection name="Delta" filename="Delta"/>
<subcollection name="en-GB" filename="en-GB"/>
# into:
<subcollection name="meta" filename="meta"/>
# remove the data collections for SNOMED CT data
rm -Rf Delta/
rm -Rf en-GB/
# note that we leave the meta collection intact.
</syntaxhighlight>

===Importing the modified backup data ===
In this step we will import the modified backup data from eXist 2.1 into eXist-db 2.2.
We need root permissions to restore data into the server, so we use sudo.

(optional) we use the linux application '''screen''' to continue the backup process if the SSH-session disconnects.

(optional) we use the linux application '''script''' to record any errors.

<syntaxhighlight lang="bash">
# Start a screen session
sudo yum install screen
screen
# change directory
cd /usr/local/exist_atp/
# Start a script session
sudo script $(date '+%Y%m%d%H')_log.txt
# restore data into eXist-db 2.2. Use the admin account and password. Replace the location for the backup with your own useraccount.
bin/backup.sh -u admin -p <pass> -r /home/user/$(date '+%Y%m%d')_exist_data/db/__contents__.xml -ouri=xmldb:exist://127.0.0.1:8877/xmlrpc
# exit the script session
exit
# output shows:
Script done, file is 201510301913_log.txt
</syntaxhighlight>

Some notes on the backup.sh script:

-u: the user we run the import process as

-r: the location on the filesystem where the backup data is located. We need to provide the location to the db/__contents__.xml file that is located in unzipped backup directory.

-ouri: on which server to perform this action on. We use 127.0.0.1: this linux instance.

* Check for errors during the import phase:
<syntaxhighlight lang="bash">
grep -i 'Error\|warn\|fail' $(date '+%Y%m%d')*_log.txt
</syntaxhighlight>
No output expected.

<syntaxhighlight lang="bash">
grep -v '^Restored\|^Processing\|^Creating' $(date '+%Y%m%d')*_log.txt|less
</syntaxhighlight>
Here we expect output like:
Starting restore of backup...
(possible blank lines)
Finished restore of backup.

<syntaxhighlight lang="bash">
# exit the screen session
exit
# output shows:
[screen is terminating]
</syntaxhighlight>

* Next we need to rebuild the application repository. The reason is that we imported ART-DECOR packages inside /db/apps, rather then installing them, so they don't show up as installed.
By rebuilding the application repository the imported packages also show up as installed:
[[Repair_application_repository_xquery|Repair application repository xquery]]

===Create user groups===
* During the export process the file '''accounts.xml''' with user groups, (hashed) user passwords and meta-information was created.
In the new eXist-db instance we upload this file as: /db/apps/accounts.xml.
* First login at the eXist-db dashboard:
** Open a webbrowser, surf to: http://<server>:8877
** Click on ‘dashboard’.
** Log in as ‘admin’.

Now upload the resource by performing the following steps:
* Click '''Collections'''
* Doubleclick on '''apps''' the enter the apps collection
* Click '''upload resources'''
* Select the file from the local filesystem
* After uploading the resource '''accounts.xml''' should be listed in /db/apps

* Next run the xquery to import user group information per user:
[[Create_account_data_xquery|Create account data xquery]]

Delete the resource: /db/apps/accounts.xml:
* First login at the eXist-db dashboard (see above)
* Click '''Collections'''
* Doubleclick on '''apps''' the enter the apps collection
* Click '''accounts.xml'''
* Click '''delete resources'''
* After deleting the resource '''accounts.xml''' should no longer be listed in /db/apps

===Deinstall, then install SNOMED CT data===
* Deinstall '''SNOMED CT data'''
Skip this step if you did not have this package installed.
Since we deleted the data from this package when we modified backup data, we first do a deinstall for this package and then install a full version with data.

To deinstall the package '''SNOMED CT data''' on the server:
* First login at the eXist-db dashboard (see above)
* Open '''Package Manager'''.
* Click '''installed'''
* Find the package listed as '''SNOMED CT Data'''
* Hover over it and click '''uninstall'''
[[File:Package_manager_deinstall_snomed.jpg|750px]]

Install the package '''SNOMED CT data''

Please note that during installation of this package, a full (re)index is done on the SNOMED CT data that typically takes about one hour during which the server is unavailable to users.
We install the SNOMED CT data package with the following steps:
* First login at the eXist-db dashboard (see above)
* Open '''Package Manager'''.
* Click '''installed'''
* Select the plus icon '''add a package'''
* Select the file from the local filesystem. Note that the installation process will start immediatly after selecting the file.

@@ Line 74: / Line 74: @@
 # remove the data from the package SNOMED CT data, skip this step if you do not have this package installed
 # change directory
-cd apps/terminology-data/snomed-data
+cd terminology-data/snomed-data
 # first edit the contents file which contains information on the exported data:
 vi __contents__.xml

@@ Line 66: / Line 66: @@
    <subcollection name="dashboard" filename="dashboard"/>
    <subcollection name="eXide" filename="eXide"/>
-   <subcollection name="eXide" filename="monex"/>
+   <subcollection name="monex" filename="monex"/>
    <subcollection name="shared-resources" filename="shared-resources"/>
 # exist vi and save the file:

@@ Line 70: / Line 70: @@
 <ESC>:wq
 # remove the data folders for the dashboard, eXide and shared-resources applications
-rm eXide/ -Rf
+rm -Rf eXide dashboard shared-resources
 # remove the data from the package SNOMED CT data, skip this step if you do not have this package installed
 # change directory
@@ Line 85: / Line 83: @@
    <subcollection name="meta" filename="meta"/>
 # remove the data collections for SNOMED CT data
-rm -Rf Delta/
+rm -Rf Delta en-GB
 # note that we leave the meta collection intact.
 </syntaxhighlight>

@@ Line 59: / Line 59: @@
 unzip full*
 # change directory
-cd ./db
+cd ./db/apps
 # remove dashboard, eXide and shared-resources applications
 # first edit the contents file which contains information on the exported data: