Administration

From reBiND Documentation
Revision as of 11:48, 30 October 2014 by LornaMorris (talk | contribs) (Modifying the correction configuration file)
Jump to: navigation, search

Administration

An XML document can be uploaded to the eXist database even if the document is not valid according to the schema used. However it is mandatory that the document is well formed, otherwise errors will occur when trying to store the document.

But before the documents can be stored, a closer look at the collection structure within eXist is needed. The reBiND Framework depends on two collections for managing the different data projects. One of the collections is for the unpublished data projects, which are still being corrected, reviewed or otherwise prepared for publication. The other collection is for the published projects, which can be publicly searched and accessed. Within this document they will be referred to as unpublished and published. Both of these collections are located in the root collection of eXist. Within these collections are the collection for the individual data projects. Having the two collections for unpublished and published data projects makes the security configuration also quite easy, since the security settings for these two collections are automatically inherited by data projects within these.

Instructions on how to create projects via the reBiND user interface have been desctibed in the [Data_upload_to_rebind_framework data archiving section].

The collection structure of eXist, showing the unpublished and published project collections is shown below:

/db/
 ├─ (eXist default)
 ├─ unpublished/
 │   ├─ (data-project-name 1)/
 │   │   ├─ data.xml
 │   │   ├─ metadata.xml
 │   │   ├─ original-data.xls
 │   │   └─ images/
 │   │       ├─ image_001.jpg
 │   │       ├─ image_002.jpg
 │   │       └─ ...
 │   └─ (data-project-name 2)/
 │       └─ ...
 └─ published/
     └─ (data-project-name 3)/
         ├─ data.xml
         ├─ metadata.xml
         ├─ original-data.xls
         ├─ images/
         │   ├─ image_001.jpg
         │   ├─ image_002.jpg
         │   └─ ...
         └─ multimedia/
             ├─ movie1.mov
             ├─ movie2.avi
             └─ ...

The majority of the database administration can be done through the eXist Java webstart client. For example, the client can be used to add new user accounts, modify permissions to different database collections and delete/edit/query the data.

User management

Users can be created via the eXist Java admin client. For example creating a new user for the contributing scientist is shown in the screenshot below.

Create exist user.PNG

In the above example the 'contributing scientist' is allocated to the group 'users', so the /db/unpublished/ directory within the eXist database should be made accessible to the group 'users'.

Correction Manager

Though all of the corrections and modifications to the data document could be done using XQuery and the XQuery Update Facility, it was decided to not have the corrections run in XQuery directly. The Correction Manager is written in Java. It is only loosely coupled to eXist in order to make the Framework more modular. Instead of a document being directly accessed within eXist by the Correction Manager, it will be exported by the custom XQuery function to a regular XML file on the file system of the server. This file is then handed over to the Correction Manager. The source code for the eXist module that interacts with the Correction Manager is available from our subversion repository.

The actual corrections are not done by the Correction Manager, but by individual Correction Modules which are managed by the Correction Manager. The source code for the Correction Manager and correction modules is available from our subversion repository.

Writing Custom Correction Modules

A Correction Module is a Java Class which implements a specific Java Interface. It is possible to extend the correction manager by adding a new module (this should implement the class Module. A new module can be added to take care of a specific problem for the XML documents of the particular reBiND Instance, by writing a new class implementing the interface and putting the compiled class into a specific folder. The eXist server should be restarted but does not have to be rebuilt, which makes it very easy to add new correction modules. This was also one of the reasons why the corrections are done in Java code and not in XQuery. Also Java is much more widely used and understood than XQuery. Furthermore the Correction Manager could be used in stand-alone mode, either by creating an independent GUI tool or by calling the appropriate correction functions via the Command Line. This would require the ABCD data file and the configuration file (specifiying the correction modules to be run) to be specified and then calling the startCorrection method in the CorrectionManager class.

Modifying the correction configuration file

Modules for the XML Correction Manager

When the user starts the correction of an XML file, a specific correction configuration has to be selected. The web interface of the Correction Manager will offer a list of all correction config files in the correction config directory. A correction config file, is an XML file which is valid to a specific schema. It specifies what correction modules will be called, in what order and with what parameters. The structure of such a file is relatively simple:

<modules xmlns="http://rebind.bgbm.org/modules" name="configuration-name">
    <module name="module-name" description="module-description">
        <setting name="module-setting-name" value="module-setting-value"/>
    </module>
    <setting name="general-setting-name" value="general-setting-value"/>
</modules>

The root element is modules. The name attribute is optional. It is the name with which the module will be displayed in the web interface of the Correction Manager.

There can be several module elements within the element modules.

Each module element must have a name attribute which specifies the name of the module to be loaded. This should either be the complete name of a Java class which implements the Module interface or the name specified within the method getName() of that class.

The description is optional and is used to distinguish different instances of the same module which are run with different settings.

Each module element can have any number of setting elements. Each setting element has mandatory attributes for the setting name and value. What settings are used for each module is specified in the module descriptions below.

The modules element can also have setting elements. These general settings are also accessible by the module and are overwritten if a setting with the same name is specified in the module element.

ElementTextReplacer

What it does: Replaces the text content of specific elements according to specific rules.

Full Name: org.bgbm.rebind.correction.modules.ElementTextReplacer

Settings:

address
The name (including element prefix) of the element whose text should be replaced. Or an XPath expression pointing to the element. If the value is interpreted as a name or as an XPath depends on the attribute isXPath.
Mandatory: yes
Example Values: abcd:Sex or //abcd:RecordBasis[matches(.,'^Specimen$')]
isXPath
A flag indicating if the address element contains an XPath expression or just the name of an element.
Mandatory: no
Default Value: false
Allowed Values: true or false
key
The part of the content that should be replaced. This could either be plain text or a RegEx, depending on the attribute isRegEx. Regardless whether it is plain text or an attribute, it could several keys to be replaced or just one, depending on the attribute isBatch. If the batch mode is used, the character or string with which the different parts are separated can be specified in the attribute splitter.
Mandatory: yes
Example Values: Hello World (plain text), (H[ea]llo World)(\!?) (RegEx), Hello World;Lorem Ipsum (plain text, Batch mode with ';' as splitter), H[ea]llo World\!?;[Ll]orem [iI]psum (RegEx, Batch mode with ';' as splitter),
value
The new content with which the content specified in key will be replaced. This could either be plain text or a RegEx, depending on the attribute isRegEx. Regardless whether it is plain text or an attribute, it could several keys to be replaced or just one, depending on the attribute isBatch. If the batch mode is used, the character or string with which the different parts are separated can be specified in the attribute splitter. If the batch mode is used, the key fragments will be replaced by the corresponding value fragments (e.g. the third key fragment will be replaced by third value fragment). Therefore the number of fragments must be the same for key and value, otherwise the replacement will stop after the number of fragments in the smaller one.
Mandatory: yes
Example Values: Hello World again (plain text), $1 again $2 (RegEx), Hello World again;Lorem ipsum dolor sit amet (plain text, Batch mode with ';' as splitter), $1 again $2;$& dolor sit amet (RegEx, Batch mode with ';' as splitter),
isRegEx
A flag indicating if the key and the value elements are regular expressions or just plain text.
Mandatory: no
Default Value: false
Allowed Values: true or false
isBatch
A flag indicating if the key and the value elements contain just one fragment which is supposed to be replaced, or several. If it is true, the character or string with which the different fragments of key and the value elements are separated can be specified in the attribute splitter.
Mandatory: no
Default Value: false
Allowed Values: true or false
splitter
The character or string with which the key and the value elements are broken into their fragments, if they are in batch mode. The splitting is done using the function String.split(String), which interprets the parameter string as a regular expression. This could cause errors when the splitter contains characters with syntactical meaning in RegEx, like <setting name="splitter" value="."/> which would cause any character to be matched and therefor only returning empty fragments.
Mandatory: no
Default Value: ;
Example Values: , or \.

Examples:

    <module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="replaces 'Specimen'">
        <setting name="address" value="//abcd:RecordBasis[matches(.,'^Specimen$')]"/>
        <setting name="isXPath" value="true"/>
        <setting name="key" value="^(Specimen)$"/>
        <setting name="value" value="Preserved$1"/>
        <setting name="isRegEx" value="true"/>
        <setting name="isBatch" value="false"/>
        <setting name="splitter" value=";"/>
    </module>
    <module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="corrects abcd:Sex">
        <setting name="address" value="abcd:Sex"/>
        <setting name="isXPath" value="false"/>
        <setting name="key" value="female;male;hermaphrodite"/>
        <setting name="value" value="F;M;X"/>
        <setting name="isRegEx" value="false"/>
        <setting name="isBatch" value="true"/>
        <setting name="splitter" value=";"/>
    </module>
    <module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="corrects abcd:Rank">
        <setting name="address" value="abcd:Rank"/>
        <setting name="isXPath" value="false"/>
        <setting name="key" value="[f.];[subvar.];[var.]"/>
        <setting name="value" value="f.;subvar.;var."/>
        <setting name="isRegEx" value="false"/>
        <setting name="isBatch" value="true"/>
        <setting name="splitter" value=";"/>
    </module>
    <module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="corrects abcd:Rank">
        <setting name="address" value="//abcd:Rank[matches(.,'^(f|var)$')]"/>
        <setting name="isXPath" value="true"/>
        <setting name="key" value="^f$;^var$"/>
        <setting name="value" value="f.;var."/>
        <setting name="isRegEx" value="true"/>
        <setting name="isBatch" value="true"/>
        <setting name="splitter" value=";"/>
    </module>


EmptyElementDeleter

What it does: Removes empty elements which have neither text content (except white spaces) nor child elements nor attributes. Currently there is a hardcoded exception regarding the attributes. If the only attribute is abcd:language then the element will be deleted as well. Such exceptions will be adjustable via the settings in the future.

Full Name: org.bgbm.rebind.correction.modules.EmptyElementDeleter

Settings: none

Example:

    <module name="org.bgbm.rebind.correction.modules.EmptyElementDeleter" description="first iteration"/>


ElementDeleter

What it does: Deletes specific elements including all its content and child elements.

Full Name: org.bgbm.rebind.correction.modules.ElementDeleter

Settings:

xpath
The XPath address of the element(s) to be removed. Also works for attributes.
Mandatory: yes
Example Values: //abcd:LogoURI or //abcd:TelephoneNumber[abcd:Device="Fax"]

Examples:

    <module name="org.bgbm.rebind.correction.modules.ElementDeleter" description="delete abcd:language attributes">
        <setting name="xpath" value="//*/@abcd:language"/>
    </module>


ElementRenamer

What it does:

Full Name: org.bgbm.rebind.correction.modules.ElementRenamer

Settings:

xpath
The XPath address of the element(s) to be renamed. Also works for attributes.
Mandatory: yes
Example Values: //abcd:LogoURI or //abcd:TelephoneNumber[abcd:Device="Fax"]
newName
The new name of the element, without namepsace prefix.
Mandatory: yes
Example Values: newElementName
useOldNamespace
A flag indicating if the old namespace (and namespace prefix) of the element should be used after renaming as well.
Mandatory: no
Default Value: true
Allowed Values: true or false
newNamespace
The namespace url of the new namespace, if useOldNamespace is set to false.
Mandatory: no
Default Value: (empty string)
Example Values: http://example.com/ns/xyz
newNamespacePrefix
The namespace prefix of the new namespace, if useOldNamespace is set to false. If the colon at the end is missing, it will be added automatically.
Mandatory: no
Default Value: (empty string)
Example Values: xyz:

Examples:

    <module name="org.bgbm.rebind.correction.modules.ElementRenamer" description="rename abcd:language attributes">
        <setting name="xpath" value="//*/@abcd:language"/>
        <setting name="newName" value="language"/>
        <setting name="useOldNamespace" value="false"/>
        <setting name="newNamespace" value=""/>
        <setting name="newNamespacePrefix" value=""/>
    </module>
    <module name="org.bgbm.rebind.correction.modules.ElementRenamer" description="rename wrong ISO Dates">
        <setting name="xpath" value="//abcd:ISODateTimeBegin[not(matches(.,'^(\d\d\d\d(\-(0[1-9]|1[012])(\-((0[1-9])|1\d|2\d|3[01])(T(0\d|1\d|2[0-3])(:[0-5]\d){0,2})?)?)?|\-\-(0[1-9]|1[012])(\-(0[1-9]|1\d|2\d|3[01]))?|\-\-\-(0[1-9]|1\d|2\d|3[01]))$'))]"/>
        <setting name="newName" value="DateText"/>
        <setting name="useOldNamespace" value="true"/>
    </module>


DummyModule

What it does: Waits 1-11 seconds before returning a quote from either the homicidal computer HAL 9000 from the movie "2001: A Space Odyssey" or the maniacally depressed robot Marvin from the book/movie "The Hitchhiker's Guide to the Galaxy". This module does not alter the XML code in any way, it only sends the quote back to the Correction Manager. It is only used for testing purposes.

Full Name: org.bgbm.rebind.correction.modules.DummyModule

Settings: none

Examples:

    <module name="org.bgbm.rebind.correction.modules.DummyModule" description="just wait a bit for a snappy robot remark" />
    <module name="org.bgbm.rebind.correction.modules.DummyModule" description="Play it once again, Marvin, for old times' sake." />


Work in Progress

These modules are currently in the making. So some of these descriptions might not reflect the current state of development.

ABCDDateCorrector

What it does: Checks the dates in the elements abcd:ISODateTimeBegin within abcd:Date and abcd:DateTime. If they are not formatted according to the ISO norm it tries to parse and fix them or renames the element to abcd:DateText. However if there are any abcd:DateText elements which are correctly formatted or can be converted, it will make them into abcd:ISODateTimeBegin elements.

Full Name: org.bgbm.rebind.correction.modules.ABCDDateCorrector

Settings: none

Examples:

    <module name="org.bgbm.rebind.correction.modules.ABCDDateCorrector" description="fixing the dates" />


SimpleCountryCodeChecker

What it does: Compares the content of the abcd:ISO3166Code element with a list of Country Codes provided by Java and warns if it doesn't occur there. Also has some hardcoded exceptions for the commonly used but unspecified codes:

ZZ	Unknown
XA	Unknown or unspecified Africa
XB	Unknown or unspecified Middle and South America
XC	Unknown or unspecified Asia
XD	Unknown or unspecified Australia and Oceania
XE	Unknown or unspecified Europe
XF	Unknown or unspecified North America

Full Name: org.bgbm.rebind.correction.modules.SimpleCountryCodeChecker

Settings: none

Examples:

    <module name="org.bgbm.rebind.correction.modules.SimpleCountryCodeChecker" description="checking the country codes" />

Correction Modules