reBiND Documentation - User contributions [en]

Reading 5.25" Floppy Disks

2014-05-08T09:57:27Z

DavidFichtmueller: /* Reading the data from the floppy */ added precedure to mount USB flash drive

This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI).

== Setting up the drive and the computer ==

1. Open the computer and position the drive in an empty slot in the drive rack. You might need to remove a small metal sheet and the corresponding plastic cover from the case.
[[File:Comparison_Connector.jpg|thumb|right|500px]]
2. Connect the drive to the main board using the connector cable. The cable itself can only be connected in one way to the drive because of the plastic division bar in the connector. However, it might be possible to connect the cable in the wrong position to the motherboard.

: The image shows the ends of two different floppy drive connectors. These ends are used to connect the cable to the motherboard. One of the connectors has a plastic knob on the long side, whereas the other one has none. This means that the connector on the right could be plugged in wrongly. If the connector doesn't have a plastic knob, but the socket has a notch for connectors that do have one, then the connector has to be plugged in with the two dents (which are present at both models) facing the side of the notch.

: '''Warning:''': Connecting the cable in the wrong way can destroy the floppy disk and make it unreadable and unusable.
3. Connect the electricity cable to the drive.

4. Close the computer case, connect all the necessary cables, including a network cable (to load the rescued files to a different computer later on) and start the computer.

5. Setting the drive in the BIOS of the computer. This step greatly depends on the manufacturer and the model of the computer. The BIOS can usually be entered by either pressing "Del", "Esc", or one of the function keys (F1-F12). The new floppy drive has to be selected as one of the floppy drives of this computer. It is important to note whether it is selected as the primary or secondary floppy drive, as this might has an influence on the later steps. For this tutorial the 5.25" floppy drive was selected as the secondary floppy drive, since the already present 3.5" floppy drive remains selected as the primary floppy drive.

6. The following steps for reading the data from the floppy drive where done using a Linux distribution called [http://www.damnsmalllinux.org/ Damn Small Linux (DSL)]. It is a small Linux system which can be run from a CD without installing any additional software. The following steps can however also be done using any other Linux system, regardless of whether it is run from a Live-CD or installed).

7. Once Linux has booted, the user should open the Linux shell and create a directory "data" within the current home directory. The shell shortcut for the current home directory is <code>~</code> and on DSL the home directory is <code>/home/dsl/</code>. The command to create that directory would therefore be:

: <pre>mkdir /home/dsl/data</pre>

8. If it doesn't already exist, one should also create a directory in which the floppy drive can be mounted. On DSL the directory <code>/mnt/floppy</code> already exists, but any other empty directory will work as well, e.g. <code>/home/dsl/floppy</code>. So if one wants to use a different directory or the default directory doesn't exist, one can create a new directory analog to previous command.

== Reading the data from the floppy ==
9. Mount the floppy drive.
: <pre>sudo mount -r -t msdos /dev/fd1 /mnt/floppy</pre>

::* <code>'''sudo'''</code> means that the command will be executed as the superuser.
::* <code>'''mount'''</code> is the actual mount command.
::* <code>'''-r'''</code> means that the drive is read-only.
::* <code>'''-t msdos'''</code> means that the floppy drive is DOS formatted. The alternative would be <code>-f ext2</code> for Linux formatted floppies.
::* <code>'''/dev/fd1'''</code> is the floppy device, if the floppy drive is selected as the secondary floppy drive. If the floppy drive is selected as the primary floppy drive, one has to use <code>/dev/fd0</code>.
::* <code>'''/mnt/floppy'''</code> is the directory to which the floppy disk will be mounted.

10. Go to the directory where the floppy is mounted
: <pre>cd /mnt/floppy</pre>

11. Show the content of the floppy
: <pre>ls -l</pre>

12. Copy the content of the floppy to the <code>data</code> directory created previously.
: <pre>cp -r -p * /home/dsl/data</pre>
::* <code>'''-r'''</code> is for the recursive mode, so subdirectories will be copied as well
::* <code>'''-p'''</code> is for the preserve mode, which means that the timestamps, ownership status and access rights flags of the files will be preserved
::* <code>'''*'''</code> means that all the files will be copied

13. Go to the home directory
: <pre>cd /home/dsl/</pre>

14. Create a zip file of the <code>data</code> data directory
: <pre>zip -r data01.zip /home/dsl/data</pre>

15. If further floppies need to be read, then remove all the files in the data directory
: <pre>rm -fr /home/dsl/data/*</pre>

16. Unmount the floppy drive.
: <pre>sudo umount /dev/fd1</pre>

In order to process further floppies, repeat the steps 9 to 16 and rename the zip file in step 14 accordingly.

17. There are several ways to get the zip-files with the rescued data to another computer.
One could save the files on a flash drive or on the regular hard drive of the computer used (in both cases the drives need to be mounted with writing permission first).
* 17.1 Plug in the USB flash drive
* 17.2 figure out the adress of the drive, using the <code>dmesg</code> command
: <pre>dmesg</pre>
: at the end of the print out, there should be the log of the operating system recognizing the flash drive, which should look something like this:
<pre>usb 1-5: new high speed USB device using ehci_hcd and address 2
scsi0 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 2
usb-storage: waiting for device to settle before scanning
Vendor: USB 2.0 Model: USB Flash Drive Rev: 1100
Type: Direct-Access ANSI SCSI revision: 04
SCSI device sda: 7975296 512-byte hdwr sectors (4083 MB)
sda: Write Protect is off
sda: Mode Sense: 43 00 00 00
sda: assuming drive cache: write through
SCSI device sda: 7975296 512-byte hdwr sectors (4083 MB)
sda: Write Protect is off
sda: Mode Sense: 43 00 00 00
sda: assuming drive cache: write through
sda: sda1
Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0
usb-storage: device scan complete</pre>
: in this case, the drive is attached to the block address '''<code>sda1</code>'''

* 17.4 create a directory to mount the USB drive to
: <pre>sudo mkdir /mnt/usb</pre>
* 17.5 mount the usb drive
: <pre>sudo mount -t vfat /dev/sda1 /mnt/usb</pre>
* 17.6 copy the data to the flash drive
: <pre>cp data*.zip /mnt/usb/</pre>
* 17.7 unmount the flash drive
: <pre>sudo umount /dev/sda1</pre>

* An alternative would be to upload the data to another computer via the LAN or the Internet. This could for example be done via FTP. The easiest way however will probably be just to upload the files via a web form to a web server from where the data can be retrieved later. This could be a specifically designed page for that purpose or a third party site, like webmail providers or file storage sites.

'''Warning:''': Once the Linux running from a Live-CD is shut down, all the data of the current session (including the data within the home directory) is lost unless it has been saved to a specified permanent storage drive (e.g. a flash drive or the regular hard drive of the current computer). By default running Linux from a Live-CD will not alter the host system. So do not shut the system down, unless the data has been copied or otherwise saved.

Reading 5.25" Floppy Disks

2014-05-07T07:45:08Z

DavidFichtmueller: /* Reading the data from the floppy */ command: this time it is correct

This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI).

== Setting up the drive and the computer ==

1. Open the computer and position the drive in an empty slot in the drive rack. You might need to remove a small metal sheet and the corresponding plastic cover from the case.
[[File:Comparison_Connector.jpg|thumb|right|500px]]
2. Connect the drive to the main board using the connector cable. The cable itself can only be connected in one way to the drive because of the plastic division bar in the connector. However, it might be possible to connect the cable in the wrong position to the motherboard.

: The image shows the ends of two different floppy drive connectors. These ends are used to connect the cable to the motherboard. One of the connectors has a plastic knob on the long side, whereas the other one has none. This means that the connector on the right could be plugged in wrongly. If the connector doesn't have a plastic knob, but the socket has a notch for connectors that do have one, then the connector has to be plugged in with the two dents (which are present at both models) facing the side of the notch.

: '''Warning:''': Connecting the cable in the wrong way can destroy the floppy disk and make it unreadable and unusable.
3. Connect the electricity cable to the drive.

4. Close the computer case, connect all the necessary cables, including a network cable (to load the rescued files to a different computer later on) and start the computer.

5. Setting the drive in the BIOS of the computer. This step greatly depends on the manufacturer and the model of the computer. The BIOS can usually be entered by either pressing "Del", "Esc", or one of the function keys (F1-F12). The new floppy drive has to be selected as one of the floppy drives of this computer. It is important to note whether it is selected as the primary or secondary floppy drive, as this might has an influence on the later steps. For this tutorial the 5.25" floppy drive was selected as the secondary floppy drive, since the already present 3.5" floppy drive remains selected as the primary floppy drive.

6. The following steps for reading the data from the floppy drive where done using a Linux distribution called [http://www.damnsmalllinux.org/ Damn Small Linux (DSL)]. It is a small Linux system which can be run from a CD without installing any additional software. The following steps can however also be done using any other Linux system, regardless of whether it is run from a Live-CD or installed).

7. Once Linux has booted, the user should open the Linux shell and create a directory "data" within the current home directory. The shell shortcut for the current home directory is <code>~</code> and on DSL the home directory is <code>/home/dsl/</code>. The command to create that directory would therefore be:

: <pre>mkdir /home/dsl/data</pre>

8. If it doesn't already exist, one should also create a directory in which the floppy drive can be mounted. On DSL the directory <code>/mnt/floppy</code> already exists, but any other empty directory will work as well, e.g. <code>/home/dsl/floppy</code>. So if one wants to use a different directory or the default directory doesn't exist, one can create a new directory analog to previous command.

== Reading the data from the floppy ==
9. Mount the floppy drive.
: <pre>sudo mount -r -t msdos /dev/fd1 /mnt/floppy</pre>

::* <code>'''sudo'''</code> means that the command will be executed as the superuser.
::* <code>'''mount'''</code> is the actual mount command.
::* <code>'''-r'''</code> means that the drive is read-only.
::* <code>'''-t msdos'''</code> means that the floppy drive is DOS formatted. The alternative would be <code>-f ext2</code> for Linux formatted floppies.
::* <code>'''/dev/fd1'''</code> is the floppy device, if the floppy drive is selected as the secondary floppy drive. If the floppy drive is selected as the primary floppy drive, one has to use <code>/dev/fd0</code>.
::* <code>'''/mnt/floppy'''</code> is the directory to which the floppy disk will be mounted.

10. Go to the directory where the floppy is mounted
: <pre>cd /mnt/floppy</pre>

11. Show the content of the floppy
: <pre>ls -l</pre>

12. Copy the content of the floppy to the <code>data</code> directory created previously.
: <pre>cp -r -p * /home/dsl/data</pre>
::* <code>'''-r'''</code> is for the recursive mode, so subdirectories will be copied as well
::* <code>'''-p'''</code> is for the preserve mode, which means that the timestamps, ownership status and access rights flags of the files will be preserved
::* <code>'''*'''</code> means that all the files will be copied

13. Go to the home directory
: <pre>cd /home/dsl/</pre>

14. Create a zip file of the <code>data</code> data directory
: <pre>zip -r data01.zip /home/dsl/data</pre>

15. If further floppies need to be read, then remove all the files in the data directory
: <pre>rm -fr /home/dsl/data/*</pre>

16. Unmount the floppy drive.
: <pre>sudo umount /dev/fd1</pre>

In order to process further floppies, repeat the steps 9 to 16 and rename the zip file in step 14 accordingly.

17. There are several ways to get the zip-files with the rescued data to another computer. One could save the files on a flash drive or on the regular hard drive of the computer used (in both cases the drives need to be mounted with writing permission first). An alternative would be to upload the data to another computer via the LAN or the Internet. This could for example be done via FTP. The easiest way however will probably be just to upload the files via a web form to a web server from where the data can be retrieved later. This could be a specifically designed page for that purpose or a third party site, like webmail providers or file storage sites.

'''Warning:''': Once the Linux running from a Live-CD is shut down, all the data of the current session (including the data within the home directory) is lost unless it has been saved to a specified permanent storage drive (e.g. a flash drive or the regular hard drive of the current computer). By default running Linux from a Live-CD will not alter the host system. So do not shut the system down, unless the data has been copied or otherwise saved.

Reading 5.25" Floppy Disks

2014-05-07T07:43:45Z

DavidFichtmueller: /* Reading the data from the floppy */ command fixed

This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI).

== Setting up the drive and the computer ==

1. Open the computer and position the drive in an empty slot in the drive rack. You might need to remove a small metal sheet and the corresponding plastic cover from the case.
[[File:Comparison_Connector.jpg|thumb|right|500px]]
2. Connect the drive to the main board using the connector cable. The cable itself can only be connected in one way to the drive because of the plastic division bar in the connector. However, it might be possible to connect the cable in the wrong position to the motherboard.

: The image shows the ends of two different floppy drive connectors. These ends are used to connect the cable to the motherboard. One of the connectors has a plastic knob on the long side, whereas the other one has none. This means that the connector on the right could be plugged in wrongly. If the connector doesn't have a plastic knob, but the socket has a notch for connectors that do have one, then the connector has to be plugged in with the two dents (which are present at both models) facing the side of the notch.

: '''Warning:''': Connecting the cable in the wrong way can destroy the floppy disk and make it unreadable and unusable.
3. Connect the electricity cable to the drive.

4. Close the computer case, connect all the necessary cables, including a network cable (to load the rescued files to a different computer later on) and start the computer.

5. Setting the drive in the BIOS of the computer. This step greatly depends on the manufacturer and the model of the computer. The BIOS can usually be entered by either pressing "Del", "Esc", or one of the function keys (F1-F12). The new floppy drive has to be selected as one of the floppy drives of this computer. It is important to note whether it is selected as the primary or secondary floppy drive, as this might has an influence on the later steps. For this tutorial the 5.25" floppy drive was selected as the secondary floppy drive, since the already present 3.5" floppy drive remains selected as the primary floppy drive.

6. The following steps for reading the data from the floppy drive where done using a Linux distribution called [http://www.damnsmalllinux.org/ Damn Small Linux (DSL)]. It is a small Linux system which can be run from a CD without installing any additional software. The following steps can however also be done using any other Linux system, regardless of whether it is run from a Live-CD or installed).

7. Once Linux has booted, the user should open the Linux shell and create a directory "data" within the current home directory. The shell shortcut for the current home directory is <code>~</code> and on DSL the home directory is <code>/home/dsl/</code>. The command to create that directory would therefore be:

: <pre>mkdir /home/dsl/data</pre>

8. If it doesn't already exist, one should also create a directory in which the floppy drive can be mounted. On DSL the directory <code>/mnt/floppy</code> already exists, but any other empty directory will work as well, e.g. <code>/home/dsl/floppy</code>. So if one wants to use a different directory or the default directory doesn't exist, one can create a new directory analog to previous command.

== Reading the data from the floppy ==
9. Mount the floppy drive.
: <pre>sudo mount -r -f msdos /dev/fd1 /mnt/floppy</pre>

::* <code>'''sudo'''</code> means that the command will be executed as the superuser.
::* <code>'''mount'''</code> is the actual mount command.
::* <code>'''-r'''</code> means that the drive is read-only.
::* <code>'''-f msdos'''</code> means that the floppy drive is DOS formatted. The alternative would be <code>-f ext2</code> for Linux formatted floppies.
::* <code>'''/dev/fd1'''</code> is the floppy device, if the floppy drive is selected as the secondary floppy drive. If the floppy drive is selected as the primary floppy drive, one has to use <code>/dev/fd0</code>.
::* <code>'''/mnt/floppy'''</code> is the directory to which the floppy disk will be mounted.

10. Go to the directory where the floppy is mounted
: <pre>cd /mnt/floppy</pre>

11. Show the content of the floppy
: <pre>ls -l</pre>

12. Copy the content of the floppy to the <code>data</code> directory created previously.
: <pre>cp -r -p * /home/dsl/data</pre>
::* <code>'''-r'''</code> is for the recursive mode, so subdirectories will be copied as well
::* <code>'''-p'''</code> is for the preserve mode, which means that the timestamps, ownership status and access rights flags of the files will be preserved
::* <code>'''*'''</code> means that all the files will be copied

13. Go to the home directory
: <pre>cd /home/dsl/</pre>

14. Create a zip file of the <code>data</code> data directory
: <pre>zip -r data01.zip /home/dsl/data</pre>

15. If further floppies need to be read, then remove all the files in the data directory
: <pre>rm -fr /home/dsl/data/*</pre>

16. Unmount the floppy drive.
: <pre>sudo umount /dev/fd1</pre>

In order to process further floppies, repeat the steps 9 to 16 and rename the zip file in step 14 accordingly.

17. There are several ways to get the zip-files with the rescued data to another computer. One could save the files on a flash drive or on the regular hard drive of the computer used (in both cases the drives need to be mounted with writing permission first). An alternative would be to upload the data to another computer via the LAN or the Internet. This could for example be done via FTP. The easiest way however will probably be just to upload the files via a web form to a web server from where the data can be retrieved later. This could be a specifically designed page for that purpose or a third party site, like webmail providers or file storage sites.

'''Warning:''': Once the Linux running from a Live-CD is shut down, all the data of the current session (including the data within the home directory) is lost unless it has been saved to a specified permanent storage drive (e.g. a flash drive or the regular hard drive of the current computer). By default running Linux from a Live-CD will not alter the host system. So do not shut the system down, unless the data has been copied or otherwise saved.

Best Practice Handbook

2012-12-21T14:46:35Z

DavidFichtmueller: a bit more specific on some chapters

==Introduction==
*about reBiND
*how to use this book

==System Installation==
* Installing BioCASe Provider Software
* Installing eXist
* Installing reBiND Framework
* [[ReBiND Framework Configuration Steps|Configuring eXist/reBiND Framework]]
* Using a custom data format
* Using the Correction Tool
** Documentation of [[Correction Modules]]
* Writing Custom Correction Modules
* Using a different metadata format

==Technical Background==
*eXist
*XQuery
*ABCD
*EML

==Data Archiving==
*workflow
** Data Preparation
** XML Export
** Import to reBiND Framework
** Run Corrections
** data review by the scientist
** enter metadata
** publish

==Best Practice Documentation==
* [[Retrieving Geocoordinates From Points In Images]]

==Data Rescue, Preparation==
*data formats
** [[Identifying Unknown File Formats]]
** Conversion for [[File Type Overview|selected file formats]]
*** [[Export DataPerfect|DataPerfect]]
*** [[Export Paradox Data|Paradox]]
*** [[Export dBase Data|dBase]]
*** Harvard Graphics
*** Word 3 for DOS
** Document new data formats for others
* hardware
** 3.5" Floppies
** [[Reading 5.25" Floppy Disks|5.25 " Floppies]]
*preparation

==Case Studies==

== Related Software Products ==
* Data Splitter
* Character Encoding Correcter
* Stand-alone Correction Manager

==Glossary==

==References==

File Type Overview

2012-12-04T13:49:52Z

DavidFichtmueller: Harvard Graphics and Word 4 described

This is a collection of various file types and how they can be treated and opened.
== Harvard Graphics ==
Harvard Graphics used to be the predominant software for creating presentations. The latest version of the software is still sold: http://www.harvardgraphics.com/. On the webpage the company also offers a free tool to open more recent Harvard Graphics files there: [http://www.harvardgraphics.com/hg-viewer-download.asp Harvard Graphics Viewer]. However it only works for some file types.

Files from Harvard Graphics DOS 2.0 and 3.0 can be opened with the original software, which can be downloaded as abandonware at http://vetusware.com/download/Harvard%20Graphics%203.0/?id=4017 and installed using [http://www.dosbox.com/ DOSBOX] (see also the article [[Export_DataPerfect#Install_and_run_DOSBox|Export DataPerfect]]). However, be aware that [[http://en.wikipedia.org/wiki/Abandonware abondonware]] is a legal gray area.

== Word 4 ==
Word 4 (DOS) files can't be opened natively with the current versions of Microsoft Word anymore. However it is possible to import Word 4 files with an additional converter for word.
# Download the file ftp://ftp.microsoft.com/Softlib/MSLFILES/WDSUPCNV.EXE, open it (it is a self extracting zip file) and select a directory to save the files.
# Copy all the resulting *.cnv files (but most importantly <code>Doswrd32.cnv</code>) to <code>C:\Program Files (x86)\Common Files\microsoft shared\TextConv</code> (For users with a 32-bit Windows it is just <code>C:\Program Files\Common Files\microsoft shared\TextConv</code>)
# (Re)Start Microsoft Word
# Open the Word 4 file via the Open-dialog within Word.
# Word will show a prompt informing you that a text converter has to be started and that this might impose a security risk which you should only do if you trust the source where you got the files from. Press OK (if you trust the source of the files).
# Word will most likely show a prompt like "Style Sheet D:/STANDARD.DFV not found". Press Ok. Now a file selector dialog will open asking you to select a style sheet (*.sty) file. If you do have a style sheet for the file (somewhere among the rescued data) then select this. Otherwise create a new empty file in the regular Windows Explorer, rename it "empty.sty" and select it in the file selector. Selecting such an empty file could cause opened file to loose some general properties like print margins etc.
# Now you should be able to see the Word 4 file within the modern Microsoft Word

Reading 5.25" Floppy Disks

2012-12-04T13:01:30Z

DavidFichtmueller: small corrections

This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI).

== Setting up the drive and the computer ==

1. Open the computer and position the drive in an empty slot in the drive rack. You might need to remove a small metal sheet and the corresponding plastic cover from the case.
[[File:Comparison_Connector.jpg|thumb|right|500px]]
2. Connect the drive to the main board using the connector cable. The cable itself can only be connected in one way to the drive because of the plastic division bar in the connector. However, it might be possible to connect the cable in the wrong position to the motherboard.

: The image shows the ends of two different floppy drive connectors. These ends are used to connect the cable to the motherboard. One of the connectors has a plastic knob on the long side, whereas the other one has none. This means that the connector on the right could be plugged in wrongly. If the connector doesn't have a plastic knob, but the socket has a notch for connectors that do have one, then the connector has to be plugged in with the two dents (which are present at both models) facing the side of the notch.

: '''Warning:''': Connecting the cable in the wrong way can destroy the floppy disk and make it unreadable and unusable.
3. Connect the electricity cable to the drive.

4. Close the computer case, connect all the necessary cables, including a network cable (to load the rescued files to a different computer later on) and start the computer.

5. Setting the drive in the BIOS of the computer. This step greatly depends on the manufacturer and the model of the computer. The BIOS can usually be entered by either pressing "Del", "Esc", or one of the function keys (F1-F12). The new floppy drive has to be selected as one of the floppy drives of this computer. It is important to note whether it is selected as the primary or secondary floppy drive, as this might has an influence on the later steps. For this tutorial the 5.25" floppy drive was selected as the secondary floppy drive, since the already present 3.5" floppy drive remains selected as the primary floppy drive.

6. The following steps for reading the data from the floppy drive where done using a Linux distribution called [http://www.damnsmalllinux.org/ Damn Small Linux (DSL)]. It is a small Linux system which can be run from a CD without installing any additional software. The following steps can however also be done using any other Linux system, regardless of whether it is run from a Live-CD or installed).

7. Once Linux has booted, the user should open the Linux shell and create a directory "data" within the current home directory. The shell shortcut for the current home directory is <code>~</code> and on DSL the home directory is <code>/home/dsl/</code>. The command to create that directory would therefore be:

: <pre>mkdir /home/dsl/data</pre>

8. If it doesn't already exist, one should also create a directory in which the floppy drive can be mounted. On DSL the directory <code>/mnt/floppy</code> already exists, but any other empty directory will work as well, e.g. <code>/home/dsl/floppy</code>. So if one wants to use a different directory or the default directory doesn't exist, one can create a new directory analog to previous command.

== Reading the data from the floppy ==
9. Mount the floppy drive.
: <pre>sudo mount -r -t msdos /dev/fd1 /mnt/floppy</pre>

::* <code>'''sudo'''</code> means that the command will be executed as the superuser.
::* <code>'''mount'''</code> is the actual mount command.
::* <code>'''-r'''</code> means that the drive is read-only.
::* <code>'''-f msdos'''</code> means that the floppy drive is DOS formatted. The alternative would be <code>-f ext2</code> for Linux formatted floppies.
::* <code>'''/dev/fd1'''</code> is the floppy device, if the floppy drive is selected as the secondary floppy drive. If the floppy drive is selected as the primary floppy drive, one has to use <code>/dev/fd0</code>.
::* <code>'''/mnt/floppy'''</code> is the directory to which the floppy disk will be mounted.

10. Go to the directory where the floppy is mounted
: <pre>cd /mnt/floppy</pre>

11. Show the content of the floppy
: <pre>ls -l</pre>

12. Copy the content of the floppy to the <code>data</code> directory created previously.
: <pre>cp -r -p * /home/dsl/data</pre>
::* <code>'''-r'''</code> is for the recursive mode, so subdirectories will be copied as well
::* <code>'''-p'''</code> is for the preserve mode, which means that the timestamps, ownership status and access rights flags of the files will be preserved
::* <code>'''*'''</code> means that all the files will be copied

13. Go to the home directory
: <pre>cd /home/dsl/</pre>

14. Create a zip file of the <code>data</code> data directory
: <pre>zip -r data01.zip /home/dsl/data</pre>

15. If further floppies need to be read, then remove all the files in the data directory
: <pre>rm -fr /home/dsl/data/*</pre>

16. Unmount the floppy drive.
: <pre>sudo umount /dev/fd1</pre>

In order to process further floppies, repeat the steps 9 to 16 and rename the zip file in step 14 accordingly.

17. There are several ways to get the zip-files with the rescued data to another computer. One could save the files on a flash drive or on the regular hard drive of the computer used (in both cases the drives need to be mounted with writing permission first). An alternative would be to upload the data to another computer via the LAN or the Internet. This could for example be done via FTP. The easiest way however will probably be just to upload the files via a web form to a web server from where the data can be retrieved later. This could be a specifically designed page for that purpose or a third party site, like webmail providers or file storage sites.

'''Warning:''': Once the Linux running from a Live-CD is shut down, all the data of the current session (including the data within the home directory) is lost unless it has been saved to a specified permanent storage drive (e.g. a flash drive or the regular hard drive of the current computer). By default running Linux from a Live-CD will not alter the host system. So do not shut the system down, unless the data has been copied or otherwise saved.

Reading 5.25" Floppy Disks

2012-12-04T13:00:04Z

DavidFichtmueller: numbering corrected

This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI).

== Setting up the drive and the computer ==

1. Open the computer and position the drive in an empty slot in the drive rack. You might need to remove a small metal sheet and the corresponding plastic cover from the case.
[[File:Comparison_Connector.jpg|thumb|right|500px]]
2. Connect the drive to the main board using the connector cable. The cable itself can only be connected in one way to the drive because of the plastic division bar in the connector. However, it might be possible to connect the cable in the wrong position to the motherboard.

: The image shows the ends of two different floppy drive connectors. These ends are used to connect the cable to the motherboard. One of the connectors has a plastic knob on the long side, whereas the other one has none. This means that the connector on the right could be plugged in wrongly. If the connector doesn't have a plastic knob, but the socket has a notch for connectors that do have one, then the connector has to be plugged in with the two dents (which are present at both models) facing the side of the notch.

: '''Warning:''': Connecting the cable in the wrong way can destroy the floppy disk and make it unreadable and unusable.
3. Connect the electricity cable to the drive.

4. Close the computer case, connect all the necessary cables, including a network cable (to load the rescued files to a different computer later on) and start the computer.

5. Setting the drive in the BIOS of the computer. This step greatly depends on the manufacturer and the model of the computer. The BIOS can usually be entered by either pressing "Del", "Esc", or one of the function keys (F1-F12). The new floppy drive has to be selected as one of the floppy drives of this computer. It is important to note whether it is selected as the primary or secondary floppy drive, as this might has an influence on the later steps. For this tutorial the 5.25" floppy drive was selected as the secondary floppy drive, since the already present 3.5" floppy drive remains selected as the primary floppy drive.

6. The following steps for reading the data from the floppy drive where done using a Linux distribution called [http://www.damnsmalllinux.org/ Damn Small Linux (DSL)]. It is a small Linux system which can be run from a CD without installing any additional software. The following steps can however also be done using any other Linux system, regardless of whether it is run from a Live-CD or installed).

7. Once Linux has booted, the user should open the Linux shell and create a directory "data" within the current home directory. The shell shortcut for the current home directory is <code>~</code> and on DSL the home directory is <code>/home/dsl/</code>. The command to create that directory would therefore be:

: <pre>mkdir /home/dsl/data</pre>

8. If it doesn't already exist, one should also create a directory in which the floppy drive can be mounted. On DSL the directory <code>/mnt/floppy</code> already exists, but any other empty directory will work as well, e.g. <code>/home/dsl/floppy</code>. So if one wants to use a different directory or the default directory doesn't exist, one can create a new directory analog to previous command.

== Reading the data from the floppy ==
9. Next the step would be to mount the floppy drive.
: <pre>sudo mount -r -t msdos /dev/fd1 /mnt/floppy</pre>

::* <code>'''sudo'''</code> means that the command will be executed as the superuser.
::* <code>'''mount'''</code> is the actual mount command.
::* <code>'''-r'''</code> means that the drive is read-only.
::* <code>'''-f msdos'''</code> means that the floppy drive is DOS formatted. The alternative would be <code>-f ext2</code> for Linux formatted floppies.
::* <code>'''/dev/fd1'''</code> is the floppy device, if the floppy drive is selected as the secondary floppy drive. If the floppy drive is selected as the primary floppy drive, one has to use <code>/dev/fd0</code>.
::* <code>'''/mnt/floppy'''</code> is the directory to which the floppy disk will be mounted.

10. Go to the directory where the floppy is mounted
: <pre>cd /mnt/floppy</pre>

11. show the content of the floppy
: <pre>ls -l</pre>

12. copy the content of the floppy to the <code>data</code> directory created previously.
: <pre>cp -r -p * /home/dsl/data</pre>
::* <code>'''-r'''</code> is for the recursive mode, so subdirectories will be copied as well
::* <code>'''-p'''</code> is for the preserve mode, which means that the timestamps, ownership status and access rights flags of the files will be preserved
::* <code>'''*'''</code> means that all the files will be copied

13. go to the home directory
: <pre>cd /home/dsl/</pre>

14. create a zip file of the <code>data</code> data directory
: <pre>zip -r data01.zip /home/dsl/data</pre>

15. If further floppies need to be read, then remove all the files in the data directory
: <pre>rm -fr /home/dsl/data/*</pre>

16. unmount the floppy drive.
: <pre>sudo umount /dev/fd1</pre>

In order to process further floppies, repeat the steps 9 to 16 and rename the zip file in step 14 accordingly.

17. There are several ways to get the zip-files with the rescued data to another computer. One could save the files on a flash drive or on the regular hard drive of the computer used (in both cases the drives need to be mounted with writing permission first). An alternative would be to upload the data to another computer via the LAN or the Internet. This could for example be done via FTP. The easiest way however will probably be just to upload the files via a web form to a web server from where the data can be retrieved later. This could be a specifically designed page for that purpose or a third party site, like webmail providers or file storage sites.

'''Warning:''': Once the Linux running from a Live-CD is shut down, all the data of the current session (including the data within the home directory) is lost unless it has been saved to a specified permanent storage drive (e.g. a flash drive or the regular hard drive of the current computer). By default running Linux from a Live-CD will not alter the host system. So do not shut the system down, unless the data has been copied or otherwise saved.

Reading 5.25" Floppy Disks

2012-12-04T12:59:31Z

DavidFichtmueller: added section: saving the rescued files

This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI).

== Setting up the drive and the computer ==

1. Open the computer and position the drive in an empty slot in the drive rack. You might need to remove a small metal sheet and the corresponding plastic cover from the case.
[[File:Comparison_Connector.jpg|thumb|right|500px]]
2. Connect the drive to the main board using the connector cable. The cable itself can only be connected in one way to the drive because of the plastic division bar in the connector. However, it might be possible to connect the cable in the wrong position to the motherboard.

: The image shows the ends of two different floppy drive connectors. These ends are used to connect the cable to the motherboard. One of the connectors has a plastic knob on the long side, whereas the other one has none. This means that the connector on the right could be plugged in wrongly. If the connector doesn't have a plastic knob, but the socket has a notch for connectors that do have one, then the connector has to be plugged in with the two dents (which are present at both models) facing the side of the notch.

: '''Warning:''': Connecting the cable in the wrong way can destroy the floppy disk and make it unreadable and unusable.
3. Connect the electricity cable to the drive.

4. Close the computer case, connect all the necessary cables, including a network cable (to load the rescued files to a different computer later on) and start the computer.

5. Setting the drive in the BIOS of the computer. This step greatly depends on the manufacturer and the model of the computer. The BIOS can usually be entered by either pressing "Del", "Esc", or one of the function keys (F1-F12). The new floppy drive has to be selected as one of the floppy drives of this computer. It is important to note whether it is selected as the primary or secondary floppy drive, as this might has an influence on the later steps. For this tutorial the 5.25" floppy drive was selected as the secondary floppy drive, since the already present 3.5" floppy drive remains selected as the primary floppy drive.

6. The following steps for reading the data from the floppy drive where done using a Linux distribution called [http://www.damnsmalllinux.org/ Damn Small Linux (DSL)]. It is a small Linux system which can be run from a CD without installing any additional software. The following steps can however also be done using any other Linux system, regardless of whether it is run from a Live-CD or installed).

7. Once Linux has booted, the user should open the Linux shell and create a directory "data" within the current home directory. The shell shortcut for the current home directory is <code>~</code> and on DSL the home directory is <code>/home/dsl/</code>. The command to create that directory would therefore be:

: <pre>mkdir /home/dsl/data</pre>

8. If it doesn't already exist, one should also create a directory in which the floppy drive can be mounted. On DSL the directory <code>/mnt/floppy</code> already exists, but any other empty directory will work as well, e.g. <code>/home/dsl/floppy</code>. So if one wants to use a different directory or the default directory doesn't exist, one can create a new directory analog to previous command.

== Reading the data from the floppy ==
9. Next the step would be to mount the floppy drive.
: <pre>sudo mount -r -t msdos /dev/fd1 /mnt/floppy</pre>

::* <code>'''sudo'''</code> means that the command will be executed as the superuser.
::* <code>'''mount'''</code> is the actual mount command.
::* <code>'''-r'''</code> means that the drive is read-only.
::* <code>'''-f msdos'''</code> means that the floppy drive is DOS formatted. The alternative would be <code>-f ext2</code> for Linux formatted floppies.
::* <code>'''/dev/fd1'''</code> is the floppy device, if the floppy drive is selected as the secondary floppy drive. If the floppy drive is selected as the primary floppy drive, one has to use <code>/dev/fd0</code>.
::* <code>'''/mnt/floppy'''</code> is the directory to which the floppy disk will be mounted.

10. Go to the directory where the floppy is mounted
: <pre>cd /mnt/floppy</pre>

11. show the content of the floppy
: <pre>ls -l</pre>

12. copy the content of the floppy to the <code>data</code> directory created previously.
: <pre>cp -r -p * /home/dsl/data</pre>
::* <code>'''-r'''</code> is for the recursive mode, so subdirectories will be copied as well
::* <code>'''-p'''</code> is for the preserve mode, which means that the timestamps, ownership status and access rights flags of the files will be preserved
::* <code>'''*'''</code> means that all the files will be copied

13. go to the home directory
: <pre>cd /home/dsl/</pre>

14. create a zip file of the <code>data</code> data directory
: <pre>zip -r data01.zip /home/dsl/data</pre>

15. If further floppies need to be read, then remove all the files in the data directory
: <pre>rm -fr /home/dsl/data/*</pre>

16. unmount the floppy drive.
: <pre>sudo umount /dev/fd1</pre>

In order to process further floppies, repeat the steps 8 to 15 and rename the zip file in step 13 accordingly.

17. There are several ways to get the zip-files with the rescued data to another computer. One could save the files on a flash drive or on the regular hard drive of the computer used (in both cases the drives need to be mounted with writing permission first). An alternative would be to upload the data to another computer via the LAN or the Internet. This could for example be done via FTP. The easiest way however will probably be just to upload the files via a web form to a web server from where the data can be retrieved later. This could be a specifically designed page for that purpose or a third party site, like webmail providers or file storage sites.

'''Warning:''': Once the Linux running from a Live-CD is shut down, all the data of the current session (including the data within the home directory) is lost unless it has been saved to a specified permanent storage drive (e.g. a flash drive or the regular hard drive of the current computer). By default running Linux from a Live-CD will not alter the host system. So do not shut the system down, unless the data has been copied or otherwise saved.

Reading 5.25" Floppy Disks

2012-12-04T12:44:05Z

DavidFichtmueller: initial version finished

Reading 5.25" Floppy Disks

2012-12-04T11:04:57Z

DavidFichtmueller: more details

Reading 5.25" Floppy Disks

2012-11-27T16:50:26Z

DavidFichtmueller: Created page with "This article presumes that the reader has a working 5.25" floppy drive, a connector cable and a computer that has a corresponding connector on the motherboard (34 pin ATA/ATAPI)...."

Export dBase Data

2012-11-09T17:04:24Z

DavidFichtmueller: initial version

This article describes how to export data out of a dBase Table File (*.dbf). For details about the file format see [http://www.nationalarchives.gov.uk/pronom/x-fmt/9] and [http://www.digitalpreservation.gov/formats/fdd/fdd000325.shtml].

=== Opening the data with Libre Office ===
It is possible to import *.dbf files into Libre Office (or Open Office) Calc or Base: http://help.libreoffice.org/Calc/Importing_and_Exporting_dBASE_Files . Libre Office will try to recognize the character encoding of the data and preselect this option in the import window. Once a file imported, it can be saved in/exported to a variety of different formats.

=== Importing the data directly into a mySQL database ===
Alternatively the data can directly be imported into a running mySQL database using the linux program <code>[http://manpages.ubuntu.com/manpages/lucid/man1/dbf2mysql.1.html dbf2mysql]</code>. Importing the data directly however could cause some problems with the character encoding for non ASCII characters.

If the mysql server is already running, the following commands will import the data:
<syntaxhighlight lang="bash">
user@linux:~$ mysql -u root -p
mysql> create database database_name;
mysql> exit
user@linux:~$ sudo apt-get install dbf2mysql
user@linux:~$ dbf2mysql -vvv -c -f -d database_name -U root -P password DATA_FILE.DBF
</syntaxhighlight>
The data can then be exported using <code>mysqldump</code>
<syntaxhighlight lang="bash">
user@linux:~$ mysqldump -u root -p database_name > data_export.sql
</syntaxhighlight>

Retrieving Geocoordinates From Points In Images

2012-07-26T13:45:52Z

DavidFichtmueller: Useful links added

= Retrieving Geocoordinates From Points In Images =
This manual will show how to get the geocoordinates from points in an image.

__TOC__

The source image we use, shows six locations in the western part of Nicaragua. Though the image is of poor quality, luckily the lines for latitude and longitude are visible in the picture.
[[File:JOSM Mapping 00 Original Image.png|frame|none|The Source Image with six locations marked with circled dots]]
To start download the program JOSM from [http://josm.openstreetmap.de/ http://josm.openstreetmap.de/]. JOSM is a Java based Editor for [http://www.oepnstreetmap.org OpenStreetMap].
[[File:JOSM Mapping 01 Initial Window.png|frame|none|The Initial Screen of JOSM]]
== Download PicLayer Plugin ==
* Start JOSM
* Press F12 to get to the Preferences Window
* Go to the Plugin-Tab
* Click on '''Download List'''
* Search for '''PicLayer'''
* Check the checkbox for the PicLayer plugin
* Click OK and restart JOSM
[[File:JOSM Mapping 02 Preferences Plugins.png|frame|none|Selecting the PicLayer Plugin]]

== Select the OpenStreetMap Image Layer ==
* Go on '''Imagery''' and select '''OpenStreetMap (Mapnik)'''
* navigate to the area of interest
[[File:JOSM Mapping 03 Area Of Interest.png|frame|none|The area of interest is selected]]

== Draw a reference rectangle ==
''This step can be skipped if the image does not show the lines for latitude and longitude, like the example image above does.''
* Open a new Data Layer by pressing '''Ctrl + N'''
* Insert a new Point by pressing '''Shift + D''' and entering its coordinates and press OK. In this example we start with 13.0 -87.0. Repeat for the other points as well: 12.0 -87,0; 12.0 -86,0 and 13.0 -86,0.

[[File:JOSM Mapping 04 Entering Coordinates.png|frame|none|The window for entering the coordinates.]]
* Switch to the drawing tool by pressing '''A'''
* Connect the four points to a rectangle.
[[File:JOSM Mapping 05 Grid Rectangle.png|frame|none|The rectangle on the map]]

== Loading the reference image ==
* In the Menu go on '''PicLayer''' and select '''New picture layer from file ...'''
* Select the prepared image.
* The Picture Layer may be displayed below the OpenStreetMap Layer. If so, move it up in the Layers Box, so that the image becomes visible.
[[File:JOSM Mapping 06 Image Loaded.png|frame|none|The image loaded as a PicLayer]]
* Also make the Picture Layer the active layer, by clicking on the third button in the Layers Box Toolbar, when the Picture Layer is selected. By doing so, the tools for calibrating the picture will become visible in the toolbar at the left side.
[[File:JOSM Mapping 07 Image Layer Selected.png|frame|none|The image layer is now the selected layer]]

== Calibrating the reference image ==
* Click on the button with the green arrow to add three reference points on the image, each on one of the confluence points (where the latitude and longitude lines cross).
* Click on the button with the red arrow to move the reference points until they align with the corners of the rectangle.
* If you now make the image layer semi-transparent, you can see how the image aligns with the map.
* If you do not have the latitude and longitude lines, you should make the image layer semi transparent and try to align distinct points in the image with the map.

[[File:JOSM Mapping 08 Image Layer Calibrated.png|frame|none|The image is now calibrated. The gray arrows of the rectangle of the data layer are now over the grid lines in the image.]]
[[File:JOSM Mapping 09 Image Layer Semi-Transparent.png|frame|none|Setting the image layer as semi transparent.]]

== Mapping the points in the image ==
* Open a new data layer by pressing '''Ctrl + N'''
* Select the drawing tool by pressing '''A'''
* Draw points on the positions of the image from which you want to know the coordinates.
* If there are several points, you should add a name for each point in the Properties Box on the right.
[[File:JOSM Mapping 10 Points Mapped And Named.png|frame|none|Mapping the points. They are visible as the light small rectangles in the black dots on the image.]]
* After all the points are mapped, right-click on the new data layer and select '''Save As ...''' and save it as XML file on your local file system.
[[File:JOSM Mapping 11 Exporting Points.png|frame|none|Exporting the mapped points.]]
* The file contains the coordinates of the selected points for further processing.
<syntaxhighlight lang="xml">
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' upload='true' generator='JOSM'>
<node id='-95' action='modify' visible='true' lat='12.881180287556537' lon='-86.11194760698338'>
<tag k='name' v='1' />
</node>
<node id='-94' action='modify' visible='true' lat='12.79249813630681' lon='-86.10761639569195'>
<tag k='name' v='2' />
</node>
<node id='-93' action='modify' visible='true' lat='12.403621057420668' lon='-86.22455910056097'>
<tag k='name' v='3' />
</node>
<node id='-92' action='modify' visible='true' lat='12.084582576302516' lon='-86.11790302250911'>
<tag k='name' v='6' />
</node>
<node id='-91' action='modify' visible='true' lat='12.20261349095152' lon='-86.46277572159048'>
<tag k='name' v='5' />
</node>
<node id='-90' action='modify' visible='true' lat='12.40062989709886' lon='-86.83219726377988'>
<tag k='name' v='4' />
</node>
</osm>
</syntaxhighlight>

== Closing JOSM ==
* Close JOSM. '''WARNING''': When you try to close JOSM, it will ask you if you want to upload the two data layers to OpenStreetMap and selects these options for you. '''DO NOT UPLOAD THE DATA!''' Click on '''Exit Now.'''
* JOSM will then ask if the calibration file for the image should be saved. You can do so, if you want to. It will be stored in the same directory as the image. The next time you select the image to be used by PicLayer it will detect this file and put the image in the correct position.

== Useful Links ==
* [http://boulter.com/gps/ GPS Coordinate Converter] Converts coordinates from the various form to various other forms, shows them on the map and links to different maps.
* [http://www.movable-type.co.uk/scripts/latlong-vincenty.html Inverse Vincenty Formula] Calculate the distance between two coordinates
* [http://www.movable-type.co.uk/scripts/latlong-vincenty-direct.html Direct Vincenty Formula] Enter GPS coordinates, a direction and a distance and get the resulting coordinates

Correction Modules

2012-07-26T12:22:12Z

DavidFichtmueller: more modules added

= Modules for the XML Correction Manager =
When the user starts the correction of an XML file, a specific correction configuration has to be selected. The web interface of the Correction Manager will offer a list of all correction config files in the correction config directory. A correction config file, is an XML file which is valid to a specific schema. It specifies what correction modules will be called, in what order and with what parameters. The structure of such a file is relatively simple:
<syntaxhighlight>
<modules xmlns="http://rebind.bgbm.org/modules" name="configuration-name">
<module name="module-name" description="module-description">
<setting name="module-setting-name" value="module-setting-value"/>
</module>
<setting name="general-setting-name" value="general-setting-value"/>
</modules>
</syntaxhighlight>

The root element is <code>modules</code>. The <code>name</code> attribute is optional. It is the name with which the module will be displayed in the web interface of the Correction Manager.

There can be several <code>module</code> elements within the element <code>modules</code>.

Each <code>module</code> element must have a <code>name</code> attribute which specifies the name of the module to be loaded. This should either be the complete name of a Java class which implements the <code>Module</code> interface or the name specified within the method <code>getName()</code> of that class.

The <code>description</code> is optional and is used to distinguish different instances of the same module which are run with different settings.

Each <code>module</code> element can have any number of <code>setting</code> elements. Each <code>setting</code> element has mandatory attributes for the setting <code>name</code> and <code>value</code>. What settings are used for each module is specified in the module descriptions below.

The <code>modules</code> element can also have <code>setting</code> elements. These general settings are also accessible by the module and are overwritten if a setting with the same name is specified in the <code>module</code> element.
__TOC__
== ElementTextReplacer ==
''What it does'': Replaces the text content of specific elements according to specific rules.

''Full Name'': <code>org.bgbm.rebind.correction.modules.ElementTextReplacer</code>

''Settings'':
: '''address'''
:: The name (including element prefix) of the element whose text should be replaced. Or an XPath expression pointing to the element. If the value is interpreted as a name or as an XPath depends on the attribute <code>isXPath</code>.
:: ''Mandatory'': yes
:: ''Example Values'': <code>abcd:Sex</code> or <code>//abcd:RecordBasis[matches(.,'^Specimen$')]</code>

: '''isXPath'''
:: A flag indicating if the <code>address</code> element contains an XPath expression or just the name of an element.
:: ''Mandatory'': no
:: ''Default Value'': <code>false</code>
:: ''Allowed Values'': <code>true</code> or <code>false</code>

: '''key'''
:: The part of the content that should be replaced. This could either be plain text or a RegEx, depending on the attribute <code>isRegEx</code>. Regardless whether it is plain text or an attribute, it could several keys to be replaced or just one, depending on the attribute <code>isBatch</code>. If the batch mode is used, the character or string with which the different parts are separated can be specified in the attribute <code>splitter</code>.
:: ''Mandatory'': yes
:: ''Example Values'': <code>Hello World</code> (plain text), <code>(H[ea]llo World)(\!?)</code> (RegEx), <code>Hello World;Lorem Ipsum</code> (plain text, Batch mode with ';' as splitter), <code>H[ea]llo World\!?;[Ll]orem [iI]psum</code> (RegEx, Batch mode with ';' as splitter),

: '''value'''
:: The new content with which the content specified in <code>key</code> will be replaced. This could either be plain text or a RegEx, depending on the attribute <code>isRegEx</code>. Regardless whether it is plain text or an attribute, it could several keys to be replaced or just one, depending on the attribute <code>isBatch</code>. If the batch mode is used, the character or string with which the different parts are separated can be specified in the attribute <code>splitter</code>. If the batch mode is used, the key fragments will be replaced by the corresponding value fragments (e.g. the third key fragment will be replaced by third value fragment). Therefore the number of fragments must be the same for key and value, otherwise the replacement will stop after the number of fragments in the smaller one.
:: ''Mandatory'': yes
:: ''Example Values'': <code>Hello World again</code> (plain text), <code>$1 again $2</code> (RegEx), <code>Hello World again;Lorem ipsum dolor sit amet</code> (plain text, Batch mode with ';' as splitter), <code>$1 again $2;$& dolor sit amet</code> (RegEx, Batch mode with ';' as splitter),

: '''isRegEx'''
:: A flag indicating if the <code>key</code> and the <code>value</code> elements are regular expressions or just plain text.
:: ''Mandatory'': no
:: ''Default Value'': <code>false</code>
:: ''Allowed Values'': <code>true</code> or <code>false</code>

: '''isBatch'''
:: A flag indicating if the <code>key</code> and the <code>value</code> elements contain just one fragment which is supposed to be replaced, or several. If it is true, the character or string with which the different fragments of <code>key</code> and the <code>value</code> elements are separated can be specified in the attribute <code>splitter</code>.
:: ''Mandatory'': no
:: ''Default Value'': <code>false</code>
:: ''Allowed Values'': <code>true</code> or <code>false</code>

: '''splitter'''
:: The character or string with which the <code>key</code> and the <code>value</code> elements are broken into their fragments, if they are in batch mode. The splitting is done using the function [http://docs.oracle.com/javase/6/docs/api/java/lang/String.html#split%28java.lang.String%29 String.split(String)], which interprets the parameter string as a regular expression. This could cause errors when the splitter contains characters with syntactical meaning in RegEx, like <code><setting name="splitter" value="."/></code> which would cause any character to be matched and therefor only returning empty fragments.
:: ''Mandatory'': no
:: ''Default Value'': <code>;</code>
:: ''Example Values'': <code>,</code> or <code>\.</code>

''Examples'':
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="replaces 'Specimen'">
<setting name="address" value="//abcd:RecordBasis[matches(.,'^Specimen$')]"/>
<setting name="isXPath" value="true"/>
<setting name="key" value="^(Specimen)$"/>
<setting name="value" value="Preserved$1"/>
<setting name="isRegEx" value="true"/>
<setting name="isBatch" value="false"/>
<setting name="splitter" value=";"/>
</module>
<module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="corrects abcd:Sex">
<setting name="address" value="abcd:Sex"/>
<setting name="isXPath" value="false"/>
<setting name="key" value="female;male;hermaphrodite"/>
<setting name="value" value="F;M;X"/>
<setting name="isRegEx" value="false"/>
<setting name="isBatch" value="true"/>
<setting name="splitter" value=";"/>
</module>
<module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="corrects abcd:Rank">
<setting name="address" value="abcd:Rank"/>
<setting name="isXPath" value="false"/>
<setting name="key" value="[f.];[subvar.];[var.]"/>
<setting name="value" value="f.;subvar.;var."/>
<setting name="isRegEx" value="false"/>
<setting name="isBatch" value="true"/>
<setting name="splitter" value=";"/>
</module>
<module name="org.bgbm.rebind.correction.modules.ElementTextReplacer" description="corrects abcd:Rank">
<setting name="address" value="//abcd:Rank[matches(.,'^(f|var)$')]"/>
<setting name="isXPath" value="true"/>
<setting name="key" value="^f$;^var$"/>
<setting name="value" value="f.;var."/>
<setting name="isRegEx" value="true"/>
<setting name="isBatch" value="true"/>
<setting name="splitter" value=";"/>
</module>
</syntaxhighlight>

== EmptyElementDeleter ==
''What it does:'' Removes empty elements which have neither text content (except white spaces) nor child elements nor attributes. Currently there is a hardcoded exception regarding the attributes. If the only attribute is <code>abcd:language</code> then the element will be deleted as well. Such exceptions will be adjustable via the settings in the future.

''Full Name:'' <code>org.bgbm.rebind.correction.modules.EmptyElementDeleter</code>

''Settings:'' none

''Example:''
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.EmptyElementDeleter" description="first iteration"/>
</syntaxhighlight>

== ElementDeleter ==
''What it does:'' Deletes specific elements including all its content and child elements.

''Full Name:'' <code>org.bgbm.rebind.correction.modules.ElementDeleter</code>

''Settings:''
: '''xpath'''
:: The XPath address of the element(s) to be removed. Also works for attributes.
:: ''Mandatory:'' yes
:: ''Example Values:'' <code>//abcd:LogoURI</code> or <code>//abcd:TelephoneNumber[abcd:Device="Fax"]</code>

''Examples:''
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.ElementDeleter" description="delete abcd:language attributes">
<setting name="xpath" value="//*/@abcd:language"/>
</module>
</syntaxhighlight>

== ElementRenamer ==
''What it does:''

''Full Name:'' <code>org.bgbm.rebind.correction.modules.ElementRenamer</code>

''Settings:''
: '''xpath'''
:: The XPath address of the element(s) to be renamed. Also works for attributes.
:: ''Mandatory:'' yes
:: ''Example Values:'' <code>//abcd:LogoURI</code> or <code>//abcd:TelephoneNumber[abcd:Device="Fax"]</code>

: '''newName'''
:: The new name of the element, without namepsace prefix.
:: ''Mandatory:'' yes
:: ''Example Values:'' <code>newElementName</code>

: '''useOldNamespace'''
:: A flag indicating if the old namespace (and namespace prefix) of the element should be used after renaming as well.
:: ''Mandatory:'' no
:: ''Default Value:'' <code>true</code>
:: ''Allowed Values:'' <code>true</code> or <code>false</code>

: '''newNamespace'''
:: The namespace url of the new namespace, if <code>useOldNamespace</code> is set to <code>false</code>.
:: ''Mandatory:'' no
:: ''Default Value:'' ''(empty string)''
:: ''Example Values:'' <code><nowiki>http://example.com/ns/xyz</nowiki></code>

: '''newNamespacePrefix'''
:: The namespace prefix of the new namespace, if <code>useOldNamespace</code> is set to <code>false</code>. If the colon at the end is missing, it will be added automatically.
:: ''Mandatory:'' no
:: ''Default Value:'' ''(empty string)''
:: ''Example Values:'' <code>xyz:</code>

''Examples:''
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.ElementRenamer" description="rename abcd:language attributes">
<setting name="xpath" value="//*/@abcd:language"/>
<setting name="newName" value="language"/>
<setting name="useOldNamespace" value="false"/>
<setting name="newNamespace" value=""/>
<setting name="newNamespacePrefix" value=""/>
</module>
<module name="org.bgbm.rebind.correction.modules.ElementRenamer" description="rename wrong ISO Dates">
<setting name="xpath" value="//abcd:ISODateTimeBegin[not(matches(.,'^(\d\d\d\d(\-(0[1-9]|1[012])(\-((0[1-9])|1\d|2\d|3[01])(T(0\d|1\d|2[0-3])(:[0-5]\d){0,2})?)?)?|\-\-(0[1-9]|1[012])(\-(0[1-9]|1\d|2\d|3[01]))?|\-\-\-(0[1-9]|1\d|2\d|3[01]))$'))]"/>
<setting name="newName" value="DateText"/>
<setting name="useOldNamespace" value="true"/>
</module>
</syntaxhighlight>

== DummyModule ==
''What it does:'' Waits 1-11 seconds before returning a quote from either the homicidal computer HAL 9000 from the movie "2001: A Space Odyssey" or the maniacally depressed robot Marvin from the book/movie "The Hitchhiker's Guide to the Galaxy". This module does not alter the XML code in any way, it only sends the quote back to the Correction Manager. It is only used for testing purposes.

''Full Name:'' <code>org.bgbm.rebind.correction.modules.DummyModule</code>

''Settings:'' none

''Examples:''
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.DummyModule" description="just wait a bit for a snappy robot remark" />
<module name="org.bgbm.rebind.correction.modules.DummyModule" description="Play it once again, Marvin, for old times' sake." />
</syntaxhighlight>

== Work in Progress ==
These modules are currently in the making. So some of these descriptions might not reflect the current state of development.

=== ABCDDateCorrector ===
''What it does:'' Checks the dates in the elements <code>abcd:ISODateTimeBegin</code> within <code>abcd:Date</code> and <code>abcd:DateTime</code>. If they are not formatted according to the ISO norm it tries to parse and fix them or renames the element to <code>abcd:DateText</code>. However if there are any <code>abcd:DateText</code> elements which are correctly formatted or can be converted, it will make them into <code>abcd:ISODateTimeBegin</code> elements.

''Full Name:'' <code>org.bgbm.rebind.correction.modules.ABCDDateCorrector</code>

''Settings:'' none

''Examples:''
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.ABCDDateCorrector" description="fixing the dates" />
</syntaxhighlight>

=== SimpleCountryCodeChecker ===
''What it does:'' Compares the content of the <code>abcd:ISO3166Code</code> element with a list of Country Codes provided by Java and warns if it doesn't occur there. Also has some hardcoded exceptions for the commonly used but unspecified codes:
ZZ Unknown
XA Unknown or unspecified Africa
XB Unknown or unspecified Middle and South America
XC Unknown or unspecified Asia
XD Unknown or unspecified Australia and Oceania
XE Unknown or unspecified Europe
XF Unknown or unspecified North America

''Full Name:'' <code>org.bgbm.rebind.correction.modules.SimpleCountryCodeChecker</code>

''Settings:'' none

''Examples:''
<syntaxhighlight>
<module name="org.bgbm.rebind.correction.modules.SimpleCountryCodeChecker" description="checking the country codes" />
</syntaxhighlight>

Correction Modules

2012-07-26T10:14:14Z

DavidFichtmueller: TOC moved

Correction Modules

2012-07-26T10:13:34Z

DavidFichtmueller: Empty templates for other modules

Correction Modules

2012-07-26T10:08:40Z

DavidFichtmueller: /* ElementTextReplacer */ formatting

Correction Modules

2012-07-26T10:06:58Z

DavidFichtmueller: Introduction and ElementTextReplacer

Retrieving Geocoordinates From Points In Images

2012-07-25T08:04:42Z

DavidFichtmueller: two images added, others renamed

This manual will show how to get the geocoordinates from points in an image.

The source image we use, shows six locations in the western part of Nicaragua. Though the image is of poor quality, luckily the lines for latitude and longitude are visible in the picture.
[[File:JOSM Mapping 00 Original Image.png|frame|none|The Source Image with six locations marked with circled dots]]
To start download the program JOSM from [http://josm.openstreetmap.de/ http://josm.openstreetmap.de/]. JOSM is a Java based Editor for [http://www.oepnstreetmap.org OpenStreetMap].
[[File:JOSM Mapping 01 Initial Window.png|frame|none|The Initial Screen of JOSM]]
== Download PicLayer Plugin ==
* Start JOSM
* Press F12 to get to the Preferences Window
* Go to the Plugin-Tab
* Click on '''Download List'''
* Search for '''PicLayer'''
* Check the checkbox for the PicLayer plugin
* Click OK and restart JOSM
[[File:JOSM Mapping 02 Preferences Plugins.png|frame|none|Selecting the PicLayer Plugin]]

== Select the OpenStreetMap Image Layer ==
* Go on '''Imagery''' and select '''OpenStreetMap (Mapnik)'''
* navigate to the area of interest
[[File:JOSM Mapping 03 Area Of Interest.png|frame|none|The area of interest is selected]]

== Draw a reference rectangle ==
''This step can be skipped if the image does not show the lines for latitude and longitude, like the example image above does.''
* Open a new Data Layer by pressing '''Ctrl + N'''
* Insert a new Point by pressing '''Shift + D''' and entering its coordinates and press OK. In this example we start with 13.0 -87.0. Repeat for the other points as well: 12.0 -87,0; 12.0 -86,0 and 13.0 -86,0.

[[File:JOSM Mapping 04 Entering Coordinates.png|frame|none|The window for entering the coordinates.]]
* Switch to the drawing tool by pressing '''A'''
* Connect the four points to a rectangle.
[[File:JOSM Mapping 05 Grid Rectangle.png|frame|none|The rectangle on the map]]

== Loading the reference image ==
* In the Menu go on '''PicLayer''' and select '''New picture layer from file ...'''
* Select the prepared image.
* The Picture Layer may be displayed below the OpenStreetMap Layer. If so, move it up in the Layers Box, so that the image becomes visible.
[[File:JOSM Mapping 06 Image Loaded.png|frame|none|The image loaded as a PicLayer]]
* Also make the Picture Layer the active layer, by clicking on the third button in the Layers Box Toolbar, when the Picture Layer is selected. By doing so, the tools for calibrating the picture will become visible in the toolbar at the left side.
[[File:JOSM Mapping 07 Image Layer Selected.png|frame|none|The image layer is now the selected layer]]

== Calibrating the reference image ==
* Click on the button with the green arrow to add three reference points on the image, each on one of the confluence points (where the latitude and longitude lines cross).
* Click on the button with the red arrow to move the reference points until they align with the corners of the rectangle.
* If you now make the image layer semi-transparent, you can see how the image aligns with the map.
* If you do not have the latitude and longitude lines, you should make the image layer semi transparent and try to align distinct points in the image with the map.

[[File:JOSM Mapping 08 Image Layer Calibrated.png|frame|none|The image is now calibrated. The gray arrows of the rectangle of the data layer are now over the grid lines in the image.]]
[[File:JOSM Mapping 09 Image Layer Semi-Transparent.png|frame|none|Setting the image layer as semi transparent.]]

== Mapping the points in the image ==
* Open a new data layer by pressing '''Ctrl + N'''
* Select the drawing tool by pressing '''A'''
* Draw points on the positions of the image from which you want to know the coordinates.
* If there are several points, you should add a name for each point in the Properties Box on the right.
[[File:JOSM Mapping 10 Points Mapped And Named.png|frame|none|Mapping the points. They are visible as the light small rectangles in the black dots on the image.]]
* After all the points are mapped, right-click on the new data layer and select '''Save As ...''' and save it as XML file on your local file system.
[[File:JOSM Mapping 11 Exporting Points.png|frame|none|Exporting the mapped points.]]
* The file contains the coordinates of the selected points for further processing.
<syntaxhighlight lang="xml">
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' upload='true' generator='JOSM'>
<node id='-95' action='modify' visible='true' lat='12.881180287556537' lon='-86.11194760698338'>
<tag k='name' v='1' />
</node>
<node id='-94' action='modify' visible='true' lat='12.79249813630681' lon='-86.10761639569195'>
<tag k='name' v='2' />
</node>
<node id='-93' action='modify' visible='true' lat='12.403621057420668' lon='-86.22455910056097'>
<tag k='name' v='3' />
</node>
<node id='-92' action='modify' visible='true' lat='12.084582576302516' lon='-86.11790302250911'>
<tag k='name' v='6' />
</node>
<node id='-91' action='modify' visible='true' lat='12.20261349095152' lon='-86.46277572159048'>
<tag k='name' v='5' />
</node>
<node id='-90' action='modify' visible='true' lat='12.40062989709886' lon='-86.83219726377988'>
<tag k='name' v='4' />
</node>
</osm>
</syntaxhighlight>

== Closing JOSM ==
* Close JOSM. '''WARNING''': When you try to close JOSM, it will ask you if you want to upload the two data layers to OpenStreetMap and selects these options for you. '''DO NOT UPLOAD THE DATA!''' Click on '''Exit Now.'''
* JOSM will then ask if the calibration file for the image should be saved. You can do so, if you want to. It will be stored in the same directory as the image. The next time you select the image to be used by PicLayer it will detect this file and put the image in the correct position.

Retrieving Geocoordinates From Points In Images

2012-07-24T16:03:41Z

DavidFichtmueller: Images added

This manual will show how to get the geocoordinates from points in an image.

The source image we use, shows six locations in the western part of Nicaragua. Though the image is of poor quality, luckily the lines for latitude and longitude are visible in the picture.
[[File:JOSM Mapping 00 Original Image.png|frame|none|The Source Image with six locations marked with circled dots]]
To start download the program JOSM from [http://josm.openstreetmap.de/ http://josm.openstreetmap.de/]. JOSM is a Java based Editor for [http://www.oepnstreetmap.org OpenStreetMap].
[[File:JOSM Mapping 01 Initial Window.png|frame|none|The Initial Screen of JOSM]]
== Download PicLayer Plugin ==
* Start JOSM
* Press F12 to get to the Preferences Window
* Go to the Plugin-Tab
* Click on '''Download List'''
* Search for '''PicLayer'''
* Check the checkbox for the PicLayer plugin
* Click OK and restart JOSM
[[File:JOSM Mapping 02 Preferences Plugins.png|frame|none|Selecting the PicLayer Plugin]]

== Select the OpenStreetMap Image Layer ==
* Go on '''Imagery''' and select '''OpenStreetMap (Mapnik)'''
* navigate to the area of interest

== Draw a reference rectangle ==
''This step can be skipped if the image does not show the lines for latitude and longitude, like the example image above does.''
* Open a new Data Layer by pressing '''Ctrl + N'''
* Insert a new Point by pressing '''Shift + D''' and entering its coordinates and press OK. In this example we start with 13.0 -87.0. Repeat for the other points as well: 12.0 -87,0; 12.0 -86,0 and 13.0 -86,0.
* Switch to the drawing tool by pressing '''A'''
* Connect the four points to a rectangle.
[[File:JOSM Mapping 03 Grid Rectangle.png|frame|none|The rectangle on the map]]

== Loading the reference image ==
* In the Menu go on '''PicLayer''' and select '''New picture layer from file ...'''
* Select the prepared image.
* The Picture Layer may be displayed below the OpenStreetMap Layer. If so, move it up in the Layers Box, so that the image becomes visible.
[[File:JOSM Mapping 04 Image Loaded.png|frame|none|The image loaded as a PicLayer]]
* Also make the Picture Layer the active layer, by clicking on the third button in the Layers Box Toolbar, when the Picture Layer is selected. By doing so, the tools for calibrating the picture will become visible in the toolbar at the left side.
[[File:JOSM Mapping 05 Image Layer Selected.png|frame|none|The image layer is now the selected layer]]

== Calibrating the reference image ==
* Click on the button with the green arrow to add three reference points on the image, each on one of the confluence points (where the latitude and longitude lines cross).
* Click on the button with the red arrow to move the reference points until they align with the corners of the rectangle.
* If you now make the image layer semi-transparent, you can see how the image aligns with the map.
* If you do not have the latitude and longitude lines, you should make the image layer semi transparent and try to align distinct points in the image with the map.

[[File:JOSM Mapping 06 Image Layer Calibrated.png|frame|none|The image is now calibrated. The gray arrows of the rectangle of the data layer are now over the grid lines in the image.]]
[[File:JOSM Mapping 07 Image Layer Semi-Transparent.png|frame|none|Setting the image layer as semi transparent.]]

== Mapping the points in the image ==
* Open a new data layer by pressing '''Ctrl + N'''
* Select the drawing tool by pressing '''A'''
* Draw points on the positions of the image from which you want to know the coordinates.
* If there are several points, you should add a name for each point in the Properties Box on the right.
[[File:JOSM Mapping 08 Points Mapped And Named.png|frame|none|Mapping the points. They are visible as the light small rectangles in the black dots on the image.]]
* After all the points are mapped, right-click on the new data layer and select '''Save As ...''' and save it as XML file on your local file system.
[[File:JOSM Mapping 09 Exporting Points.png|frame|none|Exporting the mapped points.]]
* The file contains the coordinates of the selected points for further processing.
<syntaxhighlight lang="xml">
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' upload='true' generator='JOSM'>
<node id='-95' action='modify' visible='true' lat='12.881180287556537' lon='-86.11194760698338'>
<tag k='name' v='1' />
</node>
<node id='-94' action='modify' visible='true' lat='12.79249813630681' lon='-86.10761639569195'>
<tag k='name' v='2' />
</node>
<node id='-93' action='modify' visible='true' lat='12.403621057420668' lon='-86.22455910056097'>
<tag k='name' v='3' />
</node>
<node id='-92' action='modify' visible='true' lat='12.084582576302516' lon='-86.11790302250911'>
<tag k='name' v='6' />
</node>
<node id='-91' action='modify' visible='true' lat='12.20261349095152' lon='-86.46277572159048'>
<tag k='name' v='5' />
</node>
<node id='-90' action='modify' visible='true' lat='12.40062989709886' lon='-86.83219726377988'>
<tag k='name' v='4' />
</node>
</osm>
</syntaxhighlight>

== Closing JOSM ==
* Close JOSM. '''WARNING''': When you try to close JOSM, it will ask you if you want to upload the two data layers to OpenStreetMap and selects these options for you. '''DO NOT UPLOAD THE DATA!''' Click on '''Exit Now.'''
* JOSM will then ask if the calibration file for the image should be saved. You can do so, if you want to. It will be stored in the same directory as the image. The next time you select the image to be used by PicLayer it will detect this file and put the image in the correct position.

Retrieving Geocoordinates From Points In Images

2012-07-24T15:13:18Z

DavidFichtmueller: description of the process

This manual will show how to get the geocoordinates from points in an image.

The source image we use, shows six locations in the western part of Nicaragua. Though the image is of poor quality, luckily the lines for latitude and longitude are visible in the picture.

To start download the program JOSM from [http://josm.openstreetmap.de/ http://josm.openstreetmap.de/]. JOSM is a Java based Editor for [http://www.oepnstreetmap.org OpenStreetMap].

== Download PicLayer Plugin ==
* Start JOSM
* Press F12 to get to the Preferences Window
* Go to the Plugin-Tab
* Click on '''Download List'''
* Search for '''PicLayer'''
* Check the checkbox for the PicLayer plugin
* Click OK and restart JOSM

== Select the OpenStreetMap Image Layer ==
* Go on '''Imagery''' and select '''OpenStreetMap (Mapnik)'''
* navigate to the area of interest

== Draw a reference rectangle ==
''This step can be skipped if the image does not show the lines for latitude and longitude, like the example image above does.''
* Open a new Data Layer by pressing '''Ctrl + N'''
* Insert a new Point by pressing '''Shift + D''' and entering its coordinates and press OK. In this example we start with 13.0 -87.0. Repeat for the other points as well: 12.0 -87,0; 12.0 -86,0 and 13.0 -86,0.
* Switch to the drawing tool by pressing '''A'''
* Connect the four points to a rectangle.

== Loading the reference image ==
* In the Menu go on '''PicLayer''' and select '''New picture layer from file ...'''
* Select the prepared image.
* The Picture Layer may be displayed below the OpenStreetMap Layer. If so, move it up in the Layers Box, so that the image becomes visible.
* Also make the Picture Layer the active layer, by clicking on the third button in the Layers Box Toolbar, when the Picture Layer is selected. By doing so, the tools for calibrating the picture will become visible in the toolbar at the left side.

== Calibrating the reference image ==
* Click on the button with the green arrow to add three reference points on the image, each on one of the confluence points (where the latitude and longitude lines cross).
* Click on the button with the red arrow to move the reference points until they align with the corners of the rectangle.
* If you now make the image layer semi-transparent, you can see how the image aligns with the map.
* If you do not have the latitude and longitude lines, you should make the image layer semi transparent and try to align distinct points in the image with the map.

== Mapping the points in the image ==
* Open a new data layer by pressing '''Ctrl + N'''
* Select the drawing tool by pressing '''A'''
* Draw points on the positions of the image from which you want to know the coordinates.
* If there are several points, you should add a name for each point in the Properties Box on the right.
* After all the points are mapped, right-click on the new data layer and select '''Save As ...''' and save it as XML file on your local file system.
* The file contains the coordinates of the selected points for further processing.
<syntaxhighlight lang="xml">
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' upload='true' generator='JOSM'>
<node id='-95' action='modify' visible='true' lat='12.881180287556537' lon='-86.11194760698338'>
<tag k='name' v='1' />
</node>
<node id='-94' action='modify' visible='true' lat='12.79249813630681' lon='-86.10761639569195'>
<tag k='name' v='2' />
</node>
<node id='-93' action='modify' visible='true' lat='12.403621057420668' lon='-86.22455910056097'>
<tag k='name' v='3' />
</node>
<node id='-92' action='modify' visible='true' lat='12.084582576302516' lon='-86.11790302250911'>
<tag k='name' v='6' />
</node>
<node id='-91' action='modify' visible='true' lat='12.20261349095152' lon='-86.46277572159048'>
<tag k='name' v='5' />
</node>
<node id='-90' action='modify' visible='true' lat='12.40062989709886' lon='-86.83219726377988'>
<tag k='name' v='4' />
</node>
</osm>
</syntaxhighlight>

== Closing JOSM ==
* Close JOSM. '''WARNING''': When you try to close JOSM, it will ask you if you want to upload the two data layers to OpenStreetMap and selects these options for you. '''DO NOT UPLOAD THE DATA!''' Click on '''Exit Now.'''
* JOSM will then ask if the calibration file for the image should be saved. You can do so, if you want to. It will be stored in the same directory as the image. The next time you select the image to be used by PicLayer it will detect this file and put the image in the correct position.

Export Paradox Data

2012-07-23T08:41:35Z

DavidFichtmueller: /* Further Fixing Character Encoding */ Link Correktion

This article describes how to export data from Paradox Database files, based on the data set provided by the Museum Koenig.

== Introduction ==

We received three dataset with data in the Paradox database format and were asked if we could extract the data and convert it to SQL.
Paradox is a file based database system. Each table has at least two (usually more) files with the same based name and different file ending.

Further Reading
* [http://www.sundialservices.com/articles/paradox/paradox_file_structures A Brief Tour of Paradox Database Files]
* [http://www.sundialservices.com/articles/paradox/understanding_filesystem_databases Understanding File-System Databases]

== Viewing The Data ==
In order to view the data we first used the software [http://www.sportamok.com/development/delphi/8-paradox-dbase-reader Paradox dBase Reader]. Though all the structural information of the table was clearly visible, all the content was only displayed as garbage (Image 1). As it turns out, the files were password protected, the Paradox dBase Reader actually showed that they were protected in the general information page of the table (the checked checkbox at the bottom of the second image)
{|
|[[File:Paradox dBase Reader with Password Protected Data.png|frame|none|Viewing Password Encrypted Files in Paradox dbase Reader]]
|[[File:Paradox dBase Reader with Password Protected Data 2.png|frame|none|Information about the file in Paradox dbase Reader]]
|}

We then used the software [http://www.scalabium.com/pdx/ Paradox Viewer]. It promoted a dialog box asking to input the password. All encrypted Paradox files have a default password, depending on the version of Paradox used. For Paradox 5.0 and 7.0 this is either <code>jIGGAe</code> or <code>cupcdvum</code> and for Paradox 4.0 it is <code>nx66ppx</code>. The files were saved by Paradox 7 (as can be see on the general information page of the Paradox dBase Reader in the image above). In our case <code>jIGGAe</code> was the correct password. Now the content was displayed correctly. However in the unregistered version Paradox Viewer only shows the first 100 entries and has no possibility to remove the password protection.

== Removing Password Protection ==
The [http://www.mitec.cz/pde.html Paradox Data Editor] offers the functionality of removing the password protection from a table. It also opens the encrypted tables without even prompting for a password. In properties panel on the left, it still shows that the file is protected. Above you can also see the entry "Code Page: 437" which will become important later on. To remove the password protection, click on the key icon in the toolbar.
[[File:Paradox Data Editor.png|frame|none|Data already decrypted in Paradox Data Editor]]

== Exporting the data ==
To export the data from the Paradox files, we used the software [http://jan.kneschke.de/projects/pxtools/ PXTools]. It runs under Linux but first it has be compiled and installed.

=== Installing PXTools ===
After downloading the *.tar.gz file from the homepage of PXTools, run the following commands to install the software.

<syntaxhighlight lang="bash" enclose="pre">
tar -zxvf pxtools-0.0.20.tar.gz
cd pxtools-0.0.20/
./configure
sudo make
sudo make install
sudo make clean
</syntaxhighlight>

To see information about a table use the command
<syntaxhighlight lang="bash" enclose="pre">
pxinfo -f COLEOPTE_DATA.DB
</syntaxhighlight>

=== Using PXTools ===
To export the data from one table into the mysql file use the command
<syntaxhighlight lang="bash" enclose="pre">
pxsqldump -d mysql -f COLEOPTE_DATA.DB -b COLEOPTE_DATA.MB -d coleopte_data> COLEOPTE_DATA.sql
</syntaxhighlight>

PXTools support export to CSV, XML, PostgreSQL and mySQL. In this example we will be exporting the data to mySQL.

=== Using PXTools With A Script ===
Calling the tool as shown above, means that the command has to be adjusted and executed for each table individually. A more convenient way is use a script that will run for all of the tables automatically. The following code does just this as well as a few more things, which will be explained later on.

<syntaxhighlight lang="bash" enclose="pre">
#!/bin/sh
# parameters: $1=database_name $2=Paradox_File_Encoding

if [ ! -d "sql" ]; then
echo "creating 'sql' directory"
mkdir sql
fi
DBFILES=`find -maxdepth 1 $ -name "*.DB" ! -name "BDS_*" $ `

for i in $DBFILES
do
echo
echo "#############################"
echo "# Exporting $i"
echo "#############################"
pxsqldump -d mysql -f $i -b $(echo "$i" | sed -e 's/\.DB$/\.MB/') -n $1.$(echo "$i" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///') > sql/$(echo "$i" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///').sql
done

cd sql
if [ ! -d "converted" ]; then
mkdir converted
fi
SQLFILES=`find -maxdepth 1 $ -name "*.sql" ! -name "BDS_*" $`

for i in $SQLFILES
do
echo "Converting $i"
iconv -f $2 -t UTF-8 < $i > converted/$i
done

cd converted
echo
echo "Joining SQL Files"
echo "CREATE DATABASE IF NOT EXISTS $1 ;" > $1.sql.tmp
echo "USE $1 ;" >> $1.sql.tmp
echo >> $1.sql.tmp
echo >> $1.sql.tmp

SQLFILES=`find $ -name "*.sql" ! -name "BDS_*" ! -name "$1.sql" $`

for i in $SQLFILES
do
cat $i >> $1.sql.tmp
echo >> $1.sql.tmp
echo >> $1.sql.tmp
echo $i
done
echo "" >> $1.sql.tmp
mv $1.sql.tmp ../../$1.sql
</syntaxhighlight>

Save the script above in a file called 'px2sql-all.sh' and save it in the folder ~/bin/. Then run the following commands

PATH=$PATH:~/bin
chmod +x ~/bin/px2sql-all.sh

Here are the different processing steps the script does:
# creates a directory with the name 'sql'
# goes through all the Paradox table files in the current directory and export them to individual sql files in the ''sql'' directory
# creates a subdirectory within the directory called 'converted'
# goes through all the exported sql files and converts them to UTF-8. For this to work the script needs to know the Character Encoding of the original data. This is where the Code Page entry from the properties page in the Paradox Data Editor comes in play. In this example the Code Page was 437, so the character encoding is CP437. This will be used as a parameter when calling the script. ''See the man-page for the program iconv to see what other character encodings are supported. To learn more about character encoding, please read the article ''[http://www.joelonsoftware.com/articles/Unicode.html The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)]'' by Joel Spolsky.''
# creates one sql file with code to create a database within the mySQL database engine. The content of all the individual sql files is then also added to this sql file. The resulting file is then copied in the directory in which the original Paradox files are.

To run the script go into the directory where the Paradox data files are located and call the command

px2sql-all.sh <database name> <original character encoding>

In this example the name of the database is 'ornithol' and the character encoding is CP437, so resulting command will be

px2sql-all.sh ornithol CP437

The resulting file will be ornithol.sql which can then be imported into a mySQL database.

== Additional Steps In Processing The Data ==

=== Documenting Table Relations ===
PXTools is not able to export the relations between the different columns of different tables (Foreign Key Relations). With the Paradox Data Editor it is however possible to view those relations. When a table is opened, click on the Icon "Structure Information" in the toolbar (the forth icon from the right, marked with a red rectangle in the Paradox Data Editor image above). This will open an info window with the structure information of the table (This window will not open, if the name of the Paradox file or the path to it contains any special characters). Go to the tab "Referential Integrity" and you will see a list of all the relations from this table. With these information it is possible to remodel the relations in mySQL.
[[File:Paradox Data Editor Structure Information.png|frame|none|Structural Information about the References to other tables]]

=== Further Fixing Character Encoding ===
Though most of the data in the tables we received were encoded in CP437, some parts of it used different character encodings. This was fixed using the same method as described in [[Export_DataPerfect#converting_characters_individually|Export_DataPerfect (Section 'Converting Characters Individually')]].

=== Turning BLOB Entries Into TEXT Entries ===
In our dataset a lot of large texts were stored in Paradox as BLOB objects. After exporting the data from Paradox these texts where in plain text in the mxSQL files, however the data type for the column was still BLOB. Though this works with mySQL we changed the data type for these columns to TEXT, so the text content is more easily editable and searchable within mySQL.

=== Optimized For Import Speed ===
The way PXTools exports the data, makes it slow to import it in mySQL database again. When trying to import it using phpMyAdmin, several timeout occured. In some cased phpMyAdmin saved the last position and it was easy to continue importing, on other cased, the files had to be adjusted manually in order to continue the import at the correct position. When importing the files through the command line interface of mySQL it worked without errors, however, it still took quite a while. A dataset with 76000 entries took almost 50 minutes to import. After the data was imported, it was exported again. This time the data was optimized for a reimport. The same dataset now only took 30 seconds to import. However during the initial import some data points where there was no data from Paradox but were required by mySQL were filled out using default values. This therefore altered the data and so the reexported data is not equal to the initially exported data.

Export Paradox Data

2012-07-20T15:37:41Z

DavidFichtmueller: description and images

This article describes how to export data from Paradox Database files, based on the data set provided by the Museum Koenig.

== Introduction ==

We received three dataset with data in the Paradox database format and were asked if we could extract the data and convert it to SQL.
Paradox is a file based database system. Each table has at least two (usually more) files with the same based name and different file ending.

Further Reading
* [http://www.sundialservices.com/articles/paradox/paradox_file_structures A Brief Tour of Paradox Database Files]
* [http://www.sundialservices.com/articles/paradox/understanding_filesystem_databases Understanding File-System Databases]

== Viewing The Data ==
In order to view the data we first used the software [http://www.sportamok.com/development/delphi/8-paradox-dbase-reader Paradox dBase Reader]. Though all the structural information of the table was clearly visible, all the content was only displayed as garbage (Image 1). As it turns out, the files were password protected, the Paradox dBase Reader actually showed that they were protected in the general information page of the table (the checked checkbox at the bottom of the second image)
{|
|[[File:Paradox dBase Reader with Password Protected Data.png|frame|none|Viewing Password Encrypted Files in Paradox dbase Reader]]
|[[File:Paradox dBase Reader with Password Protected Data 2.png|frame|none|Information about the file in Paradox dbase Reader]]
|}

We then used the software [http://www.scalabium.com/pdx/ Paradox Viewer]. It promoted a dialog box asking to input the password. All encrypted Paradox files have a default password, depending on the version of Paradox used. For Paradox 5.0 and 7.0 this is either <code>jIGGAe</code> or <code>cupcdvum</code> and for Paradox 4.0 it is <code>nx66ppx</code>. The files were saved by Paradox 7 (as can be see on the general information page of the Paradox dBase Reader in the image above). In our case <code>jIGGAe</code> was the correct password. Now the content was displayed correctly. However in the unregistered version Paradox Viewer only shows the first 100 entries and has no possibility to remove the password protection.

== Removing Password Protection ==
The [http://www.mitec.cz/pde.html Paradox Data Editor] offers the functionality of removing the password protection from a table. It also opens the encrypted tables without even prompting for a password. In properties panel on the left, it still shows that the file is protected. Above you can also see the entry "Code Page: 437" which will become important later on. To remove the password protection, click on the key icon in the toolbar.
[[File:Paradox Data Editor.png|frame|none|Data already decrypted in Paradox Data Editor]]

== Exporting the data ==
To export the data from the Paradox files, we used the software [http://jan.kneschke.de/projects/pxtools/ PXTools]. It runs under Linux but first it has be compiled and installed.

=== Installing PXTools ===
After downloading the *.tar.gz file from the homepage of PXTools, run the following commands to install the software.

<syntaxhighlight lang="bash" enclose="pre">
tar -zxvf pxtools-0.0.20.tar.gz
cd pxtools-0.0.20/
./configure
sudo make
sudo make install
sudo make clean
</syntaxhighlight>

To see information about a table use the command
<syntaxhighlight lang="bash" enclose="pre">
pxinfo -f COLEOPTE_DATA.DB
</syntaxhighlight>

=== Using PXTools ===
To export the data from one table into the mysql file use the command
<syntaxhighlight lang="bash" enclose="pre">
pxsqldump -d mysql -f COLEOPTE_DATA.DB -b COLEOPTE_DATA.MB -d coleopte_data> COLEOPTE_DATA.sql
</syntaxhighlight>

PXTools support export to CSV, XML, PostgreSQL and mySQL. In this example we will be exporting the data to mySQL.

=== Using PXTools With A Script ===
Calling the tool as shown above, means that the command has to be adjusted and executed for each table individually. A more convenient way is use a script that will run for all of the tables automatically. The following code does just this as well as a few more things, which will be explained later on.

<syntaxhighlight lang="bash" enclose="pre">
#!/bin/sh
# parameters: $1=database_name $2=Paradox_File_Encoding

if [ ! -d "sql" ]; then
echo "creating 'sql' directory"
mkdir sql
fi
DBFILES=`find -maxdepth 1 $ -name "*.DB" ! -name "BDS_*" $ `

for i in $DBFILES
do
echo
echo "#############################"
echo "# Exporting $i"
echo "#############################"
pxsqldump -d mysql -f $i -b $(echo "$i" | sed -e 's/\.DB$/\.MB/') -n $1.$(echo "$i" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///') > sql/$(echo "$i" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///').sql
done

cd sql
if [ ! -d "converted" ]; then
mkdir converted
fi
SQLFILES=`find -maxdepth 1 $ -name "*.sql" ! -name "BDS_*" $`

for i in $SQLFILES
do
echo "Converting $i"
iconv -f $2 -t UTF-8 < $i > converted/$i
done

cd converted
echo
echo "Joining SQL Files"
echo "CREATE DATABASE IF NOT EXISTS $1 ;" > $1.sql.tmp
echo "USE $1 ;" >> $1.sql.tmp
echo >> $1.sql.tmp
echo >> $1.sql.tmp

SQLFILES=`find $ -name "*.sql" ! -name "BDS_*" ! -name "$1.sql" $`

for i in $SQLFILES
do
cat $i >> $1.sql.tmp
echo >> $1.sql.tmp
echo >> $1.sql.tmp
echo $i
done
echo "" >> $1.sql.tmp
mv $1.sql.tmp ../../$1.sql
</syntaxhighlight>

Save the script above in a file called 'px2sql-all.sh' and save it in the folder ~/bin/. Then run the following commands

PATH=$PATH:~/bin
chmod +x ~/bin/px2sql-all.sh

Here are the different processing steps the script does:
# creates a directory with the name 'sql'
# goes through all the Paradox table files in the current directory and export them to individual sql files in the ''sql'' directory
# creates a subdirectory within the directory called 'converted'
# goes through all the exported sql files and converts them to UTF-8. For this to work the script needs to know the Character Encoding of the original data. This is where the Code Page entry from the properties page in the Paradox Data Editor comes in play. In this example the Code Page was 437, so the character encoding is CP437. This will be used as a parameter when calling the script. ''See the man-page for the program iconv to see what other character encodings are supported. To learn more about character encoding, please read the article ''[http://www.joelonsoftware.com/articles/Unicode.html The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)]'' by Joel Spolsky.''
# creates one sql file with code to create a database within the mySQL database engine. The content of all the individual sql files is then also added to this sql file. The resulting file is then copied in the directory in which the original Paradox files are.

To run the script go into the directory where the Paradox data files are located and call the command

px2sql-all.sh <database name> <original character encoding>

In this example the name of the database is 'ornithol' and the character encoding is CP437, so resulting command will be

px2sql-all.sh ornithol CP437

The resulting file will be ornithol.sql which can then be imported into a mySQL database.

== Additional Steps In Processing The Data ==

=== Documenting Table Relations ===
PXTools is not able to export the relations between the different columns of different tables (Foreign Key Relations). With the Paradox Data Editor it is however possible to view those relations. When a table is opened, click on the Icon "Structure Information" in the toolbar (the forth icon from the right, marked with a red rectangle in the Paradox Data Editor image above). This will open an info window with the structure information of the table (This window will not open, if the name of the Paradox file or the path to it contains any special characters). Go to the tab "Referential Integrity" and you will see a list of all the relations from this table. With these information it is possible to remodel the relations in mySQL.
[[File:Paradox Data Editor Structure Information.png|frame|none|Structural Information about the References to other tables]]

=== Further Fixing Character Encoding ===
Though most of the data in the tables we received were encoded in CP437, some parts of it used different character encodings. This was fixed using the same method as described in [[Export_DataPerfect (Section 'Converting Characters Individually')||Export_DataPerfect#converting_characters_individually]].

=== Turning BLOB Entries Into TEXT Entries ===
In our dataset a lot of large texts were stored in Paradox as BLOB objects. After exporting the data from Paradox these texts where in plain text in the mxSQL files, however the data type for the column was still BLOB. Though this works with mySQL we changed the data type for these columns to TEXT, so the text content is more easily editable and searchable within mySQL.

=== Optimized For Import Speed ===
The way PXTools exports the data, makes it slow to import it in mySQL database again. When trying to import it using phpMyAdmin, several timeout occured. In some cased phpMyAdmin saved the last position and it was easy to continue importing, on other cased, the files had to be adjusted manually in order to continue the import at the correct position. When importing the files through the command line interface of mySQL it worked without errors, however, it still took quite a while. A dataset with 76000 entries took almost 50 minutes to import. After the data was imported, it was exported again. This time the data was optimized for a reimport. The same dataset now only took 30 seconds to import. However during the initial import some data points where there was no data from Paradox but were required by mySQL were filled out using default values. This therefore altered the data and so the reexported data is not equal to the initially exported data.

Export Paradox Data

2012-06-11T12:59:45Z

DavidFichtmueller: complete script added

This article describes how to export data from Paradox Database files, based on the data set provided by the Museum König.

== Introduction ==

Further Reading
* [http://www.sundialservices.com/articles/paradox/paradox_file_structures A Brief Tour of Paradox Database Files]
* [http://www.sundialservices.com/articles/paradox/understanding_filesystem_databases Understanding File-System Databases]

== Viewing the Data ==
* [http://www.sportamok.com/development/delphi/8-paradox-dbase-reader Paradox dbase Reader]
* [http://www.scalabium.com/pdx/ Paradox Viewer] 30 day trial version
Paradox 5.0 and 7.0: <code>jIGGAe</code> or <code>cupcdvum</code>
Paradox 4.0: <code>nx66ppx</code>

== Removing Password Protection ==
* [http://www.mitec.cz/pde.html Paradox Data Editor]

== Documenting Table Relations ==

== Exporting the data ==

=== Installing PX Tools ===
* http://jan.kneschke.de/projects/pxtools/

<syntaxhighlight lang="bash" enclose="pre">
tar -zxvf pxtools-0.0.20.tar.gz
cd pxtools-0.0.20/
./configure
sudo make
sudo make install
sudo make clean
pxinfo -f COLEOPTE_DATA.DB
pxsqldump -d mysql -f COLEOPTE_DATA.DB -b COLEOPTE_DATA.MB -d coleopte_data> COLEOPTE_DATA.sql

</syntaxhighlight>

pxconvert.c.477: Read less than requested
pxconvert.c.483: Extract failed: `��A�f,<�� !=
pxconvert.c.477: Read less than requested
Speicherzugriffsfehler (Speicherabzug geschrieben)

<syntaxhighlight lang="bash" enclose="pre">
#!/bin/sh
if [ ! -d "sql" ]; then
mkdir sql
fi
echo
echo "#############################"
echo "# Exporting $1"
echo "#############################"
pxsqldump -d mysql -f $1 -b $(echo "$1" | sed -e 's/\.DB$/\.MB/') -n $(echo "$1" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///') > sql/$1.sql
</syntaxhighlight>

PATH=$PATH:~/bin
chmod +x ~/bin/px2sql.sh
find -name '*.DB' -exec px2sql.sh {} \;

//TODO SQL Join script
//TODO encoding

<syntaxhighlight lang="bash" enclose="pre">
#!/bin/sh
# parameters: $1=database_name $2=Paradox_File_Encoding

if [ ! -d "sql" ]; then
echo "creating 'sql' directory"
mkdir sql
fi
DBFILES=`find -maxdepth 1 $ -name "*.DB" ! -name "BDS_*" $ `

for i in $DBFILES
do
echo
echo "#############################"
echo "# Exporting $i"
echo "#############################"
pxsqldump -d mysql -f $i -b $(echo "$i" | sed -e 's/\.DB$/\.MB/') -n $1.$(echo "$i" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///') > sql/$(echo "$i" | tr '[:upper:]' '[:lower:]'| sed -e 's/\.db$//' | sed -e 's/.*\///').sql
done

cd sql
if [ ! -d "converted" ]; then
mkdir converted
fi
SQLFILES=`find -maxdepth 1 $ -name "*.sql" ! -name "BDS_*" $`

for i in $SQLFILES
do
echo "Converting $i"
iconv -f $2 -t UTF-8 < $i > converted/$i
done

cd converted
echo
echo "Joining SQL Files"
echo "CREATE DATABASE IF NOT EXISTS $1 ;" > $1.sql.tmp
echo "USE $1 ;" >> $1.sql.tmp
echo >> $1.sql.tmp
echo >> $1.sql.tmp

SQLFILES=`find $ -name "*.sql" ! -name "BDS_*" ! -name "$1.sql" $`

for i in $SQLFILES
do
cat $i >> $1.sql.tmp
echo >> $1.sql.tmp
echo >> $1.sql.tmp
echo $i
done
echo "" >> $1.sql.tmp
mv $1.sql.tmp ../../$1.sql
</syntaxhighlight>

px2sql-all.sh ornithol CP437

Export Paradox Data

2012-06-04T15:04:33Z

DavidFichtmueller: save working copy

Export Paradox Data

2012-06-04T10:22:15Z

DavidFichtmueller: first overview

Export DataPerfect

2012-03-19T17:24:03Z

DavidFichtmueller: last steps

This article describes how to export data from a DataPerfect database, based on the Rohwer data set.

Data Perfect is a DOS based database system. It's latest release is 2.6Y from June 2008. It can be downloaded via http://dataperfect.nl/ .

=== Install and run DOSBox ===
: To run DataPerfect a DOS emulator is needed. The free emulator DOSBox works quite well. Go to http://www.dosbox.com/download.php?main=1 or http://sourceforge.net/projects/dosbox/, download the latest release and install it. For this articles DOSBox 0.74 will be used and installed under Windows 7.

=== Download DataPerfect ===
: Go to http://dataperfect.nl/ and download the DataPerfect. Unzip the files into a specific folder. The folder used for this article is ''C:/DOS/DP26Y''.

=== Copy your data files in the DataPerfect Folder===
: You can either copy them directly in the folder or create a subfolder for your data files and copy them in there. The data files for this article are located in the directory ''DIAS/'' within the DataPerfect folder.

=== Start DOSBox===
: When starting DOSBox a second console window is opened. When running several instances of DOSBox these additional windows can bloat the window bar quite a bit. To avoid this just start the version ''DOXBox-0.74/Extras/DOSBox 0.74 (noconsole)'' from the Program Menu.
[[File:DPE01_DOSBox.png|frame|none|The DOSBOX Start Window]]

=== Mount the DataPerfect Folder===
: After starting DOSBox the directory of the DataPerfect files needs to be mounted as a virtual drive. Type '''mount c C:\DOS\DP27Y''' to mount the folder and '''C:''' to change to the drive.
[[File:DPE02_DOSBox_mount.png|frame|none|Mounting the DataPerfect Folder]]

=== Start DataPerfect===
: Start DataPerfect by typing '''DP26YU''' and pressing Enter.
{|
|[[File:DPE03_DataPerfect_start.png|frame|none|Switch to C: and start DataPerfect]]
|[[File:DPE04_DataPerfect_welcome.png|frame|none|The Welcome Screen of DataPerfect]]
|}

=== Change to the folder of your data files===
: If the data files are not in the directory of DataPerfect you need to change the directory. Press '''2''' to change the directory and type in the path to the directory of your data.
{|
|[[File:DPE05_DataPerfect_open.png|frame|none| The initial screen of DataPerfect]]
|[[File:DPE06_DataPerfect_change_directory.png|frame|none|Change to the data directory]]
|}

=== Open the data files===
: If you are in the correct directory you see the DataPerfect datasets within the folder. Use the cursor keys to select the correct data set and press enter.
[[File:DPE07_DataPerfect_select_DIAS.png|frame|none|Selecting the DIAS data set.]]

=== Select the table to export ===
You can now see the tables within this database project. Select the table you want to export by using the up or down keys and pressing enter.
{|
|[[File:DPE08_DIAS_Overview.png|frame|none|The list of available tables.]]
|[[File:DPE09_DIAS_Table1.png|frame|none|The first entry in the first table of DIAS]]
|}
=== Navigate the Table ===
''This these steps are not necessary for the export, but will be documented here anyway.''
Here are important keys for navigating thought a table:
* '''Tab''' the next field is highlighted
* '''Down''' on a field which has references to entries in another table, that referenced entry is displayed
* '''Up''' The list of entries in this table where the content of the current field is displayed. Navigate though this list using the '''Up''' and '''Down''' keys. When typing characters this list the focus will jump to the entry whose unique key column is like the typed characters. For example in the table displayed in the image below, typing a number will focus on the entry with that id. The corresponding entry is automatically displayed. To edit this entry, press '''Enter'''.
* '''F7''' Goes to the next upper level in the hierarchy. For example if the list of entries is opened, it will jump back to the next upper level in the hierarchy.
[[File:DPE10_DIAS_Table1_browse.png|frame|none|Browsing through the first table of DIAS]]

=== Select Report ===
From the list of available reports select the entry at the top: '''Build-In Short Reports'''.
[[File:DPE11_DIAS_Table1_Report_List.png|frame|none|The list of available reports.]]

=== Set Export Properties ===
Set the export settings as shown in the first image below. Press the number of the property you want to change. For example, to change the file name of the output file, press '''2''' and the file options will appear, as can be seen in the second image. Press '''1''' to create a new file and enter the name of the file (image 3).
The filename of the export file must not be longer than 8 characters without file extension and the file extension itself (without the dot) must not be longer than 3 characters.
{|
|[[File:DPE12_DIAS_Table1_Export_1.png|frame|none|Export Settings Overview]]
|[[File:DPE13_DIAS_Table1_Export_2.png|frame|none|Changing the File Options]]
|[[File:DPE14_DIAS_Table1_Export_3.png|frame|none|Changing the File Name]]
|}

=== Run Export ===
To start the export press '''Shift + F7''' again. The screen now shows a counter of how many elements from the table have already been exported. DOSBox starts with a limited CPU speed for the programs running in it. The speed can be increased by pressing '''Ctrl + F12'''. Increasing the speed to much, however creates a big overhead which will also result in a slower export. For some reason when increasing the speed the CPU load will jump from around 5% of one core to 100% within just 2-3 increase steps. So the CPU load should be watched when increasing the speed of DOSBox. On a DualCore 3 GHz Processor a speed of 50000-55000 cycles (can be seen in the title bar of the DOSBox Window) appears to be a good export speed without overhead.
[[File:DPE15_DIAS_Export.png|frame|none|The export is running.]]

After the export is done, you will see the list of available reports again. To step to the next higher level of the hierarchy press '''F7'''. Repeat this until you see the list of tables within this database project again. Now repeat the export process for all the other tables you want to export.

=== Converting special characters ===
The best way to handle special characters is to know the character encoding by the original file system. If it is a DOS based system, the command '''CHCP''' will display what Character Code Page used. With this knowledge the file can be easily converted. In DOSBox the Command is '''KEYB''' however, it only helps if all of the special characters are displayed correctly within the DataPerfect. In the case of the Rohwer data set, the KEYB command showed Codepage 437, however Codepage 850 was actually used, must of the special characters are however identical between the two sets.

Once the character encoding is known the exported file converted by a program that is able to read that encoding. Under Windows Notepad++ does a good job. After opening the file (the special characters are probably distorted), one must select the original character encoding as the encoding of the file and then convert the file to UTF-8.

Google Refine is another software that is able to read the CP 850 encoding and allows for further processing of the files.

Additional Infomation about Codepages can be found on the Wikipedia Article ''[http://en.wikipedia.org/wiki/Code_page Code page]''. Also the article ''[http://www.joelonsoftware.com/articles/Unicode.html The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)]'' by Joel Spolsky is very helpful.

In retrospective we now know that the Rohwer data set is encoded using the Codepage 850 (also called OEM 850), however some single characters are in the wrong encoding. This however is the result of the process where we parsed the file, got a list of all the different characters used and looked them up and converted them by hand. Though this process is a lot more work, but it works even in the case of mixed character sets. It is documented in detail below.

==== converting characters individually ====
The export out of DataPerfect has problems with special characters which are not converted correctly in a proper character set. Though all of the occurrences of such special characters can be replaced automatically, each of the character to be replaced has to be defined manually one. The files as they are generated by DataPerfect use the ANSI Character Encoding. This needs to be converted to UTF8. Under Windows the free program Notepad++ is very suited for this. When selecting the menu item '''Encoding''' (''Kodierung'' in the image, because it is the German version of Notepad++), the entry ''ANSI'' should be marked as the current encoding. Now click on '''Convert to UTF-8''' and save the file.
[[File:DPE16 Notepad CharSet.png|frame|none|Converting the Character Set in Notepad++]]
The next step is to run the file through a program that converts the characters. The small Java program CharReplacer was written just to do that. There are two different files to that program: CharReplacer.class (the acutual program) and CharReplacer.settings (the settings file, which specifies, what characters to replace. The first line in ''CharReplacer.settings'' are all the characters which will not be replaced. All the following lines have the number value of the character which will be replaced, followed by a tab and the character it will be replaced with.

abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789",.-; ()?\':/=&[]!`<>#+%
8222 ä
8221 ö
129 ü
225 ß
381 Ä
8482 Ö
353 Ü
Excerpt from the ''CharReplacer.settings'' file.

To run the CharReplacer program, the Java Runtime Envirement (JRE) must be installed on a computer. It must be run via the command line interface (also known as console).

Under Windows the Command Prompt (cmd.exe) can be used, but it needs to be adjusted to show the UTF-8 characters.

''(The following steps displayed in italics only need to be done if the Windows Command Prompt is used.)''
* Copy the files CharReplacer.class and CharReplacer.settings in the folder in which the exported files are (in the example from above it is C:\DOS\DP26Y\DIAS\)
* Start the console.
* Change to the directory in which the files are.
* ''Set the Font for the Command Prompt to '''Lucida Console'''''
* ''Switch the Character Set for the console by typing '''chcp 65001'''''
* run the program by typing '''java -Dfile.encoding=UTF-8 CharReplacer''' followed by the name or names of the files you want to run the program on, e.g. ''java -Dfile.encoding=UTF-8 CharReplacer OUTPUT1.EXP'' and hit Enter.

{|
|[[File:DPE17 Console change Font.png|frame|none|Changing the Font in the Windows Command Prompt (German version)]]
|[[File:DPE18 Console start CharReplacer.png|frame|none|Running the CharReplacer]]
|}

The output of the program will be a file with the name of the input file, but with an additional '''.csv''' extension at the end. So '''OUTPUT1.EXP''' will become '''OUTPUT1.EXP.csv'''.

If the program find characters which are not in the list of allowed characters (the first line of the file) and for which no replacement rule exists, it will not replace this character, but it will show a message, informing the user that an unknown character was found, where it was found (file, line and column), what it looks like in UTF-8 and what its code is. If this occurs it is necessary to review the file at this position and create a new rule on how to handle that character. If the program prints out an unknown character, open the file and look at the given position. If the character is correct and represents precisely what was meant in the original dataset, then add this character at the end of the first line in the settings file. If it is a wrong character however, try to figure out what character it is meant to be. This could be derived from the context in which this unknown character appears. In the example below, the unknown character is displayed as '''†''' and appears in the work '''"R†dhusplassen"'''. People how are familiar with Norwegian might recognize the word as '''"Rådhusplassen"''' (the Norwegian word for "City Hall Place" or "Town Hall Place"). In other cased it could be not so clear. So it becomes necessary to look up the entry in the original DataPerfect table and see, what character was originally entered. In the third image, it can be seen that in this case the unknown character is indeed a '''å'''. So a new line can be added at the end of the settings file with the following content:
8224 å
The 8224 is the code for the character, as shown in the message of the CharReplacer program in the first image.

To edit the ''CharReplacer.settings'' file, use a regular text editor (like Notepad++ under Windows).

{|
|[[File:DPE19 Console run CharReplacer.png‎|frame|none|some unknown characters were found]]
|[[File:DPE20_Notepad_Special_Characters.png‎|frame|none|Viewing the unconverted character in Notepad++]]
|[[File:DPE21 DataPerfect Special Characters.png|frame|none|Viewing the unconverted character in the original data entry in DataPerfect.]]
|}

=== Documenting Foreign Key Relations ===
When the data is exported out of DataPerfect the associations between the tables are lost. So it is important to take a look at the exported data and see the relation between the tables. What columns are unique keys for their table and what columns are foreign key links to other tables. Looking at the tables in DataPerfect is helpful in this context, as the UI of DataPerfect sometimes shows columns which are not part of the export of that table, so they must be loaded from a different table. Sometimes when selecting a field in DataPerfect it also opens the window to another table, making it obvious that this column is a foreign key connection.

=== Importing into a modern relational database ===
Once the data is exported, converted into UTF-8 and the foreign key relations have been analyzed, it is possible to import the data in a modern relational data base, so it can be accessed by the BioCASe Provider Software. To do this, one must first create a database for it and the respective tables. The columns of the tables must already be prepared. It is important that the maximal length allowed for the columns is not smaller then the longest entry in this columns. Also the data formats must be correct. Though it is possible to store any kind of data in a text field, it will be more useful if the data is actually stored using the format it actually is.

=== Next Steps ===
Depending on how the tables are structured, it might become necessary to do a controlled denormalization so that the entries can be easier mapped to ABCD. This is described on the [http://wiki.bgbm.org/bps/index.php/Preparation Preparation page] of the [http://wiki.bgbm.org/bps BioCASe Provider Software Wiki].

The other following steps are also described at the BioCASe Provider Software Wiki, like the mapping of the ABCD concepts.

=== Alternative Ways of exporting data from a DataPerfect database ===
Another way of exporting DataPerfect files is the [http://dans-dp-lib.sourceforge.net/ DANS DataPerfect Library]. There is a reference implementation of a [http://dans-dp-lib.sourceforge.net/Dp2MySqlExport.java DP2MySQLConverter]. After testing it on small sample databases, it worked fine, but it ran intro problems exporting the DIAS database, causing huge and corrupt output files. For other DataPerfect files it could still be useful, especially since it already handles the special characters correctly.

Export DataPerfect

2012-03-19T17:10:12Z

DavidFichtmueller: /* Documenting Foreign Key Relations */

This article describes how to export data from a DataPerfect database, based on the Rohwer data set.

Data Perfect is a DOS based database system. It's latest release is 2.6Y from June 2008. It can be downloaded via http://dataperfect.nl/ .

=== Install and run DOSBox ===
: To run DataPerfect a DOS emulator is needed. The free emulator DOSBox works quite well. Go to http://www.dosbox.com/download.php?main=1 or http://sourceforge.net/projects/dosbox/, download the latest release and install it. For this articles DOSBox 0.74 will be used and installed under Windows 7.

=== Download DataPerfect ===
: Go to http://dataperfect.nl/ and download the DataPerfect. Unzip the files into a specific folder. The folder used for this article is ''C:/DOS/DP26Y''.

=== Copy your data files in the DataPerfect Folder===
: You can either copy them directly in the folder or create a subfolder for your data files and copy them in there. The data files for this article are located in the directory ''DIAS/'' within the DataPerfect folder.

=== Start DOSBox===
: When starting DOSBox a second console window is opened. When running several instances of DOSBox these additional windows can bloat the window bar quite a bit. To avoid this just start the version ''DOXBox-0.74/Extras/DOSBox 0.74 (noconsole)'' from the Program Menu.
[[File:DPE01_DOSBox.png|frame|none|The DOSBOX Start Window]]

=== Mount the DataPerfect Folder===
: After starting DOSBox the directory of the DataPerfect files needs to be mounted as a virtual drive. Type '''mount c C:\DOS\DP27Y''' to mount the folder and '''C:''' to change to the drive.
[[File:DPE02_DOSBox_mount.png|frame|none|Mounting the DataPerfect Folder]]

=== Start DataPerfect===
: Start DataPerfect by typing '''DP26YU''' and pressing Enter.
{|
|[[File:DPE03_DataPerfect_start.png|frame|none|Switch to C: and start DataPerfect]]
|[[File:DPE04_DataPerfect_welcome.png|frame|none|The Welcome Screen of DataPerfect]]
|}

=== Change to the folder of your data files===
: If the data files are not in the directory of DataPerfect you need to change the directory. Press '''2''' to change the directory and type in the path to the directory of your data.
{|
|[[File:DPE05_DataPerfect_open.png|frame|none| The initial screen of DataPerfect]]
|[[File:DPE06_DataPerfect_change_directory.png|frame|none|Change to the data directory]]
|}

=== Open the data files===
: If you are in the correct directory you see the DataPerfect datasets within the folder. Use the cursor keys to select the correct data set and press enter.
[[File:DPE07_DataPerfect_select_DIAS.png|frame|none|Selecting the DIAS data set.]]

=== Select the table to export ===
You can now see the tables within this database project. Select the table you want to export by using the up or down keys and pressing enter.
{|
|[[File:DPE08_DIAS_Overview.png|frame|none|The list of available tables.]]
|[[File:DPE09_DIAS_Table1.png|frame|none|The first entry in the first table of DIAS]]
|}
=== Navigate the Table ===
''This these steps are not necessary for the export, but will be documented here anyway.''
Here are important keys for navigating thought a table:
* '''Tab''' the next field is highlighted
* '''Down''' on a field which has references to entries in another table, that referenced entry is displayed
* '''Up''' The list of entries in this table where the content of the current field is displayed. Navigate though this list using the '''Up''' and '''Down''' keys. When typing characters this list the focus will jump to the entry whose unique key column is like the typed characters. For example in the table displayed in the image below, typing a number will focus on the entry with that id. The corresponding entry is automatically displayed. To edit this entry, press '''Enter'''.
* '''F7''' Goes to the next upper level in the hierarchy. For example if the list of entries is opened, it will jump back to the next upper level in the hierarchy.
[[File:DPE10_DIAS_Table1_browse.png|frame|none|Browsing through the first table of DIAS]]

=== Select Report ===
From the list of available reports select the entry at the top: '''Build-In Short Reports'''.
[[File:DPE11_DIAS_Table1_Report_List.png|frame|none|The list of available reports.]]

=== Set Export Properties ===
Set the export settings as shown in the first image below. Press the number of the property you want to change. For example, to change the file name of the output file, press '''2''' and the file options will appear, as can be seen in the second image. Press '''1''' to create a new file and enter the name of the file (image 3).
The filename of the export file must not be longer than 8 characters without file extension and the file extension itself (without the dot) must not be longer than 3 characters.
{|
|[[File:DPE12_DIAS_Table1_Export_1.png|frame|none|Export Settings Overview]]
|[[File:DPE13_DIAS_Table1_Export_2.png|frame|none|Changing the File Options]]
|[[File:DPE14_DIAS_Table1_Export_3.png|frame|none|Changing the File Name]]
|}

=== Run Export ===
To start the export press '''Shift + F7''' again. The screen now shows a counter of how many elements from the table have already been exported. DOSBox starts with a limited CPU speed for the programs running in it. The speed can be increased by pressing '''Ctrl + F12'''. Increasing the speed to much, however creates a big overhead which will also result in a slower export. For some reason when increasing the speed the CPU load will jump from around 5% of one core to 100% within just 2-3 increase steps. So the CPU load should be watched when increasing the speed of DOSBox. On a DualCore 3 GHz Processor a speed of 50000-55000 cycles (can be seen in the title bar of the DOSBox Window) appears to be a good export speed without overhead.
[[File:DPE15_DIAS_Export.png|frame|none|The export is running.]]

After the export is done, you will see the list of available reports again. To step to the next higher level of the hierarchy press '''F7'''. Repeat this until you see the list of tables within this database project again. Now repeat the export process for all the other tables you want to export.

=== Converting special characters ===
The best way to handle special characters is to know the character encoding by the original file system. If it is a DOS based system, the command '''CHCP''' will display what Character Code Page used. With this knowledge the file can be easily converted. In DOSBox the Command is '''KEYB''' however, it only helps if all of the special characters are displayed correctly within the DataPerfect. In the case of the Rohwer data set, the KEYB command showed Codepage 437, however Codepage 850 was actually used, must of the special characters are however identical between the two sets.

Once the character encoding is known the exported file converted by a program that is able to read that encoding. Under Windows Notepad++ does a good job. After opening the file (the special characters are probably distorted), one must select the original character encoding as the encoding of the file and then convert the file to UTF-8.

Google Refine is another software that is able to read the CP 850 encoding and allows for further processing of the files.

Additional Infomation about Codepages can be found on the Wikipedia Article ''[http://en.wikipedia.org/wiki/Code_page Code page]''. Also the article ''[http://www.joelonsoftware.com/articles/Unicode.html The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)]'' by Joel Spolsky is very helpful.

In retrospective we now know that the Rohwer data set is encoded using the Codepage 850 (also called OEM 850), however some single characters are in the wrong encoding. This however is the result of the process where we parsed the file, got a list of all the different characters used and looked them up and converted them by hand. Though this process is a lot more work, but it works even in the case of mixed character sets. It is documented in detail below.

==== converting characters individually ====
The export out of DataPerfect has problems with special characters which are not converted correctly in a proper character set. Though all of the occurrences of such special characters can be replaced automatically, each of the character to be replaced has to be defined manually one. The files as they are generated by DataPerfect use the ANSI Character Encoding. This needs to be converted to UTF8. Under Windows the free program Notepad++ is very suited for this. When selecting the menu item '''Encoding''' (''Kodierung'' in the image, because it is the German version of Notepad++), the entry ''ANSI'' should be marked as the current encoding. Now click on '''Convert to UTF-8''' and save the file.
[[File:DPE16 Notepad CharSet.png|frame|none|Converting the Character Set in Notepad++]]
The next step is to run the file through a program that converts the characters. The small Java program CharReplacer was written just to do that. There are two different files to that program: CharReplacer.class (the acutual program) and CharReplacer.settings (the settings file, which specifies, what characters to replace. The first line in ''CharReplacer.settings'' are all the characters which will not be replaced. All the following lines have the number value of the character which will be replaced, followed by a tab and the character it will be replaced with.

abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789",.-; ()?\':/=&[]!`<>#+%
8222 ä
8221 ö
129 ü
225 ß
381 Ä
8482 Ö
353 Ü
Excerpt from the ''CharReplacer.settings'' file.

To run the CharReplacer program, the Java Runtime Envirement (JRE) must be installed on a computer. It must be run via the command line interface (also known as console).

Under Windows the Command Prompt (cmd.exe) can be used, but it needs to be adjusted to show the UTF-8 characters.

''(The following steps displayed in italics only need to be done if the Windows Command Prompt is used.)''
* Copy the files CharReplacer.class and CharReplacer.settings in the folder in which the exported files are (in the example from above it is C:\DOS\DP26Y\DIAS\)
* Start the console.
* Change to the directory in which the files are.
* ''Set the Font for the Command Prompt to '''Lucida Console'''''
* ''Switch the Character Set for the console by typing '''chcp 65001'''''
* run the program by typing '''java -Dfile.encoding=UTF-8 CharReplacer''' followed by the name or names of the files you want to run the program on, e.g. ''java -Dfile.encoding=UTF-8 CharReplacer OUTPUT1.EXP'' and hit Enter.

{|
|[[File:DPE17 Console change Font.png|frame|none|Changing the Font in the Windows Command Prompt (German version)]]
|[[File:DPE18 Console start CharReplacer.png|frame|none|Running the CharReplacer]]
|}

The output of the program will be a file with the name of the input file, but with an additional '''.csv''' extension at the end. So '''OUTPUT1.EXP''' will become '''OUTPUT1.EXP.csv'''.

If the program find characters which are not in the list of allowed characters (the first line of the file) and for which no replacement rule exists, it will not replace this character, but it will show a message, informing the user that an unknown character was found, where it was found (file, line and column), what it looks like in UTF-8 and what its code is. If this occurs it is necessary to review the file at this position and create a new rule on how to handle that character. If the program prints out an unknown character, open the file and look at the given position. If the character is correct and represents precisely what was meant in the original dataset, then add this character at the end of the first line in the settings file. If it is a wrong character however, try to figure out what character it is meant to be. This could be derived from the context in which this unknown character appears. In the example below, the unknown character is displayed as '''†''' and appears in the work '''"R†dhusplassen"'''. People how are familiar with Norwegian might recognize the word as '''"Rådhusplassen"''' (the Norwegian word for "City Hall Place" or "Town Hall Place"). In other cased it could be not so clear. So it becomes necessary to look up the entry in the original DataPerfect table and see, what character was originally entered. In the third image, it can be seen that in this case the unknown character is indeed a '''å'''. So a new line can be added at the end of the settings file with the following content:
8224 å
The 8224 is the code for the character, as shown in the message of the CharReplacer program in the first image.

To edit the ''CharReplacer.settings'' file, use a regular text editor (like Notepad++ under Windows).

{|
|[[File:DPE19 Console run CharReplacer.png‎|frame|none|some unknown characters were found]]
|[[File:DPE20_Notepad_Special_Characters.png‎|frame|none|Viewing the unconverted character in Notepad++]]
|[[File:DPE21 DataPerfect Special Characters.png|frame|none|Viewing the unconverted character in the original data entry in DataPerfect.]]
|}

=== Documenting Foreign Key Relations ===
When the data is exported out of DataPerfect the associations between the tables are lost. So it is important to take a look at the exported data and see the relation between the tables. What columns are unique keys for their table and what columns are foreign key links to other tables.

=== Importing into a modern relational database ===

=== Next Steps ===

=== Alternative Ways of exporting data from a DataPerfect database ===
Another way of exporting DataPerfect files is the [http://dans-dp-lib.sourceforge.net/ DANS DataPerfect Library]. There is a reference implementation of a [http://dans-dp-lib.sourceforge.net/Dp2MySqlExport.java DP2MySQLConverter]. After testing it on small sample databases, it worked fine, but it ran intro problems exporting the DIAS database, causing huge and corrupt output files. For other DataPerfect files it could still be useful, especially since it already handles the special characters correctly.

Export DataPerfect

2012-03-19T17:07:42Z

DavidFichtmueller: /* Converting special characters */ simple automated conversion based on the known character encoding

This article describes how to export data from a DataPerfect database, based on the Rohwer data set.

Data Perfect is a DOS based database system. It's latest release is 2.6Y from June 2008. It can be downloaded via http://dataperfect.nl/ .

=== Install and run DOSBox ===
: To run DataPerfect a DOS emulator is needed. The free emulator DOSBox works quite well. Go to http://www.dosbox.com/download.php?main=1 or http://sourceforge.net/projects/dosbox/, download the latest release and install it. For this articles DOSBox 0.74 will be used and installed under Windows 7.

=== Download DataPerfect ===
: Go to http://dataperfect.nl/ and download the DataPerfect. Unzip the files into a specific folder. The folder used for this article is ''C:/DOS/DP26Y''.

=== Copy your data files in the DataPerfect Folder===
: You can either copy them directly in the folder or create a subfolder for your data files and copy them in there. The data files for this article are located in the directory ''DIAS/'' within the DataPerfect folder.

=== Start DOSBox===
: When starting DOSBox a second console window is opened. When running several instances of DOSBox these additional windows can bloat the window bar quite a bit. To avoid this just start the version ''DOXBox-0.74/Extras/DOSBox 0.74 (noconsole)'' from the Program Menu.
[[File:DPE01_DOSBox.png|frame|none|The DOSBOX Start Window]]

=== Mount the DataPerfect Folder===
: After starting DOSBox the directory of the DataPerfect files needs to be mounted as a virtual drive. Type '''mount c C:\DOS\DP27Y''' to mount the folder and '''C:''' to change to the drive.
[[File:DPE02_DOSBox_mount.png|frame|none|Mounting the DataPerfect Folder]]

=== Start DataPerfect===
: Start DataPerfect by typing '''DP26YU''' and pressing Enter.
{|
|[[File:DPE03_DataPerfect_start.png|frame|none|Switch to C: and start DataPerfect]]
|[[File:DPE04_DataPerfect_welcome.png|frame|none|The Welcome Screen of DataPerfect]]
|}

=== Change to the folder of your data files===
: If the data files are not in the directory of DataPerfect you need to change the directory. Press '''2''' to change the directory and type in the path to the directory of your data.
{|
|[[File:DPE05_DataPerfect_open.png|frame|none| The initial screen of DataPerfect]]
|[[File:DPE06_DataPerfect_change_directory.png|frame|none|Change to the data directory]]
|}

=== Open the data files===
: If you are in the correct directory you see the DataPerfect datasets within the folder. Use the cursor keys to select the correct data set and press enter.
[[File:DPE07_DataPerfect_select_DIAS.png|frame|none|Selecting the DIAS data set.]]

=== Select the table to export ===
You can now see the tables within this database project. Select the table you want to export by using the up or down keys and pressing enter.
{|
|[[File:DPE08_DIAS_Overview.png|frame|none|The list of available tables.]]
|[[File:DPE09_DIAS_Table1.png|frame|none|The first entry in the first table of DIAS]]
|}
=== Navigate the Table ===
''This these steps are not necessary for the export, but will be documented here anyway.''
Here are important keys for navigating thought a table:
* '''Tab''' the next field is highlighted
* '''Down''' on a field which has references to entries in another table, that referenced entry is displayed
* '''Up''' The list of entries in this table where the content of the current field is displayed. Navigate though this list using the '''Up''' and '''Down''' keys. When typing characters this list the focus will jump to the entry whose unique key column is like the typed characters. For example in the table displayed in the image below, typing a number will focus on the entry with that id. The corresponding entry is automatically displayed. To edit this entry, press '''Enter'''.
* '''F7''' Goes to the next upper level in the hierarchy. For example if the list of entries is opened, it will jump back to the next upper level in the hierarchy.
[[File:DPE10_DIAS_Table1_browse.png|frame|none|Browsing through the first table of DIAS]]

=== Select Report ===
From the list of available reports select the entry at the top: '''Build-In Short Reports'''.
[[File:DPE11_DIAS_Table1_Report_List.png|frame|none|The list of available reports.]]

=== Set Export Properties ===
Set the export settings as shown in the first image below. Press the number of the property you want to change. For example, to change the file name of the output file, press '''2''' and the file options will appear, as can be seen in the second image. Press '''1''' to create a new file and enter the name of the file (image 3).
The filename of the export file must not be longer than 8 characters without file extension and the file extension itself (without the dot) must not be longer than 3 characters.
{|
|[[File:DPE12_DIAS_Table1_Export_1.png|frame|none|Export Settings Overview]]
|[[File:DPE13_DIAS_Table1_Export_2.png|frame|none|Changing the File Options]]
|[[File:DPE14_DIAS_Table1_Export_3.png|frame|none|Changing the File Name]]
|}

=== Run Export ===
To start the export press '''Shift + F7''' again. The screen now shows a counter of how many elements from the table have already been exported. DOSBox starts with a limited CPU speed for the programs running in it. The speed can be increased by pressing '''Ctrl + F12'''. Increasing the speed to much, however creates a big overhead which will also result in a slower export. For some reason when increasing the speed the CPU load will jump from around 5% of one core to 100% within just 2-3 increase steps. So the CPU load should be watched when increasing the speed of DOSBox. On a DualCore 3 GHz Processor a speed of 50000-55000 cycles (can be seen in the title bar of the DOSBox Window) appears to be a good export speed without overhead.
[[File:DPE15_DIAS_Export.png|frame|none|The export is running.]]

After the export is done, you will see the list of available reports again. To step to the next higher level of the hierarchy press '''F7'''. Repeat this until you see the list of tables within this database project again. Now repeat the export process for all the other tables you want to export.

=== Converting special characters ===
The best way to handle special characters is to know the character encoding by the original file system. If it is a DOS based system, the command '''CHCP''' will display what Character Code Page used. With this knowledge the file can be easily converted. In DOSBox the Command is '''KEYB''' however, it only helps if all of the special characters are displayed correctly within the DataPerfect. In the case of the Rohwer data set, the KEYB command showed Codepage 437, however Codepage 850 was actually used, must of the special characters are however identical between the two sets.

Once the character encoding is known the exported file converted by a program that is able to read that encoding. Under Windows Notepad++ does a good job. After opening the file (the special characters are probably distorted), one must select the original character encoding as the encoding of the file and then convert the file to UTF-8.

Google Refine is another software that is able to read the CP 850 encoding and allows for further processing of the files.

Additional Infomation about Codepages can be found on the Wikipedia Article ''[http://en.wikipedia.org/wiki/Code_page Code page]''. Also the article ''[http://www.joelonsoftware.com/articles/Unicode.html The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)]'' by Joel Spolsky is very helpful.

In retrospective we now know that the Rohwer data set is encoded using the Codepage 850 (also called OEM 850), however some single characters are in the wrong encoding. This however is the result of the process where we parsed the file, got a list of all the different characters used and looked them up and converted them by hand. Though this process is a lot more work, but it works even in the case of mixed character sets. It is documented in detail below.

==== converting characters individually ====
The export out of DataPerfect has problems with special characters which are not converted correctly in a proper character set. Though all of the occurrences of such special characters can be replaced automatically, each of the character to be replaced has to be defined manually one. The files as they are generated by DataPerfect use the ANSI Character Encoding. This needs to be converted to UTF8. Under Windows the free program Notepad++ is very suited for this. When selecting the menu item '''Encoding''' (''Kodierung'' in the image, because it is the German version of Notepad++), the entry ''ANSI'' should be marked as the current encoding. Now click on '''Convert to UTF-8''' and save the file.
[[File:DPE16 Notepad CharSet.png|frame|none|Converting the Character Set in Notepad++]]
The next step is to run the file through a program that converts the characters. The small Java program CharReplacer was written just to do that. There are two different files to that program: CharReplacer.class (the acutual program) and CharReplacer.settings (the settings file, which specifies, what characters to replace. The first line in ''CharReplacer.settings'' are all the characters which will not be replaced. All the following lines have the number value of the character which will be replaced, followed by a tab and the character it will be replaced with.

abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789",.-; ()?\':/=&[]!`<>#+%
8222 ä
8221 ö
129 ü
225 ß
381 Ä
8482 Ö
353 Ü
Excerpt from the ''CharReplacer.settings'' file.

To run the CharReplacer program, the Java Runtime Envirement (JRE) must be installed on a computer. It must be run via the command line interface (also known as console).

Under Windows the Command Prompt (cmd.exe) can be used, but it needs to be adjusted to show the UTF-8 characters.

''(The following steps displayed in italics only need to be done if the Windows Command Prompt is used.)''
* Copy the files CharReplacer.class and CharReplacer.settings in the folder in which the exported files are (in the example from above it is C:\DOS\DP26Y\DIAS\)
* Start the console.
* Change to the directory in which the files are.
* ''Set the Font for the Command Prompt to '''Lucida Console'''''
* ''Switch the Character Set for the console by typing '''chcp 65001'''''
* run the program by typing '''java -Dfile.encoding=UTF-8 CharReplacer''' followed by the name or names of the files you want to run the program on, e.g. ''java -Dfile.encoding=UTF-8 CharReplacer OUTPUT1.EXP'' and hit Enter.

{|
|[[File:DPE17 Console change Font.png|frame|none|Changing the Font in the Windows Command Prompt (German version)]]
|[[File:DPE18 Console start CharReplacer.png|frame|none|Running the CharReplacer]]
|}

The output of the program will be a file with the name of the input file, but with an additional '''.csv''' extension at the end. So '''OUTPUT1.EXP''' will become '''OUTPUT1.EXP.csv'''.

If the program find characters which are not in the list of allowed characters (the first line of the file) and for which no replacement rule exists, it will not replace this character, but it will show a message, informing the user that an unknown character was found, where it was found (file, line and column), what it looks like in UTF-8 and what its code is. If this occurs it is necessary to review the file at this position and create a new rule on how to handle that character. If the program prints out an unknown character, open the file and look at the given position. If the character is correct and represents precisely what was meant in the original dataset, then add this character at the end of the first line in the settings file. If it is a wrong character however, try to figure out what character it is meant to be. This could be derived from the context in which this unknown character appears. In the example below, the unknown character is displayed as '''†''' and appears in the work '''"R†dhusplassen"'''. People how are familiar with Norwegian might recognize the word as '''"Rådhusplassen"''' (the Norwegian word for "City Hall Place" or "Town Hall Place"). In other cased it could be not so clear. So it becomes necessary to look up the entry in the original DataPerfect table and see, what character was originally entered. In the third image, it can be seen that in this case the unknown character is indeed a '''å'''. So a new line can be added at the end of the settings file with the following content:
8224 å
The 8224 is the code for the character, as shown in the message of the CharReplacer program in the first image.

To edit the ''CharReplacer.settings'' file, use a regular text editor (like Notepad++ under Windows).

{|
|[[File:DPE19 Console run CharReplacer.png‎|frame|none|some unknown characters were found]]
|[[File:DPE20_Notepad_Special_Characters.png‎|frame|none|Viewing the unconverted character in Notepad++]]
|[[File:DPE21 DataPerfect Special Characters.png|frame|none|Viewing the unconverted character in the original data entry in DataPerfect.]]
|}

=== Documenting Foreign Key Relations ===

=== Importing into a modern relational database ===

=== Next Steps ===

=== Alternative Ways of exporting data from a DataPerfect database ===
Another way of exporting DataPerfect files is the [http://dans-dp-lib.sourceforge.net/ DANS DataPerfect Library]. There is a reference implementation of a [http://dans-dp-lib.sourceforge.net/Dp2MySqlExport.java DP2MySQLConverter]. After testing it on small sample databases, it worked fine, but it ran intro problems exporting the DIAS database, causing huge and corrupt output files. For other DataPerfect files it could still be useful, especially since it already handles the special characters correctly.

Export DataPerfect

2012-03-07T15:06:44Z

DavidFichtmueller: special characters

This article describes how to export data from a DataPerfect database, based on the Rohwer data set.

Data Perfect is a DOS based database system. It's latest release is 2.6Y from June 2008. It can be downloaded via http://dataperfect.nl/ .

=== Install and run DOSBox ===
: To run DataPerfect a DOS emulator is needed. The free emulator DOSBox works quite well. Go to http://www.dosbox.com/download.php?main=1 or http://sourceforge.net/projects/dosbox/, download the latest release and install it. For this articles DOSBox 0.74 will be used and installed under Windows 7.

=== Download DataPerfect ===
: Go to http://dataperfect.nl/ and download the DataPerfect. Unzip the files into a specific folder. The folder used for this article is ''C:/DOS/DP26Y''.

=== Copy your data files in the DataPerfect Folder===
: You can either copy them directly in the folder or create a subfolder for your data files and copy them in there. The data files for this article are located in the directory ''DIAS/'' within the DataPerfect folder.

=== Start DOSBox===
: When starting DOSBox a second console window is opened. When running several instances of DOSBox these additional windows can bloat the window bar quite a bit. To avoid this just start the version ''DOXBox-0.74/Extras/DOSBox 0.74 (noconsole)'' from the Program Menu.
[[File:DPE01_DOSBox.png|frame|none|The DOSBOX Start Window]]

=== Mount the DataPerfect Folder===
: After starting DOSBox the directory of the DataPerfect files needs to be mounted as a virtual drive. Type '''mount c C:\DOS\DP27Y''' to mount the folder and '''C:''' to change to the drive.
[[File:DPE02_DOSBox_mount.png|frame|none|Mounting the DataPerfect Folder]]

=== Start DataPerfect===
: Start DataPerfect by typing '''DP26YU''' and pressing Enter.
{|
|[[File:DPE03_DataPerfect_start.png|frame|none|Switch to C: and start DataPerfect]]
|[[File:DPE04_DataPerfect_welcome.png|frame|none|The Welcome Screen of DataPerfect]]
|}

=== Change to the folder of your data files===
: If the data files are not in the directory of DataPerfect you need to change the directory. Press '''2''' to change the directory and type in the path to the directory of your data.
{|
|[[File:DPE05_DataPerfect_open.png|frame|none| The initial screen of DataPerfect]]
|[[File:DPE06_DataPerfect_change_directory.png|frame|none|Change to the data directory]]
|}

=== Open the data files===
: If you are in the correct directory you see the DataPerfect datasets within the folder. Use the cursor keys to select the correct data set and press enter.
[[File:DPE07_DataPerfect_select_DIAS.png|frame|none|Selecting the DIAS data set.]]

=== Select the table to export ===
You can now see the tables within this database project. Select the table you want to export by using the up or down keys and pressing enter.
{|
|[[File:DPE08_DIAS_Overview.png|frame|none|The list of available tables.]]
|[[File:DPE09_DIAS_Table1.png|frame|none|The first entry in the first table of DIAS]]
|}
=== Navigate the Table ===
''This these steps are not necessary for the export, but will be documented here anyway.''
Here are important keys for navigating thought a table:
* '''Tab''' the next field is highlighted
* '''Down''' on a field which has references to entries in another table, that referenced entry is displayed
* '''Up''' The list of entries in this table where the content of the current field is displayed. Navigate though this list using the '''Up''' and '''Down''' keys. When typing characters this list the focus will jump to the entry whose unique key column is like the typed characters. For example in the table displayed in the image below, typing a number will focus on the entry with that id. The corresponding entry is automatically displayed. To edit this entry, press '''Enter'''.
* '''F7''' Goes to the next upper level in the hierarchy. For example if the list of entries is opened, it will jump back to the next upper level in the hierarchy.
[[File:DPE10_DIAS_Table1_browse.png|frame|none|Browsing through the first table of DIAS]]

=== Select Report ===
From the list of available reports select the entry at the top: '''Build-In Short Reports'''.
[[File:DPE11_DIAS_Table1_Report_List.png|frame|none|The list of available reports.]]

=== Set Export Properties ===
Set the export settings as shown in the first image below. Press the number of the property you want to change. For example, to change the file name of the output file, press '''2''' and the file options will appear, as can be seen in the second image. Press '''1''' to create a new file and enter the name of the file (image 3).
The filename of the export file must not be longer than 8 characters without file extension and the file extension itself (without the dot) must not be longer than 3 characters.
{|
|[[File:DPE12_DIAS_Table1_Export_1.png|frame|none|Export Settings Overview]]
|[[File:DPE13_DIAS_Table1_Export_2.png|frame|none|Changing the File Options]]
|[[File:DPE14_DIAS_Table1_Export_3.png|frame|none|Changing the File Name]]
|}

=== Run Export ===
To start the export press '''Shift + F7''' again. The screen now shows a counter of how many elements from the table have already been exported. DOSBox starts with a limited CPU speed for the programs running in it. The speed can be increased by pressing '''Ctrl + F12'''. Increasing the speed to much, however creates a big overhead which will also result in a slower export. For some reason when increasing the speed the CPU load will jump from around 5% of one core to 100% within just 2-3 increase steps. So the CPU load should be watched when increasing the speed of DOSBox. On a DualCore 3 GHz Processor a speed of 50000-55000 cycles (can be seen in the title bar of the DOSBox Window) appears to be a good export speed without overhead.
[[File:DPE15_DIAS_Export.png|frame|none|The export is running.]]

After the export is done, you will see the list of available reports again. To step to the next higher level of the hierarchy press '''F7'''. Repeat this until you see the list of tables within this database project again. Now repeat the export process for all the other tables you want to export.

=== Converting special characters ===
The export out of DataPerfect has problems with special characters which are not converted correctly in a proper character set. Though all of the occurrences of such special characters can be replaced automatically, each of the character to be replaced has to be defined manually one. The files as they are generated by DataPerfect use the ANSI Character Encoding. This needs to be converted to UTF8. Under Windows the free program Notepad++ is very suited for this. When selecting the menu item '''Encoding''' (''Kodierung'' in the image, because it is the German version of Notepad++), the entry ''ANSI'' should be marked as the current encoding. Now click on '''Convert to UTF-8''' and save the file.
[[File:DPE16 Notepad CharSet.png|frame|none|Converting the Character Set in Notepad++]]
The next step is to run the file through a program that converts the characters. The small Java program CharReplacer was written just to do that. There are two different files to that program: CharReplacer.class (the acutual program) and CharReplacer.settings (the settings file, which specifies, what characters to replace. The first line in ''CharReplacer.settings'' are all the characters which will not be replaced. All the following lines have the number value of the character which will be replaced, followed by a tab and the character it will be replaced with.

abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789",.-; ()?\':/=&[]!`<>#+%
8222 ä
8221 ö
129 ü
225 ß
381 Ä
8482 Ö
353 Ü
Excerpt from the ''CharReplacer.settings'' file.

To run the CharReplacer program, the Java Runtime Envirement (JRE) must be installed on a computer. It must be run via the command line interface (also known as console).

Under Windows the Command Prompt (cmd.exe) can be used, but it needs to be adjusted to show the UTF-8 characters.

''(The following steps displayed in italics only need to be done if the Windows Command Prompt is used.)''
* Copy the files CharReplacer.class and CharReplacer.settings in the folder in which the exported files are (in the example from above it is C:\DOS\DP26Y\DIAS\)
* Start the console.
* Change to the directory in which the files are.
* ''Set the Font for the Command Prompt to '''Lucida Console'''''
* ''Switch the Character Set for the console by typing '''chcp 65001'''''
* run the program by typing '''java -Dfile.encoding=UTF-8 CharReplacer''' followed by the name or names of the files you want to run the program on, e.g. ''java -Dfile.encoding=UTF-8 CharReplacer OUTPUT1.EXP'' and hit Enter.

{|
|[[File:DPE17 Console change Font.png|frame|none|Changing the Font in the Windows Command Prompt (German version)]]
|[[File:DPE18 Console start CharReplacer.png|frame|none|Running the CharReplacer]]
|}

The output of the program will be a file with the name of the input file, but with an additional '''.csv''' extension at the end. So '''OUTPUT1.EXP''' will become '''OUTPUT1.EXP.csv'''.

If the program find characters which are not in the list of allowed characters (the first line of the file) and for which no replacement rule exists, it will not replace this character, but it will show a message, informing the user that an unknown character was found, where it was found (file, line and column), what it looks like in UTF-8 and what its code is. If this occurs it is necessary to review the file at this position and create a new rule on how to handle that character. If the program prints out an unknown character, open the file and look at the given position. If the character is correct and represents precisely what was meant in the original dataset, then add this character at the end of the first line in the settings file. If it is a wrong character however, try to figure out what character it is meant to be. This could be derived from the context in which this unknown character appears. In the example below, the unknown character is displayed as '''†''' and appears in the work '''"R†dhusplassen"'''. People how are familiar with Norwegian might recognize the word as '''"Rådhusplassen"''' (the Norwegian word for "City Hall Place" or "Town Hall Place"). In other cased it could be not so clear. So it becomes necessary to look up the entry in the original DataPerfect table and see, what character was originally entered. In the third image, it can be seen that in this case the unknown character is indeed a '''å'''. So a new line can be added at the end of the settings file with the following content:
8224 å
The 8224 is the code for the character, as shown in the message of the CharReplacer program in the first image.

To edit the ''CharReplacer.settings'' file, use a regular text editor (like Notepad++ under Windows).

{|
|[[File:DPE19 Console run CharReplacer.png‎|frame|none|some unknown characters were found]]
|[[File:DPE20_Notepad_Special_Characters.png‎|frame|none|Viewing the unconverted character in Notepad++]]
|[[File:DPE21 DataPerfect Special Characters.png|frame|none|Viewing the unconverted character in the original data entry in DataPerfect.]]
|}

=== Documenting Foreign Key Relations ===

=== Importing into a modern relational database ===

=== Next Steps ===

=== Alternative Ways of exporting data from a DataPerfect database ===
Another way of exporting DataPerfect files is the [http://dans-dp-lib.sourceforge.net/ DANS DataPerfect Library]. There is a reference implementation of a [http://dans-dp-lib.sourceforge.net/Dp2MySqlExport.java DP2MySQLConverter]. After testing it on small sample databases, it worked fine, but it ran intro problems exporting the DIAS database, causing huge and corrupt output files. For other DataPerfect files it could still be useful, especially since it already handles the special characters correctly.

Export DataPerfect

2012-03-06T11:15:30Z

DavidFichtmueller: additional information

Export DataPerfect

2012-03-06T09:57:14Z

DavidFichtmueller: images added

Export DataPerfect

2012-03-05T17:31:08Z

DavidFichtmueller: first steps