BioCASe Provider Software - User contributions [en]

VersionHistory

2026-01-30T09:59:49Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Version 3.8.9 [2026-01-30] ===
* DarwinCore archive transformation
** DwC terms ''recordedByID'', ''identifiedByID'', ''scientificNameID'', ''associatedSequences'' and ''preparations'' are now filled from respective ABCD fields.
** ''License/Text'' and ''License/URI'' from ABCD are now stored in ''intellectualRights'' paragraph in EML document, so that license can be parsed from GBIF (and the DwC archives successfully pass the GBIF Data Validator test).
* New ABCD subversion 2.06g added. This subversion allows semantic annotation of scientificName, identifierPerson and gatheringAgent (using ''ResourceURI'' data elements).

=== Version 3.8.8 [2025-08-05] ===
* Usage of cgi-traceback module removed (dropped in Python v3.13)
* Legcacy-cgi included in BPS (dropped in Python v3.13)
* Default webroot changed to "/" in Docker image

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2026-01-30T09:54:46Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* DarwinCore archive transformation
** DwC terms ''recordedByID'', ''identifiedByID'', ''scientificNameID'', ''associatedSequences'' and ''preparations'' are now filled from respective ABCD fields.
** ''License/Text'' and ''License/URI'' from ABCD are now stored in ''intellectualRights'' paragraph in EML document, so that license can be parsed from GBIF (and the DwC archives successfully pass the GBIF Data Validator test).
* New ABCD subversion 2.06g added. This subversion allows semantic annotation of scientificName, identifierPerson and gatheringAgent (using ''ResourceURI'' data elements).

=== Version 3.8.8 [2025-08-05] ===
* Usage of cgi-traceback module removed (dropped in Python v3.13)
* Legcacy-cgi included in BPS (dropped in Python v3.13)
* Default webroot changed to "/" in Docker image

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-12-11T17:03:32Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* DarwinCore archive transformation
** DwC terms ''recordedByID'', ''identifiedByID'', ''scientificNameID'', ''associatedSequences'' and ''preparations'' are now filled from respective ABCD fields.
** ''License/Text'' and ''License/URI'' from ABCD are now stored in ''intellectualRights'' paragraph in EML document, so that license can be parsed from GBIF (and the DwC archives successfully pass the GBIF Data Validator test).

=== Version 3.8.8 [2025-08-05] ===
* Usage of cgi-traceback module removed (dropped in Python v3.13)
* Legcacy-cgi included in BPS (dropped in Python v3.13)
* Default webroot changed to "/" in Docker image

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-12-11T09:42:50Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Upcoming ===
* DarwinCore archive transformation: DwC terms recordedByID, identifiedByID, scientificNameID, associatedSequences and preparations are now filled from respective ABCD fields.

=== Version 3.8.8 [2025-08-05] ===
* Usage of cgi-traceback module removed (dropped in Python v3.13)
* Legcacy-cgi included in BPS (dropped in Python v3.13)
* Default webroot changed to "/" in Docker image

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

Installation

2025-08-05T11:53:44Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181'' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Tell BioCASe your dockerhost''' Last thing to do is to tell BioCASe the name of your dockerhost: On the start page of BioCASe, choose Config Tool, then System Administration. Enter the password you just set and click Submit. Type in the URL you just used in your browser (''http://<dockerhost>'' or ''http://<dockerhost>:8181'' if you used port 8181) in the first line as Webserver domain.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-08-05T11:51:51Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181'' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Tell BioCASe your dockerhost''' Last thing to do is to tell BioCASe the name of your dockerhost: On the start page of BioCASe, choose Config Tool, then System Administration. Enter the password you just set and click Submit. Find the line Base URL and type in the URL you just used in your browser (''http://<dockerhost>'' or ''http://<dockerhost>:8181'' if you used port 8181).

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-08-05T11:51:15Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181'' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

'''Tell BioCASe your dockerhost''' Last thing to do is to tell BioCASe the name of your dockerhost: On the start page of BioCASe, choose Config Tool, then System Administration. Enter the password you just set and click Submit. Find the line Base URL and type in the URL you just used in your browser (''http://<dockerhost>'' or ''http://<dockerhost>:8181'' if you used port 8181).

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-08-05T11:42:50Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.
Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:
[[File:DockerInstall.png|650px]]
Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181'' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

VersionHistory

2025-08-05T11:17:41Z

JoergHoletschek: /* Version 3.8.8 [2025-08-04] */

== Current Version 3 ==

=== Version 3.8.8 [2025-08-05] ===
* Usage of cgi-traceback module removed (dropped in Python v3.13)
* Legcacy-cgi included in BPS (dropped in Python v3.13)
* Default webroot changed to "/" in Docker image

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-08-04T10:08:15Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Version 3.8.8 [2025-08-04] ===
* Usage of cgi-traceback module removed (dropped in Python v3.13)
* Legcacy-cgi ncluded in BPS (dropped in Python v3.13)

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

Installation

2025-03-10T08:48:31Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181'' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-03-10T08:48:11Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>'' (or ''http://<dockerhost>:8181' if you used port 8181). Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-03-10T08:46:05Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://<dockerhost>/''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

VersionHistory

2025-03-06T10:04:55Z

JoergHoletschek: /* Version 3.8.7 [Upcoming] */

== Current Version 3 ==

=== Version 3.8.7 [2025-03-06] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-03-05T10:23:25Z

JoergHoletschek: /* Version 3.8.7 [2025-03-04] */

== Current Version 3 ==

=== Version 3.8.7 [Upcoming] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-03-04T14:03:28Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Version 3.8.7 [2025-03-04] ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-03-04T12:16:42Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Code repo moved to git: https://git.bgbm.org/biocase/bps
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!
* Miscellaneous optimisations in Docker image (multistage build & chained RUN statements for smaller image size, separate Apache config file, no root user) thanks to Petr Novotný from Prague university!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-02-13T12:07:19Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Update hint now uses link to new Git repo instead of Subversion
* Use alternative DB connector pymysql if importing MySQLdb fails (apparently on latest Ubuntu distributions). Thanks to Kjell-Åke Lundblad from Nordic Genetic Resource Center for this!

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

Installation

2025-02-11T15:06:51Z

JoergHoletschek: /* Downloading and unpacking the Provider Software */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.0''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-02-11T15:05:05Z

JoergHoletschek: /* Downloading and unpacking the Provider Software */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system. It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-02-11T15:03:55Z

JoergHoletschek: /* Downloading and unpacking the Provider Software */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [https://github.com/pentaho/pentaho-kettle Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system e.g. ''/opt'' on Linux). It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-02-11T15:02:59Z

JoergHoletschek: /* Downloading and unpacking the Provider Software */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [https://git.bgbm.org/biocase/bps/-/releases/permalink/latest Git page]. There are two archives available for download:
* The [https://git.bgbm.org/biocase/bps/-/archive/3.8.6/bps-3.8.6.zip default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [Kettle libraries] are included in the download.
* The [https://www.biocase.org/products/provider_software/files/bps-3.8.6-minimal.zip minimal package] doesn't contain the [https://github.com/pentaho/pentaho-kettle Kettle libraries], hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system e.g. ''/opt'' on Linux). It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

Installation

2025-02-11T14:57:26Z

JoergHoletschek: /* Downloading the Provider Software */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading and unpacking the Provider Software ===
You can find the latest release of BioCASe on our [http://www.example.com Git page]. There are two archives available for download:
* The [ default package (recommended)] contains the full installation and allows the generation of DarwinCore archives. For this feature, the [Kettle libraries] are included in the download.
* The [ minimal package] doesn't contain the Kettle libraries, hence it's not possible to create DarwinCore archives with such an installation. Choose this download only if you have a good reason to do so.

Download the file to your server and unpack it to the desired location of your file system e.g. ''/opt'' on Linux). It contains the BioCASe Provider Software in a folder that contains the version number, e.g. ''bps-3.8.6''. We recommend to rename it to something that doesn't include the version number, so you don't need to touch your web server's configuration during future updates.

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

VersionHistory

2025-02-11T11:03:16Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Update hint now uses link to new Git repo instead of Subversion

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2025-01-21T09:11:38Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Upcoming ===

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2024-12-19T14:51:03Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Upcoming ===
* Dockerfile adapted to latest Debian version and Python 3.13

=== Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2024-12-19T14:50:44Z

JoergHoletschek: /* Version Version 3.8.6 [2024-12-12] */

Well written!
I’ll immediately clutch your rss feed as I can’t in finding your
e-mail subscription hyperlink or newsletter service.
Do you’ve any? Kindly permit me recognize in order that I may just subscribe.

VersionHistory

2024-12-12T08:51:52Z

JoergHoletschek: /* Version Version 3.8.5 [2023-11-30] [2024-12-12] */

== Current Version 3 ==

=== Version Version 3.8.6 [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2024-12-12T08:51:32Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Version Version 3.8.5 [2023-11-30] [2024-12-12] ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2024-11-14T15:06:43Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml
* DwC archive transformation: BasisOfRecord="AbsenceObservation" will be translated into "HumanObservation", since AbsenceObservation is not accepted by GBIF.

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2024-11-14T10:40:08Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly
* Fix for DwC archives: occurrenceStatus and occurrencID had been swapped in descriptor file meta.xml

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2024-11-04T11:51:26Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2
* QueryTool: Changed default schema from ABCD 2.1 to ABCD 2.06, since this is used more commonly

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

FAQ

2024-08-15T09:26:44Z

JoergHoletschek: /* Installation */

== General ==

===What is the BioCASe Provider Software and what can I use it for?===
See the [[BeginnersGuide|Beginner's Guide]].

===What do I need to use the BioCASe Provider Software?===
See the requirements in the [[BeginnersGuide|Beginner's Guide]].

===I’m not sure whether the Provider Software is the right choice for me. How do I find out?===
Just [[Special:Contact|contact us]] and describe your situation and what you’re planning to use BioCASe for.

===How much is the BioCASe Provider Software?===
It’s Open Source Software, so you can use it for free.

=== I am not affiliated with any institution. Can I use the Provider Software in spite of that? ===
Sure, you can also use the Provider Software as a private person.

===Can I use the Provider Software on my operating system?===
I guess. As long as a web server (such as Apache) and Python are available for your operating system, you can – and they are for most operating systems.

===I cannot meet all of the requirements for installing the BPS. What options do I have?===
Basically, there are two options: If you still want to use BioCASe (maybe because you owe very rich data and need the comprehensive ABCD schema for publication), you might ask a partner institution already using BioCASe to host your data. In this case, you would send the database (or Excel file) to this institution, where it would be published using the existing BioCASe installation. You would still remain owner of the data and attributed accordingly on the data portals; the partner institution would act merely as a technical host (like a commercial internet host).

Another option would be to use one of the tools offered by GBIF. With the Integrated Publishing Toolkit (IPT, http://www.gbif.org/ipt) you can create a DarwinCore Archive (see http://rs.tdwg.org/dwc/terms/guides/text/index.htm for more on DwC Archives) from your database, which you would send to GBIF for publication. The GBIF Spreadsheet Processor (http://code.google.com/p/gbif-spreadsheet-processor/) is a similar tool intended for smaller occurrence datasets stored in Excel files. The downside for both is that they support only the DarwinCore standard, which is rather limited in comparison to ABCD.

===I have some recommendations/bug report. How can I file it?===
You can either leave a message on the respective ''Discussion'' page or get directly in [[Special:Contact|contact with us]].

== Installation ==

===After installing BioCASe, I get a "500: Internal Server Error"===
# Make sure you've [[Installation#Running_the_Setup_Script | run the setup script]] after downloading or checking out BioCASe with the right Python version. To check this, open the file ''www/index.cgi'' of your installation in a text editor and check the first line: This is the Python executable used by Python.
# Empty the folder ''cache'' of your installation - please remove all folders and files you find here.
# If the problem persists, please contact us!

===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an empty page or a “URL not found” error. What’s wrong?===
If you get a blank page, that’s probably because Apache (or the web server you use) is not running. Start it and retry. If you get a “The requested URL /biocase was not found on this server” or similar message, the web server doesn’t find your BioCASe installation. Go to the section [[Installation#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]] of the [[Installation|Installation guide]] and follow the instructions.

===When typing in the URL for my new installation (<nowiki>http://localhost/biocase</nowiki>), I get an “Internal Server Error: The server encountered an internal error or misconfiguration and was unable to complete your request” message.===
You’re probably using Apache, which cannot find your Python installation. Make sure you’ve installed Python correctly and did run the setup script (see sections [[Installation#Installing_Apache.2FIIS_and_Python|Installing Apache/IIS and Python]] and [[Installation#Running_the_Setup_Script|Running the Setup Script]] in the [[Installation|Installation guide]]).

===I'm using Internet Information Server, and after creating the Virtual Directory for BioCASe, I get a “502.2 - Bad Gateway. The specified CGI application misbehaved by not returning a complete set of HTTP headers” error.===
This happens when you didn't correctly associate .cgi and .py files with Python. Make sure you've linked both files with Python and that the path contains the placeholders ''%s %s'' (see [[Installation#Internet_Information_Server|Linking IIS]]).

===I cannot find the installation package of the database connectivity module for the DBMS and the Python version I am using.===
You’re probably using one of the newer Python versions which are not yet supported by the DB package you want to use (that’s the reason Python 2.5 is recommended). Just install an older version (2.5 or 2.6). You can have several Python versions installed on the same machine, so you can keep the newer version for other applications and install the older version for BioCASe.

===I’ve installed the Python package for my DBMS, but it doesn’t show up in the Test libs page. What’s wrong?===
Python packages are Python version specific and you’ve probably installed the DB package for the wrong Python version. If you have several Python versions installed on your machine, make sure you installed the package for the version that is used by BioCASe.

===I’m using Postgres and installed the psycopg2 lib for that. When connecting, I get a pink error message “A problem occurred in a Python script”; the last line of the error messages reads “psycopg2._psycopg.connection object has no attribute server_version”.===
This is caused by an outdated psycopg2 version. Please install version 2.0.12 or later of psycopg2.

===I'm trying to use BioCASe with Postgres; for this, I tried to install psycopg on Windows. But it doesn't work - either the install crashes, or it finishes, but the test libs page still reports psycopg2 as not installed. In the debug log I can find the error message "ImportError: DLL load failed (...)". What's wrong?===
We suspect this is caused by a bug in recent psycopg2 releases (find more [http://psycopg.lighthouseapp.com/projects/62710/tickets/20 here]). On our test installations, we managed to get round this issue by installing pycopg2 with administration rights. Do do this, right-click the installer file and choose "Run as administrator".

===What's the difference between the DBMS packages odbc_tsql and odbc_tsql_2012 and between pymssql/pymssql_2012, respectively?===
The dbmods ending in "2012" use the new OFFSET clause that was introduced in MS SQL Server 2012. This will make paging faster and increase performance for larger datasets. They work only with SQL Server 2012 or later, and if you use SQL Server 2012, you should use them for best performance. Dbmods without a trailing "2012" work for both older versions and SQL Server 2012, but will be slower in paging large datasets.

===I have problems installing GraphViz. Do I really need it?===
If installed, Graphviz will create a graph of the table you've configured with their primary/foreign keys and the relations between them on the ''Database Structure'' page. If you have more than just a few tables and lots of relations between them, this can be very useful. If you don't need this feature, you don't need to bother installing GraphViz.

===I’ve installed GraphViz, but it doesn’t show up as being installed on the Test libs page.===
You need to tell BioCASe about the location of the GraphViz files, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk. Go to the System Administration and enter the path to the binary executable into the text box labelled Graphviz dot binary. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:\Program Files\Graphviz2.26.3\bin\dot.exe''. Press update config to save the path.

===I want to update my BioCASe installation. How do I do this?===
See [[Installation#Updating_an_existing_BioCASe_Installation|Updating an existing BioCASe Installation]].

===I've updated my installation from a version prior to 2.6. Now, when I enter the DB structure, some of the table and column names are gone, the same happens in the mapping editor.===
Starting with version 2.6, BioCASe retrieves table and column names from the database for some database management systems (namely MySQL, SQL Server, Postgres, Foxpro, Access and Oracle). In older versions, you had to enter table/column names manually, and when you did the setup with such an early version, you probably didn't care for upper/lower cases. Since some DB management systems are case-sensitive, BioCASe can't ignore the differences and won't find these technically incorrect names in the lists retrieved from the database. The respective drop down boxes will therefore be empty. If that happens, just select the correct names from the drop down lists.

===When I try to create an archive, I get a "Permission denied" error. Why is that?===
For creating archives, you need to grant write permissions for the folders ''archive'' and ''www/downloads'' of your BioCASe installation to Python. You can check if permissions are set correctly in the test libs page (Start --> Utilities --> Test Libs).

===I'm trying to install BioCASe on Ubuntu (or another Linux). Even though I've run the setup script and added the snippet to the Apache configuration file, I see source code when I try out my installation instead of the pages described in the documentation===
Apparently CGI is not enabled on your machine. Google how to enable CGI for your distribution and follow the instructions you'll find.

===When trying to run BioCASe after installation, I get an internal server error (500). But the log folder of BioCASe is empty, nothing to find.===
This is probably because BioCASe - or the web server that runs BioCASe - can't write to this folder. Make sure that the web server (user ''www-data'' for Apache) has write permissions for this folder.

===I'm running BioCASe in a Docker container and I can't connect to a local MS SQL Server (on the same machine, but not in a Docker)===
Do not use host.docker.internal:host-gateway (e.g 172.17.0.1) to connect, but the external LAN IP, often like 192.168.0.x or 10.x.x.x.

== Debugging ==

===I’ve mapped ABCD2. In the result document, each record of the dataset becomes a single dataset, even though there is just one dataset in the database – what’s wrong?===
That’s probably because you’ve mapped a metadata element to a table that holds several rows for this dataset, so for each row a different dataset gets created. It doesn’t matter whether the values for that field are effectively different from each other; rows are distinguished from each other only by their primary keys.

===I’ve mapped ABCD2, and in the result datasets I get several units for each record, each with the same UnitID.===
That happens when you map a non-repeatable element of ABCD to a database column that has several rows for one record. In order to create a valid ABCD document, for each of these values a new unit gets created by the Provider Software. To solve this, remove the multiple values or map them to a repeatable element.

===I've mapped ABCD2. When testing the web service, I get a negative number for "RecordsDropped" in the response document. What does that mean?===
Same problem as above - it means you have an error in your mapping. Most probably you've mapped a non-repeatable ABCD concept to a table/column in your database that holds several entries per unit (occurrence) record. The BioCASe Provider Software will duplicate the respective unit records to be able to create valid ABCD documents - which is indicated by a negative number of "records dropped". Make sure the cardinality of your database relations match the structure (in respects of element repeatability) of ABCD to avoid this problem.

===I'm using pymssql to connect to a SQL Server database. Char(n) and Varchar(n) strings get truncated to 255 characters - what's wrong?===
This is known limitation of TDS protocol used by pymssql. Just change the data type for columns storing long string to TEXT or create a view that converts Char/Varchar columns to TEXT columns on the fly to work around this issue.

===I'm using pymssql to connect to a SQL Server database, BioCASe is running on a Linux distribution. When I do a Search or Scan request, I don't get any results (0 hits), even though there are definitely records in the database.===
That problem occurs on old versions of pymssql (1.0.2 or earlier) on Linux distributions. Update to version 2.0 or later to solve this. If you still get empty result sets, these posts might be helpful:
* http://open-up.cybertaxonomy.africamuseum.be/forum_topic/important_if_you_use_biocase_bps_linux_platform_connect_ms_sql_server
* http://code.google.com/p/pymssql/issues/detail?id=76.

===The database connection status is No Connection, even though I’ve selected the correct DBMS in the connection settings dialog and entered the credentials into the text boxes.===
Check whether you’ve installed Python package for the DBMS you’re using (test libs page). If you still can’t get a connection, turn on debugging (in the System Administration dialog) and go to the database settings dialog again, press Save to retry connecting. Then see the log files (in the ''log'' folder of your biocase installation) for more detailed error messages.

===I am using BioCASe on a Windows Server. For some queries with the Local QueryForm, I get an error page that is obviously produced by the web server and not BioCASe. It reads "Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)"===
This happens when BioCASe is run behind a Microsoft ISA Server/Microsoft Forefront Threat Management Gateway. To solve this problem, follow the instructions given in this article: http://support.microsoft.com/kb/837865.

===The information that can be found in the log files folder is too sparse – what can I do?===
Turn on debugging in the System Administration.

===How do I find out if my BioCASe web service works right?===
Use the Query Form to send a BioCASe request to your web server and check the result document (see the [[Debugging|Debugging guide]] and the [[SampleABCDDocument|Sample ABCD document]] for more).

===I’ve copied a schema mapping for a datasource from an older installation by placing the cmf_ABCD_2.06.xml in the datasource configuration folder, but when querying the datasource, I get a “schema is not supported by this provider” error.===
The Provider Software keeps a list of supported schema mappings in the provider setup file. Just open go to the mapping editor for the schema in question, press Save (without having modified anything), then go back to the overview page of the datasource configuration. The Provider Software will then update the list of the supported schemas.

===When using the local Query Form for debugging, I don’t get any results, even though my web service is working. On pressing Submit, I get a timeout after about a minute.===
That’s probably because the domain configured in the BioCASe System administration is not correct. The default localhost works only if you’re using the query form directly on the server machine. If you’re using them in the browser window of another machine, the domain needs to be configured correctly with the IP address or the full server name (e.g. ''ww3.bgbm.org'').

===I mapped the ABCD concept FullScientificNameString. But when running the Search test (which includes a filter on the scientific name per default), I still get the error message "The requested concept /DataSets/DataSet/Units/Unit/Identifications/Identification/Result/TaxonIdentified/ScientificName/FullScientificNameString is not searchable for this provider! Please do a capabilities request to see all searchable concepts." What's wrong?===
The element ''FullScientificNameString'' exists twice in ABCD 2 (as the whole ''TaxonIdentified'' tree that surrounds it): once for the ''Identification'' object under the path above, and then in the ''Synecology'' branch (/DataSets/DataSet/Units/Unit/Gathering/Synecology). This can be a bit confusing, and probably you've accidentally mapped the wrong concept. Change the mapping, and it should work.

===I'm using BioCASe in a Vagrant Virtual Box on MacOS with Synced Folders. When using the Manual Query Form or the Archiving page, I get an "ImportError: No module named wrapper.errorclasses."===
We don't know exactly what causes this, but apparently it happens when BioCASe is run in a Vagrant Virtual Box on MacOS and Vagrant's synced folder are used. We do have a workaround for this, but it is not easy. Please [[Special:Contact|contact us]] for assistance on this.

===I'm using BioCASe on a Windows server and odbc_tsql to connect to an MS SQL Server. Even though I do get results for a SCAN request, I cannot do a SEARCH (invalid document received).===
Please check your encoding settings in the DB connection configuration. If it is set to UTF_8, try Latin_1 instead.

== Trouble Shooting ==

===I’ve forgotten the password of my BioCASe installation. What should I do?===
Just go to the ''config'' folder of your installation and have a look into the file ''config.ini''.

===In the mapping editor, I am trying to insert a literal that contains a non ASCII character. When saving, the system directs me to a page that allows me to change the system encoding to UTF-8. Unfortunately, on my installation there is no such file as printed in the ''technical description''; as a result, I get an enoding error in the Provider Software later on.===
First of all, read the paragraphs on the disadvantages of mapping literals in the [[ABCD2Mapping#Adding_Mandatory_ABCD_Elements|Beginners Guide]] and consider storing the information in a database table instead of mapping it as literals. Then, just move the data to a table and map the database columns. If, after thinking it over again, you still want to map literals with non-ASCII characters, you need to do the following:
* Go to your Python installation (for example ''/usr/lib/python2.5''),
* Create the file ''sitecustomize.py'' in the subfolder ''/site-packages'' with the following two lines:
<pre>
import sys
sys.setdefaultencoding('utf-8')
</pre>
* Save the file and retry. Maybe you'll have to restart your machine for the changes to take effect.

===I get a nice pink “A problem occurred in a Python script” error. What’s wrong?===
Probably something happened we didn’t foresee, or you might have just found a bug. Please take a screenshot of the screen and write down what you did to make this happen. Then [[Special:Contact|contact the BioCASe team]].

===I run into problems when installing/configuring the Provider Software. Neither the tutorials nor these FAQ I am looking at now can help me.===
Just [[Special:Contact|contact us]]; we’ll try our best to help you.

VersionHistory

2024-07-09T07:46:28Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Timeout added for the request to retrieve latest version from subversion repository
* Archiving page: List schemas in dropdown box in descending order, so that ABCD 2.1 shows before 2.06 before 1.2

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

Installation

2024-06-11T13:55:21Z

JoergHoletschek: /* Installing BioCASe from a Docker image */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# If you're not using Linux, install the ''make'' tool from https://gnuwin32.sourceforge.net/packages/make.htm.
# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading the Provider Software ===
You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository.

==== Downloading the Archive File ====
Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder.

==== Downloading from the Subversion Repository ====
If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled.

The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0.

In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose Checkout. Fill in the URL, change the destination folder name into something meaningful and press OK:

[[image:instTortoiseMenu.png|300px]][[image:instTortoiseDialog.png|450px]]

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

VersionHistory

2024-04-15T13:50:29Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Upcoming ===
* Timeout added for the request to retrieve latest version from sbuversion repository

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-11-30T11:14:04Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Version 3.8.5 [2023-11-30] ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-10-02T10:18:57Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Bugfix for ConfigTool login page (didn't work for empty webroots like ''biocase.xxxx.de'')
* Fix for CMF upgrade function

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-09-29T13:50:29Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Upcoming ===
* Bugfix for empty webroots (like biocase.xxxx.de)

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-08-18T12:03:21Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Version 3.8.4 [2023-08-18] ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-08-18T11:59:51Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
* Fix for login page of configtool (didn't used the webroot from configuration)
* Use SQL statements with parameters for most DB mods

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-04-18T10:19:01Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Upcoming ===
* Fix for login page of configtool (didn't used the webroot from configuration)

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-01-13T12:12:54Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Version 3.8.3 [2023-01-13]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2023-01-13T09:52:48Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== 3.8.3 [2023-01-xx]===
* Fix in Database connection for Python versions with two-digits minor version (e.g. 3.10)

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

Installation

2022-06-10T09:33:01Z

JoergHoletschek: /* Installing Apache/IIS and Python */

This part of the wiki explains how to install the BioCASe Provider Software. Depending on your skills, it should take between 15 and 60 minutes. If you’re a beginner, please read the [[BeginnersGuide|Beginner’s Guide]] first.

== Requirements and Considerations ==
Before installing, please make sure that the system you’re intending to install the BPS on meets the following requirements (If you find out one of these items is not met, please ask your system administrator for help, read the [[FAQ]] or get in [[Special:Contact|contact with the BioCASe Team]].):

# You have administrative privileges on this machine, i.e. the user rights to install new software packages.
# It is accessible from outside your institution. This means it either needs to be outside any institutional firewall or, more likely, the HTTP port is open for an existing firewall.
# A direct connection can be established from this machine to the database to be published – either because there’s no firewall in between, or the port used by the DBMS is open (e. g. 3306 for MySQL).

In addition to the regular installation, you can install BioCASe from a Docker image. Docker is an operating-system-level virtualization software that allows easy software deployment in so-called containers. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide]. '''If you're famliar with Docker, we'd suggest to use the Docker path for installing BioCASe''' - read more in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]] below.'''

== Installing BioCASe from a Docker image ==

Good choice! Installing BioCASe from a Docker image is much easer than installing the required components step-by-step. In addition, it's easier to update an existing installation. If you don't know yet what Docker is, you should read the [https://docs.docker.com/get-started/ Get Started Guide] first.

'''Important note:''' BioCASe requires Docker version 17 or later. If you already have Docker installed, run ''docker --version'' in a terminal and make sure the version number is 17+.

For installing BioCASe from the Docker image, please follow these steps:

# Install Docker Community Edition (Docker CE). For detailed instructions, see [https://docs.docker.com/install/ About Docker CE] and choose the tutorial for your platform. To make sure Docker is properly installed, run ''docker --version'' in a terminal and make sure the version number is 17+.
# Download the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] that contains the commands for installing and updating BioCASe. Use "Save as..." to store it to your home folder on the server machine.
# Open a terminal window and change the work directory to the location you've just saved the Makefile to. Then type in ''make install'' and press enter. This should download the latest BioCASe image, create a BioCASe container and run it.

Per default, Docker will use port 80 for BioCASe. If this port is already used by another application on your server, you can instruct Docker to use another port. For using port 8181, for example, type in ''make install PORT=8181'' instead:

[[File:DockerInstall.png|650px]]

Voilà, that's it! To check your BioCASe installation, use your Browser to go to ''http://localhost/biocase''. Don't forget to [[Installation#Changing_the_Default_Password | change the default password]].

You can also use the Makefile for stopping and starting the BioCASe container (for example, if you restart your server each night automatically) - just type ''make stop'' and ''make start''. For removing the container, use ''make remove'', for updating your BioCASe installation ''make update''.

'''Important note:''' The configuration and the log files of your BioCASe installation, as well as any XML and DwC archives created, will be stored in Docker hosted volumes. That means they are stored outside of the BioCASe container and will be preserved, even when the container is removed (actually, that's what happens if you update your installation). Docker hosted volumes are stored in the ''volumes'' folder of your docker installation, e.g. /var/lib/docker/volumes. If you want to backup the configuration of your BioCASe installation, just backup the folder /var/lib/docker/volumes/bps_config. To learn more about docker hosted volumes, please refer to the [https://docs.docker.com/storage/volumes/ Docker guide].

== Regular BioCASe Installation ==
This is the path to choose if you haven't heard of [https://www.docker.com/ Docker] yet. If you have experience with this virtualisation software, we'd suggest to install BioCASe from the Docker image (instructions in section [[Installation#Installing_BioCASe_from_a_Docker_image | Installing from a Docker image]]).

BioCASe runs on both Windows and Linux machines. Two thoughts to consider for you:
* If you want to use ODBC to connect to your database (for example to an Access database), you should make sure that an ODBC driver is available for the OS of your choice. For an Access database, for example, such a ODBC driver is available only for Windows - so you should go for Windows.
* If you don't wanna use ODBC, because you plan to connect to a wide-spread DB system such as MySQL, Postgres or MS SQL Server, we recommend Linux (Debian).

=== Installing Apache/IIS and Python ===
If you’re using a Linux system, you probably don’t need to install either, since Apache and Python are usually preinstalled. Same holds true for MacOS, but you should have a look at one of the articles on the web to find out about the file locations.

Even though the BPS runs with both Apache and Internet Information Server (IIS), we recommend using Apache, which is much easier to configure. However, if you’re already using IIS, you might want to stick with that.

The BPS requires Python 3.7+ to run, it doesn't work with Python 2, which ended its life cycle in 2020 (same is true for Python 3 versions < 3.7). On most Linux systems, Python 3 will be pre-installed. If not, you should be able to install Python using ''apt-get update'' and ''apt-get install python''. (By the way: You can install several different Python versions on the same machine without any conflicts.)

If you’re installing on a Windows Vista (poor you) or Windows 7 machine, we recommend not to install Apache in the default program files folder, since there is some magic spell on that which can get you into trouble when changing Apache’s configuration files. Just choose another path (Apache Software Foundation?) to spare that hassle.

Once you’ve successfully installed Apache and Python, you can proceed to the next step.

=== Downloading the Provider Software ===
You can download the BPS either as a g-zipped archive or directly from our Subversion repository. First option is the easiest if you’re not familiar with Subversion; the second option makes it easier for you to update your installation later. So if you’ve never used Subversion before and plan not to keep your installation up-to-date, choose the archive. If you plan to set up a productive installation, which should be kept up-to-date, you should opt for the subversion repository.

==== Downloading the Archive File ====
Just download the archive from the [http://www.biocase.org/products/provider_software Download section of the BioCASe site] and unpack the contents to the destination folder.

==== Downloading from the Subversion Repository ====
If you’re using Windows, you need to install a Subversion client first (try [http://tortoisesvn.net Tortoise], which comes with a nice user interface). Linux and MacOS usually have Subversion preinstalled.

The URL for the latest stable version is http://ww2.biocase.org/svn/bps2/branches/stable, which we recommend to use. Updating your installation to the latest BPS version is then as easy as just running the Subversion update command. If you want to download a defined version, the URL would be http://ww2.biocase.org/svn/bps2/tags/release_3.8.0.

In order to download the BPS from subversion with the Subversion command line client (preinstalled on Linux/MacOS), just go to the destination and type in <pre>svn co http://ww2.biocase.org/svn/bps2/branches/stable biocase</pre>This will connect to our server with anonymous access and download all files to the folder ''biocase''. In case you’re using Tortoise on Windows, right-click on the destination in the Windows explorer and choose Checkout. Fill in the URL, change the destination folder name into something meaningful and press OK:

[[image:instTortoiseMenu.png|300px]][[image:instTortoiseDialog.png|450px]]

For the remainder of this guide, we will assume you’ve installed the BPS into a folder named ''biocase'', which should look like this by now:

[[image:instBiocaseFolder.png|650px]]

=== Running the Setup Script ===
The setup script will initiate the configuration files with values you provide and point the script files to your Python installation (you can re-run the setup script in case that location changes). If you want to modify any of settings you made during the setup, you can change the values in the configuration tool later.

In order to run the script, open a terminal window (or “command prompt” in Windows). Change the working directory to the folder you’ve downloaded the BPS to (''biocase'') and type in:<pre>python setup.py</pre> In case Python is not found, you’ll have to add the full path name to the Python executable. Make sure the Python version given in the fifth line of the output is the one you desire (otherwise add the path to the correct executable):

[[image:instSetupScriptStart.png|550px]]

The questions asked by the setup script do have defaults. If you want to keep them, just press Return.

Webserver domain: This is the domain name of the web server you’re installing BioCASe on, e.g. ''http://www.myinstitution.org''. You can keep the default ''localhost'' as long as you’re configuring and testing the installation locally; for production you’ll need to change that into the correct domain name (or IP address) later.

Base URL: This will become the second part of the URL your installation will use. If you keep the default biocase, your installation will be accessible later at ''www.myinstitution.org/biocase'', for example.

After answering these two questions, the script will update your files. At the end of the output you can find the lines you'll need later to create the link between BioCASe and your web server, so do not close the window!

[[image:instSetupScriptDone.png|550px]]

=== Linking BioCASe and Your Web Server ===

The public part of the provider software is stored in the ''www'' subfolder of your BioCASe installation and must be configured in your web server as a public folder. You should always make this folder public using a virtual directory: This means that you should never install BioCASe under the default root folder of your web server. Doing so can compromise your server security and will allow people to see your configuration files (which include passwords). So since we installed BioCASe in ''c:\biocase'', the folder that must be published with a virtual directory is ''c:\biocase\www'' and will be accessible later through ''http://www.myinstitution.org/biocase''.

==== Apache ====
If you use Apache, adding the BioCASe folder to your web server environment is as simple as copying the lines printed at the end of the setup script to the end of the Apache configuration file ''httpd.conf''. This can typically be found in the ''conf'' folder of your Apache installation. If you’re adding BioCASe to an already productive web server, please ask the server admin before doing any changes!

[[image:instHttpdConf.png|550px]]

Save the file and restart Apache for the changes to take effect.

==== Internet Information Server ====

If you use Internet Information Server, you need to create a virtual directory for the folder ''c:\biocase\www'' using the IIS Server Manager. Grant all rights for this directory (Read, Write, Script Execution, Directory listing). You should also add ''index.html'' to the list of default documents (Virtual Directory > Properties > Documents > Default).

In order to associate Python scripts with Python in IIS 5/6, you need to use the Configure button on the virtual directory tab and associate the file extensions .cgi and .py with Python (see screenshot below). In IIS 7, you can open this dialog by choosing Add Script Map. Note the placeholders ''%s %s'' after the Python executable!

[[File:instAddScriptMap.png|330px]]

If you run into problems configuring IIS, you should have a look at the tutorials available on the web, for example
* http://haishibai.blogspot.de/2011/02/setting-up-python-on-iis-7_01.html
* http://support.microsoft.com/kb/276494

==== Testing the Link ====

Once you’re done, you can test your BioCASe Installation by opening a browser and typing in ''http://localhost/biocase'' (replace ''localhost'' with the domain name of the machine you installed BioCASe on, if you’re not working locally). You should then see the start page of the BPS:

[[image:instBiocaseStartpage.png|550px]]

=== Installing Additional Python Packages/Java/Graphviz ===

Before you can start configuring your installation, you need to install additional packages, depending on the management system used for the database to be published and the functions of the BPS you intend to use.

On the BPS start page, click on Utilities, then library test in order to navigate to the library test page that shows the status of installed libraries. On a fresh BPS installation, only the BioCASe libraries should be installed, since they're part of the archive file you’ve downloaded (or the Subversion repository you checked out from):

[[image:instLibrariesVanilla.png|550px]]

If you plan to use the Local Query Tool (a small portal for querying and browsing your BioCASe webservice), you need to install the lxml library. You can do this easily by running the Python Package Installer (PIP): ''pip install lxml''.

For each DBMS you plan to use with BioCASe (i.e. you have a database you want to publish with BioCASe), you’ll have to install the corresponding package. So if your collection is stored in a MySQL database, you'll need the MySQLdb library; the table shows you the command to run for installing it using pip: ''pip install mysqlclient''. Once pip has installed the package, reload the libraries test page. The according list entry should turn from an orange ''not installed'' into a green version number. Redo this process for all DBMS you want to connect to.

The Graphviz library listed under optional external binaries is used by the BPS to create a graph of your database’s data model in the table setup (you will learn in the [[DatasourceSetup#Setting_up_the_Database_Structure|Table Setup]] about this). If the data model is very simple (i.e., not more than 3 or 4 tables), you probably can spare the effort of installing Graphviz. Otherwise we recommend installing it: Simply click on the Download link and install the package using an installer file or a package manager on Linux.

If you want to create DarwinCore Archives for your web services, you also need to install a Java Runtime Environment (JRE). BioCASe requires Java 1.8 or later for creating DarwinCore archives, so please check the version listed on the Testlibs page if it's already installed on the server.

=== Changing the Default Password ===

The configuration tool is a browser interface that allows you to configure almost every aspect of the BPS. As a last step of the setup, you should change the password.

Go back to the Start page of the BPS (by clicking Start at the top of the page, or by retyping ''http://localhost/biocase'' if the server is running on your computer), choose Config Tool, then System Administration. You will then be prompted for a password. Type in the default ''ACDC'', then click Submit.

The top of the configuration page you’ll be directed to should look similar to this:

[[image:instConfigTool.png|500px]]

In the first two lines, you’ll see the values you entered during the setup script (in this case, the defaults ''localhost'' and ''biocase''). In the fourth line, enter a new password. Please be careful when typing (if you forget the password some day, you can reset it by directly editing the configuration file). Press update config to save the new password to the configuration file.

In case you’ve installed Graphviz, you need to tell the BPS the location of the binaries, since it is not a Python package that will be registered automatically. In order to do so, locate the Graphviz installation on your local disk and enter the path to the binary executable into the text box labelled Path to Graphviz executable. On Linux machines it might look similar to the default (''/usr/local/bin/dot''), on Windows machines it will be probably something like ''C:/Program Files/Graphviz 2.44.1/bin/''. Press update config to save the path.

=== Checking Your Installation ===

Before you proceed to the configuration, you should make sure everything is installed correctly:

# Typing in ''http://localhost/biocase'' (or the appropriate values of your installation) takes you to the entry page of the provider software (as shown in the section [[#Linking_BioCASe_and_Your_Web_Server|Linking BioCASe and Your Web Server]]).
# The libraries test page (Utilities > library test) shows you’ve installed the packages for your DBMS, lxml (if you want to use the Local Query Tool), Java (if you want to create DarwinCore Archives) and the Graphviz dot (if you decided so).
# Status of writeable directories shows a green writeable for all four directories. If not, the path to problematic file will be given – please grant write access to Python.

If you want to connect BioCASe to a variety of different database management systems, create DarwinCore archives and use the QueryTool, the libraries test pages should look like this:

[[image:instLibrariesDone.png|650px]]

== Updating an existing BioCASe Installation ==

=== Updating a BioCASe Docker installation ===
This is very simple with the [http://ww2.biocase.org/svn/bps2/trunk/Makefile Makefile] you downloaded for installing BioCASe. Just go to the folder where you stored the file and run ''make update''. This will stop and remove the BioCASe container, download the latest BioCASe image and create/start a new BioCASe container. Since the configuration files are not stored within the container, all your data sources will be preserved.

=== Updating a regular BioCASe installation ===

'''Important Note''': If you have a running BioCASe installation, you shouldn’t touch it until you’ve successfully installed a new version. So keep the old installation instead of replacing it!

==== Updating from version 2.5 or later ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version.
# Download the new version as described above into a directory with the name you just memorised.
# Change directory to the new installation and run setup script. When asked whether to import the configuration from an old installation, answer yes. The setup script will now import the configuration of the old installation, including the data sources.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

==== Updating from older versions < 2.5 ====

# Rename the folder of the old installation to something meaningful, for example ''biocase_old'', and memorise its old name. By doing this, you can always revert to this working version; moreover, you will need the configuration files later.
# Download the new version as described above into a directory with the name you just memorised.
# [[Installation#Running_the_Setup_Script|Run the setup script]]. When asked whether to import the configuration from an old installation, just press Return (defaults to no). Make sure to provide the the same base URL that was used by the old installation (for example ''/biocase'').
# [[Installation#Changing_the_Default_Password|Change the default password]] of the new installation to the one you used in the old installation.
# Open the file ''config/config.ini'' of your old installation in a text editor. Open the config tool of the new installation and navigate to the System Administration page. Compare the settings in the section Server configuration with the values you can find in the old configuration file and change them, if necessary (leave values unchanged if you don't find them in the old file). Press Update config.
# Last, copy the configuration of the BioCASe web services: Open the folder ''config/datasources'' of your old installation and copy all subfolders to ''config/datasources'' of your new installation.
# Done! [[Installation#Checking_Your_Installation|Try out your new installation]] to make sure it is working.

VersionHistory

2022-06-08T14:51:37Z

JoergHoletschek: /* Version 3.8.2 [2021-06-08] */

== Current Version 3 ==

=== Version 3.8.2 [2022-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2022-06-08T14:51:26Z

JoergHoletschek: /* Current Version 3 */

== Current Version 3 ==

=== Version 3.8.2 [2021-06-08]===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2021-12-22T09:51:13Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.
* Check for existence of Kettle libraries before starting a DwC-A transformation - and issue a hint in case they're missing because a BioCASe version without Kettle is installed.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2021-12-21T16:13:38Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduced in that version. Python 3.6 doesn't receive any security fixes anymore.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool

VersionHistory

2021-12-21T16:10:49Z

JoergHoletschek: /* Upcoming */

== Current Version 3 ==

=== Upcoming ===
'''Important: BioCASe now requires Python version 3.7+'''
* Fixes
** Unicode problem with MySQL data sources fixed (by setting the enoding explicitly in connection string)
** Check if file system supports Unicode; if not, convert archive file names to ASCII.
** Always use utf8 encoding for the archiving logs. This requires Python 3.7, since the method for reconfiguring stream encodings was introduces in that version.

=== Version 3.8.1 [2021-09-24] ===
* DarwinCore Archives
** Term ''occurrenceStatus'' added (''present/absent'', set from ''abcd:basisOfRecord'')
** Extension ''MeasurementOrFact'' added; will be filled from ''abcd:MeasurementOrFact''
** Invalid term ''occurrenceDetails'' removed
* Smaller fixes
** Apache configuration snippet is now compatible with Apache 2.4
** If installed, the lxml library will now be shown as installed on Testlibs page (workaround for non-exposed __version__ attribute in versions < 4.5.0)
** Fix for Ubuntu 18.04: QueryTool didn't work because OS reported a wrong file encoding (encoding now explicitly set to UTF8 upon template loading)
** Changes in Postgres dbmod for Version 3.8 reverted (used to cause problems for case-sensitive file systems)

=== Version 3.8.0 [2021-06-04] ===
* BioCASe now runs on Python 3 - yeah!
* Advantages
** Removal of external dependencies: consistent use of internal Python packages
** Up to 50% faster due to code clean-up and Python optimizations (tested on MySQL and SQL Server databases)
** Better handling of invalid database values (control characters used to break requests)
** Easier installation: All database-dependent packages can be installed by Python itself
** DB connection setting "encoding" removed (will be detected automatically)
** Pre/post 2012 versions for SQL Server merged: server version will now be detected automatically and SQL syntax adapted accordingly

=== Version 3.7.4 [2021-05-17] ===
* Last version for Python 2!
* Fixes
** in DwC transformation if there is only one identification without preferred flag (ends up now correctly in occurrence core file, not identification extension).
** for archiving page in config tool: Also allow unicode chars in dataset titles (broken since introduction of filtered downloads).
** in Dockerfile: Remove apache2.pid before starting Apache in order to prevent problems in case the Docker container wasn't shut down correctly.
** in Local QueryTool, record list page: The map now also works if some of the records lack coordinates (used to break the leaflet map).

=== Version 3.7.3 [2019-12-02]===
* Skin unitid_int added for UnitIDs of integer type (necessary for Postgres databases, since it doesn't accept the like operator on integers; MySQL and SQL Server don't object if the operand doesn't contain wildcards).
* ABCD 2.1 template in manual query form moved right next to ABCD 2.06
* Bugfix for mapping editor: If column retrieval is not available, potential dots are removed from column names, since they're not allowed.
* Method sanitiseFieldStorage added that removes any suspicious characters from CGI paramaters.

=== Version 3.7.2 [2019-07-10]===
* Bugfix for Archiving/section "Existing archives": Link for converting XML archives into DwC archives fixed.
* Bugfixes in Local Query Tool
** Error page on "Grouping page" now displayed correctly.
** for SSL handshake error on certain installations if HTTPS is used.
** Skin configuration in querytool_prefs.xml is now read correctly (used to be ignored).
* PyWrapper Wiki logo moved to local resources in order to prevent "not secure" message on HTTPS installations.
* QueryTool: Preview added in detail view for multimedia types video and sound
* Template for connecting to an [https://www.irisbg.com IrisBG database] added.

=== Version 3.7.1 [2018-10-02] ===
* XML Archiving allows now to set a BioCASe filter in order to archive and publish just a subset of the records published by a data source.
* New feature ''Filtered Export'' allows to export and download a subset of the records in a datasource into an XML archive without touching the XML and DwC archives that will be published to network harvesters by the dataset inventory.
* GoogleMaps in QueryTool replaced with OpenStreetMaps because of changes to Google's billing policy.
* Bugfix for AnnoSys retrieval (used to cause SSL handshake errors on certain installation).
* Bugfix for path handling of sqlite databases (now always relative to BioCASe home directory).

=== Version 3.7 [2018-06-12]===
* Dockerfile and Makefile added for dockerisation
* ODBC modules now use pyodbc or ceODBC, whichever is installed.
* DBmod_sqlite overhauled; uses Python-integrated module ''sqlite3'' and supports metadata retrieval.
* Local QueryTool: If a typified name is given for a type, it is now listed in brackets after the type status.
* Java version now shows type of JRE installed (for example OpenJDK)
* If configuration files cannot be loaded, raise an exception right away (makes it easier for the user to identify the problem if the setup script is run with missing configuration files).
* Testlibs page: Graphviz check now executes the file and retrieves version info instead of just checking for file existence.
* Default value for graphviz path in config.ini set to Debian default for easier dockerisation.
* Apache snippet in setup script adapted to Apache 2.4
* Fully functional demo data source included (uses included Sqlite database).
* Parameter ''url'' removed from manual query form for security reasons; target URL can now be changed manually in the form.
* Fixes
** in dbmod_oracle for metadata retrieval
** in DB structure editor of ConfigTool: If a table is removed from the setup, the second confirmation dialog has been removed. This keeps the user from rendering the configuration invalid by keeping invalid FKs.
** in DB structure editor of ConfigTool: If a table is purged from the setup because of a missing primary key or table name, a more verbose message is being printed.
** for XML archiving: Hashtag character removed from dataset title when generating the archive names; the download link for the archiving page of the config tool is now correctly URL-encoded.
** for layout of ''dsa_info'' page for new Firefox Quantum (versions 57+).
** in DwC transformation for the case that several identifications exist per unit and one of them has a preferred flag=true, while others have no preferred flag set (= empty).

=== Version 3.6.4 [2017-07-04] ===
* QueryTool now lists annotations created with [https://annosys.bgbm.fu-berlin.de/ AnnoSys] and allows to create an annotation.
* ''System administration'' page now allows to view log files in browser
* Changes to LIDO schema (small version): Element ''Point'' of complex type ''gml'' now expanded to full GML type; elements ''LineString'' and ''Polygon'' removed
* Fixes
** Bugfix in wrapper core for multi-column primary keys
** Bugfix for "remove cached objects" in system administration
** Bugfix in QueryTool for search fields with no label defined (XPath used instead now)
** Bugfix in metadata retrieval for dbmod_pymssql_2012+

=== Version 3.6.3 [2016-08-08] ===
* DwC archive generation: ABCD node ''Licenses'' will now be tranferred into an ''intellectualRights'' paragraph in the EML document if provided.
* Fix/change for dbmod_oracle.py: Paging is now faster
* New "reset" function for archiving: If the archiving thread is killed for some reason, this allows to reset the archiving process by removing the proc file

=== Version 3.6.2 [2016-07-21] ===
* Debug output now contains INFO items on time required for SQL execution and XML creation.
* Changes in DarwinCore transformation:
** Transformation streamlined
** ''IdentificationQualifier'' is now correctly filled from ABCD field
** GatheringAgents are now ordered by sequence (if they are mapped as separate gathering persons and the ''sequence'' attribute is mapped)
** If type status information are provided with a typified name, they are now assigned to the respective identification result (if no typified name is given, they are grouped and assigned to the preferred identification in the occurrence core)
** Fixed the issue that one of the identifications was lost if identical scientific names were provided for both a preferred and a non-preferred identification
** Fixed the issue that higherClassification contained double entries if higher taxa were provided for identifications with empty scientific name
* Fixes
** Bugfix in metadata retrieval for dbmod_pymssql
** Duplicate slash in access point URL removed in case a slash was specified at the end of base URL

=== Version 3.6.1 [2016-03-14] ===
* New experimental dbmod added for text files: Allows to publish data directly from csv files (for example, the contents of a DarwinCore archive).
* DarwinCore Archives: Field ''higherClassification'' is now also filled with higher taxa that don't have a rank specified.
* Fixes
** Unhandled exceptions in manual query form removed (occurred when incorrect data source name or no parameter at all was given).
** Bugfix for XML archiving (occurred when configuration parameter xml_retries not present in config.ini).

=== Version 3.6.0 [2015-10-27] ===
* Support for [http://www.lido-schema.org/ LIDO schema] added (including smaller changes to wrapper core).
* Various changes to mapping editor in preparation for the LIDO schema (e.g. dealing with mandatory nodes that do not have mandatory leaf elements)
* XML archiving improved
** uses less memory now (and will be probably faster with larger datasets)
** is error tolerant now: If a query fails during the archiving, a configurable number of retries will be done after a short waiting period.
* Support for ABCD 2.1 added to QueryTool and archiving.
* ''SchemaLocation'' is now set for most schemas (ABCD 2.06, ABCD 2.1, LIDO, GGBN).
* DB mod for Excel not experimental any more.
* Fixes
** DarwinCore Archives: Fields ''dc:identifier'' and ''dc:references'' in multimedia extension are now correctly created from ''abcd:FileURI'' and ''abcd:ProductURI''.
** DarwinCore Archives: Deprecated namespace for multimedia extension replaced with namespace for Simple Multimedia Extension
** Namespace only added for all attributes if attributeForm=qualified in schema definition.
** base_dbmod: verifyTable now retrieves the table list from the DB only once.
** dbmod_pymssql/2012: Order tables alphabetically; convert table/column names to unicode explicitly.
** dbmod_pymssql2012/odbc_tsql_2012 renamed to 2012+, because they also work with SQL Server 2014.
** Warning ''Destination hostname could not be determined'' removed.
** Archives that have been manually moved/renamed/deleted in the file system will now be automatically removed from the Provider Software.
** Mapping editor doesn't run into an exception if a mapping contains more than five elements and issues a warning instead.
** In the DB structure editor, trailing ''$'' characters in table/view names are removed for creating the alias.

=== Version 3.5.3 [2015-01-23] ===
* QueryTool now uses libxml2/libxslt or 4Suite XML, whichever is installed. This allows the QueryTool to be used with Python versions later than 2.5, for which 4Suite XML is not available.
* New experimental dbmod for Excel (ODBC).
* Slight changes and fixes in mapping editor (pop-up window can now be closed with Return or Escape) and datasource overview of the configuration tool.
* Fixes
** In wrapper core for mandatory attributes: If unmapped, the respective element is now removed (used to be removed only if the mandatory attribute was mapped to an empty, i.e. NULL field).
** for dbmod_odbc_tsql/dbmod_odbc_tsql_2012: In certain cases, SQL statements used to return no results for no reason. Parameters are now integrated into the statements instead of being passed separately, that fixed the issue.
** CMF upgrades now also keep the static table definition.

=== Version 3.5.2 [2014-07-18] ===
* Fixes
** For metadata retrieval in DB package pymssql (table lists used to be empty for SQL Server 2005 installations).
** Admin password removed from configuration dump in system administration.
** For update hint (for the case that an installation was updated before the hint advised this).
** DwC archiving
*** For duplicate identical identifications in the source ABCD document (used to quadruplicate the affected occurrences in the result DarwinCore archive).
*** EML document now also valid if some metadata elements in ABCD are empty.
* Namespace prefix for ABCD 2.1 added.
* Some content added to dummy files to force Winzip into extracting empty folders from the download archive.
* Warning added to table setup if configured PKs/FKs do not exist in database any more.

=== Version 3.5.1 [2014-04-10] ===
* DwC Archives
** Terms ''verbatimCoordinates'', ''coordinatePrecision'', ''georeferenceProtocol'', ''verbatimElevation'', ''verbatimDepth'', ''minimumDistanceAboveSurfaceInMeters'', ''maximumDistanceAboveSurfaceInMeters'', ''locationRemarks'', ''identificationVerificationStatus'', ''recordNumber'' added.
** Term ''verbatimLocality'' removed (ABCD element AreaDetail now stored as ''locationRemarks''), term ''eventID'' removed (ABCD Gathering/Code now stored as ''fieldNumber'').
** ABCD element CollectorsFieldNumber is now correctly mapped to ''recordNumber''; ''fieldNumber'' is filled with ABCD elemen Gathering/Code.
* DB connectivity
** Paging mechanism optimised for Oracle (rownum replaced with row_number).
** Unicode handling changed for MySQL.
** Firebird modules (native + ODBC) revised; works now with current Firebird installation and supports efficient paging.
** Sybase module revised; works with current versions of Sybase Adaptive Server; metadata retrieval implemented.
** When Postgres is used, another port than the default 5432 can be used.
* Update Mechanism
** Provider Software checks for updates once a week and shows update hint if a new version is available.
** Installation script now supports importing the configuration of an old installation, including mapped data sources.
* QueryForm now also shows Dataset Inventory.
* Testlibs page now shows detailed error message in hover text for packages that are not installed.
* Data source lists are now sorted alphabetically.
* Fixes
** If no data source parameter is passed in the URL or if data source is invalid, an error message is now being returned instead of an empty response when Apache is used.
** Wrong error messages caused by empty strings returned from DB for columns mapped to int and float XML data types removed.
** Search request with count=true doesn't fail with exception now if count SQL statement doesn't return a row.

=== Version 3.5 [2013-12-05] ===
* XML archiving now shows progress in percent.
* Mapping editor concept tree
** Overhauled (collapsable subtrees thanks to Falko Glöckler)
** Fix for attributes (used to be misplaced when ''show all concepts'' was checked)
** Now has a floating ''Save'' button
* New dbmods for SQL Server 2012 added, supporting new OFFSET clause.
* Fixes
** Term ''recordedBy'' is now filled correctly in DarwinCore archives.
** Mapping editor now works even if there is no connection to the database.
** Selective XML archiving now also works for datasets with multiple subsequent spaces in dataset title.
** XML archiving now works for datasets with characters ''&'', ''<'' or ''>'' in title.
* Cosmetics
** Datasource page: List of target schemas for possible upgrades now hides ''cmf_'' prefix.

=== Version 3.4.2 [2013-07-15] ===
* XML Archiving can now be limited to certain datasets within one data source.
* Optional attribute ''expires'' for archives added to dataset inventory.
* Fixes:
** On Archiving page for datasets with titles containing non-ASCII characters (Internet Explorer).
** Archiving page now also works with Safari.
** DarwinCore Archives: Term verbatimCoordinateSystem removed (was misapplied), verbatimSRS replaced with geodeticDatum.
** Namespace prefixes added for DNA and EFG extensions.
** More verbose error messages for dbmods odbc_access and odbc_tsql.

=== Version 3.4.1 [2013-04-04] ===
* Fixes:
** in dbmod_psycopg2 (Postgres) for non-ASCII characters in primary keys.
** in dbmod_mysql (MySQL) for unicode databases.
** in dbmod_oracle (Oracle): SCAN results now only yield values that are referenced by at least one root record.
** Search request with "count=true" now returns the correct number when filters on repeatable elements are used (by doing a COUNT DISTINCT on the primary key).
** for non-ASCII characters in element attributes (used to break the request).

=== Version 3.4 [2013-02-20] ===
* XML archiving re-implemented:
** Now directly calls BioCASe libraries instead of using the web service. Results in less complex code and significantly improved archiving performance.
** XML documents are not pretty-printed any more, but stored without line breaks and leading spaces for indentation, which results in smaller documents and smaller archives.
** Archiving is now dataset-aware for ABCD 2.06 and HISPID 5 schemas, i.e. each datset published by the webservice will end up in a separate archive.
* DwC archive transformation simplified.
* Archiving page overhauled.
* Archiving information shown in datasource configuration page overhauled.
* New dataset inventory allows discovery of published datasets and their XML/DwC archives for schemas ABCD 2.06 and HISPID 5.
* Corrections in layout of DB structure page in datasource configuration tool.
* Default record limit for web service set to 1000.
* Change in dbmod_tsql: Do not display tables/views of schemas ''INFORMATION_SCHEMA'' and ''sys''.
* Improvements to the QueryTool (thanks to Franck Theeten): Images will now be viewed on the record details page, coodinates printed on a map; several smaller changes.
* Bugfix for diagnostic messages with non-ASCII characters (used to bring up a malformed response document).
* Bugfix for filters containing single quotes for dbmods odbc_tsql and odbc_access.
* Fix for dbmod_orcale: Synonyms for tables now included in table lists.
* Fix for dbmod_odbc_access: Python error when SQL statement failed removed.
* Fix for dbmod_pymssql: Problems with non-ASCII characters in primary key columns solved.

=== Version 3.3.2 [2012-11-26]===
* Test version of new dataset inventory added: If ABCD 2.06/HISPID5 is mapped, it will list all datasets published by the web service with the respective XML/DwC archives, if present.
* Bugfix: When on the DB Connection page of the config tool the Save button was pressed after the user had been logged out because of a long period of inactivity, the settings were lost. This has been fixed.

=== Version 3.3.1 [2012-11-09] ===
* Archiving: Archive name is now set automatically by BioCASe; various smaller changes.
* Upgrade feature changed; now possible to a schema with the same namespace. For example, an existing ABCD 2.06 schema mapping can now be upgraded to ABCD-EFG or ABCD-DNA and vice versa. More upgrade paths added.
* Fixes
** dbmod_odbc_tsql now lists tables of all schemas (not only dbo)
** dbmod_oracle/oracle now use correct format for date/datetime conversion
** base_dbmod: Bugfix for filter parameters with unicode characters (used to break the request in certain cases)
** Bugfix for conversion of XML data type "date"

=== Version 3.3 [2012-10-04] ===
* Archiving can now be triggered from outside the UI through a deep link (allows scheduling)
* Datasource Overview now lists DarwinCore Archives; several smaller changes in UI.
* Bugfixes
** Ordering the result of a Scan request is now done in the database. This prevents problems when the collation of the DBMS is different from Python's.
** Concept count in config tool (datasource overview) corrected: Now includes attributes of hierarchy elements. The count now includes leaf elements that can store data and all attributes.
** Dummy files added to empty folders to prevent WinZip from ignoring them.
** Local QueryTool: Umlauts in scientific names are now replaced with placeholder *

=== Version 3.2 [2012-07-09] ===
* DarwinCore Archives fully integrated into config tool interface
* Changes for XML archiving:
** Log files are kept for old archives and can be viewed later even if a XML archive generation for the same file name was started, but was not finished successfully (in the past, this used to overwrite the old log file)
** Log files of failed or cancelled archiving runs can be viewed later
** If errors occur during archiving, the log output is less arcane now (e.g. when ceODBC is used)
** Format of output changed
* Fixes for page ''DB Structure'':
** DB graph generation works now also when IIS is used
** No exception thrown in case there is no connection to the database
* Other fixes:
** Removing a schema mapping in the config tool doesn't break the Capabilities any more (PSF is updated to reflect this change)
** install/readme files updated

=== Version 3.1 [2012-04-23] ===
* Fixes for XML archives:
** Compatibility issue with IIS solved (root directory returned by webserver is different from Apache's, see http://osdir.com/ml/python.web/2005-10/msg00008.html)
** Webinterface overhauled; reacts faster now because server interaction has been reduced
** If webinterface is left open in the browser during archive generation, less load is put on the server for showing the progress
** XML declaration added to documents
** If documents contain elements from several schemas (e.g. an extension is used), all namespaces are added
** Bugfix for the case that no results are returned for the chosen mapping.
** Archiving binary moved from archive working folder to archive lib folder
* Metadata retrieval added for pymssql (SQL Server connectivity package)
* Error message for dbmods that don't support parameters (such as pymssql) is less arcane now (dbmod_base changed)
* "Wrong" error messages removed for unicode datasources and in mapping editor
* Setup script now fixes write permissions for archive folders (working/download directory)
* Hint on the usage of literals and concatenation added to mapping editor
* Bugfix in library test page (for the case that Apache has different access privileges than Python)
* Bugfix in Mapping Editor for literals containing unicode characters
* Bugfixes for deleting a datasource:
** Archive folders will now be correctly removed (both working and download directory)
** Deleting a datasource with the same name as a previously deleted datasource that is still in the recycling bin now works; it will replace the old version in the recycling bin
* Bugfix for start page: Leftover debug code removed that always logged an error ("in index")
* Bugfix for IIS: Error message "No handlers could be found for logger lib" removed

===Version 3.0.1 [2012-04-05]===
replaced by 3.1

===Version 3.0 [2011-12-14]===
* New feature: XML Archives (integrated into UI), DwC Archives (separate application)
* List of table/column names now also available for Oracle databases
* "Report a bug" links added
* Fixes in wrapper core:
** ORDER BY clause now always used in first query when paging (necessary also when OFFSET is used)
** Python-sorting of ID list returned by first query removed (already done by DB; caused problems under certain circumstances when paging because Python sort order might be different from DB sort order)
* Optimization in wrapper core:
** Check for uniqueness of IDs in the list returned by first query only done when this is not alreay done by DISTINCT clause in query.
** Db_mod Oracle now uses paging optimization.

===Version 3.0RC2 [2011-11-24]===

===Version 3.0RC1 [2011-11-01]===

== Version 2 History ==

===Version 2.6.1 [2011-08-23]===
* Test for write-protected files in Testlibs page now ignores Subversion directories, thus avoiding false alarm.
* ''Help'' links now point to the respective sections of the new PyWrapper Wiki.
* Link to FAQ added to the encoding error page (which can be used to alter system enocding).
* Non-existing upgrade function removed from setup script.
* Bugfix for HISPID5: Missing value "unknown" added to the CoordinateMethod vocabulary.
* Bugfix in XML Handler for unary operators (isNull, isNotNull).

===Version 2.6.0 [2011-06-07]===
New Features:
* Table Setup completely overhauled: Tables/Columns will now be retrieved from the database and shown in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* Mapping Editor now shows table columns in drop-down-boxes when Postgres, MySQL, MS SQL Server, Access or Foxpro are used.
* When MySQL or Postgres is used, for Search requests the OFFSET SQL clause is used now for paging; this improves performance for harvesting large datasets considerably.
* Support for DNA extension of ABCD (data model for the DNA Bank network) added
* Mapping Editor shows XML Path for current element
* Deleted datasources and schemas will now be moved to a recycle bin and can be restored later.
* Several confirmation dialogs and smaller changes in UI.
* mxODBC package (not free for academic use any more) replaced with Open Source package ceODBC
* "Mapping Test" in mapping editor now uses the Query Forms, which are more reliable.

Bugs fixed:
* Session management: User gets logged out only when inactive for more than 30 mins.
* Bugfix for unicode characters in mapped literals.
* Mapping editor: Added elements do not disappear temporarily any more.
* Subversion users: BPS now can handle write-protected svn folders when removing a datasource.
* Graphviz dot binary now works also when installed in a folder with a name containing spaces
* Creating multiple mapping files for same namespace will be prevented now
* Display issues for paths containing "\b" and "\t"
* Filters combining two comparison operators that always evaluate to FALSE with OR will now be compressed correctly to FALSE, not TRUE. This prevents the Provider Software from returning any results if, for example, the filter of a Search request combines two LIKE-operators on unmapped concepts with the logical operator OR.

=== Version 2.5.4 [2010-09-14] ===
Withdrawn; changes included in version 2.6

=== Version 2.5.3 [2010-05-12] ===
* Support for HISPID5 added
* Bugfix CMF upgrade
* Bugfix QueryTool (Stylesheet transformation error)

=== Version 2.5.2 [2009-10-01] ===
* Bugfix QueryTool (Unicode conversion error)

=== Version 2.5.1 [2009-01-25] ===
* Security Bugfix (database credentials removed from debug output)

=== Version 2.5.0 [2008-04-29] ===
* replaced XSLT library, mx.DateTime & PyXML with 4suite & python native libraries (ElementTree, datetime). No external libraries are required anymore, just 4suite for xslt processing if you want to use the querytool (and you dont need to install the horrible libxml2 anymore)
* removed all metakit lib dependencies, thereby removing the concept finder window in the configtool. To select a new concept for mapping you can list all concepts of a schema or enter the xpath to one.
* Introduced a new ''cache'' directory where all temporary files go like user session data, pickled CMFs or temporary configtool files. To remove any cached files you can now simply empty this one folder!
* remove many old relicts that were never used like datasource metadata, system metadata, database autodiscovery, etc (all nice features to have for a future release, but they were never implemented so far)
* Use a new config singleton based on some simple ini text files. Incl a new parameter "prettyprint" to format the output nicely
* When creating a new datasource you can now select template datasources like Brahms6, a training db, ABCD2.06 metadata tables or NoDIT2
* Replaced the entire logging system with python file logging. Separate debug logs for each datasource plus global request & error logging
* new sesion based login that stays
* updated setup.py script
* changed the library test page
* and last but not least for developers: cleaned up quite some code

=== Version 2.4.2 [2006-11-16] ===
* configtool mapping pane. use anchors to go back to last editing position
* typeconverter: some dates were not correctly converted to xs:dateTime. Adding a "T00:00:00" solved it.
* ABCD2.06 CMF template updated to contain the allowed enumeration of ranks!

=== Version 2.4.1 [2006-06-26] ===
* Football Worldcup 2006 logos included
* minor bugfixes (dead links)
* admin password management fixed

=== Version 2.4.0 [2006-04-06] ===
* move most of the querytool_prefs to the new skin.xml preference file within a skin folder. Leave only list of concepts for form in old querytool_pref.xml. Modified querytool_prefs schema
* webapps log into rotating files instead of screen
* updated CMF parser service link
* configtool catches errors for unknown static aliases
* querytool XSLTs for abcd & abcdefg2.06
* remove wrapper_config.xml
* removed character encoding bug in CMF pickling.

=== Version 2.3.2 [2006-03-06] ===
* ABCDEFG extension 1.0 for Earth sciences included
* List available character encodings for python2.4 and 64bit systems
* QueryTool capable of querying DiGIR providers for details.
* new synthesys/biocase QT skin

=== Version 2.3.1 [2005-11-30] ===
* new 4th dimension dbmod!
* record filters inside CMFs fixed
* works with NoDIT v2 again
* GCP Passport 1.03 added

=== Version 2.3.0 [2005-11-03] ===
* compatible with Python 2.4 /!\ Using a new metakit compilation for windows.
* Graphviz's dot is being used to visualize the database structure.
* ABCD 2.06 included instead of ABCD2.05

=== Version 2.2.4 [2005-10-13] ===
* include PonTaurus example db
* scans without a filter crashed with some dbmods. bug fixed.
* use select distinct in access for scans
* ConfigTool includes graphical view of the db structure graph through graphviz dot images
* CMF filters with scan filters crashed the wrapper. solved.
* typeconverter: fixed bug in transforming zero-length strings to integers.

=== Version 2.2.3 [2005-09-01] ===
* "upgrade" from abcd 2.05 to abcd 1.20 crashes the config tool no more
* abcd2.05 debugform templates included
* fixed scan problem with oracle8 module
* fixed connection problem with native firebird module
* Querytool allows to view raw xml
* Querytool can handle unicode queries now
* Concatenation with pure whitespace works now in wrapper output
* database module is logged again into the protocol header. was broken before
* if request filter evaluates to False (because concept is not mapped for example), dont query database with an SQL "... WHERE False ..." but return no results immediately.
* configtool removes now non mapped concepts after pressing "revert" or re-entering the mapping section

=== Version 2.2.2 [2005-07-22] ===
* Include a button to show all concepts in the mapping tools directly without the need to use the concept retrieval tool. Works wit all CMFs.
* Update MCPD CMF template to include w3c types.
* Include new page to create your own CMFs from custom XML schemas with the help of a BGBM TAPIR service.

=== Version 2.2.1 ===
* Alternative root tables for SPICE wrappers work now correctly.
* Hierarchy requests in SPICE parse correct ID parameters now.
* corrected SPICE1.3 CMF template
* full Berlin Model SPICE CMF example

=== Version 2.2.0 [2005-07-18] ===
* System section of configtool can modify .configuration (domain, baseURL, admin password) file and gives overview about DSA passwords.
* setup script included to handle basic configuration, system adaptation and an upgrade facility for existing BPS2 installations.
* SPICE 1.3 supported with a single CMF now! Search is being done on seperate genus, species and infraspecies fields now. Spice queryforms and IOPI/Berlin model example mapping updated.
* ABCD 2.05 included (template,upgrade,example), the candidate for the final TDWG ABCD standard.
* Bugfixes in configtool, mapping section:
** Internet Explorer compatability fixed (no tablealias names were saved)
** CMF filter display fixed. Colored green now
** Allow special mappings (TotalMatchedRecords) to be displayed and saved correctly
** Fixed minor display problems with attributes
* fixed a serious new bug of the configtool that does not allow to add new tables to the db structure
* Allow case insensitive like searches in oracle if wanted.
* The performance debug logging works again

=== Version 2.1.0 ===
* CMF/schema version upgrade facility integrated into the configtool. By using Altovas mapforce to graphically map 2 schemas or versions of the same schema (e.g. ABCD 1.20 to ABCD 2.0, or ABCD1.20 to DarwinCOre and vice versa), it is now possible to upgrade your datasource configuration to support similar schemas. Thanks to Synthesys!
* SPICE search: bug removed with genus search including the wildcard
* SPICE: empty root element returned if no result was found. Debugging as html comments output was broken - fixed!
* optional XML attributes bug removed. optional attributes were sometimes treated as mandatory ones. relevant for SPICE
* cfgtool attr display: sort order in configtool now shows attributes always below its element and displays the attributes name only with @ instead of the entire element name
* oracle dbmod improved. Paging has to use subqueries to guarantee that ordering takes place BEFORE the rownum limit. See Oracle FAQ http://www.orafaq.com/faqsql.htm#ROWXtoY
* attribute mapping synching repaired. It is possible now to sync the mappings of an element with several attribute mappings
* bug in local conditional mapping removed.
* logging fixed (error and request)
* only use "text" instead of "string" and "unicode" everywhere now in the code.
* ConfigTool: deletion of primary keys, foreign keys and tables works again [Ben Richardson].

=== Version 2.0.3 ===
* SPICE search: Modify search string for genera only by adding " *" and normalize whitespace.

=== Version 2.0.2 ===
* Limit for Oracle via ROWNUM implemented for ora8,9&10
* Include a Linux, Windows & MacOSX binary build of metakit. Solaris users still have to build it themselves.
* SQLLite module added

=== Version 2.0.1 ===
* ABCD 2.0 CMF included

Bugfixes
* Configtool
** Lot of little bugs solved.
** Make possible to map attributes
** Be able to delete mappings
** Alert user when several CMFs with the same namespace exist for a datasource
** Allow full xpaths to be entered directly into the add path form
** allow datasource admin to delete all serialized objects
* QueryTool
** More inteligent XSLT for processing
* PyWrapper
** Solved error with wrapper not presenting attributes
** Empty elements are now excluded in the output.

=== Version 2.0.0 ===
* SPICE protocol supported for Species2000 CDM 1.20
* completely rewritten config tool including a concept retrieval interface to search for concepts
* completely rewritten an adaptable query tool with skin support

== Version 1 history ==
=== Version 1.6b ===
* needs at least version 1.4b of the pywrapper
* added pywrapper skeleton for new species2000 protocol
* cleaned the querytool sourcecode (thanks to Doreen Pahlke)
* added little script "cmfupdater.py" to biocase/cgi that updates all existing CMFs of the installation to the latest CMF version.

=== Version 1.56 ===
* Removed authorization bug when entering non ascii characters as the password.
* added more database modules for (upcoming) new pywrapper version: Sybase, Firebird, PostgreSQL via psycopg

=== Version 1.55 ===
* NoDIT v2 metaprofile templates and preconfigured database corrected
* added a GBIF registration method in the configtool

=== Version 1.54 ===
* added a query engine prototype for CMF concepts to the configtool
* more database modules supported (Oracle8, FoxPro)

=== Version 1.52 ===
* added URL encoding service in cgi-test.
* added AltText to images.
* removed bugs in config-tool when editing dbType for foreign key attributes and added FoxPro dbmod to menu.
* Updating the cgi-test script to fully support Unix systems.

=== Version 1.5 ===
* Includes the fully working version 1.0 of the config-tool to edit WCF, PSF and CMF files. Structural CMF editing is not yet supported as well as guidance for ABCD.
* template subdirectories renamed. Subdir "wrapper-skeleton" now contains the skeleton for new database directories, which comes without any CMF template. CMF templates are now hosted in the config-files subdirectory.
* private config directory used now for wrapper_config.xml, providerlist.xml and all other future configuration files that should be private.

=== Version 1.41 ===
* first release including the query-tool