https://wiki.bgbm.org/collectionworkflows/api.php?action=feedcontributions&feedformat=atom&user=AntonGuentsch 2026-07-09T22:44:41Z User contributions MediaWiki 1.43.6 https://wiki.bgbm.org/collectionworkflows/index.php?title=MediaWiki:Sidebar&diff=176 2018-09-25T11:39:50Z

<p>AntonGuentsch: </p> <hr /> <div><br /> * navigation<br /> ** mainpage|mainpage-description<br /> * Collection Workflows<br /> ** Collection Data Pipeline| Data Pipeline<br /> * Software<br /> ** Collection Data Form | Collection Data Form<br /> ** Open Refine Data Cleaning | Data Cleaning<br /> ** http://search.biocase.org/bgbm | BGBM collection portal<br /> ** http://www.ggbn.org | DNA collection portal<br /> ** Portal_Architecture | Architecture overview<br /> * Documentation<br /> ** Collection Manuals | Collection Manuals<br /> ** Barcodes für Sammlungsobjekte | Barcodes at B<br /> ** Question and Answer | Q &amp; A<br /> <br /> ** Internal:Main Page|Internal<br /> ** recentchanges-url|recentchanges<br /> <br /> * SEARCH<br /> * TOOLBOX</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Portal_Architecture&diff=175 2018-09-25T11:34:09Z

<p>AntonGuentsch: </p> <hr /> <div>{| class = &quot;wikitable&quot;<br /> | [[File:PortalArchitecture_2.JPG|border|800px]]<br /> |-<br /> |style=&quot;width:800px&quot;| Specimen portals such as [http://www.ggbn.org/ggbn_portal/search/index GGBN], the [http://search.biocase.org/bgbm/ BGBM Specimen Portal], and various other Special Interest Network Portals (SINs) are based on a range of open source software components and services most of which developed by or in co-operation with the BGBM Biodiversity Informatics Research Group. The main components are:<br /> <br /> * [http://www.biocase.org/products/provider_software BioCASE Provider Software] and the [https://www.gbif.org/ipt Integrated Publishing Tookit]<br /> * [https://github.com/tdwg/abcd ABCD] and [http://rs.tdwg.org/dwc/ DarwinCore] data standards<br /> * [https://wiki.bgbm.org/bhit B-HIT harvesting software]<br /> * Index database and Solr<br /> * Portal software<br /> * [https://annosys.bgbm.fu-berlin.de/ AnnoSys] annotation software<br /> <br /> By using the same software platform for all system, we ensure that new developments and improvements can be applied to all portals developed and hosted by the BGBM. Recent developments are an interface to the [https://cybertaxonomy.eu/ EDIT Platform for Cybertaxonomy] as well as a rights management function for &quot;confidential&quot; collection data.<br /> |}</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=File:PortalArchitecture_2.JPG&diff=174 2018-09-25T11:32:38Z

<p>AntonGuentsch: </p> <hr /> <div></div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Portal_Architecture&diff=167 2018-09-25T08:02:35Z

<p>AntonGuentsch: </p> <hr /> <div>{| class = &quot;wikitable&quot;<br /> | [[File:PortalArchitecture.JPG|border|1000px]]<br /> |-<br /> |style=&quot;width:800px&quot;| Specimen portals such as GGBN, the BGBM Specimen Portal, and various other Special Interest Network Portals (SINs) are based on a range of open source software components and services most of which developed by or in co-operation with the BGBM Biodiversity Informatics Research Group. The main components are:<br /> <br /> * BioCASE provider software and the Integrated Publishing Tookit<br /> * ABCD and DarwinCore data standards<br /> * B-HIT harvesting software<br /> * Index database and Solr<br /> * Portal software<br /> * AnnoSys annotation software<br /> <br /> By using the same software platform for all system, we ensure that new developments and improvements can be applied to all portals developed and hosted by the BGBM. Recent developments are an interface to the EDIT Platform for Cybertaxonomy as well as a rights management function for &quot;confidential&quot; collection data.<br /> |}</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Portal_Architecture&diff=166 2018-09-25T07:21:41Z

<p>AntonGuentsch: </p> <hr /> <div>{| class = &quot;wikitable&quot;<br /> | [[File:PortalArchitecture.JPG|border|400px]]<br /> |-<br /> |style=&quot;width:500px&quot;| This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data.&lt;br&gt;It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics].&lt;br&gt;Please use the menu on the left to find more information.&lt;br&gt;&lt;br&gt;A good starting point to get an overview of the BGBM collection data pipeline is provided by this [[Collection_Data_Pipeline|workflow diagram]].<br /> |}</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=File:PortalArchitecture.JPG&diff=165 2018-09-25T07:21:07Z

<p>AntonGuentsch: BGBM Specimen Portal Architecture</p> <hr /> <div>BGBM Specimen Portal Architecture</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Portal_Architecture&diff=164 2018-09-25T07:19:34Z

<p>AntonGuentsch: Created page with &quot;{| class = &quot;wikitable&quot; | 400px |- |style=&quot;width:500px&quot;| This wiki serves as a central knowledge base for standards, documents, software, and workf...&quot;</p> <hr /> <div>{| class = &quot;wikitable&quot;<br /> | [[File:BGBM Col.jpg|border|400px]]<br /> |-<br /> |style=&quot;width:500px&quot;| This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data.&lt;br&gt;It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics].&lt;br&gt;Please use the menu on the left to find more information.&lt;br&gt;&lt;br&gt;A good starting point to get an overview of the BGBM collection data pipeline is provided by this [[Collection_Data_Pipeline|workflow diagram]].<br /> |}</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=106 2018-09-18T08:03:49Z

<p>AntonGuentsch: /* (Very) first steps */</p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes, for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-Button and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|1100px]]<br /> &lt;br&gt;<br /> Fig.1: selection of worksheets and header row.<br /> <br /> == Ressources ==<br /> The OpenRefine website (http://openrefine.org/) provides a very nice compilation of tutorial videos, which will help you to understand the basic concepts of data exploration, cleaning, and transformation. There is also a written manual in the form of a wiki-site with common use cases and links to external tutorials in different languages (https://github.com/OpenRefine/OpenRefine/wiki).</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=105 2018-09-18T08:02:49Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes, for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|1100px]]<br /> &lt;br&gt;<br /> Fig.1: selection of worksheets and header row.<br /> <br /> == Ressources ==<br /> The OpenRefine website (http://openrefine.org/) provides a very nice compilation of tutorial videos, which will help you to understand the basic concepts of data exploration, cleaning, and transformation. There is also a written manual in the form of a wiki-site with common use cases and links to external tutorials in different languages (https://github.com/OpenRefine/OpenRefine/wiki).</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=BGBM_Collection_Workflows&diff=98 2018-09-18T07:47:25Z

<p>AntonGuentsch: </p> <hr /> <div>{| class = &quot;wikitable&quot;<br /> | [[File:BGBM Col.jpg|border|400px]]<br /> | This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data.&lt;br&gt;It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics].&lt;br&gt;Please use the menu on the left to find more information.&lt;br&gt;&lt;br&gt;A good starting point to get an overview of the BGBM collection data pipeline is provided by this [https://wiki.bgbm.org/collectionworkflows/index.php/Collection_Data_Pipeline workflow diagram].<br /> |}</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=BGBM_Collection_Workflows&diff=97 2018-09-18T07:46:40Z

<p>AntonGuentsch: </p> <hr /> <div>{| class = &quot;wikitable&quot;<br /> | [[File:BGBM Col.jpg|border|400px]]<br /> | This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data.&lt;br&gt;It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics].&lt;br&gt;Please use the menu on the left to find more information.&lt;br&gt;A good starting point to get an overview of the BGBM collection data pipeline is provided by this [https://wiki.bgbm.org/collectionworkflows/index.php/Collection_Data_Pipeline workflow diagram].<br /> |}</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=BGBM_Collection_Workflows&diff=96 2018-09-18T07:21:10Z

<p>AntonGuentsch: </p> <hr /> <div>[[File:BGBM Col.jpg|border|400px]]<br /> <br /> This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data. It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics]. Please use the menu on the left to find more information. A good starting point to get an overview of the BGBM collection data pipeline is provided by this [https://wiki.bgbm.org/collectionworkflows/index.php/Collection_Data_Pipeline workflow diagram].</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=BGBM_Collection_Workflows&diff=95 2018-09-18T07:20:04Z

<p>AntonGuentsch: </p> <hr /> <div>[[File:BGBM Col.jpg]]<br /> <br /> This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data. It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics]. Please use the menu on the left to find more information. A good starting point to get an overview of the BGBM collection data pipeline is provided by this [https://wiki.bgbm.org/collectionworkflows/index.php/Collection_Data_Pipeline workflow diagram].</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=File:BGBM_Col.jpg&diff=94 2018-09-18T07:19:01Z

<p>AntonGuentsch: BGBM collections overview</p> <hr /> <div>BGBM collections overview</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=BGBM_Collection_Workflows&diff=93 2018-09-18T07:06:11Z

<p>AntonGuentsch: </p> <hr /> <div>[[File:BGBM Herbarium.JPG]]<br /> <br /> This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data. It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the [https://www.bgbm.org/en/biodiversity-informatics BGBM Biodiversity Informatics]. Please use the menu on the left to find more information. A good starting point to get an overview of the BGBM collection data pipeline is provided by this [https://wiki.bgbm.org/collectionworkflows/index.php/Collection_Data_Pipeline workflow diagram].</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=63 2018-09-17T12:48:00Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|1100px]]<br /> &lt;br&gt;<br /> Fig.1: selection of worksheets and header row.<br /> <br /> == Ressources ==<br /> The OpenRefine website (http://openrefine.org/) provides a very nice compilation of tutorial videos, which will help you to understand the basic concepts of data exploration, cleaning, and transformation. There is also a written manual in the form of a wiki-site with common use cases and links to external tutorials in different languages (https://github.com/OpenRefine/OpenRefine/wiki).</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=62 2018-09-17T12:46:56Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|1100px]]<br /> &lt;br&gt;<br /> Fig.1: selection of worksheets and header row.<br /> <br /> == Ressources ==<br /> The OpenRefine website (http://openrefine.org/) provides a very nice compilation of tutorial videos, which will help you to understand the basic concepts of data exploration, cleaning, and transformation. There is also a written manual in the form of a wiki-site with common use cases and links to external tutorials in different languages (https://github.com/OpenRefine/OpenRefine/wiki).</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=61 2018-09-17T12:45:39Z

<p>AntonGuentsch: /* */</p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|1100px]]<br /> Fig.1: selection of worksheets and header row.<br /> <br /> &lt;br&gt;<br /> <br /> == Ressources ==<br /> The OpenRefine website (http://openrefine.org/) provides a very nice compilation of tutorial videos, which will help you to understand the basic concepts of data exploration, cleaning, and transformation. There is also a written manual in the form of a wiki-site with common use cases and links to external tutorials in different languages (https://github.com/OpenRefine/OpenRefine/wiki).</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=59 2018-09-17T12:45:13Z

<p>AntonGuentsch: /* (Very) first steps */</p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|1100px]]<br /> Fig.1: selection of worksheets and header row.<br /> <br /> &lt;br&gt;<br /> <br /> == Ressources ==<br /> The OpenRefine website (http://openrefine.org/) provides a very nice compilation of tutorial videos, which will help you to understand the basic concepts of data exploration, cleaning, and transformation. There is also a written manual in the form of a wiki-site with common use cases and links to external tutorials in different languages (https://github.com/OpenRefine/OpenRefine/wiki).<br /> <br /> == ==</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=58 2018-09-17T12:43:27Z

<p>AntonGuentsch: /* (Very) first steps */</p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|150px]]<br /> <br /> == ==</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=57 2018-09-17T12:43:00Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|border|30px]]<br /> <br /> == ==</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=56 2018-09-17T12:41:43Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|frame|50px]]<br /> <br /> == ==</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=55 2018-09-17T12:40:38Z

<p>AntonGuentsch: /* (Very) first steps */</p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> &lt;br&gt;<br /> [[File:OpRef Screenshot.jpg|frame|100px]]<br /> <br /> == ==</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=File:OpRef_Screenshot.jpg&diff=54 2018-09-17T12:36:51Z

<p>AntonGuentsch: Open Refine Screenshot</p> <hr /> <div>Open Refine Screenshot</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=52 2018-09-17T12:35:59Z

<p>AntonGuentsch: /* (Very) first steps */</p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.<br /> <br /> [[File:Example.jpg]]<br /> <br /> == ==</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=47 2018-09-17T12:34:06Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form ([[Collection_Data_Form]]):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=44 2018-09-17T12:26:41Z

<p>AntonGuentsch: </p> <hr /> <div>== Introduction ==<br /> Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information ([mailto:biodiversitydata@bgbm.org email]).<br /> <br /> == (Very) first steps ==<br /> OpenRefine is extremely powerful and we will definitely not be able to compete with dozens of already existing excellent manuals and tutorial videos. Please refer to the Resources section for further information. Here, we would just like to guide you through the very first steps with OpenRefine using the example of a BGBM Collection Data Form (CDF, zzzLinkzzz):<br /> <br /> # Open your web browser and navigate to the BGBM OpenRefine installation at http://services.bgbm.org/ws/OpenRefine/.<br /> # Press the “Durchsuchen”-B utton and select a CDF-File of your choice.<br /> # Press “Next”.<br /> # You will now be greeted with a first preview of your data. Since your CDF-File consisted of several (Excel) worksheets, OpenRefine needs to know, which of them need to be considered for importing data. In this case, de-select all worksheets except the collection dataform worksheet (see fig. 1). OpenRefine also needs to know, which row contains the relevant header of your worksheet. For the CDF you will have to “ignore first 3 line(s)” and “parse next 1 line(s)”.<br /> # Finally, press the “create project” button and you are ready to go.</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=Open_Refine_Data_Cleaning&diff=39 2018-09-17T12:16:09Z

<p>AntonGuentsch: Created page with &quot;Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for...&quot;</p> <hr /> <div>Data cleaning is an important part of the scientific data management and curation process and involves activities such as identification and correction of spelling errors (for example on scientific names), mapping data to agreed controlled vocabularies (e.g. ISO country names), setting correct references to external files (e.g. digital images), as well as re-arranging data so that they comply with standards required for specific data processing steps (e.g. imports into the JACQ collection management system at the BGBM).<br /> <br /> The BGBM scientific data workflows are based on the assumption that scientists are cleaning their data themselves to meet the BGBM data formats and standards. To support this important step of the scientific process, we provide an open installation of OpenRefine (formerly known as Google Refine) on a BGBM server at http://services.bgbm.org/ws/OpenRefine/.<br /> <br /> '''!! Please note that the BGBM OpenRefine server is only intended for the use as a tool for working on your data. It does not offer functions for storing data permanently and safely. We will however not delete any data without notice !!'''<br /> <br /> Like Microsoft Excel, OpenRefine operates primarily on tabular data with rows representing a measurement, occurrence record, or specimen for example and columns representing specific attributes (e.g. collectors, locality, longitude, latitude). Unlike Microsoft Excel, OpenRefine focusses specifically on functions and operations which are particularly useful for cleaning and arranging scientific data. This includes for example, the automated detection and merging of (potential) duplicate data items such as collector names, which have been captured using slightly different spellings but are referring to the same persons or person teams. The BGBM BDI team is constantly adding functions to the OpenRefine server installation, which are particularly useful in the context of botanic collection data processing. Please contact us for further information (biodiversitydata@bgbm.org).</div>

AntonGuentsch https://wiki.bgbm.org/collectionworkflows/index.php?title=BGBM_Collection_Workflows&diff=33 2018-09-12T11:00:43Z

<p>AntonGuentsch: </p> <hr /> <div><br /> '''This wiki serves as a central knowledge base for standards, documents, software, and workflows related to BGBM collection processes and networking of collection data. It is constantly updated with results of the BGBM &quot;collection workflows&quot; working group as well as the networking group of the BGBM Biodiversity Informatics.'''</div>

AntonGuentsch