https://wiki.bgbm.org/collectionworkflows/api.php?action=feedcontributions&feedformat=atom&user=GabiDroegeBGBM Collection Workflows - User contributions [en]2026-07-08T01:46:19ZUser contributionsMediaWiki 1.43.6https://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=579Research Data Management2026-05-21T08:37:35Z<p>GabiDroege: /* At a glance: What is research data and what isn’t */</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what is not =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[Evolutionary Analysis Results]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[Experimental Data and Lab Methodology Data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-based Data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[Derivatives of Image-based Data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scripts:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=578Research Data Management2026-05-21T08:35:58Z<p>GabiDroege: /* At a glance: What is research data and what isn’t */</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[Evolutionary Analysis Results]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[Experimental Data and Lab Methodology Data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-based Data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[Derivatives of Image-based Data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scripts:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=577Research Data Management2026-05-06T13:04:26Z<p>GabiDroege: /* At a glance: What is research data and what isn’t */</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[Evolutionary Analysis Results]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[Experimental Data and Lab Methodology Data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-based Data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[Derivatives of Image-based Data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Derivatives_of_Image-based_Data&diff=576Derivatives of Image-based Data2026-05-06T13:03:14Z<p>GabiDroege: Created page with "'''Nature:''' Derivatives of image-based data include for example * morphometry data derived from measurements of images and statistical analysis thereof * chromosome counts * tables containing characters matrices compiled from image analysis and analyses of these tables * phenotypic data recorded from living plants due to their similar nature and purpose. '''File formats:''' various formats. CSV or Excel for numerical data, or use special software, e.g. for character..."</p>
<hr />
<div>'''Nature:'''<br />
<br />
Derivatives of image-based data include for example<br />
<br />
* morphometry data derived from measurements of images and statistical analysis thereof<br />
* chromosome counts<br />
* tables containing characters matrices compiled from image analysis and analyses of these tables<br />
* phenotypic data recorded from living plants due to their similar nature and purpose.<br />
<br />
'''File formats:''' various formats. CSV or Excel for numerical data, or use special software, e.g. for character matrices. Analysis output format based on software.<br />
<br />
'''Storage / folder organization:'''<br />
<br />
* Store in (a) subfolder(s) within a project folder.<br />
* Organize folder(s) by type of derivative data and analyses.<br />
* Create subfolders for each analysis with a date and indicating the analysis method. These folders contain all files used for the analysis.<br />
* Store visualizations alongside the analysis data<br />
<br />
'''Naming convention:''' Use descriptive and consistent naming, include the type of analysis, date and relevant identifiers in the file name.<br />
<br />
'''Version control:''' Include version numbers or dates in the file names to keep track of different versions.<br />
<br />
'''Metadata:''' Document the source images, analysis method, software used and any other relevant parameters or settings. Store metadata in a README file within each folder.<br />
<br />
'''Link to source data:''' Maintain clear links to source images, and to samples or specimens through consistent naming and metadata documentation.<br />
<br />
'''Retention:''' Retain key data and final versions of analysis results indefinitely. Periodically review and delete intermediate or obsolete versions that are not necessary for future reference or publications.<br />
<br />
'''Publication:''' Publish data tables, character matrices or similar data as supplementary materials.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=575Research Data Management2026-05-06T13:01:54Z<p>GabiDroege: </p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[Evolutionary Analysis Results]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[Experimental Data and Lab Methodology Data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-based Data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[DerivativesOfImage-basedData | Derivatives of image-based data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Evolutionary_Analysis_Results&diff=574Evolutionary Analysis Results2026-05-06T13:01:02Z<p>GabiDroege: </p>
<hr />
<div>'''Nature:''' Analysis results derived from DNA sequence data, including phylogenetic analyses, networks, biogeography reconstruction., ancestral character state reconstructions, and more. Includes all files related to an analysis, such as matrices, command blocks, scripts, analysis logfiles and outputs, phylogenetic trees, networks and documentation of each analysis.<br />
<br />
'''File formats:''' Includes various formats provided by analysis software.<br />
<br />
'''Storage / folder organization:'''<br />
<br />
'''- Subfolder for analysis results:''' Store all analysis results in a dedicated subfolder within the relevant project folder.<br />
<br />
'''Organize by datasets:''' Each dataset should have its folder within the analysis results folder.<br />
<br />
'''Subfolders for each analysis:''' Create subfolders for each individual analysis, clearly labelled with the date and the analysis method. These subfolders should contain all files related to that specific analysis, like matrices, command files or blocks, log files and resulting trees. Include any further relevant information into a README file.<br />
<br />
'''Tree visualizations:''' Store any tree visualizations alongside the corresponding analysis data.<br />
<br />
'''Example folder structure'''<br />
<br />
Dianthus_phylogeny [project folder]<br />
<br />
└── Phylogenetic_analyses<br />
<br />
└── ITS<br />
<br />
└── 2024-01-15_MaximumLikelihood<br />
<br />
└── 2024-01-20_BayesianInference<br />
<br />
└── trnK-matK<br />
<br />
└── 2024-02-10_ MaximumLikelihood<br />
<br />
└── 2024-02-11_ BayesianInference<br />
<br />
'''Naming convention:''' Use short and descriptive file names to reflect the content. The analysis type and the data can also be included in the file name.<br />
<br />
Examples:<br />
<br />
Jurinea_ITS_datasetC.nex<br />
<br />
Arenaria_ITS_120taxa.nex<br />
<br />
Dianthus_trnKmatK_2024-01-15.nex<br />
<br />
Dianthus_ITS_2024-01-15_BI.nex<br />
<br />
'''Link to source data:''' Ensure that matrices and analysis outputs are clearly linked by maintaining consistent file names.<br />
<br />
Jurinea_ITS_datasetC.nex [matrix]<br />
<br />
Jurinea_ITS_datasetC.con [tree]<br />
<br />
'''Version control:''' Include dates in the file names to track different versions of the matrices or analyses.<br />
<br />
Example:<br />
<br />
Pyrus_plastid_combined_2017-04-05.nex<br />
<br />
'''Metadata:''' Document analysis parameters, software, version and any other relevant parameters or notes not included in the analysis logfiles. Store in README files for each analysis.<br />
<br />
'''Retention:''' As logfiles tend to be large, retain only latest analyses, unless earlier versions are needed. At the end of the project, retain only the final published version.<br />
<br />
'''Publication:'''<br />
* Publish trees in *.nex format along with the manuscript or upload to a repository<br />
* Store final/published version of the analyses, matrices, command files, trees files and any other relevant files along with the manuscript files, ensuring the to the publication is maintained.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Image-based_Data&diff=573Image-based Data2026-05-06T12:57:46Z<p>GabiDroege: Created page with "'''Nature:''' Image-based data can be classified by the method used to produce them and include * Scanning Electron Microscopy (SEM) images * Light microscopy images * Photographic data: field photos, sample photos, macro photos, etc. * Herbarium image data: Photograph of specimens, labels, spirit collections, fruits, etc. taken during herbarium visits or provided by herbaria that don’t have associated online records or online scans. '''Storage / folder organization'..."</p>
<hr />
<div>'''Nature:''' Image-based data can be classified by the method used to produce them and include<br />
<br />
* Scanning Electron Microscopy (SEM) images<br />
* Light microscopy images<br />
* Photographic data: field photos, sample photos, macro photos, etc.<br />
* Herbarium image data: Photograph of specimens, labels, spirit collections, fruits, etc. taken during herbarium visits or provided by herbaria that don’t have associated online records or online scans.<br />
<br />
'''Storage / folder organization'''<br />
<br />
'''SEM images and LM images'''<br />
<br />
* Store all image data under a centralized location, organized by project and type of imaging<br />
* Main folders for each project, with subfolders for each type of image data.<br />
* Subfolders within each type to further organize by date, specimen, or other relevant categories.<br />
<br />
Naming conventions:<br />
* Use descriptive names reflecting the content, include relevant details in the file name<br />
<br />
'''Herbarium images'''<br />
<br />
'''Folder structure:''' Store in a dedicated folder for &quot;Herbarium image data&quot; within a project folder. If necessary, create subfolders for different herbaria or taxa.<br />
<br />
'''Naming convention:''' Include the herbarium acronym, the taxon and date of the photograph. Ensure that the collection metadata referring to the specimen are linked for pictures showing only a part of the specimen.<br />
<br />
'''Publication:''' Respect any copyright or usage restrictions imposed by the herbarium from which the image was taken. Ensure proper citation and acknowledgment of the source herbaria.<br />
<br />
'''Metadata documentation:'''<br />
<br />
Each set of images should include metadata detailing<br />
<br />
* Imaging conditions (e.g., lighting, settings)<br />
* magnification level<br />
* Sample preparation method<br />
* Image creator (photographer or technician)<br />
* Usage rights and CC-license<br />
<br />
Store metadata in a README file or as part of the image file properties.<br />
<br />
'''Retention:'''<br />
<br />
Retain key images used in publications and other important images. Periodically review and delete redundant or low-quality images that are not necessary for future reference or publications.<br />
<br />
'''Publication:'''<br />
<br />
Ensure that image creator and copyright are indicated.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=572Research Data Management2026-05-06T12:54:22Z<p>GabiDroege: /* At a glance: What is research data and what isn’t */</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[Evolutionary Analysis Results]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[Experimental Data and Lab Methodology Data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-basedData | Image-based data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[DerivativesOfImage-basedData | Derivatives of image-based data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Experimental_Data_and_Lab_Methodology_Data&diff=571Experimental Data and Lab Methodology Data2026-05-06T12:50:28Z<p>GabiDroege: Created page with "'''Nature:''' Lab-related data encompass various types of experimental and methodological data, such as: * PCR protocols * information on primers * PCR parameters * gel electrophoresis pictures * DNA concentration or quality measures results * flow cytometry results * genome size measurements results * supporting data, e.g. Macrogen order tables, lab progress tracked in a spreadsheet, spreadsheets used to produce dilutions, etc. '''Storage / folder organization:''' St..."</p>
<hr />
<div>'''Nature:''' Lab-related data encompass various types of experimental and methodological data, such as:<br />
<br />
* PCR protocols<br />
* information on primers<br />
* PCR parameters<br />
* gel electrophoresis pictures<br />
* DNA concentration or quality measures results<br />
* flow cytometry results<br />
* genome size measurements results<br />
* supporting data, e.g. Macrogen order tables, lab progress tracked in a spreadsheet, spreadsheets used to produce dilutions, etc.<br />
<br />
'''Storage / folder organization:'''<br />
<br />
Store lab data in subfolders under the main project folders for each type of data<br />
<br />
'''Naming convention:'''<br />
<br />
Use clear, descriptive file names that reflect the content and purpose of the file.<br />
<br />
'''Link to source data:'''<br />
<br />
Ensure all lab data files are linked to source samples through the labcode. This can be done for example by including the labcode in the file names or maintaining a list that maps files to their respective labcodes.<br />
<br />
'''Retention:'''<br />
<br />
<u>Retained indefinitely</u><br />
<br />
* DNA gel electrophoresis pictures, concentration measurements and other parameters of genomic DNA, curated by the DNA bank. Document when samples are used up.<br />
<br />
<u>Retained temporarily</u><br />
<br />
* PCR gel electrophoresis, long gel images, Macrogen order sheets and other non-essential data are deleted after successful sequencing, latest after the publication of the relevant study.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=570Research Data Management2026-05-06T10:15:57Z<p>GabiDroege: </p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[Evolutionary Analysis Results]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[ExperimentalData/LabMethodologyData | Experimental data / lab data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-basedData | Image-based data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[DerivativesOfImage-basedData | Derivatives of image-based data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Evolutionary_Analysis_Results&diff=569Evolutionary Analysis Results2026-05-06T10:14:36Z<p>GabiDroege: Created page with "'''Nature:''' Analysis results derived from DNA sequence data, including phylogenetic analyses, networks, biogeography reconstruction., ancestral character state reconstructions, and more. Includes all files related to an analysis, such as matrices, command blocks, scripts, analysis logfiles and outputs, phylogenetic trees, networks and documentation of each analysis. '''File formats:''' Includes various formats provided by analysis software. '''Storage / folder organi..."</p>
<hr />
<div>'''Nature:''' Analysis results derived from DNA sequence data, including phylogenetic analyses, networks, biogeography reconstruction., ancestral character state reconstructions, and more. Includes all files related to an analysis, such as matrices, command blocks, scripts, analysis logfiles and outputs, phylogenetic trees, networks and documentation of each analysis.<br />
<br />
'''File formats:''' Includes various formats provided by analysis software.<br />
<br />
'''Storage / folder organization:'''<br />
<br />
'''- Subfolder for analysis results:''' Store all analysis results in a dedicated subfolder within the relevant project folder.<br />
<br />
'''Organize by datasets:''' Each dataset should have its folder within the analysis results folder.<br />
<br />
'''Subfolders for each analysis:''' Create subfolders for each individual analysis, clearly labelled with the date and the analysis method. These subfolders should contain all files related to that specific analysis, like matrices, command files or blocks, log files and resulting trees. Include any further relevant information into a README file.<br />
<br />
'''Tree visualizations:''' Store any tree visualizations alongside the corresponding analysis data.<br />
<br />
'''Example folder structure'''<br />
<br />
Dianthus_phylogeny [project folder]<br />
<br />
└── Phylogenetic_analyses<br />
<br />
└── ITS<br />
<br />
└── 2024-01-15_MaximumLikelihood<br />
<br />
└── 2024-01-20_BayesianInference<br />
<br />
└── trnK-matK<br />
<br />
└── 2024-02-10_ MaximumLikelihood<br />
<br />
└── 2024-02-11_ BayesianInference<br />
<br />
'''Naming convention:''' Use short and descriptive file names to reflect the content. The analysis type and the data can also be included in the file name.<br />
<br />
Examples:<br />
<br />
Jurinea_ITS_datasetC.nex<br />
<br />
Arenaria_ITS_120taxa.nex<br />
<br />
Dianthus_trnKmatK_2024-01-15.nex<br />
<br />
Dianthus_ITS_2024-01-15_BI.nex<br />
<br />
'''Link to source data:''' Ensure that matrices and analysis outputs are clearly linked by maintaining consistent file names.<br />
<br />
Jurinea_ITS_datasetC.nex [matrix]<br />
<br />
Jurinea_ITS_datasetC.con [tree]<br />
<br />
'''Version control:''' Include dates in the file names to track different versions of the matrices or analyses.<br />
<br />
Example:<br />
<br />
Pyrus_plastid_combined_2017-04-05.nex<br />
<br />
'''Metadata:''' Document analysis parameters, software, version and any other relevant parameters or notes not included in the analysis logfiles. Store in README files for each analysis.<br />
<br />
'''Retention:''' As logfiles tend to be large, retain only latest analyses, unless earlier versions are needed. At the end of the project, retain only the final published version.<br />
<br />
'''Publication:'''<br />
<br />
- Publish trees in *.nex format along with the manuscript or upload to a repository<br />
<br />
- Store final/published version of the analyses, matrices, command files, trees files and any other relevant files along with the manuscript files, ensuring the to the publication is maintained.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=DNA_Sequences_and_Derivatives&diff=568DNA Sequences and Derivatives2026-05-06T10:12:35Z<p>GabiDroege: /* DNA sequences */</p>
<hr />
<div>DNA sequence data and their derivatives are primarily organized by sequencing methods (Sanger vs. NGS) and by lab code and DNA region, with folders for raw data, processed data and analysis results<br />
<br />
= DNA sequences =<br />
'''Nature:''' pherograms / reads from Sanger sequencing or NGS platforms<br />
<br />
'''File formats:'''<br />
- Sanger sequences: Pherograms: *.scf for own sequences, *.abi format (can be converted in scf)<br />
- NGS: *.fastq.gz<br />
<br />
'''Storage / folder organization:'''<br />
<br />
- Sanger sequences are organized by taxonomic group and the DNA region.<br />
<br />
'''Version control:''' multiple reads derived from the same sample are distinguished by consecutive numbering of the PCR products they result from.<br />
<br />
'''Metadata:''' Linked to source data under section 1. by labcode (own data) or INSDC accession number (published data)<br />
<br />
- Sample metadata: as described under section 1.<br />
<br />
- Sequencing details: for NGS data, the sequencing platform or method and relevant parameters<br />
<br />
'''Retention:'''<br />
<br />
'''- Sanger sequences:''' Pherograms are stored indefinitely. Bad pherograms resulting from failed sequencing are deleted once sequencing is successful, or latest after completion of a study.<br />
<br />
- ab1, txt and pdf files from Macrogen are not retained.<br />
<br />
'''- NGS raw data''': A backup copy of the raw data as received from the sequencing company is saved in BGBM NGS raw data storage. In addition, the raw data files (usually renamed for further processing) are stored in a separate subdirectory of the project directory.<br />
<br />
'''Publication:''' pherograms are normally not published<br />
<br />
== Sanger Sequences ==<br />
'''Sanger sequences are organized by DNA region, by study group or project, and by labcode.'''<br />
* '''Main folder:''' The folder containing pherograms and contigs is named “sequences” or includes the word “sequences”.<br />
<br />
Example: CAR_sequences<br />
* '''Folder naming:''' the folder name must contain the unambiguous name of the DNA region sequenced, e.g. trnL-F. Naming folders with person’s names is not permitted, even if multiple people are working on the same project.<br />
<br />
Example: CAR_trnK-matK<br />
* '''New sequences:''' Store in a subfolder named “new_sequences” until they are processed.<br />
* '''No separation by study:''' Sequences under one labcode are not separated by studies or projects. For example, all Caryophyllaceae sequences (labcode CAR) are stored in one folder CAR_sequences, not separated by the genera studied.<br />
* '''Batch organization:''' Depending on the number of sequences, they can either be stored in one folder or in subfolders of batches of 100 sequences. The second option is highly recommended.<br />
* '''Labcode / DB number:''' For sequences that have no labcode but a DB number, the structure is the same, using DB numbers instead of labcodes.<br />
* '''Deprecated or old sequences:''' Move old or deprecated sequences to an archive folder within the same structure, clearly labelled with the date of archival.<br />
<br />
Example: Archive_CA_trnL-F_Jan2024<br />
<br />
Example structure:<br />
<br />
Cactaceae<br />
<br />
└── CA_sequences<br />
<br />
└── CA_trnL-F<br />
<br />
└── CA_trnK-matK<br />
<br />
└── CA_new_sequences<br />
<br />
'''Naming conventions'''<br />
<br />
'''Pherograms'''<br />
<br />
&lt;Lab code&gt;_&lt;PCR product number&gt;_&lt;primer&gt;.scf<br />
<br />
Example: CAR001_ NK304_ITS4.scf<br />
<br />
This naming is automatically generated by Macrogen if the order sheet is filled out in this way.<br />
<br />
DB numbers are used if the sample has no labcode.<br />
<br />
'''Contigs'''<br />
<br />
The name always contains the lab code and the DNA region name. The species / taxon name is optional.<br />
<br />
&lt;Lab code&gt;_&lt;region name&gt;.pde<br />
<br />
Example: CRD095_ITS.pde<br />
<br />
&lt;Lab code&gt;_&lt;taxon&gt;_&lt;region name&gt;.pde<br />
<br />
Example: CRD095_Jurinea-arachnoidea_ITS.pde<br />
<br />
'''File structure within the folders'''<br />
<br />
When working with PhyDE, the contig files and the pherograms have to be either<br />
<br />
<blockquote>a. in the same folder, or<br />
<br />
b. the pherograms must be in a subfolder of the folder containing the contig files.<br />
</blockquote><br />
This ensures a stable link between pherograms and contig. This link will remain when migrating data (all folders); but will disappear if the folders holding pherograms are renamed.<br />
<br />
'''Option 1.''' Contigs and pherograms in the same folder (convenient for small datasets)<br />
<br />
📂 CAR_ITS<br />
<br />
CAR001_ITS.pde [contig]<br />
<br />
CAR001_NK304_ITS4.scf [pherogram]<br />
<br />
CAR001_NK304_ITS5.scf [pherogram]<br />
<br />
'''Option 2.''' Pherograms in a subfolder (better for larger datasets)<br />
<br />
📂 CAR_ITS<br />
<br />
CAR001_ITS.pde<br />
<br />
📂 pherograms<br />
<br />
CAR001_ NK304_ITS4.scf<br />
<br />
CAR001_ NK304_ITS5.scf<br />
<br />
'''Batch organization.''' Batches of 100 for large datasets and trnK-matK data<br />
<br />
📂 CA_rpl16<br />
<br />
CA101_NK205_rpl16.pde<br />
<br />
📂 CA001-099<br />
<br />
📂 CA100-199<br />
<br />
CA101_NK205_CArps3F.pde<br />
<br />
CA101_NK205_rpl16R.pde<br />
<br />
'''Additional guidelines'''<br />
<br />
'''Genbank sequences:''' Store alongside contigs. Alternatively, include directly in an alignment with the sequence accession number. Documenting the accession numbers and publication is crucial for accurate source citation.<br />
<br />
'''Sequences from external collaborators:''' Store separately from own sequences, and the folder name can contain the collaborator’s name. Save any e-mails or other information regarding these sequences in the same folder and clearly document all associated information.<br />
<br />
'''Sequences with different lab codes for one study:''' In case of multiple lab codes for one taxonomic group (like ERS and CRD for Cardueae), store them together in a folder under the labcode that refers to the taxonomic group (CRD).<br />
<br />
= Derivatives of DNA sequences =<br />
<br />
'''Nature:'''<br />
<br />
(a) contigs or assembled reads; assembled cp genomes; cleaned and deduped NGS reads (including those from partners or downloaded published reads; reads mapped to Hyb-Seq targets.<br />
<br />
(b) alignments: master alignments, single-locus alignments, concatenated alignments.<br />
<br />
'''File formats:''' various formats, including Fasta, Phylip, Nexus, PhyDe.<br />
<br />
'''Storage / folder organization:''' Sequences are organized by taxonomic group or project, and the DNA region. Sequences obtained from other sources are stored in the same location as own contigs.<br />
<br />
'''Naming convention:''' see separate document.<br />
<br />
'''Version control:''' For alignments, include version numbers or dates in the file names, if relevant, also include the number of terminals or any other dataset parameter.<br />
<br />
'''Metadata:'''<br />
<br />
'''Sample metadata:''' As described above<br />
<br />
'''Data source:''' For sequences obtained from other sources (e.g., collaborators), include the origin, accession numbers, citation and any other relevant information. For GenBank sequences, include accession number and citations.<br />
<br />
'''Link to source data:''' Linked to source data under section 1. by labcode (own data) or INSDC accession number (published data).<br />
<br />
'''Link to sample metadata:''' through labcode or DB number<br />
<br />
Accession numbers for published sequences are included in the DNA master list.<br />
<br />
'''Retention:'''<br />
<br />
'''- Contigs:''' stored indefinitely<br />
<br />
'''- Alignments:''' published alignments are always retained and linked to the publication. Master alignments are stored as versions; outdated versions can be deleted.<br />
<br />
'''Publication:'''<br />
<br />
- Sequences used in a manuscript are submitted to GenBank or ENA, along with the sample metadata, using a specified structure including standardized locality metadata, identifiers, and marker designations.<br />
<br />
- Final alignments are published along with the manuscript, either as supplement, or uploaded to a repository.<br />
<br />
'''Sequence submission files:''' Project info text file, metadata file(s) (original and formatted for submission), alignments files, taxonomic info files (e.g., for unpublished names), flat files for submission (ENA), other ENA-specific files, and the file(s) containing accession numbers are archived along with the manuscript files.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=567Research Data Management2026-05-06T10:06:23Z<p>GabiDroege: /* At a glance: What is research data and what isn’t */</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNA Sequences and Derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[EvolutionaryAnalysisResults | Evolutionary analyses]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[ExperimentalData/LabMethodologyData | Experimental data / lab data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-basedData | Image-based data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[DerivativesOfImage-basedData | Derivatives of image-based data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=DNA_Sequences_and_Derivatives&diff=566DNA Sequences and Derivatives2026-05-06T10:01:57Z<p>GabiDroege: Created page with "DNA sequence data and their derivatives are primarily organized by sequencing methods (Sanger vs. NGS) and by lab code and DNA region, with folders for raw data, processed data and analysis results = DNA sequences = '''Nature:''' pherograms / reads from Sanger sequencing or NGS platforms '''File formats:''' - Sanger sequences: Pherograms: *.scf for own sequences, *.abi format (can be converted in scf) - NGS: *.fastq.gz '''Storage / folder organization:''' - Sanger se..."</p>
<hr />
<div>DNA sequence data and their derivatives are primarily organized by sequencing methods (Sanger vs. NGS) and by lab code and DNA region, with folders for raw data, processed data and analysis results<br />
<br />
= DNA sequences =<br />
'''Nature:''' pherograms / reads from Sanger sequencing or NGS platforms<br />
<br />
'''File formats:'''<br />
- Sanger sequences: Pherograms: *.scf for own sequences, *.abi format (can be converted in scf)<br />
- NGS: *.fastq.gz<br />
<br />
'''Storage / folder organization:'''<br />
<br />
- Sanger sequences are organized by taxonomic group and the DNA region.<br />
<br />
'''Version control:''' multiple reads derived from the same sample are distinguished by consecutive numbering of the PCR products they result from.<br />
<br />
'''Metadata:''' Linked to source data under section 1. by labcode (own data) or INSDC accession number (published data)<br />
<br />
- Sample metadata: as described under section 1.<br />
<br />
- Sequencing details: for NGS data, the sequencing platform or method and relevant parameters<br />
<br />
'''Retention:'''<br />
<br />
'''- Sanger sequences:''' Pherograms are stored indefinitely. Bad pherograms resulting from failed sequencing are deleted once sequencing is successful, or latest after completion of a study.<br />
<br />
- ab1, txt and pdf files from Macrogen are not retained.<br />
<br />
'''- NGS raw data''': A backup copy of the raw data as received from the sequencing company is saved in BGBM NGS raw data storage. In addition, the raw data files (usually renamed for further processing) are stored in a separate subdirectory of the project directory.<br />
<br />
'''Publication:''' pherograms are normally not published<br />
<br />
== Sanger Sequences ==<br />
'''Sanger sequences are organized by DNA region, by study group or project, and by labcode.'''<br />
* '''Main folder:''' The folder containing pherograms and contigs is named “sequences” or includes the word “sequences”.<br />
<br />
Example: CAR_sequences<br />
* '''Folder naming:''' the folder name must contain the unambiguous name of the DNA region sequenced, e.g. trnL-F. Naming folders with person’s names is not permitted, even if multiple people are working on the same project.<br />
<br />
Example: CAR_trnK-matK<br />
* '''New sequences:''' Store in a subfolder named “new_sequences” until they are processed.<br />
* '''No separation by study:''' Sequences under one labcode are not separated by studies or projects. For example, all Caryophyllaceae sequences (labcode CAR) are stored in one folder CAR_sequences, not separated by the genera studied.<br />
* '''Batch organization:''' Depending on the number of sequences, they can either be stored in one folder or in subfolders of batches of 100 sequences. The second option is highly recommended.<br />
* '''Labcode / DB number:''' For sequences that have no labcode but a DB number, the structure is the same, using DB numbers instead of labcodes.<br />
* '''Deprecated or old sequences:''' Move old or deprecated sequences to an archive folder within the same structure, clearly labelled with the date of archival.<br />
<br />
Example: Archive_CA_trnL-F_Jan2024<br />
<br />
Example structure:<br />
<br />
Cactaceae<br />
<br />
└── CA_sequences<br />
<br />
└── CA_trnL-F<br />
<br />
└── CA_trnK-matK<br />
<br />
└── CA_new_sequences<br />
<br />
'''Naming conventions'''<br />
<br />
'''Pherograms'''<br />
<br />
&lt;Lab code&gt;_&lt;PCR product number&gt;_&lt;primer&gt;.scf<br />
<br />
Example: CAR001_ NK304_ITS4.scf<br />
<br />
This naming is automatically generated by Macrogen if the order sheet is filled out in this way.<br />
<br />
DB numbers are used if the sample has no labcode.<br />
<br />
'''Contigs'''<br />
<br />
The name always contains the lab code and the DNA region name. The species / taxon name is optional.<br />
<br />
&lt;Lab code&gt;_&lt;region name&gt;.pde<br />
<br />
Example: CRD095_ITS.pde<br />
<br />
&lt;Lab code&gt;_&lt;taxon&gt;_&lt;region name&gt;.pde<br />
<br />
Example: CRD095_Jurinea-arachnoidea_ITS.pde<br />
<br />
'''File structure within the folders'''<br />
<br />
When working with PhyDE, the contig files and the pherograms have to be either<br />
<br />
<blockquote>a. in the same folder, or<br />
<br />
b. the pherograms must be in a subfolder of the folder containing the contig files.<br />
</blockquote><br />
This ensures a stable link between pherograms and contig. This link will remain when migrating data (all folders); but will disappear if the folders holding pherograms are renamed.<br />
<br />
'''Option 1.''' Contigs and pherograms in the same folder (convenient for small datasets)<br />
<br />
📂 CAR_ITS<br />
<br />
CAR001_ITS.pde [contig]<br />
<br />
CAR001_NK304_ITS4.scf [pherogram]<br />
<br />
CAR001_NK304_ITS5.scf [pherogram]<br />
<br />
'''Option 2.''' Pherograms in a subfolder (better for larger datasets)<br />
<br />
📂 CAR_ITS<br />
<br />
CAR001_ITS.pde<br />
<br />
📂 pherograms<br />
<br />
CAR001_ NK304_ITS4.scf<br />
<br />
CAR001_ NK304_ITS5.scf<br />
<br />
'''Batch organization.''' Batches of 100 for large datasets and trnK-matK data<br />
<br />
📂 CA_rpl16<br />
<br />
CA101_NK205_rpl16.pde<br />
<br />
📂 CA001-099<br />
<br />
📂 CA100-199<br />
<br />
CA101_NK205_CArps3F.pde<br />
<br />
CA101_NK205_rpl16R.pde<br />
<br />
'''Additional guidelines'''<br />
<br />
'''Genbank sequences:''' Store alongside contigs. Alternatively, include directly in an alignment with the sequence accession number. Documenting the accession numbers and publication is crucial for accurate source citation.<br />
<br />
'''Sequences from external collaborators:''' Store separately from own sequences, and the folder name can contain the collaborator’s name. Save any e-mails or other information regarding these sequences in the same folder and clearly document all associated information.<br />
<br />
'''Sequences with different lab codes for one study:''' In case of multiple lab codes for one taxonomic group (like ERS and CRD for Cardueae), store them together in a folder under the labcode that refers to the taxonomic group (CRD).<br />
<br />
== NGS Sequences ==<br />
<br />
= Derivatives of DNA sequences =<br />
<br />
'''Nature:'''<br />
<br />
(a) contigs or assembled reads; assembled cp genomes; cleaned and deduped NGS reads (including those from partners or downloaded published reads; reads mapped to Hyb-Seq targets.<br />
<br />
(b) alignments: master alignments, single-locus alignments, concatenated alignments.<br />
<br />
'''File formats:''' various formats, including Fasta, Phylip, Nexus, PhyDe.<br />
<br />
'''Storage / folder organization:''' Sequences are organized by taxonomic group or project, and the DNA region. Sequences obtained from other sources are stored in the same location as own contigs.<br />
<br />
'''Naming convention:''' see separate document.<br />
<br />
'''Version control:''' For alignments, include version numbers or dates in the file names, if relevant, also include the number of terminals or any other dataset parameter.<br />
<br />
'''Metadata:'''<br />
<br />
'''Sample metadata:''' As described above<br />
<br />
'''Data source:''' For sequences obtained from other sources (e.g., collaborators), include the origin, accession numbers, citation and any other relevant information. For GenBank sequences, include accession number and citations.<br />
<br />
'''Link to source data:''' Linked to source data under section 1. by labcode (own data) or INSDC accession number (published data).<br />
<br />
'''Link to sample metadata:''' through labcode or DB number<br />
<br />
Accession numbers for published sequences are included in the DNA master list.<br />
<br />
'''Retention:'''<br />
<br />
'''- Contigs:''' stored indefinitely<br />
<br />
'''- Alignments:''' published alignments are always retained and linked to the publication. Master alignments are stored as versions; outdated versions can be deleted.<br />
<br />
'''Publication:'''<br />
<br />
- Sequences used in a manuscript are submitted to GenBank or ENA, along with the sample metadata, using a specified structure including standardized locality metadata, identifiers, and marker designations.<br />
<br />
- Final alignments are published along with the manuscript, either as supplement, or uploaded to a repository.<br />
<br />
'''Sequence submission files:''' Project info text file, metadata file(s) (original and formatted for submission), alignments files, taxonomic info files (e.g., for unpublished names), flat files for submission (ENA), other ENA-specific files, and the file(s) containing accession numbers are archived along with the manuscript files.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=565Research Data Management2026-05-06T09:55:51Z<p>GabiDroege: /* At a glance: What is research data and what isn’t */</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNAsequencesAndDerivatives | DNA sequences and their derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[EvolutionaryAnalysisResults | Evolutionary analyses]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[ExperimentalData/LabMethodologyData | Experimental data / lab data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-basedData | Image-based data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[DerivativesOfImage-basedData | Derivatives of image-based data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Sample_Metadata_and_Sampling_Information&diff=564Sample Metadata and Sampling Information2026-05-06T09:54:21Z<p>GabiDroege: Created page with "'''Nature:''' Sample origin data (e.g., locality) and various identifiers associated with it, e.g., labcode, tissue ID (BGT- number), DB number, voucher ID (herbarium barcode, internal or external if available). This category also encompasses sampling lists for subprojects, sampling wish lists, sample lists obtained from other sources (e.g., collaborators, datasets from herbaria, records downloaded from databases, BoGart exports, and similar). Tables that track lab work..."</p>
<hr />
<div>'''Nature:'''<br />
<br />
Sample origin data (e.g., locality) and various identifiers associated with it, e.g., labcode, tissue ID (BGT- number), DB number, voucher ID (herbarium barcode, internal or external if available). This category also encompasses sampling lists for subprojects, sampling wish lists, sample lists obtained from other sources (e.g., collaborators, datasets from herbaria, records downloaded from databases, BoGart exports, and similar).<br />
Tables that track lab work progress (e.g., status or success of DNA extraction, PCR success, sequencing results) do not fall in this category as they serve a different purpose; they are part of lab methodology data.<br />
<br />
'''Format:'''<br />
<br />
- spreadsheets: curated standardised DNA lists, collection data forms (CDF) curated in MS Excel<br />
<br />
- The format should follow an established standard; if data come from external sources, they should be formatted to fit this standard. For locality metadata, the established standard is the JACQ database field structure.<br />
<br />
'''Storage / folder organization:'''<br />
<br />
- stored on <DRIVENAME>:/DNA-lists and organized by labcode (i.e. corresponding to a taxonomic group)<br />
<br />
- long-term-goal: consolidate all records in a central database or core that effectively links different data sources<br />
<br />
'''Naming convention:'''<br />
<br />
Example: MASTER_CAR_Caryophyllaceae_DNA_CuratedList_Jan2024.xlsx<br />
<br />
'''Version control:'''<br />
<br />
Version regularly, approximately every three months or after each major update/addition, including the date into the file name.<br />
<br />
'''Access control:'''<br />
<br />
Access is restricted to maintain data integrity and consistency. One person is responsible for curating each list; others may have writing permissions. All other persons get read-only access.<br />
<br />
'''Retention:'''<br />
<br />
Retained indefinitely. Old or obsolete versions identified as such by the responsible curator may be deleted.<br />
<br />
'''Publication:'''<br />
<br />
- Sample metadata are published through online portals (e.g., JACQ, GGBN) ensuring links between sample, tissue, etc., providing the associated identifiers.<br />
<br />
- Sample metadata for published sequences are published through GenBank/ENA.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=MediaWiki:Sidebar&diff=563MediaWiki:Sidebar2026-05-06T09:51:09Z<p>GabiDroege: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
** Contacts| Contacts<br />
* Collection Workflows<br />
** Collection Data Pipeline| Data Pipeline<br />
* Software<br />
** Collection Data Form | Collection Data Form<br />
** Open Refine Data Cleaning | Data Cleaning<br />
** https://www.gbif.org/occurrence/search?institution_code=B&institution_code=BGBM&occurrence_status=present | BGBM collections in GBIF<br />
** http://www.ggbn.org | DNA collection portal<br />
** Portal_Architecture | Architecture overview<br />
* Documentation<br />
** Research Data Management | Research Data Management<br />
** Collection Manuals | Collection Manuals<br />
** Barcodes für Sammlungsobjekte | Barcodes at B<br />
** Question and Answer | Q & A<br />
<br />
** Internal:Main Page|Internal<br />
** recentchanges-url|recentchanges<br />
<br />
* SEARCH<br />
* TOOLBOX</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=MediaWiki:Sidebar&diff=562MediaWiki:Sidebar2026-05-06T09:50:42Z<p>GabiDroege: added RDM page</p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
** Contacts| Contacts<br />
* Collection Workflows<br />
** Collection Data Pipeline| Data Pipeline<br />
* Software<br />
** Collection Data Form | Collection Data Form<br />
** Open Refine Data Cleaning | Data Cleaning<br />
** https://www.gbif.org/occurrence/search?institution_code=B&institution_code=BGBM&occurrence_status=present | BGBM collections in GBIF<br />
** http://www.ggbn.org | DNA collection portal<br />
** Portal_Architecture | Architecture overview<br />
* Documentation<br />
** Research Data Management at Bo Berlin | Research Data Management<br />
** Collection Manuals | Collection Manuals<br />
** Barcodes for Herbarium specimens | Barcodes at B<br />
** Question and Answer | Q & A<br />
<br />
** Internal:Main Page|Internal<br />
** recentchanges-url|recentchanges<br />
<br />
* SEARCH<br />
* TOOLBOX</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Research_Data_Management&diff=561Research Data Management2026-05-06T09:46:33Z<p>GabiDroege: Created page with "= Introduction = Research Data Management (RDM) is an integral part of the research process. It encompasses: * Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable. * Storage: Storing data in a manner that ensures its integrity and accessibility over time. * Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation. * Sharing and publication: Making d..."</p>
<hr />
<div>= Introduction =<br />
Research Data Management (RDM) is an integral part of the research process. It encompasses:<br />
* Organisation and documentation: Systematic arrangement and thorough documentation of data to ensure it is understandable and useable.<br />
* Storage: Storing data in a manner that ensures its integrity and accessibility over time.<br />
* Backup and archiving: Regular backups to prevent data loss and archiving for long-term preservation.<br />
* Sharing and publication: Making data available in accordance with ethical, legal and institutional guidelines.<br />
<br />
= Legal and formal framework =<br />
<br />
== In Germany ==<br />
In Germany, research data management is governed by several legal and formal frameworks. The primary framework is the '''[https://www.dfg.de/en/basics-topics/basics-and-principles-of-funding/good-research-practice/code-of-conduct Code of Conduct of the Deutsche Forschungsgemeinschaft (DFG)]'''. This code sets the standards for good scientific practice and mandates that universities implement these standards in a legally binding manner.<br />
<br />
== At Freie Universität Berlin ==<br />
At Freie Universität Berlin, the following regulations are in place:<br />
<br />
'''[https://www.fu-berlin.de/sites/gwp/informationen/dokumente/GWP_Satzung_FUB_2024_EN.pdf Freie Universität Berlin’s Statutes for Safeguarding Good Research Practice] ([https://www.fu-berlin.de/service/zuvdocs/amtsblatt/2024/ab022024.pdf Satzung zur Sicherung der guten wissenschaftlichen Praxis])''': These statutes are the legally binding implementation of the DFG Code of Conduct and govern principles of good scientific practice, including:<br />
<br />
* Obligations of researchers<br />
* Supervision responsibilities<br />
* Usage rights and authorship<br />
* Prevention of abuse of power<br />
* Addressing scientific misconduct<br />
<br />
'''[https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/policy/index.html Research Data Policy]''': This policy outlines the requirements for handling research data, ensuring it is managed in line with legal and ethical standards.<br />
<br />
== Usage rights ==<br />
According to German law, no single party owns research data. Instead, data usage and publication rights are regulated to ensure fairness and compliance with institutional policies. [https://www.fu-berlin.de/en/sites/forschungsdatenmanagement/in-der-praxis/planung/vereinbarungen.html Research Data Agreements FU Berlin]<br />
<br />
= At a glance: What is research data and what isn’t =<br />
Understanding what constitutes research data is crucial for proper management. In general terms, research data constitute<br />
<br />
* Raw data: Original data collected from experiments, surveys, or observations.<br />
* Processed data: Data that has been cleaned, transformed, or analysed.<br />
* Metadata: Information describing the context, content, and structure of data.<br />
* Protocols and methods: Documentation of methodologies and procedures used in data collection and analysis.<br />
* Software and code: Scripts, programs, or algorithms developed for data analysis or as part of a research project (does not encompass standard software).<br />
* Images and visualizations: Photographs, graphs, charts, and other visual representations of data.<br />
<br />
More specifically, research data in biodiversity and evolutionary research encompass:<br />
<br />
* '''[[SampleMetadataAndSamplingInfo | Sample Metadata and Sampling Information]]:''' Includes sample origin data and the various identifiers associated with it.<br />
* '''[[DNAsequencesAndDerivatives | DNA sequences and their derivatives]]:''' Includes raw data from sequencing platforms, pherograms, contigs, alignments, and assembled plastid genomes.<br />
* '''[[EvolutionaryAnalysisResults | Evolutionary analyses]]:''' Includes analysis results derived from DNA sequence data, such as matrices, command blocks, analysis logfiles and outputs, phylogenetic trees or networks.<br />
* '''[[ExperimentalData/LabMethodologyData | Experimental data / lab data]]:''' Includes lab-related data such as PCR protocols, primer information, PCR parameters, gel electrophoresis pictures, DNA concentration or quality results, flow cytometry results measures, genome size measurements, etc.<br />
* '''[[Image-basedData | Image-based data]]:''' Includes Scanning Electron Microscopy (SEM) images, light microscopy images, photographic data (e.g., field photos, sample photos, macro photos), and herbarium image data.<br />
* '''[[DerivativesOfImage-basedData | Derivatives of image-based data]]:''' Includes morphometry data derived from measurements of images and statistical analysis thereof, chromosome counts, tables containing characters matrices compiled from image analysis, and phenotypic data recorded from living plants.<br />
* '''Geospatial data, environmental data, ecological data:''' Includes coordinates, maps, plot inventories, data logger data, etc.<br />
* '''Software and scrips:''' Includes commands, scripts or pipelines developed for research as well as software packages developed in research projects.<br />
<br />
== Non-research data encompass ==<br />
== Project administration ==<br />
* '''General documents:''' Includes project proposals, work plans, exposes, budget, reports, formal collaboration agreements and any administrative or legal documents related to projects or grants.<br />
* '''Meeting minutes &amp; e-mail correspondence'''<br />
* '''Event-related documents:''' Includes agendas, participant lists, and other organizational documents related to planning and executing events.<br />
<br />
== Other non-research data ==<br />
* '''Literature and reference materials:''' Includes articles, books, software manuals, images, illustrations and other reference materials used for research.<br />
* '''Training and teaching materials:''' Includes presentations and documents for in-house training and academic education.<br />
* '''Protocols and SOPs:''' Includes lab procedures or in-house guidelines.<br />
* '''Standard software:''' Includes standard software used for analyses like MrBayes, R, Geneious, etc.<br />
<br />
'''Why this distinction is important'''<br />
<br />
'''Research data''' need to be managed for scientific integrity, accessibility and compliance with good scientific practice.<br />
<br />
'''Project administration documents''' require careful management to meet legal, financial and administrative requirement by the Freie Universität Berlin and the funders.<br />
<br />
'''Other non-research data''' are important for reference abut do not have the same stringent management and storage requirements as research data or project administration.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=533ENA Submission Pipeline2021-07-23T12:48:40Z<p>GabiDroege: /* Most common software errors */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc., replace all of them by "regular" characters<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing, add data to metadata.csv<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file, update number in nexus file<br />
|-<br />
| annonex2embl ERROR: csv-file does not contain a column labelled isolate.<br />
| You have probably forgotten, that ";" is NOT supported as delimiter; see above how to export a csv with "," as delimiter<br />
|-<br />
| ValueError: End location (1410) must be greater than or equal to start location (1417)<br />
| CharSet error in Nexus fils, e.g. 1874-2222; 2222-2335; instead of 1874-2221; 2222-2335;<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://kb.bgbm.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* '''Mandatora: create output folder''', otherwise the software will tell you it can't write to output folder<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=532ENA Submission Pipeline2021-07-23T12:03:19Z<p>GabiDroege: /* Most common software errors */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc., replace all of them by "regular" characters<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing, add data to metadata.csv<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file, update number in nexus file<br />
|-<br />
| annonex2embl ERROR: csv-file does not contain a column labelled isolate.<br />
| You have probably forgotten, that ";" is NOT supported as delimiter; see above how to export a csv with "," as delimiter<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://kb.bgbm.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* '''Mandatora: create output folder''', otherwise the software will tell you it can't write to output folder<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Collection_Data_Form&diff=529Collection Data Form2021-07-18T09:55:16Z<p>GabiDroege: </p>
<hr />
<div>{| class = "wikitable"<br />
| [[File:Collectiondataform.JPG|border|400px]]<br />
|style="width:500px"| The BGBM provides a standardized MS Excel form for rapid data entry in situations where direct access to the central collection databases is not available. The form is typically used for data capture during field trips or in collections without network connections. The form fields are derived from the ABCD standard for international collection data exchange as well as its GGBN extension for DNA/tissue data. Please adhere strictly to the specifications given in the documentation file.<br />
|}<br />
<br />
Using the CDF is mandatory for all people working at BGBM. Entering data from a herbarium label or our living collection can be difficult. [[Using the Collection Data Form|This documentation]] shall guide you through the process. If you have questions regarding the CDF, please contact Gabi Droege (g.droege@bgbm.org) or Dominik Röpert (d.roepert@bgbm.org).<br />
<br />
=How to=<br />
* [[Using the Collection Data Form | Using the Collection Data Form]]<br />
* [https://kb.bgbm.org/share/s/kGQkeNO8R0etjJaOXTJ2Lg Examples Herbarium Specimen Labels (Pics)]<br />
* [https://kb.bgbm.org/share/s/7iByJ9gmSh-Wv633XPWKLQ Examples Herbarium Specimen Labels (CDF)]<br />
<br />
= Latest version =<br />
* '''Attention: for sending CDF through email you must rename it to e.g. ".abc" instead of ".xls". Sending files that include macros through email is no longer possible. You can not convert the file to ".xlsx" as you will loose macro functions unfortunately. Alternatively please use the cloud system provided by FU or BGBM.'''<br />
* [[:Media:Collectiondataform 2.1.zip | Collection data form 2.1]]<br />
* [[:Media:Collectiondataform documentation.pdf | Dataform field documentation (pdf)]]<br />
* [[:Media:Release notes.pdf | Release notes (pdf)]]<br />
* [[:Media:Excel Macro.pdf | Excel Macro Security (pdf)]]<br />
<br />
= Prior versions=<br />
* [[:Media:Collectiondataform 2.0.zip | Collection data form 2.0]]<br />
* [[:Media:Collectiondataform_V1.11.zip | Collection data form V1.11]]<br />
* [[:Media:Collectiondataform_V1.10.zip | Collection data form V1.10]]<br />
* [[:Media:Collectiondataform_V1.9.zip | Collection data form V1.9]]<br />
* [[:Media:Collectiondataform_V1.8.zip | Collection data form V1.8]]<br />
* [[:Media:Collectiondataform_V1.7.zip | Collection data form V1.7]]<br />
* [[:Media:Collectiondataform_V1.6.2.zip | Collection data form V1.6.2]]<br />
* [[:Media:Collectiondataform_V1.5.zip | Collection data form V1.5]]<br />
* [[:Media:Collectiondataform_V1.4.xls | Collection data form V1.4]]<br />
* [[:Media:Collectiondataform_V1.0.xls | Collection data form V1.0]]</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=528ENA Submission Pipeline2021-07-18T09:54:42Z<p>GabiDroege: </p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc., replace all of them by "regular" characters<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing, add data to metadata.csv<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file, update number in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://kb.bgbm.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* '''Mandatora: create output folder''', otherwise the software will tell you it can't write to output folder<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Annotations&diff=525ENA Annotations2021-04-20T09:00:07Z<p>GabiDroege: GabiDroege moved page Internal:ENA Annotations to ENA Annotations</p>
<hr />
<div>Annotations must follow the INSDC vocabulary (http://www.insdc.org/files/feature_table.html):<br />
<br />
=trnG=<br />
<code><br />
CharSet trnG_tRNA = 1-13;<br />
<br />
CharSet trnG_exon = 1-13;<br />
<br />
CharSet trnGIGS_misc_feature = 14-1103; change this to "trnG-trnS intergenic spacer" in embl-File before submission<br />
</code><br />
<br />
=ITS=<br />
<code><br />
CharSet 18S_rRNA = 1-3;<br />
<br />
CharSet ITS1_misc_RNA = 4-303;<br />
<br />
CharSet 58S_misc_feature = 304-459; change this to "5.8S" in embl-File before submission<br />
<br />
CharSet ITS2_misc_RNA = 460-723;<br />
<br />
CharSet 28S_rRNA = 724-747;<br />
</code><br />
<br />
=rbcL=<br />
<code><br />
CharSet rbcL_CDS = 1-957; must be a multiple of 3!<br />
<br />
CharSet rbcL_gene = 1-959;<br />
</code><br />
<br />
=rpl16=<br />
<code><br />
CharSet rpl16_intron = 1-1120;<br />
<br />
CharSet rpl16_gene = 1-1120;<br />
<br />
CharSet rpl16_exon = 1121-1123;<br />
</code><br />
<br />
=trnL-trnF=<br />
<code><br />
CharSet trnL_exon = 1-17 575-624;<br />
<br />
CharSet trnL_gene = 1-17 575-624;<br />
<br />
CharSet trnL_tRNA = 1-17 575-624;<br />
<br />
CharSet trnL_intron = 18-574;<br />
<br />
CharSet trnLtrnF_misc_feature = 625-989; change this to "trnL-trnF" in embl-File before submission<br />
<br />
CharSet trnF_gene = 990-1049;<br />
<br />
CharSet trnF_tRNA = 990-1049;<br />
</code><br />
<br />
=trnK-matK=<br />
<code><br />
CharSet trnK_exon = 1-3 2563-2597;<br />
<br />
CharSet trnK_gene = 1-3 2563-2597;<br />
<br />
CharSet trnK_tRNA = 1-3 2563-2597;<br />
<br />
CharSet trnK_intron = 4-704;<br />
<br />
CharSet matK_gene = 705-2240;<br />
<br />
CharSet matK_CDS = 705-2240; (optional)<br />
<br />
CharSet trnKpsbA_misc_feature = 2598-2839; change this to "trnK-psbA" in embl-File before submission<br />
<br />
CharSet psbA_gene = 2840-2897;<br />
<br />
CharSet psbA_CDS = 2840-2897; (optional, must be a multiple of 3!)<br />
</code></div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=523ENA Submission Pipeline2021-04-15T13:18:25Z<p>GabiDroege: GabiDroege moved page Internal:ENA Submission Pipeline to ENA Submission Pipeline</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc., replace all of them by "regular" characters<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing, add data to metadata.csv<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file, update number in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* '''Mandatora: create output folder''', otherwise the software will tell you it can't write to output folder<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=MediaWiki:Sidebar&diff=522MediaWiki:Sidebar2021-04-15T11:20:49Z<p>GabiDroege: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
** Contacts| Contacts<br />
* Collection Workflows<br />
** Collection Data Pipeline| Data Pipeline<br />
* Software<br />
** Collection Data Form | Collection Data Form<br />
** Open Refine Data Cleaning | Data Cleaning<br />
** http://search.biocase.org/bgbm | BGBM collections in GBIF<br />
** http://www.ggbn.org | DNA collection portal<br />
** Portal_Architecture | Architecture overview<br />
* Documentation<br />
** Collection Manuals | Collection Manuals<br />
** Barcodes für Sammlungsobjekte | Barcodes at B<br />
** Question and Answer | Q & A<br />
<br />
** Internal:Main Page|Internal<br />
** recentchanges-url|recentchanges<br />
<br />
* SEARCH<br />
* TOOLBOX</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=MediaWiki:Sidebar&diff=521MediaWiki:Sidebar2021-04-15T11:20:41Z<p>GabiDroege: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
** Contacts|C Contacts<br />
* Collection Workflows<br />
** Collection Data Pipeline| Data Pipeline<br />
* Software<br />
** Collection Data Form | Collection Data Form<br />
** Open Refine Data Cleaning | Data Cleaning<br />
** http://search.biocase.org/bgbm | BGBM collections in GBIF<br />
** http://www.ggbn.org | DNA collection portal<br />
** Portal_Architecture | Architecture overview<br />
* Documentation<br />
** Collection Manuals | Collection Manuals<br />
** Barcodes für Sammlungsobjekte | Barcodes at B<br />
** Question and Answer | Q & A<br />
<br />
** Internal:Main Page|Internal<br />
** recentchanges-url|recentchanges<br />
<br />
* SEARCH<br />
* TOOLBOX</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=MediaWiki:Sidebar&diff=520MediaWiki:Sidebar2021-04-15T11:19:37Z<p>GabiDroege: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
** Contacts<br />
* Collection Workflows<br />
** Collection Data Pipeline| Data Pipeline<br />
* Software<br />
** Collection Data Form | Collection Data Form<br />
** Open Refine Data Cleaning | Data Cleaning<br />
** http://search.biocase.org/bgbm | BGBM collections in GBIF<br />
** http://www.ggbn.org | DNA collection portal<br />
** Portal_Architecture | Architecture overview<br />
* Documentation<br />
** Collection Manuals | Collection Manuals<br />
** Barcodes für Sammlungsobjekte | Barcodes at B<br />
** Question and Answer | Q & A<br />
<br />
** Internal:Main Page|Internal<br />
** recentchanges-url|recentchanges<br />
<br />
* SEARCH<br />
* TOOLBOX</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=Contacts&diff=518Contacts2021-04-15T11:08:45Z<p>GabiDroege: GabiDroege moved page Internal:Contacts to Contacts: can go public</p>
<hr />
<div>{|<br />
! style="text-align:left;"| Role<br />
! Name<br />
! Email<br />
|-<br />
|Collection Workflows, DNA Bank, ENA, CDF<br />
|Gabi Droege<br />
|[mailto:g.droege@bgbm.org g.droege@bgbm.org]<br />
|-<br />
|Herbarium Phanerogams<br />
|Robert Vogt<br />
|[mailto:r.vogt@bgbm.org r.vogt@bgbm.org]<br />
|-<br />
|Herbarium Cryptogams<br />
|Robert Lücking<br />
|[mailto:r.luecking@bgbm.org r.luecking@bgbm.org]<br />
|-<br />
|Herbarium Algae<br />
|Nelida Abarca<br />
|[mailto:n.abarca@bgbm.org n.abarca@bgbm.org]<br />
|-<br />
|Herbarium Collection Management<br />
|Sarah Bollendorff<br />
|[mailto:s.bollendorff@bgbm.org s.bollendorff@bgbm.org]<br />
|-<br />
|<br />
|Peter Hein<br />
|[mailto:p.hein@bgbm.org p.hein@bgbm.org]<br />
|-<br />
|Herbarium Administration<br />
|Katharina Rabe<br />
|[mailto:k.rabe@bgbm.org k.rabe@bgbm.org]<br />
|-<br />
|Living collection Tropical/Subtropical<br />
|Nils Köster<br />
|[mailto:n.koester@bgbm.org n.koester@bgbm.org]<br />
|-<br />
|Living collection Mediterranean/Temperate<br />
|Gerald Parolly<br />
|[mailto:g.parolly@bgbm.org g.parolly@bgbm.org]<br />
|-<br />
|Seed Bank<br />
|Elke Zippel<br />
|[mailto:e.zippel@bgbm.org e.zippel@bgbm.org]<br />
|-<br />
|<br />
|Thomas Dürbye<br />
|[mailto:t.duerbye@bgbm.org t.duerbye@bgbm.org]<br />
|-<br />
|Lab<br />
|Kim Govers<br />
|[mailto:k.govers@bgbm.org k.govers@bgbm.org]<br />
|-<br />
|<br />
|Julia Dietrich<br />
|[mailto:j.dietrich@bgbm.org j.dietrich@bgbm.org]<br />
|-<br />
|JACQ, BoGart, Virtual Herbarium, CDF<br />
|Dominik Röpert<br />
|[mailto:d.roepert@bgbm.org d.roepert@bgbm.org]<br />
|-<br />
|<br />
|Frank Specht<br />
|[mailto:f.specht@bgbm.org f.specht@bgbm.org]<br />
|-<br />
|GBIF, BioCASe, ABCD<br />
|Jörg Holetschek<br />
|[mailto:j.holetschek@bgbm.org j.holetschek@bgbm.org]<br />
|-<br />
|Biodiversity Informatics<br />
|Team BDI<br />
|[mailto:biodiversitydata@bgbm.org biodiversitydata@bgbm.org]<br />
|-<br />
|}</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Annotations&diff=517ENA Annotations2021-03-30T08:28:07Z<p>GabiDroege: </p>
<hr />
<div>Annotations must follow the INSDC vocabulary (http://www.insdc.org/files/feature_table.html):<br />
<br />
=trnG=<br />
<code><br />
CharSet trnG_tRNA = 1-13;<br />
<br />
CharSet trnG_exon = 1-13;<br />
<br />
CharSet trnGIGS_misc_feature = 14-1103; change this to "trnG-trnS intergenic spacer" in embl-File before submission<br />
</code><br />
<br />
=ITS=<br />
<code><br />
CharSet 18S_rRNA = 1-3;<br />
<br />
CharSet ITS1_misc_RNA = 4-303;<br />
<br />
CharSet 58S_misc_feature = 304-459; change this to "5.8S" in embl-File before submission<br />
<br />
CharSet ITS2_misc_RNA = 460-723;<br />
<br />
CharSet 28S_rRNA = 724-747;<br />
</code><br />
<br />
=rbcL=<br />
<code><br />
CharSet rbcL_CDS = 1-957; must be a multiple of 3!<br />
<br />
CharSet rbcL_gene = 1-959;<br />
</code><br />
<br />
=rpl16=<br />
<code><br />
CharSet rpl16_intron = 1-1120;<br />
<br />
CharSet rpl16_gene = 1-1120;<br />
<br />
CharSet rpl16_exon = 1121-1123;<br />
</code><br />
<br />
=trnL-trnF=<br />
<code><br />
CharSet trnL_exon = 1-17 575-624;<br />
<br />
CharSet trnL_gene = 1-17 575-624;<br />
<br />
CharSet trnL_tRNA = 1-17 575-624;<br />
<br />
CharSet trnL_intron = 18-574;<br />
<br />
CharSet trnLtrnF_misc_feature = 625-989; change this to "trnL-trnF" in embl-File before submission<br />
<br />
CharSet trnF_gene = 990-1049;<br />
<br />
CharSet trnF_tRNA = 990-1049;<br />
</code><br />
<br />
=trnK-matK=<br />
<code><br />
CharSet trnK_exon = 1-3 2563-2597;<br />
<br />
CharSet trnK_gene = 1-3 2563-2597;<br />
<br />
CharSet trnK_tRNA = 1-3 2563-2597;<br />
<br />
CharSet trnK_intron = 4-704;<br />
<br />
CharSet matK_gene = 705-2240;<br />
<br />
CharSet matK_CDS = 705-2240; (optional)<br />
<br />
CharSet trnKpsbA_misc_feature = 2598-2839; change this to "trnK-psbA" in embl-File before submission<br />
<br />
CharSet psbA_gene = 2840-2897;<br />
<br />
CharSet psbA_CDS = 2840-2897; (optional, must be a multiple of 3!)<br />
</code></div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=516ENA Submission Pipeline2021-03-18T11:20:59Z<p>GabiDroege: /* Validating EMBL flatfile */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc., replace all of them by "regular" characters<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing, add data to metadata.csv<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file, update number in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* '''Mandatora: create output folder''', otherwise the software will tell you it can't write to output folder<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Annotations&diff=515ENA Annotations2021-03-18T10:54:48Z<p>GabiDroege: /* trnK-matK */</p>
<hr />
<div>Annotations must follow the INSDC vocabulary (http://www.insdc.org/files/feature_table.html):<br />
<br />
=trnG=<br />
<code><br />
CharSet trnG_tRNA = 1-13;<br />
<br />
CharSet trnG_exon = 1-13;<br />
<br />
CharSet trnGIGS_misc_feature = 14-1103; change this to "trnG-trnS intergenic spacer" in embl-File before submission<br />
</code><br />
<br />
=ITS=<br />
<code><br />
CharSet 18S_rRNA = 1-3;<br />
<br />
CharSet ITS1_misc_RNA = 4-303;<br />
<br />
CharSet 58S_misc_feature = 304-459; change this to "5.8S" in embl-File before submission<br />
<br />
CharSet ITS2_misc_RNA = 460-723;<br />
<br />
CharSet 28S_rRNA = 724-747;<br />
</code><br />
<br />
=rpl16=<br />
<code><br />
CharSet rpl16_intron = 1-1120;<br />
<br />
CharSet rpl16_gene = 1-1120;<br />
<br />
CharSet rpl16_exon = 1121-1123;<br />
</code><br />
<br />
=trnL-trnF=<br />
<code><br />
CharSet trnL_exon = 1-17 575-624;<br />
<br />
CharSet trnL_gene = 1-17 575-624;<br />
<br />
CharSet trnL_tRNA = 1-17 575-624;<br />
<br />
CharSet trnL_intron = 18-574;<br />
<br />
CharSet trnLtrnF_misc_feature = 625-989; change this to "trnL-trnF" in embl-File before submission<br />
<br />
CharSet trnF_gene = 990-1049;<br />
<br />
CharSet trnF_tRNA = 990-1049;<br />
</code><br />
<br />
=trnK-matK=<br />
<code><br />
CharSet trnK_exon = 1-3 2563-2597;<br />
<br />
CharSet trnK_gene = 1-3 2563-2597;<br />
<br />
CharSet trnK_tRNA = 1-3 2563-2597;<br />
<br />
CharSet trnK_intron = 4-704;<br />
<br />
CharSet matK_gene = 705-2240;<br />
<br />
CharSet matK_CDS = 705-2240; (optional)<br />
<br />
CharSet trnKpsbA_misc_feature = 2598-2839; change this to "trnK-psbA" in embl-File before submission<br />
<br />
CharSet psbA_gene = 2840-2897;<br />
<br />
CharSet psbA_CDS = 2840-2897; (optional, must be a multiple of 3!)<br />
</code></div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=514ENA Submission Pipeline2021-03-18T10:29:25Z<p>GabiDroege: /* Most common software errors */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc., replace all of them by "regular" characters<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing, add data to metadata.csv<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file, update number in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=513ENA Submission Pipeline2021-03-18T10:25:06Z<p>GabiDroege: /* Check all taxon names against NCBI Taxonomy */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=512ENA Submission Pipeline2021-03-18T10:24:46Z<p>GabiDroege: </p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=511ENA Submission Pipeline2021-03-18T10:24:33Z<p>GabiDroege: /* Final submission */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
'''You will NOT receive a confirmation email, but they will be listed in the Webin user interface.'''</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=510ENA Submission Pipeline2021-03-18T10:23:49Z<p>GabiDroege: /* Add Annotations to Sequences */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see [[Internal:ENA Annotations | ENA Annotations]]'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=509ENA Submission Pipeline2021-03-18T10:22:38Z<p>GabiDroege: /* Check all taxon names against NCBI Taxonomy */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. DB 12843" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to six weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see Annotations'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=508ENA Submission Pipeline2021-03-18T10:21:37Z<p>GabiDroege: /* Überprüfung der Korrespondenz zwischen DNA-Alignment und Metadaten */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. G.Parolly et al. 15015" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to two weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see Annotations'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=507ENA Submission Pipeline2021-03-18T10:21:11Z<p>GabiDroege: /* Mandatory conventions for non-canonical names */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. G.Parolly et al. 15015" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to two weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see Annotations'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
===Mandatory conventions for non-canonical names===<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Überprüfung der Korrespondenz zwischen DNA-Alignment und Metadaten==<br />
Benötigte Software: beliebiger Software für Tabellenkalkulation (z.B. LibreOffice), beliebiger Texteditor (z.B. Geany)<br />
<br />
# Die Sequenznamen des DNA-Alignments müssen den Einträgen einer der Spalten der Metadaten-Datei exakt entsprechen.<br />
# Die Metadaten-Datei darf hierbei mehr Einträge haben als das DNA-Alignment, aber das DNA-ALignment darf nicht mehr Einträge haben als die Metadaten-Datei.<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=506ENA Submission Pipeline2021-03-18T10:20:57Z<p>GabiDroege: /* Registration of all new names at NCBI/ENA */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. G.Parolly et al. 15015" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to two weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see Annotations'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
==Mandatory conventions for non-canonical names==<br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina DB 12342<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Überprüfung der Korrespondenz zwischen DNA-Alignment und Metadaten==<br />
Benötigte Software: beliebiger Software für Tabellenkalkulation (z.B. LibreOffice), beliebiger Texteditor (z.B. Geany)<br />
<br />
# Die Sequenznamen des DNA-Alignments müssen den Einträgen einer der Spalten der Metadaten-Datei exakt entsprechen.<br />
# Die Metadaten-Datei darf hierbei mehr Einträge haben als das DNA-Alignment, aber das DNA-ALignment darf nicht mehr Einträge haben als die Metadaten-Datei.<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=505ENA Submission Pipeline2021-03-18T10:16:31Z<p>GabiDroege: /* Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. G.Parolly et al. 15015" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to two weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see Annotations'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
Email von Carol Hotton (NCBI): Here’s the GenBank format for non-canonical names: <br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina GD-2019<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* We use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters (as you have done, so that’s not a problem).<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Überprüfung der Korrespondenz zwischen DNA-Alignment und Metadaten==<br />
Benötigte Software: beliebiger Software für Tabellenkalkulation (z.B. LibreOffice), beliebiger Texteditor (z.B. Geany)<br />
<br />
# Die Sequenznamen des DNA-Alignments müssen den Einträgen einer der Spalten der Metadaten-Datei exakt entsprechen.<br />
# Die Metadaten-Datei darf hierbei mehr Einträge haben als das DNA-Alignment, aber das DNA-ALignment darf nicht mehr Einträge haben als die Metadaten-Datei.<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<br />
===Most common software errors===<br />
{|class="wikitable"<br />
!Error<br />
!Solution<br />
|-<br />
|annonex2embl ERROR: expected string or bytes-like object<br />
|csv parsing error in metadata file, typically " in strings, Umlaute, ó, é, á, í etc.<br />
|-<br />
|annonex2embl ERROR with qualifiers of `SAL169`: list index out of range<br />
|sample metadata missing<br />
|-<br />
| annonex2embl ERROR: Not enough taxa in matrix.<br />
| DIMENSIONS NTAX= doesn't match number of sequences in nexus file<br />
|}<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Submission_Pipeline&diff=504ENA Submission Pipeline2021-03-18T09:39:47Z<p>GabiDroege: /* Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata */</p>
<hr />
<div>'''Attention: BGBM submissions to ENA including registration of new names should only be done by authorised staff! Please contact the DNA Bank if you want to submit sequences SIX weeks ahead of paper submission!'''<br />
The following guide shall help you to prepare all required data and to understand the complexity of sequence submissions. The submission itself will be done by authorised staff only!<br />
<br />
'''!To Do: Text includes a Mix of German and English !'''<br />
<br />
<br />
=Researchers=<br />
==INPUT required for any submission==<br />
<br />
# DNA alignment in NEXUS format with annotations following the [http://www.insdc.org/files/feature_table.html INSDC vocabulary]<br />
## most common errors: coordinates syntax, usage of Umlaute or charcters such as °<br />
# Sample metadata in CSV format following the BGBM standard (TODO: description of this standard)<br />
# List of standardized scientific names<br />
# Project metadata (short name, title of study, abstract, release date, autors/contributors)<br />
<br />
==Check all taxon names against NCBI Taxonomy==<br />
Option A: Use the NCBI tool: https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi<br />
<br />
Option B: Use OpenRefine: TODO: documentation<br />
<br />
* Names muss follow IPNI standard (except for unidenified organisms and new described taxa), most common mistakes are blanks in authorship (e.g. H. Karst); IPNI always without blanks (H.Karst); BGBM follows this standard in all its collection databases<br />
* Names not found at ENA/NCBI Taxonomy must be registered at ENA (by BGBM DNA Bank!). Note, that you will only find names associated to <u>published</u> sequences!<br />
* ENA has rules in place for registration of new names one must follow. BGBM documents those name submissions centrally, to enable updating sequence records after publication of new names.<br />
** Unidentified or unpublished names must be unique, e.g. "Minuartia sp. G.Parolly et al. 15015" instead of "Minuartia spec. 1"<br />
<br />
'''Attention: the registration of new names might require up to two weeks, make sure ALL names are checked carefully and submitted right in time'''<br />
<br />
==Add Annotations to Sequences==<br />
* Required Software: Phyde, Geneious, every common text editor (e.g. Geany, Notepad++, Textpad)<br />
* Video Tutorial: https://www.youtube.com/watch?v=CY1e2RkULas<br />
* '''Attention: ENA requires use of certain vocabulary for annotations see Annotations'''<br />
<br />
Export your file into Nexus format!<br />
In the nexus file annotations show up at the end of the file in following format. Annotations must not contain spaces!<br />
<pre><br />
BEGIN SETS;<br />
charset 18S_rRNA = 1-378;<br />
charset 18S_gene = 1-378;<br />
<br />
END;<br />
</pre><br />
<br />
=DNA Bank=<br />
==Registration of all new names at NCBI/ENA==<br />
# Login at ENA's Webin<br />
# Click on "Taxonomy Check/Request" --> Next<br />
# Enter organism name / upload list of names<br />
## Save list of names as txt file (one per row) or enter individually<br />
## todo: Beispiel-Anfragen hinzufügen<br />
<br />
Email von Carol Hotton (NCBI): Here’s the GenBank format for non-canonical names: <br />
* Sensu stricto names are treated as canonical, e.g.: Arabis hirsuta s. str. = Arabis hirsuta<br />
* Sensu lato names are treated as cf., e.g.: Festuca ovina s. l. = Festuca cf. ovina <br />
* Cf. (and aff.) names we treat as non-unique, to which a unique string is attached, e.g.: Festuca cf. ovina GD-2019<br />
* Hybrid formulas repeat the genus, i.e.: Carex cespitosa x Carex nigra<br />
* We use an ‘x’ rather than a multiplication sign in hybrid names because not all applications can handle non-ASCII characters (as you have done, so that’s not a problem).<br />
<br />
==Create project at ENA to get project number==<br />
# Login at ENA's Webin<br />
# Click on "Register study (project)" --> Next<br />
# Fill out relevant information (Short name, Title of study, Abstract, Release date)<br />
<br />
==Überprüfung der Korrespondenz zwischen DNA-Alignment und Metadaten==<br />
Benötigte Software: beliebiger Software für Tabellenkalkulation (z.B. LibreOffice), beliebiger Texteditor (z.B. Geany)<br />
<br />
# Die Sequenznamen des DNA-Alignments müssen den Einträgen einer der Spalten der Metadaten-Datei exakt entsprechen.<br />
# Die Metadaten-Datei darf hierbei mehr Einträge haben als das DNA-Alignment, aber das DNA-ALignment darf nicht mehr Einträge haben als die Metadaten-Datei.<br />
<br />
==Converting DNA alignment (nexus file) to EMBL flate file, integration of metadata==<br />
Required Software: https://github.com/michaelgruenstaeudl/annonex2embl<br />
<br />
===Metadata File===<br />
* MUST NOT contain any empty columns or rows<br />
* '''Comma''' as only accepted '''delimiter''' (Windows -> change region to -> CSV export done with comma instead of semicolon)<br />
* '''DATE''' must be converted into '''YYYY-MM-DD''' ''Attention: when you do the export in Windows dates will be converted into US format!''<br />
* '''Coordinates''' must follow this syntax: "47.94 N 28.12 W" (no commas, keep spaces!)<br />
* MUST NOT contain any Umlaut or "strange" character such as °<br />
* Column headers must match with [http://www.insdc.org/files/feature_table.html INSDC feature table]<br />
* for '''matK''' use '''organelle = plastid'''; for others column "organelle" is not needed<br />
<br />
<pre><br />
##LINUX##<br />
INPUT=examples/DNA_Alignment.nex<br />
METAD=examples/Metadata.csv<br />
DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
EMAIL=your_email_here@bgbm.org<br />
AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n $INPUT -c $METAD -o ${INPUT%.nex*}.embl -d $DESCR -e $EMAIL -a $AUTHR<br />
<br />
##WINDOWS##<br />
SET INPUT=examples/DNA_Alignment.nex<br />
SET METAD=examples/Metadata.csv<br />
SET DESCR="description_of_alignment" // note, that spaces are unfortunately not yet supported!<br />
SET EMAIL=your_email_here@bgbm.org<br />
SET AUTHR="Your_name_here" // note, that spaces are unfortunately not yet supported!<br />
<br />
annonex2embl -n %INPUT% -c %METAD% -o output.embl -d %DESCR% -e %EMAIL% -a %AUTHR%<br />
</pre><br />
<br />
Examples for DESCR: '''note, that spaces are unfortunately not yet supported! You'll need to replace underscores in the next step'''<br />
* ITS: "18S rRNA gene (partial), ITS1, 5.8S rRNA gene, ITS2 and 28S rRNA gene (partial)"<br />
* trnLF: "tRNA-Leu (trnL) gene, partial sequence; trnL-trnF intergenic spacer, complete sequence; and tRNA-Phe (trnF) gene"<br />
* trnKmatK: "tRNA-Lys (trnK) gene and intron, partial sequence; maturase K (matK) gene, complete cds; psbA gene, partial sequence"<br />
* rpl16: "rpl16 intron, partial sequence"<br />
* 18S: "partial 18S rRNA gene"<br />
* rbcL: "partial rbcL gene"<br />
* petD: "petB gene, partial sequence; petD gene and intron, partial sequence"<br />
<br />
===Check EMBL File before you continue===<br />
* first row '''mol_type''' formatted correctly?<br />
** WRONG: ID D257_023a; SV 1; linear; '''DNA'''; ; PLN; 378 BP.<br />
** CORRECTED: ID D257_023a; SV 1; linear; '''genomic DNA'''; ; PLN; 378 BP.<br />
* all dates in ISO format?<br />
* Identifier doen't have to be changed<br />
* "isolate DB" to be replaced by "isolate DB "<br />
* "isolate=DB" to be replaced by "isolate=DB "<br />
* 58S etc. to replaced by "5 8S" etc.<br />
* replace underscores in descriptions by spaces<br />
<br />
==Validating EMBL flatfile==<br />
* Required Software: https://github.com/enasequence/webin-cli/releases -> download latest jar file<br />
* '''Mandatory: Manifest file''', file ending ".manifest", example file can be found at https://library.ggbn.org/share/s/zWNjSrM1RyW1DNyNb5c15Q<br />
* The EMBL Validator will provide you with a report<br />
* Most common errors: wrong syntax for coordinates, unregistered names, typos in isolate/first part of nexus sequence rows (e.g. annonex2embl ERROR with qualifiers of `AmCoc01`: list index out of range means AmCoc01 found in Nexus file could not be found in metadata file)<br />
<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -validate<br />
</pre><br />
<br />
<pre><br />
##Example Manifest file##<br />
STUDY PRJEB42197 // see above, you'll needd to register a study as a very first step<br />
NAME partial 18S rRNA gene // same as used in DESCR parameter before, but spaces are fine this time<br />
FLATFILE Cocconeis_crawfordii_18SV4.embl.gz // the zipped embl file name<br />
</pre><br />
<br />
==Testing submission==<br />
If validation was successul you can test the submission ('''NEVER SKIP THIS STEP''')<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ -test - submit<br />
</pre><br />
<br />
==Final submission==<br />
If test submission was successul you can do the final the submission<br />
<br />
<pre><br />
##LINUX and WINDOWS##<br />
java -jar ~/Downloads/webin-cli-3.5.0.jar -context sequence -userName TOPSECRET -password EVENMORESECRET -manifest ~/Desktop/ENA/YOUR_FOLDER/MANIFEST_FILE.manifest -outputDir ~/Desktop/ENA/YOUR_FOLDER/output -inputDir ~/Desktop/ENA/YOUR_FOLDER/ - submit<br />
</pre><br />
<br />
Command line will provide you with a submission number, e.g.:<br />
<br />
INFO : The submission has been completed successfully. The following analysis accession was assigned to the submission: ERZ1740481<br />
<br />
You will NOT receive a confirmation email.</div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Annotations&diff=503ENA Annotations2021-03-16T10:33:55Z<p>GabiDroege: </p>
<hr />
<div>Annotations must follow the INSDC vocabulary (http://www.insdc.org/files/feature_table.html):<br />
<br />
=trnG=<br />
<code><br />
CharSet trnG_tRNA = 1-13;<br />
<br />
CharSet trnG_exon = 1-13;<br />
<br />
CharSet trnGIGS_misc_feature = 14-1103; change this to "trnG-trnS intergenic spacer" in embl-File before submission<br />
</code><br />
<br />
=ITS=<br />
<code><br />
CharSet 18S_rRNA = 1-3;<br />
<br />
CharSet ITS1_misc_RNA = 4-303;<br />
<br />
CharSet 58S_misc_feature = 304-459; change this to "5.8S" in embl-File before submission<br />
<br />
CharSet ITS2_misc_RNA = 460-723;<br />
<br />
CharSet 28S_rRNA = 724-747;<br />
</code><br />
<br />
=rpl16=<br />
<code><br />
CharSet rpl16_intron = 1-1120;<br />
<br />
CharSet rpl16_gene = 1-1120;<br />
<br />
CharSet rpl16_exon = 1121-1123;<br />
</code><br />
<br />
=trnL-trnF=<br />
<code><br />
CharSet trnL_exon = 1-17 575-624;<br />
<br />
CharSet trnL_gene = 1-17 575-624;<br />
<br />
CharSet trnL_tRNA = 1-17 575-624;<br />
<br />
CharSet trnL_intron = 18-574;<br />
<br />
CharSet trnLtrnF_misc_feature = 625-989; change this to "trnL-trnF" in embl-File before submission<br />
<br />
CharSet trnF_gene = 990-1049;<br />
<br />
CharSet trnF_tRNA = 990-1049;<br />
</code><br />
<br />
=trnK-matK=<br />
<code><br />
CharSet trnK_exon = 1-3 2563-2597;<br />
<br />
CharSet trnK_gene = 1-3 2563-2597;<br />
<br />
CharSet trnK_tRNA = 1-3 2563-2597;<br />
<br />
CharSet trnK_intron = 4-2562;<br />
<br />
CharSet matK_gene = 705-2240;<br />
<br />
CharSet matK_CDS = 705-2240; (optional)<br />
<br />
CharSet trnKpsbA_misc_feature = 2598-2839; change this to "trnK-psbA" in embl-File before submission<br />
<br />
CharSet psbA_gene = 2840-2897;<br />
<br />
CharSet psbA_CDS = 2840-2897; (optional, must be a multiple of 3!)<br />
</code></div>GabiDroegehttps://wiki.bgbm.org/collectionworkflows/index.php?title=ENA_Annotations&diff=502ENA Annotations2021-03-16T10:31:43Z<p>GabiDroege: Created page with "Annotations must follow the INSDC vocabulary (http://www.insdc.org/files/feature_table.html): =trnG= CharSet trnG_tRNA = 1-13; CharSet trnG_exon = 1-13; CharSet trnGIGS_misc_..."</p>
<hr />
<div>Annotations must follow the INSDC vocabulary (http://www.insdc.org/files/feature_table.html):<br />
<br />
=trnG=<br />
CharSet trnG_tRNA = 1-13;<br />
CharSet trnG_exon = 1-13;<br />
CharSet trnGIGS_misc_feature = 14-1103; change this to "trnG-trnS intergenic spacer" in embl-File before submission<br />
<br />
=ITS=<br />
CharSet 18S_rRNA = 1-3;<br />
CharSet ITS1_misc_RNA = 4-303;<br />
CharSet 58S_misc_feature = 304-459; change this to "5.8S" in embl-File before submission<br />
CharSet ITS2_misc_RNA = 460-723;<br />
CharSet 28S_rRNA = 724-747;<br />
<br />
=rpl16=<br />
CharSet rpl16_intron = 1-1120;<br />
CharSet rpl16_gene = 1-1120;<br />
CharSet rpl16_exon = 1121-1123;<br />
<br />
=trnL-trnF=<br />
CharSet trnL_exon = 1-17 575-624;<br />
CharSet trnL_gene = 1-17 575-624;<br />
CharSet trnL_tRNA = 1-17 575-624;<br />
CharSet trnL_intron = 18-574;<br />
CharSet trnLtrnF_misc_feature = 625-989; change this to "trnL-trnF" in embl-File before submission<br />
CharSet trnF_gene = 990-1049;<br />
CharSet trnF_tRNA = 990-1049;<br />
<br />
=trnK-matK=<br />
CharSet trnK_exon = 1-3 2563-2597;<br />
CharSet trnK_gene = 1-3 2563-2597;<br />
CharSet trnK_tRNA = 1-3 2563-2597;<br />
CharSet trnK_intron = 4-2562;<br />
CharSet matK_gene = 705-2240;<br />
CharSet matK_CDS = 705-2240; (optional)<br />
CharSet trnKpsbA_misc_feature = 2598-2839; change this to "trnK-psbA" in embl-File before submission<br />
CharSet psbA_gene = 2840-2897;<br />
CharSet psbA_CDS = 2840-2897; (optional, must be a multiple of 3!)</div>GabiDroege