Platform for Data Management - Revision history

KNOW at 20:58, 20 August 2023

2023-08-20T20:58:03Z

← Older revision		Revision as of 20:58, 20 August 2023
Line 4:		Line 4:
	The design of the platform allows the handling of two different functionalities:		The design of the platform allows the handling of two different functionalities:

	# Data Discovery & Management - Handles the organization and storage of data		# '''Data Discovery & Management''' - Handles the organization and storage of data
	# Data Analytics – Focuses on extracting insights and trends from data		# '''Data Analytics''' – Focuses on extracting insights and trends from data

	Both functionalities allow the '''KDP''' to be a powerful and user-friendly data management and analytics platform that utilizes many different components. Its robust features provide researchers with an efficient and secure way to manage and analyse their data while also facilitating collaboration and discovery.		Both functionalities allow the '''KDP''' to be a powerful and user-friendly data management and analytics platform that utilizes many different components. Its robust features provide researchers with an efficient and secure way to manage and analyse their data while also facilitating collaboration and discovery.
Line 14:		Line 14:
	To provide a user-friendly interface for data management tasks, the platform uses '''Metalnx''' as the graphical user interface. Users can upload, download, and share data, as well as set permissions and manage metadata. '''Metalnx''' is highly customizable with the ability to add custom metadata fields and workflows to fit the specific needs of the platform.		To provide a user-friendly interface for data management tasks, the platform uses '''Metalnx''' as the graphical user interface. Users can upload, download, and share data, as well as set permissions and manage metadata. '''Metalnx''' is highly customizable with the ability to add custom metadata fields and workflows to fit the specific needs of the platform.

	[~~https~~:~~//www.elastic.co/de/~~ Elasticsearch] is used for data discovery, enabling users to easily search and browse data collections based on metadata and other attributes. This ensures that users can quickly find and access the data they need to support their research. The authentication system is powered by [~~https~~:~~//www.keycloak.org/~~ Keycloak] to ensure secure access to the '''KDP'''. It is common for each data discovery and management component to utilize a [~~https~~:~~//www.postgresql.org/~~ PostgreSQL] database for storing information such as user accounts, metadata on data collections and configs.		[[wikipedia:Elasticsearch\|Elasticsearch]] is used for data discovery, enabling users to easily search and browse data collections based on metadata and other attributes. This ensures that users can quickly find and access the data they need to support their research. The authentication system is powered by [[wikipedia:Keycloak\|Keycloak]] to ensure secure access to the '''KDP'''. It is common for each data discovery and management component to utilize a [[wikipedia:PostgreSQL\|PostgreSQL]] database for storing information such as user accounts, metadata on data collections and configs.

	==== 2. Data Analytics ====		==== 2. Data Analytics ====
	The analytics part of the platform exposes [https://jupyterhub.readthedocs.io/en/stable/ JupyterHub] or [https://zeppelin.apache.org/ Apache Zeppelin] (depending on the required programming languages) to support collaboration and data analysis. '''JupyterHub''' is a web-based platform with a shared environment for running Jupyter notebooks. The platform allows partners and researchers to analyze and work with shared data, promoting collaboration and efficiency. Results and findings can then be easily shared among team members and beyond. On the other hand, '''Apache Zeppelin''' has a stronger focus on data analytics with built-in support for data sources such as [[wikipedia:Apache_Spark\|Apache Spark]], [[wikipedia:SQL\|SQL]] databases and [[wikipedia:NoSQL\|NoSQL]] databases.		The analytics part of the platform exposes [https://jupyterhub.readthedocs.io/en/stable/ JupyterHub] or [https://zeppelin.apache.org/ Apache Zeppelin] (depending on the required programming languages) to support collaboration and data analysis. '''JupyterHub''' is a web-based platform with a shared environment for running Jupyter notebooks. The platform allows partners and researchers to analyze and work with shared data, promoting collaboration and efficiency. Results and findings can then be easily shared among team members and beyond. On the other hand, '''Apache Zeppelin''' has a stronger focus on data analytics with built-in support for data sources such as [[wikipedia:Apache_Spark\|Apache Spark]], [[wikipedia:SQL\|SQL]] databases and [[wikipedia:NoSQL\|NoSQL]] databases.


	'''EDIAQI''' partners that prefer to keep the data on their server have the option to setup an own '''iRODS''' zone and utilize secure federation with the '''KDP'''. The '''iRODS''' provider of the '''KDP''' then accepts the partner server as an additional '''iRODS''' consumer. This option enables secure sharing and accessing of data across multiple '''iRODS''' zones as if they were part of a single virtual file system. Authentication and authorization protocols ensure that data is only accessible to authorized users within their respective zones, maintaining control over access and security.		'''EDIAQI''' partners that prefer to keep the data on their server have the option to setup an own '''iRODS''' zone and utilize secure federation with the '''KDP'''. The '''iRODS''' provider of the '''KDP''' then accepts the partner server as an additional '''iRODS''' consumer. This option enables secure sharing and accessing of data across multiple '''iRODS''' zones as if they were part of a single virtual file system. Authentication and authorization protocols ensure that data is only accessible to authorized users within their respective zones, maintaining control over access and security.

	[[Category:Data Management]]		[[Category:Data Management]]

KNOW at 15:10, 18 August 2023

2023-08-18T15:10:21Z

← Older revision		Revision as of 15:10, 18 August 2023
Line 1:		Line 1:
	...		[https://www.know-center.at/en/ Know-Center] Data Management Platform ('''KDP''') will form the central data access point for the [https://ediaqi.eu/ EDIAQI] project and will be continually enhanced and expanded as the project advances. The '''KDP''' is designed to provide a secure, scalable, and robust system for managing and sharing data. It is built on the [https://irods.org/ iRODS] data management system for distributed and replicated storage. [https://metalnx.github.io/ Metalnx] is used as the graphical user interface for an intuitive and user-friendly sharing and searching of data. The project team at Know-Center developed the '''KDP''' with the [[wikipedia:FAIR_data\|FAIR principles]] in mind and a strong focus on reproducible research.

	~~test~~		=== Platform Architecture ===
			The design of the platform allows the handling of two different functionalities:

	~~.....~~		# Data Discovery & Management - Handles the organization and storage of data
			# Data Analytics – Focuses on extracting insights and trends from data

			Both functionalities allow the '''KDP''' to be a powerful and user-friendly data management and analytics platform that utilizes many different components. Its robust features provide researchers with an efficient and secure way to manage and analyse their data while also facilitating collaboration and discovery.

			==== 1. Data Discovery & Management ====
			The heart of the '''KDP''' is the '''iRODS''' data management system that provides a unified interface for accessing, managing, and sharing data. '''iRODS''' consists of two different server components - data catalog which contains information about the file names, locations and permissions, and catalog consumer which retrieves metadata about data objects and collections from the catalog provider. Despite consisting of multiple servers, the platform appears as a single virtual file system to users.

			To provide a user-friendly interface for data management tasks, the platform uses '''Metalnx''' as the graphical user interface. Users can upload, download, and share data, as well as set permissions and manage metadata. '''Metalnx''' is highly customizable with the ability to add custom metadata fields and workflows to fit the specific needs of the platform.

			[https://www.elastic.co/de/ Elasticsearch] is used for data discovery, enabling users to easily search and browse data collections based on metadata and other attributes. This ensures that users can quickly find and access the data they need to support their research. The authentication system is powered by [https://www.keycloak.org/ Keycloak] to ensure secure access to the '''KDP'''. It is common for each data discovery and management component to utilize a [https://www.postgresql.org/ PostgreSQL] database for storing information such as user accounts, metadata on data collections and configs.

			==== 2. Data Analytics ====
			The analytics part of the platform exposes [https://jupyterhub.readthedocs.io/en/stable/ JupyterHub] or [https://zeppelin.apache.org/ Apache Zeppelin] (depending on the required programming languages) to support collaboration and data analysis. '''JupyterHub''' is a web-based platform with a shared environment for running Jupyter notebooks. The platform allows partners and researchers to analyze and work with shared data, promoting collaboration and efficiency. Results and findings can then be easily shared among team members and beyond. On the other hand, '''Apache Zeppelin''' has a stronger focus on data analytics with built-in support for data sources such as [[wikipedia:Apache_Spark\|Apache Spark]], [[wikipedia:SQL\|SQL]] databases and [[wikipedia:NoSQL\|NoSQL]] databases.


			'''EDIAQI''' partners that prefer to keep the data on their server have the option to setup an own '''iRODS''' zone and utilize secure federation with the '''KDP'''. The '''iRODS''' provider of the '''KDP''' then accepts the partner server as an additional '''iRODS''' consumer. This option enables secure sharing and accessing of data across multiple '''iRODS''' zones as if they were part of a single virtual file system. Authentication and authorization protocols ensure that data is only accessible to authorized users within their respective zones, maintaining control over access and security.

	[[Category:Data Management]]		[[Category:Data Management]]

KNOW at 14:05, 17 August 2023

2023-08-17T14:05:55Z

← Older revision		Revision as of 14:05, 17 August 2023
Line 1:		Line 1:
	...		...

			test

			.....


	[[Category:Data Management]]		[[Category:Data Management]]

KNOW: Created page with "... Category:Data Management"

2023-08-17T14:03:10Z

Created page with "... Category:Data Management"

New page

...

[[Category:Data Management]]