3 Star 3 Fork 1

WongBin / seatunnel-web

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

Apache SeaTunnel

seatunnel logo

Backend Workflow Slack Twitter Follow


EN doc

SeaTunnel的前身是Waterdrop,并于2021年10月12日更名为SeaTunnel。


那么,SeaTunnel是什么? SeaTunnel是下一代超高性能、分布式、海量数据集成工具。它每天可以稳定高效地同步数百亿个数据,并已在许多公司的生产中使用。SeaTunnel-web为SeaTunnel提供可视化界面,方便用户使用Apache SeaTunnel。

如果你想知道更多关于SeaTunnel的设计。 👉🏻设计

如何开始

注意:一些细节请参考 docs/QuickStart.md

1 准备Apache SeaTunnel环境

因为SeaTunnel Web使用SeaTunnel Java客户端来提交作业,所以运行SeaTunnel Web需要首先准备SeaTunnel Zeta Engine服务。

根据SeaTunnel Zeta Engine的使用要求,提交作业的SeaTunnel Client节点必须与运行作业的SeaTunnel Server节点具有相同的操作系统和安装目录结构。比如如果你想在本地IDEA中运行SeaTunnel Web,你必须在与IDEA相同的机器上安装和运行SeaTunnel Zeta引擎服务器。

别担心,接下来的步骤将告诉您如何在不同情况下正确安装SeaTunnel Zeta Engine Server。

2 在IDEA中运行SeaTunnel Web

如果您想部署和运行SeaTunnel Web,请转到 [3 Run SeaTunnel Web In Server](#3 Run SeaTunnel Web In Server)

2.1 安装SeaTunnel Zeta引擎服务器

有两种方法可以获得SeaTunnel安装包。从源代码构建或从SeaTunnel网站下载。

此处使用的SeaTunnel版本仅用于编写本文档以向您展示所使用的过程,并不一定代表正确的版本。SeaTunnel Web和SeaTunnel Engine有严格的版本依赖关系,您可以通过确认具体的版本对应关系。现在只支持在本地构建SeaTunnel Web和SeaTunnel Zeta Engine,因为有必要确保SeaTunnel Web中的SeaTunnel-api与SeaTunnel Zeta Engine中的版本相同

2.1.1 从源代码构建和部署
  • https://seatunnel.apache.org/downloadhttps://github.com/apache/seatunnel.git 获取源包
  • 构建安装包使用maven命令。./mvnw -U -T 1C clean install -DskipTests -D"maven.test.skip"=true -D"maven.javadoc.skip"=true -D"checkstyle.skip"=true -D"license.skipAddThirdParty"
  • 在构建之后,需要设置一个环境变量ST_WEB_BASEDIR_PATH来表示数据源shade包的位置。将使用自定义类加载器来基于此加载数据源shade包。例如:ST_WEB_BASEDIR_PATH=/seatunnel-web-dist/target/apache-seatunnel-web-1.0.0-SNAPSHOT/ 然后你可以在${Your_code_dir}/seatunnel-dist/target下获取安装包,例如:apache-seatunnel-2.3.3-SNAPSHOT-bin.tar.gz
  • 执行tar -zxvf apache-seatunnel-2.3.3-SNAPSHOT-bin.tar.gz解压安装包。
  • 运行cd apache-seatunnel-2.3.3-SNAPSHOT & sh bin/seatunnel-cluster.sh -d运行SeaTunnel Zeta Engine Server。
  • SeaTunnel Zeta Engine Server默认占用5801端口,请确认端口5801正在被SeaTunnelServer进程占用。
2.1.2 下载安装程序包并进行部署

下载安装包并部署SeaTunnel Zeta Engine Server的另一种安装方式是从https://seatunnel.apache.org/download 下载安装包并部署。

2.2 初始化数据库

  1. 编辑 seatunnel-server/seatunnel-app/src/main/resources/script/seatunnel_server_env.sh 文件, 填写已安装的数据库 address, port, username, and password. 下面是一个例子:

    export HOSTNAME="localhost"
    export PORT="3306"
    export USERNAME="root"
    export PASSWORD="123456"
  2. 执行命令 sh seatunnel-server/seatunnel-app/src/main/resources/script/init_sql.sh 如果运行过程中没有错误,则说明初始化成功。

2.3 构建项目

sh build.sh code

2.4 配置应用程序并运行SeaTunnel Web后端服务器

  1. 编辑 seatunnel-server/seatunnel-app/src/main/resources/application.yml 写数据库连接信息

img.png

  1. 复制 apache-seatunnel-2.3.3-SNAPSHOT/connectors/plugin-mapping.properties 文件 到 seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources 目录.
  2. 运行 seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/SeatunnelApplication.java 如果没有报错,说明seatunnel web后端服务启动成功。注意,你必须设置 -DSEATUNNEL_HOME=${your_seatunnel_install_path} 像这样:

img.png

由于数据源插件是动态加载的,所以需要设置相关的环境变量:

img.png

2.3 运行SeaTunnel Web Front End

cd seatunnel-ui
npm install
npm run dev

如果操作正常,系统显示如下信息:

  ➜  Local:   http://127.0.0.1:5173/
  ➜  Network: use --host to expose
  ➜  press h to show help

在浏览器中访问 http://127.0.0.1:5173/login , 默认用户名和密码是 admin/admin

3 在服务器上运行 SeaTunnel Web

要在服务器上运行SeaTunnel Web,首先需要有一个SeaTunnel Zeta引擎服务器环境。如果还没有,可以参考以下步骤进行部署。

3.1 在Server Node中部署SeaTunnel Zeta Engine Server

有两种方法可以获得SeaTunnel安装包。从源代码构建或从SeaTunnel网站下载。

此处使用的SeaTunnel版本仅用于编写本文档以向您展示所使用的过程,并不一定代表正确的版本。SeaTunnel Web和SeaTunnel Engine有严格的版本依赖关系,您可以通过xxx确认具体的版本对应关系

3.1.1 从源码编译
  • https://seatunnel.apache.org/downloadhttps://github.com/apache/seatunnel.git 获取源码包
  • 使用maven命令编译安装包 ./mvnw -U -T 1C clean install -DskipTests -D"maven.test.skip"=true -D"maven.javadoc.skip"=true -D"checkstyle.skip"=true -D"license.skipAddThirdParty"
  • 然后您可以在${您的代码目录}/seatunnel-dist/target中获得安装包,例如:apache-seatunnel-2.3.3-SNAPSHOT-bin.tar.gz
3.1.2 下载安装包

获取 SeaTunnel Zeta 引擎服务安装包的另一种方式是从 https://seatunnel.apache.org/download 下载安装包并在线安装插件。

  • 下载并安装连接器插件(这个过程中也会自动下载和安装一些第三方依赖包,比如 hadoop jar)。您可以参考 https://seatunnel.apache.org/docs/2.3.2/start-v2/locally/deployment 获取详细步骤。
  • 完成上述步骤后,您将获得一个可以用于在服务器上安装 SeaTunnel Zeta 引擎服务的安装包。运行 tar -zcvf apache-seatunnel-2.3.3-SNAPSHOT-bin.tar.gz apache-seatunnel-2.3.3-SNAPSHOT
3.1.3 部署 SeaTunnel Zeta 服务端

在完成 3.1.1 或 3.1.2 后您可以获得一个安装包 apache-seatunnel-2.3.3-SNAPSHOT-bin.tar.gz,然后您可以将其复制到服务器节点并参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment 进行部署。

3.1.4 在 SeaTunnel Web 运行节点部署 SeaTunnel Zeta 客户端

如果您使用 SeaTunnel Web,您需要在 SeaTunnel Web 运行节点部署一个 SeaTunnel Zeta 客户端。如果 SeaTunnel Zeta 服务端和 SeaTunnel Web 在同一节点,您可以跳过此步骤

  • apache-seatunnel-2.3.3-SNAPSHOT-bin.tar.gz 复制到 SeaTunnel Web 节点并解压,解压到与 SeaTunnel Zeta 服务端节点相同的路径下
  • 和 SeaTunnel Zeta 服务端节点相同,设置 SEATUNNEL_HOME 环境变量。
  • 参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment#6-config-seatunnel-engine-client 配置 hazelcast-client.yaml
  • 运行 $SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template,如果该作业运行结束,表示客户端部署成功。

3.2 从源码构建 SeaTunnel Web 安装包

cd seatunnel-web
sh build.sh code

然后您可以在 seatunnel-web/seatunnel-web-dist/target/apache-seatunnel-web-${project.version}.tar.gz 目录中获得安装包。

3.3 安装

apache-seatunnel-web-${project.version}.tar.gz 复制到服务器节点并解压。

tar -zxvf apache-seatunnel-web-${project.version}.tar.gz
```shell
tar -zxvf apache-seatunnel-web-${project.version}.tar.gz

3.4 初始化数据库

  1. 编辑 apache-seatunnel-web-${project.version}/script/seatunnel_server_env.sh 文件, 填写已安装的数据库 address, port, username, and password. 例如:

    export HOSTNAME="localhost"
    export PORT="3306"
    export USERNAME="root"
    export PASSWORD="123456"
  2. 运行初始化脚本 sh apache-seatunnel-web-${project.version}/script/init_sql.sh 如果操作过程中没有错误,表示初始化成功。

3.5 配置应用并运行 SeaTunnel Web 后端服务

  • 编辑 apache-seatunnel-web-${project.version}/conf/application.yml 在文件中填写数据库连接信息和数据服务接口相关信息。

image

  • 复制 $SEATUNNEL_HOME/config/hazelcast-client.yamlapache-seatunnel-web-${project.version}/conf/
  • 复制 apache-seatunnel-2.3.3-SNAPSHOT/connectors/plugin-mapping.properties 文件到 apache-seatunnel-web-${project.version}/conf/ 目录.

3.6 运行 SeaTunnel Web

cd apache-seatunnel-web-${project.version}
sh bin/seatunnel-backend-daemon.sh start

在浏览器中访问 http://127.0.0.1:8801/ui/ , 默认用户名和密码是 admin/admin.

如何使用

完成所有前置工作后,我们可以打开以下网址: http://127.0.0.1:7890(请根据您的配置替换)来使用它。

现在,让我向您展示如何使用它。

用户管理

img.png

任务管理

img.png

数据源管理

img.png

虚拟表

img.png

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright {yyyy} {name of copyright owner} Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ======================================================================== Apache 2.0 licenses ======================================================================== The following components are provided under the Apache License. See project link for details. The text of each license is the standard Apache 2.0 license. tools/dependencies/checkLicense.sh files from https://github.com/apache/skywalking mvnw files from https://github.com/apache/maven-wrapper Apache 2.0

简介

seatunnel-web 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
1
https://gitee.com/coding-now/seatunnel-web.git
git@gitee.com:coding-now/seatunnel-web.git
coding-now
seatunnel-web
seatunnel-web
dev

搜索帮助