Compare commits

...

111 Commits

Author SHA1 Message Date
Icey.Zhang-张洁 d4ec3231b1 Pull request #643: 10.0.19.37版本插件代码合并 1 year ago
Afly-储泓飞 bee1a0a4ae Pull request #615: REPORT-109024 fix: 小文件不使用分片上传 1 year ago
Afly 870cd6890f Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 1 year ago
Afly 48fee39cda REPORT-109024 fix: 小文件不使用分片上传 1 year ago
superman 5f6e67e52a Merge pull request #464 in PG/plugin-repository-s3 from persist/10.0 to release/10.0 1 year ago
Icey.Zhang-张洁 3649bfc1ec Pull request #463: 10.0.19.34版本插件代码合并 1 year ago
Ocean.Hu-胡浩 aecbffaa35 Pull request #446: 无jira,回退错误提升的jartime 1 year ago
Ocean.Hu 42d6a685f0 无jira,回退错误提升的jartime 1 year ago
Afly-储泓飞 688f08bd4d Pull request #405: REPORT-100817 && REPORT-102146 fix: s3插件对于分片上传的支持;isDirectory方法某些场景判断不正确 2 years ago
Afly ee7b156bde Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 2 years ago
Afly 3232e03cbd 无jira任务 fix: 提升插件版本 2 years ago
Afly d762e63483 JSY-28624 fix: 优化下分片上传逻辑 2 years ago
Afly 66beccd286 REPORT-102146 fix: isDirectory方法某些场景判断不正确 2 years ago
Afly 13d945172f REPORT-100817 fix: s3插件对于分片上传的支持 2 years ago
Aeolus.Zhang-张敬峥 67c37b888c REPORT-99648 fix:s3 fineui兼容问题 2 years ago
Afly-储泓飞 e60cf6fa06 Pull request #330: 无jira任务 chore: 提升版本 2 years ago
Afly 41a9d67249 Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 2 years ago
Afly 649f712ab3 无jira任务 chore: 提升版本 2 years ago
Afly-储泓飞 5f9e9517c9 Pull request #290: REPORT-96529 && REPORT-97765 && REPORT-98184 fix:list接口数量限制;删除接口性能优化;s3插件支持配置连接池大小;测试连接后释放资源;list接口过滤自身目录 2 years ago
Afly 7d4a1f54e3 无jira任务 fix: 测试连接后释放资源;list接口过滤自身目录 2 years ago
Afly b1b2c507cb REPORT-97765 feat: s3插件支持配置连接池大小 2 years ago
Afly 590584026d REPORT-97765 feat: s3插件支持配置连接池大小 2 years ago
Afly 8d67ac17a2 REPORT-96529 fix: list接口数量限制;删除接口性能优化 2 years ago
Ocean.Hu-胡浩 0b8fb5f2ca Pull request #211: 无jira任务提升jartime和版本号 2 years ago
Ocean.Hu d113e73146 Merge remote-tracking branch 'origin/release/10.0' into release/10.0 2 years ago
Ocean.Hu 1313dda374 无jira任务提升jartime和版本号 2 years ago
Ocean.Hu-胡浩 375d09aee5 Pull request #207: REPORT-95454 bugfix:http配置根据endpoint进行设置 2 years ago
Ocean.Hu 6eaeaa8153 REPORT-95454 bugfix:http配置根据endpoint进行设置 2 years ago
superman 3d5ab115e5 Merge pull request #178 in PG/plugin-repository-s3 from persist/10.0 to release/10.0 2 years ago
Icey.Zhang-张洁 d4059beb55 Pull request #177: 10.0.19.29版本插件代码合并 2 years ago
Afly-储泓飞 17804cb6d0 Pull request #121: REPORT-91781 fix: 三方组件升级 2 years ago
Afly 7045fb3255 Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 2 years ago
Afly f14ad7a346 REPORT-91781 fix: 三方组件升级 2 years ago
superman f16f9c55f9 Merge pull request #90 in PG/plugin-repository-s3 from persist/10.0 to release/10.0 2 years ago
Icey.Zhang-张洁 432a0949c2 Pull request #89: Release/10.0 2 years ago
Afly-储泓飞 8533a78ee9 Pull request #87: REPORT-88094 fix: 文件数量较多时,listEntry结果不全;优化写文件性能 2 years ago
Afly 8f1227e659 Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 2 years ago
Freddy-李飞宇 e1b8a79786 Pull request #85: REPORT-87924 fix:cherry pick11.0改动 2 years ago
Freddy eac1c7f9fc REPORT-87924 fix:增加输入限制 2 years ago
Freddy 434779e030 REPORT-87924 feat:适配S3插件前台改动 2 years ago
Afly 255207d5ba REPORT-88094 fix: change-notes 2 years ago
Afly df29385343 REPORT-88094 fix: 文件数量较多时,listEntry结果不全 2 years ago
Afly 4165ca4ff1 无jira任务 perf: 优化写文件性能 2 years ago
Icey.Zhang-张洁 413ec0c3c1 Pull request #79: Release/10.0 2 years ago
Afly-储泓飞 ce965a1dd1 Pull request #76: 无jira任务 feat: 提升版本号 2 years ago
Afly b1008c32a8 Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 2 years ago
Afly eaeacfb216 无jira任务 feat: 提升版本号 2 years ago
Afly-储泓飞 4bb9f8e5da Pull request #75: RREPORT-80581 fix: 第三方组件升级 2 years ago
Afly 97699dc6b3 Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 2 years ago
Afly d62a1e4d3e RREPORT-80581 fix: 第三方组件升级 2 years ago
Icey.Zhang 14300187a7 Pull request #70: 10.0.19.17-0803版本插件代码合并 3 years ago
Ocean.Hu c5f2c17231 Pull request #69: REPORT-76271 10.0版本的S3资源仓库插件修复http无法连接的问题 3 years ago
Ocean 3e099977cd REPORT-76271 10.0版本的S3资源仓库插件修复http无法连接的问题 3 years ago
Afly 635b546d70 Pull request #66: REPORT-72874 feat: 新增两个后台配置项 3 years ago
Afly 4e6dcf5bc0 Merge branch 'release/10.0' of ssh://code.fineres.com:7999/~afly/plugin-repository-s3 into release/10.0 3 years ago
Afly 319418000b REPORT-74814 feat: 新增两个后台配置项 3 years ago
superman 5381eafb83 Merge pull request #62 in PG/plugin-repository-s3 from persist/10.0 to release/10.0 3 years ago
Icey.Zhang 628d259ea2 Pull request #60: 10.0.19.15-0630版本插件代码合并 3 years ago
Afly 1b1e13eded Pull request #53: REPORT-73356 fix: 第三方组件升级 3 years ago
Afly 1c39cf9c87 REPORT-73356 fix: 第三方组件升级 3 years ago
Afly 7696c41116 REPORT-73356 fix: 升级jar包 3 years ago
superman 80bb4bc54a Merge pull request #47 in PG/plugin-repository-s3 from persist/10.0 to release/10.0 3 years ago
Icey.Zhang 166d039938 Pull request #41: 10.0.19.10-0328维护版本插件代码合并 3 years ago
Afly 530ce35da4 Pull request #38: REPORT-67291 fix: 第三方组件升级 3 years ago
Afly 8be6dcd23f REPORT-67291 fix: 第三方组件升级 3 years ago
Afly ef4ffb73e5 REPORT-67291 fix: 第三方组件升级 3 years ago
Icey.Zhang f5cbe76616 Pull request #30: 10.0.19.1-0923版本插件代码合并 3 years ago
Dylan.Liu c1c68c6d53 Pull request #29: 无jira任务 fix: 版本号提升 3 years ago
Dylan.Liu 59369362ec 无jira任务 fix: 版本号提升 3 years ago
Dylan.Liu 7ddeabf9e9 Pull request #27: 无jira任务 fix: env-version 3 years ago
Dylan.Liu 12291a7620 无jira任务 fix: env-version 3 years ago
Icey.Zhang 609a453297 Pull request #25: 10.0.18-0802版本插件代码合并 4 years ago
ju|剧浩宇 19dc9ba92e Pull request #24: INO-9768 跑FR 10.0.18国际化脚本+插件脚本 4 years ago
Joe.Jiang 8911690304 INO-9768 跑FR 10.0.18国际化脚本+插件脚本 4 years ago
Roger.Chen 41d0c61f78 Pull request #23: DEC-19779 fix: 添加一下国际化文件给国际组翻译 4 years ago
Roger.Chen 533df83872 DEC-19775 fix: 添加一下国际化文件给国际组翻译 4 years ago
Icey.Zhang c7e9479e78 Pull request #22: 10.0.16-0526号版本插件代码合并 4 years ago
Feng b6d9f00336 Pull request #21: DEC-18310 fix:文件服务器插件接口未兼容 4 years ago
Feng 4623bd97c8 DEC-18310 fix:文件服务器插件接口未兼容 4 years ago
Feng 8834f03f2b Pull request #20: QFX-5554 fix: 【冒烟】测试环境大数据量excel上传后点击保存无法保存成功 4 years ago
Feng cc8efcd603 QFX-5554 fix: 【冒烟】测试环境大数据量excel上传后点击保存无法保存成功 4 years ago
Feng 39b51beaf8 QFX-5554 fix: 【冒烟】测试环境大数据量excel上传后点击保存无法保存成功 4 years ago
Feng 3ddfba09e2 Pull request #17: 无jira 限制插件jartime 4 years ago
Feng 222b8c93e2 无jira 限制插件jartime 4 years ago
Frank.Qiu f5146deee0 Pull request #16: DEC-17106 fix: 【迭代】【S3插件】前端协议框与配置项框没有对齐 4 years ago
Frank.Qiu afbeeb7da0 DEC-17106 fix: 【迭代】【S3插件】前端协议框与配置项框没有对齐 4 years ago
Feng 7d2a958e8a Pull request #15: DEC-17012 fix: 【S3插件】备份文件删除后,实际文件服务器上文件未删除 4 years ago
Feng 3c1eb94de7 Merge remote-tracking branch 'origin/release/10.0' into release/10.0 4 years ago
Feng 4a0f1369d1 DEC-17012 fix: 【S3插件】备份文件删除后,实际文件服务器上文件未删除 4 years ago
Feng a56cdef22c Pull request #14: DEC-17013 fix: 【S3插件】备份模板很慢,有生成多余文件,后台很多警告 4 years ago
Feng 41d31de6e6 DEC-16997 fix: 【S3插件】定时调度的结果文件预览失败 4 years ago
Feng 9395670690 Merge remote-tracking branch 'origin/release/10.0' into release/10.0 4 years ago
Feng 893bbae5b4 DEC-17013 fix: 【S3插件】备份模板很慢,有生成多余文件,后台很多警告 4 years ago
Dylan.Liu 06689bd208 Pull request #12: DEC-16834 feat: 集群组件出错优化 4 years ago
Dylan.Liu 0402aa3a9e Merge branch 'release/10.0' of https://code.fineres.com/scm/~dylan.liu/plugin-repository-s3 into release/10.0 4 years ago
Dylan.Liu ac8ee784fe DEC-16834 feat: 集群组件出错优化 4 years ago
Feng 0506f22a5d Pull request #11: DEC-16997 fix: 【S3插件】定时调度的结果文件预览失败 4 years ago
Feng b3beded0f2 Merge remote-tracking branch 'origin/release/10.0' into release/10.0 4 years ago
Feng f16a557bf0 DEC-16997 fix: 【S3插件】定时调度的结果文件预览失败 4 years ago
Feng 2c5773904b DEC-16997 fix: 【S3插件】定时调度的结果文件预览失败 4 years ago
Feng 19a499536b Pull request #10: DEC-17006 fix: 【S3插件】配置输入不存在的路径也可以连接成功 4 years ago
Feng 39e97de37f DEC-17006 fix: 【S3插件】配置输入不存在的路径也可以连接成功 4 years ago
Feng d797433f1a Pull request #9: DEC-16411 feat: S3插件变为官方插件 国际化 4 years ago
Feng 1c342df1e0 DEC-16411 feat: S3插件变为官方插件 国际化 4 years ago
Feng 5724103649 DEC-16411 feat: S3插件变为官方插件 4 years ago
Feng dee24ab76e Pull request #8: DEC-16411 feat: S3插件变为官方插件 4 years ago
Feng 4bf15d218c DEC-16411 feat: S3插件变为官方插件 4 years ago
Feng b0a6e35c7c DEC-16411 feat: S3插件变为官方插件 4 years ago
Feng bba598df12 Pull request #6: DEC-16411 feat: S3插件作为官方插件维护 4 years ago
Feng ad2da13ab2 DEC-16411 S3仓库打包脚本 4 years ago
Feng ab60e1c61f DEC-16411 S3插件维护 4 years ago
  1. 4
      .gitignore
  2. 123
      build.xml
  3. 318
      front/bundle.js
  4. 10
      front/gulpfile.js
  5. 13
      front/package.json
  6. BIN
      lib/aws-java-sdk-core-1.11.801.jar
  7. BIN
      lib/aws-java-sdk-core-1.12.435.jar
  8. BIN
      lib/aws-java-sdk-kms-1.11.801.jar
  9. BIN
      lib/aws-java-sdk-kms-1.12.435.jar
  10. BIN
      lib/aws-java-sdk-s3-1.11.801.jar
  11. BIN
      lib/aws-java-sdk-s3-1.12.435.jar
  12. BIN
      lib/httpclient-4.5.13.jar
  13. BIN
      lib/jackson-annotations-2.14.1.jar
  14. BIN
      lib/jackson-annotations-2.6.0.jar
  15. BIN
      lib/jackson-core-2.14.1.jar
  16. BIN
      lib/jackson-core-2.6.7.jar
  17. BIN
      lib/jackson-databind-2.14.1.jar
  18. BIN
      lib/jackson-databind-2.6.7.3.jar
  19. BIN
      lib/jackson-dataformat-cbor-2.14.1.jar
  20. BIN
      lib/jackson-dataformat-cbor-2.6.7.jar
  21. 22
      plugin.xml
  22. 4
      src/main/java/com/fanruan/fs/s3/repository/S3FileServerResource.java
  23. 70
      src/main/java/com/fanruan/fs/s3/repository/core/S3Config.java
  24. 29
      src/main/java/com/fanruan/fs/s3/repository/core/S3RepositoryFactory.java
  25. 390
      src/main/java/com/fanruan/fs/s3/repository/core/S3ResourceRepository.java
  26. 5
      src/main/resources/com/fanruan/fs/s3/repository/locale/s3.properties
  27. 11
      src/main/resources/com/fanruan/fs/s3/repository/locale/s3_en_US.properties
  28. 9
      src/main/resources/com/fanruan/fs/s3/repository/locale/s3_ja_JP.properties
  29. 9
      src/main/resources/com/fanruan/fs/s3/repository/locale/s3_ko_KR.properties
  30. 11
      src/main/resources/com/fanruan/fs/s3/repository/locale/s3_zh_CN.properties
  31. 9
      src/main/resources/com/fanruan/fs/s3/repository/locale/s3_zh_TW.properties
  32. 192
      src/main/resources/com/fanruan/fs/s3/repository/web/js/bundle.js

4
.gitignore vendored

@ -7,4 +7,6 @@ target/
build build
local.properties local.properties
classes/ classes/
transform-classes/ transform-classes/
node_modules
yarn.lock

123
build.xml

@ -0,0 +1,123 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project basedir="." default="jar" name="plugin-repository-s3">
<!-- JDK路径,根据自己机器上实际位置修改-->
<property name="jdk.home" value="/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home"/>
<property name="libs" value="${basedir}/lib"/>
<property name="publicLibs" value=""/>
<property name="destLoc" value="."/>
<property name="classes" value="classes"/>
<xmlproperty file="${basedir}/plugin.xml"/>
<property name="current-version" value="${plugin.version}"/>
<!-- 插件版本-->
<property name="plugin-version" value="${current-version}"/>
<!-- 插件名字-->
<property name="plugin-name" value="s3-repository"/>
<property name="plugin-jar" value="fr-plugin-${plugin-name}-${plugin-version}.jar"/>
<target name="prepare">
<delete dir="${classes}"/>
<delete dir="fr-plugin-${plugin-name}-${plugin-version}"/>
<xmlproperty file="${basedir}/plugin.xml"/>
<delete dir="${destLoc}/${plugin.name}"/>
</target>
<path id="compile.classpath">
<fileset dir="${libs}">
<include name="**/*.jar"/>
</fileset>
<fileset dir="${publicLibs}">
<include name="**/*.jar"/>
</fileset>
</path>
<patternset id="resources4Jar">
<exclude name="**/.settings/**"/>
<exclude name=".classpath"/>
<exclude name=".project"/>
<exclude name="**/*.java"/>
<exclude name="**/*.db"/>
<exclude name="**/*.g"/>
<exclude name="**/package.html"/>
</patternset>
<target name="copy_resources">
<echo message="从${resources_from}拷贝图片,JS,CSS等资源文件"/>
<delete dir="tmp"/>
<copy todir="tmp">
<fileset dir="${resources_from}/src/main/resources">
<patternset refid="resources4Jar"/>
</fileset>
</copy>
<copy todir="${classes}">
<fileset dir="tmp"/>
</copy>
<delete dir="tmp"/>
</target>
<target name="compile_javas">
<echo message="编译${compile_files}下的Java文件"/>
<javac destdir="${classes}" debug="false" optimize="on" source="${source_jdk_version}"
target="${target_jdk_version}"
fork="true" memoryMaximumSize="512m" listfiles="false" srcdir="${basedir}"
executable="${compile_jdk_version}/bin/javac">
<src path="${basedir}/src/main/java"/>
<exclude name="**/.svn/**"/>
<compilerarg line="-encoding UTF8 "/>
<classpath refid="compile.classpath"/>
</javac>
</target>
<target name="jar_classes">
<echo message="打Jar包:${jar_name}"/>
<delete file="${basedir}/${jar_name}"/>
<jar jarfile="${basedir}/${jar_name}">
<fileset dir="${classes}">
</fileset>
</jar>
</target>
<target name="super_jar" depends="prepare">
<antcall target="copy_resources">
<param name="resources_from" value="${basedir}"/>
</antcall>
<antcall target="compile_javas">
<param name="source_jdk_version" value="1.8"/>
<param name="target_jdk_version" value="1.8"/>
<param name="compile_jdk_version" value="${jdk.home}"/>
<param name="compile_files" value="${basedir}/src"/>
</antcall>
<echo message="compile plugin success!"/>
<antcall target="jar_classes">
<param name="jar_name" value="${plugin-jar}"/>
</antcall>
<delete dir="${classes}"/>
</target>
<target name="jar" depends="super_jar">
<antcall target="zip"/>
</target>
<target name="zip">
<property name="plugin-folder" value="fr-plugin-${plugin-name}-${plugin-version}"/>
<echo message="----------zip files----------"/>
<mkdir dir="${plugin-folder}"/>
<copy todir="${plugin-folder}">
<fileset dir=".">
<include name="${plugin-jar}"/>
<include name="plugin.xml"/>
</fileset>
<fileset dir="${libs}">
<include name="*.jar"/>
<include name="*.dll"/>
</fileset>
</copy>
<zip destfile="${basedir}/${plugin-folder}.zip" basedir=".">
<include name="${plugin-folder}/*.jar"/>
<include name="${plugin-folder}/*.dll"/>
<include name="${plugin-folder}/plugin.xml"/>
</zip>
<xmlproperty file="${basedir}/plugin.xml"/>
<move file="${plugin-folder}.zip" todir="${destLoc}/${plugin.name}"/>
<delete dir="${plugin-folder}"/>
</target>
</project>

318
front/bundle.js

@ -0,0 +1,318 @@
BI.config("dec.constant.intelligence.cluster.file.server", function (items) {
items.push({
value: "S3", // 地址栏显示的hash值
id: "decision-intelligence-cluster-file-s3", // id
text: "S3", // 文字
cardType: "dec.intelligence.cluster.file.s3",
workRoot: false,
});
return items;
});
!(function () {
var LABEL_WIDTH = 107, EDITOR_WIDTH = 393;
var S3 = BI.inherit(BI.Widget, {
props: {
baseCls: "dec-cluster-ftp",
value: {},
},
_store: function () {
return BI.Models.getModel("dec.model.intelligence.cluster.file.s3", {
value: this.options.value,
});
},
watch: {
isOpen: function (v) {
this.toggle.setVisible(v);
}
},
render: function () {
var self = this, o = this.options;
return {
type: "bi.vertical",
tgap: 15,
items: [
{
type: "dec.label.editor.item",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Input"),
text: BI.i18nText("Plugin-S3_End_Point"),
value: this.model.endPoint,
el: {
disabled: !o.editable,
},
ref: function (_ref) {
self.endPointRow = _ref;
},
listeners: [{
eventName: BI.Editor.EVENT_CHANGE,
action: function () {
self.store.setEndPoint(this.getValue());
},
}],
}, {
type: "dec.label.editor.item",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Input"),
text: BI.i18nText("Plugin-S3_Region"),
value: this.model.region,
el: {
disabled: !o.editable,
},
ref: function (_ref) {
self.regionRow = _ref;
},
listeners: [{
eventName: BI.Editor.EVENT_CHANGE,
action: function () {
self.store.setRegion(this.getValue());
},
}],
}, {
type: "dec.label.editor.item",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Input"),
text: BI.i18nText("Plugin-S3_Access_Key_Id"),
value: this.model.accessKeyId,
el: {
disabled: !o.editable,
},
ref: function (_ref) {
self.portRow = _ref;
},
listeners: [{
eventName: BI.Editor.EVENT_CHANGE,
action: function () {
self.store.setAccessKeyId(this.getValue());
},
}],
},
{
type: "dec.common.cipher.editor",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Access_Key_Secret"),
text: BI.i18nText("Plugin-S3_Access_Key_Secret"),
value: this.model.password,
el: {
disabled: !o.editable,
},
ref: function (_ref) {
self.passwordRow = _ref;
},
},
{
type: "dec.label.editor.item",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Bucket"),
text: BI.i18nText("Plugin-S3_Bucket"),
value: this.model.bucket,
el: {
disabled: !o.editable,
},
listeners: [{
eventName: BI.Editor.EVENT_CHANGE,
action: function () {
self.store.setBucket(this.getValue());
},
}],
},
{
type: "dec.label.editor.item",
el: {
disabled: !o.editable,
},
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Dec-Please_Input"),
text: BI.i18nText("Dec-Basic_Path"),
value: this.model.workRoot,
ref: function (_ref) {
self.filePathRow = _ref;
},
}, {
type: "bi.vertical_adapt",
items: [{
type: "bi.icon_change_button",
iconCls: this.model.isOpen ? "expander-down-font" : "expander-right-font",
ref: (_ref) => {
this.OtherConfigButton = _ref;
},
handler: () => {
this.store.setIsOpen(!this.model.isOpen);
this.OtherConfigButton.setIcon(this.model.isOpen ? "expander-down-font" : "expander-right-font");
}
}, {
type: "bi.text_button",
text: BI.i18nText('Plugin-S3_Other_Config'),
handler: () => {
this.store.setIsOpen(!this.model.isOpen);
this.OtherConfigButton.setIcon(this.model.isOpen ? "expander-down-font" : "expander-right-font");
}
}]
}, {
type: 'bi.vertical',
ref: (_ref) => {
this.toggle = _ref;
},
invisible: () => !this.model.isOpen,
items: [{
type: "dec.label.editor.item",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Input"),
text: "PathStyleAccess",
value: this.model.enablePathStyleAccess,
ref: function (_ref) {
self.enablePathStyleAccessRow = _ref;
},
el: {
disabled: !o.editable,
},
listeners: [{
eventName: BI.Editor.EVENT_CHANGE,
action: function () {
self.store.setEnablePathStyleAccess(this.getValue());
}
}]
}, {
type: "dec.label.editor.item",
textWidth: LABEL_WIDTH,
editorWidth: EDITOR_WIDTH,
watermark: BI.i18nText("Plugin-S3_Input"),
text: "SignerOverride",
value: this.model.signerOverride,
el: {
disabled: !o.editable,
},
tgap: 15,
listeners: [{
eventName: BI.Editor.EVENT_CHANGE,
action: function () {
self.store.setSignerOverride(this.getValue());
}
}]
}],
}
]
};
},
getValue: function () {
var enablePathStyleAccess = false;
if (this.model.enablePathStyleAccess === 'true') {
enablePathStyleAccess = true;
}
return {
endPoint: this.model.endPoint,
region: this.model.region,
accessKeyId: this.model.accessKeyId,
password: this.passwordRow.getCipher(),
bucket: this.model.bucket,
workRoot: this.filePathRow.getValue(),
enablePathStyleAccess,
signerOverride: this.model.signerOverride,
};
},
validation: function () {
var valid = true;
var path = this.filePathRow.getValue();
if (Dec.Utils.strLength(path) > DecCst.STRING_SHORT_TEXT_LENGTH) {
this.filePathRow.showError(BI.i18nText("Dec-Error_Length_Greater_Than_Short_Text"));
valid = false;
}
if (BI.startWith(path, "/") || !BI.endWith(path, "/")) {
this.filePathRow.showError(BI.i18nText("Dec-Error_Start_With_Slash_Or_End_Without_Slash"));
valid = false;
}
if (!BI.isKey(path)) {
this.filePathRow.showError(BI.i18nText("Dec-Error_Null"));
valid = false;
}
if (!(this.model.enablePathStyleAccess === 'false' || this.model.enablePathStyleAccess === 'true')) {
this.enablePathStyleAccessRow.showError(BI.i18nText("Plugin-S3_EnablePathStyleAccess_Error_Tip"));
valid = false;
}
return valid;
},
});
BI.shortcut("dec.intelligence.cluster.file.s3", S3);
}());
!(function () {
var Model = BI.inherit(Fix.Model, {
state: function () {
var val = this.options.value;
return {
endPoint: val.endPoint,
region: val.region,
accessKeyId: val.accessKeyId,
password: val.password,
bucket: val.bucket,
workRoot: val.workRoot,
isOpen: false,
enablePathStyleAccess: 'false',
signerOverride: "",
};
},
computed: {
encodingArray: function () {
return BI.map(DecCst.EncodeConstants.ENCODING_ARRAY, function (i, v) {
return {
value: v,
};
});
},
},
actions: {
setEndPoint: function (v) {
this.model.endPoint = v;
},
setRegion: function (v) {
this.model.region = v;
},
setAccessKeyId: function (v) {
this.model.accessKeyId = v;
},
setBucket: function (v) {
this.model.bucket = v;
},
setEnablePathStyleAccess: function (v) {
this.model.enablePathStyleAccess = v;
},
setSignerOverride: function (v) {
this.model.signerOverride = v;
},
setIsOpen: function (v) {
this.model.isOpen = v;
},
},
});
BI.model("dec.model.intelligence.cluster.file.s3", Model);
}());

10
front/gulpfile.js

@ -0,0 +1,10 @@
const gulp = require('gulp');
const uglify = require('gulp-uglify');
function build() {
return gulp.src('./bundle.js')
.pipe(uglify())
.pipe(gulp.dest('../src/main/resources/com/fanruan/fs/s3/repository/web/js'));
}
exports.default = build;

13
front/package.json

@ -0,0 +1,13 @@
{
"name": "front",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",
"scripts": {
"build": "gulp"
},
"dependencies": {
"gulp": "^4.0.2",
"gulp-uglify": "^3.0.2"
}
}

BIN
lib/aws-java-sdk-core-1.11.801.jar

Binary file not shown.

BIN
lib/aws-java-sdk-core-1.12.435.jar

Binary file not shown.

BIN
lib/aws-java-sdk-kms-1.11.801.jar

Binary file not shown.

BIN
lib/aws-java-sdk-kms-1.12.435.jar

Binary file not shown.

BIN
lib/aws-java-sdk-s3-1.11.801.jar

Binary file not shown.

BIN
lib/aws-java-sdk-s3-1.12.435.jar

Binary file not shown.

BIN
lib/httpclient-4.5.9.jar → lib/httpclient-4.5.13.jar

Binary file not shown.

BIN
lib/jackson-annotations-2.14.1.jar

Binary file not shown.

BIN
lib/jackson-annotations-2.6.0.jar

Binary file not shown.

BIN
lib/jackson-core-2.14.1.jar

Binary file not shown.

BIN
lib/jackson-core-2.6.7.jar

Binary file not shown.

BIN
lib/jackson-databind-2.14.1.jar

Binary file not shown.

BIN
lib/jackson-databind-2.6.7.3.jar

Binary file not shown.

BIN
lib/jackson-dataformat-cbor-2.14.1.jar

Binary file not shown.

BIN
lib/jackson-dataformat-cbor-2.6.7.jar

Binary file not shown.

22
plugin.xml

@ -5,12 +5,28 @@
<main-package>com.fanruan.fs</main-package> <main-package>com.fanruan.fs</main-package>
<active>yes</active> <active>yes</active>
<hidden>no</hidden> <hidden>no</hidden>
<version>1.2.2</version> <version>1.4.1</version>
<env-version>10.0</env-version> <env-version>10.0~10.0</env-version>
<jartime>2020-01-27</jartime> <jartime>2021-03-11</jartime>
<vendor>richie</vendor> <vendor>richie</vendor>
<description><![CDATA[使用支持S3协议的云存储文件系统作为文件服务器。]]></description> <description><![CDATA[使用支持S3协议的云存储文件系统作为文件服务器。]]></description>
<change-notes><![CDATA[ <change-notes><![CDATA[
[2023-11-21]小文件不使用分片上传。 <br/>
[2023-08-08]支持分片上传,模板保存问题修复。 <br/>
[2023-06-30]修复默认配置获取错误的问题,过滤有问题的路径。 <br/>
[2023-03-28]第三方组件升级。 <br/>
[2023-01-03]优化写文件性能; 修复文件太多显示不全的问题。<br/>
[2022-09-22]第三方组件升级。 <br/>
[2022-07-25]修复http无法连接,而https和缺省的时候正常<br/>
[2022-06-30]新增enablePathStyleAccess、signerOverride后台配置<br/>
[2022-06-14]第三方组件升级。 <br/>
[2022-03-10]第三方组件升级。 <br/>
[2021-07-16]提供韩文、日文、繁体等国际化文件。 <br/>
[2021-04-23]copyFile接口兼容。 <br/>
[2021-03-30]修复copy文件耗时问题。<br/>
[2021-01-28]修复备份还原S3报警告问题。<br/>
[2021-01-24]修复定时调度结果文件预览失败。<br/>
[2021-01-20]正式作为官方插件维护。<br/>
[2020-12-16]修复远程设计新增、修改模板问题。<br/> [2020-12-16]修复远程设计新增、修改模板问题。<br/>
[2020-10-29]修复连接不释放问题。<br/> [2020-10-29]修复连接不释放问题。<br/>
[2020-08-17]修复key不存在时报错的问题。<br/> [2020-08-17]修复key不存在时报错的问题。<br/>

4
src/main/java/com/fanruan/fs/s3/repository/S3FileServerResource.java

@ -1,7 +1,7 @@
package com.fanruan.fs.s3.repository; package com.fanruan.fs.s3.repository;
import com.fr.decision.fun.impl.AbstractWebResourceProvider; import com.fr.decision.fun.impl.AbstractWebResourceProvider;
import com.fr.decision.web.MainComponent; import com.fr.decision.web.CommonComponent;
import com.fanruan.fs.s3.repository.web.S3FileServerComponent; import com.fanruan.fs.s3.repository.web.S3FileServerComponent;
import com.fr.web.struct.Atom; import com.fr.web.struct.Atom;
@ -14,7 +14,7 @@ public class S3FileServerResource extends AbstractWebResourceProvider {
@Override @Override
public Atom attach() { public Atom attach() {
return MainComponent.KEY; return CommonComponent.KEY;
} }
@Override @Override

70
src/main/java/com/fanruan/fs/s3/repository/core/S3Config.java

@ -1,8 +1,8 @@
package com.fanruan.fs.s3.repository.core; package com.fanruan.fs.s3.repository.core;
import com.fanruan.api.conf.HolderKit;
import com.fr.config.Identifier; import com.fr.config.Identifier;
import com.fr.config.holder.Conf; import com.fr.config.holder.Conf;
import com.fr.config.holder.factory.Holders;
import com.fr.io.config.CommonRepoConfig; import com.fr.io.config.CommonRepoConfig;
import com.fr.io.context.ResourceModuleContext; import com.fr.io.context.ResourceModuleContext;
import com.fr.io.context.info.GetConfig; import com.fr.io.context.info.GetConfig;
@ -21,19 +21,25 @@ public class S3Config extends CommonRepoConfig {
} }
@Identifier("endPoint") @Identifier("endPoint")
private Conf<String> endPoint = Holders.simple(StringUtils.EMPTY); private Conf<String> endPoint = HolderKit.simple(StringUtils.EMPTY);
@Identifier("region") @Identifier("region")
private Conf<String> region = Holders.simple(StringUtils.EMPTY); private Conf<String> region = HolderKit.simple(StringUtils.EMPTY);
@Identifier("accessKeyId") @Identifier("accessKeyId")
private Conf<String> accessKeyId = Holders.simple(StringUtils.EMPTY); private Conf<String> accessKeyId = HolderKit.simple(StringUtils.EMPTY);
@Identifier("accessKeySecret")
private Conf<String> accessKeySecret = Holders.simple(StringUtils.EMPTY);
@Identifier("bucket") @Identifier("bucket")
private Conf<String> bucket = Holders.simple(StringUtils.EMPTY); private Conf<String> bucket = HolderKit.simple(StringUtils.EMPTY);
@Identifier("enablePathStyleAccess")
private Conf<Boolean> enablePathStyleAccess = HolderKit.simple(false);
@Identifier("signerOverride")
private Conf<String> signerOverride = HolderKit.simple(StringUtils.EMPTY);
@Identifier("maxConnections")
private Conf<Integer> maxConnections = HolderKit.simple(200);
@GetConfig("endPoint") @GetConfig("endPoint")
public String getEndPoint() { public String getEndPoint() {
@ -65,16 +71,6 @@ public class S3Config extends CommonRepoConfig {
this.accessKeyId.set(accessKeyId); this.accessKeyId.set(accessKeyId);
} }
@GetConfig("accessKeySecret")
public String getAccessKeySecret() {
return accessKeySecret.get();
}
@SetConfig("accessKeySecret")
public void setAccessKeySecret(String accessKeySecret) {
this.accessKeySecret.set(accessKeySecret);
}
@GetConfig("bucket") @GetConfig("bucket")
public String getBucket() { public String getBucket() {
return bucket.get(); return bucket.get();
@ -85,6 +81,36 @@ public class S3Config extends CommonRepoConfig {
this.bucket.set(bucket); this.bucket.set(bucket);
} }
@GetConfig("enablePathStyleAccess")
public boolean isEnablePathStyleAccess() {
return enablePathStyleAccess.get();
}
@SetConfig("enablePathStyleAccess")
public void setEnablePathStyleAccess(boolean enablePathStyleAccess) {
this.enablePathStyleAccess.set(enablePathStyleAccess);
}
@GetConfig("signerOverride")
public String getSignerOverride() {
return signerOverride.get();
}
@SetConfig("signerOverride")
public void setSignerOverride(String signerOverride) {
this.signerOverride.set(signerOverride);
}
@GetConfig("maxConnections")
public int getMaxConnections() {
return maxConnections.get();
}
@SetConfig("maxConnections")
public void setMaxConnections(int maxConnections) {
this.maxConnections.set(maxConnections);
}
@Override @Override
public void update(String key) { public void update(String key) {
super.update(key); super.update(key);
@ -93,8 +119,10 @@ public class S3Config extends CommonRepoConfig {
this.setEndPoint(newConfig.getEndPoint()); this.setEndPoint(newConfig.getEndPoint());
this.setRegion(newConfig.getRegion()); this.setRegion(newConfig.getRegion());
this.setAccessKeyId(newConfig.getAccessKeyId()); this.setAccessKeyId(newConfig.getAccessKeyId());
this.setAccessKeySecret(newConfig.getAccessKeySecret());
this.setBucket(newConfig.getBucket()); this.setBucket(newConfig.getBucket());
this.setEnablePathStyleAccess(newConfig.isEnablePathStyleAccess());
this.setSignerOverride(newConfig.getSignerOverride());
this.setMaxConnections(newConfig.getMaxConnections());
} }
} }
@ -104,8 +132,10 @@ public class S3Config extends CommonRepoConfig {
cloned.endPoint = (Conf<String>) endPoint.clone(); cloned.endPoint = (Conf<String>) endPoint.clone();
cloned.region = (Conf<String>) region.clone(); cloned.region = (Conf<String>) region.clone();
cloned.accessKeyId = (Conf<String>) accessKeyId.clone(); cloned.accessKeyId = (Conf<String>) accessKeyId.clone();
cloned.accessKeySecret = (Conf<String>) accessKeySecret.clone();
cloned.bucket = (Conf<String>) bucket.clone(); cloned.bucket = (Conf<String>) bucket.clone();
cloned.enablePathStyleAccess = (Conf<Boolean>) enablePathStyleAccess.clone();
cloned.signerOverride = (Conf<String>) signerOverride.clone();
cloned.maxConnections = (Conf<Integer>) maxConnections.clone();
return cloned; return cloned;
} }
} }

29
src/main/java/com/fanruan/fs/s3/repository/core/S3RepositoryFactory.java

@ -1,5 +1,7 @@
package com.fanruan.fs.s3.repository.core; package com.fanruan.fs.s3.repository.core;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSStaticCredentialsProvider; import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials; import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder; import com.amazonaws.client.builder.AwsClientBuilder;
@ -9,6 +11,9 @@ import com.fanruan.api.log.LogKit;
import com.fr.io.base.provider.impl.ConfigRepositoryFactory; import com.fr.io.base.provider.impl.ConfigRepositoryFactory;
import com.fr.io.context.info.RepositoryProfile; import com.fr.io.context.info.RepositoryProfile;
import com.fr.io.repository.ResourceRepository; import com.fr.io.repository.ResourceRepository;
import com.fr.stable.StringUtils;
import static com.fanruan.fs.s3.repository.core.S3ResourceRepository.HTTP;
/** /**
* @author richie * @author richie
@ -35,15 +40,33 @@ public class S3RepositoryFactory extends ConfigRepositoryFactory<S3Config> {
@Override @Override
public boolean verifyConfig(S3Config config) { public boolean verifyConfig(S3Config config) {
AmazonS3 s3 = null;
try { try {
BasicAWSCredentials credentials = new BasicAWSCredentials(config.getAccessKeyId(), config.getAccessKeySecret()); BasicAWSCredentials credentials = new BasicAWSCredentials(config.getAccessKeyId(), config.getPassword());
AmazonS3 s3 = AmazonS3ClientBuilder.standard() AmazonS3ClientBuilder amazonS3ClientBuilder = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(config.getEndPoint(), config.getRegion())) .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(config.getEndPoint(), config.getRegion()))
.withCredentials(new AWSStaticCredentialsProvider(credentials)).build(); .withCredentials(new AWSStaticCredentialsProvider(credentials)).disableChunkedEncoding();
if (config.isEnablePathStyleAccess()) {
amazonS3ClientBuilder = amazonS3ClientBuilder.enablePathStyleAccess();
}
ClientConfiguration clientConfiguration = new ClientConfiguration();
if (StringUtils.isNotEmpty(config.getSignerOverride())) {
clientConfiguration.setSignerOverride(config.getSignerOverride());
}
LogKit.debug("[S3] endpoint is {}", config.getEndPoint());
if (config.getEndPoint().startsWith(HTTP)) {
clientConfiguration.setProtocol(Protocol.HTTP);
}
amazonS3ClientBuilder = amazonS3ClientBuilder.withClientConfiguration(clientConfiguration);
s3 = amazonS3ClientBuilder.build();
s3.listObjects(config.getBucket()); s3.listObjects(config.getBucket());
} catch (Exception e) { } catch (Exception e) {
LogKit.error(e.getMessage(), e); LogKit.error(e.getMessage(), e);
return false; return false;
} finally {
if (s3 != null) {
s3.shutdown();
}
} }
return true; return true;
} }

390
src/main/java/com/fanruan/fs/s3/repository/core/S3ResourceRepository.java

@ -1,27 +1,39 @@
package com.fanruan.fs.s3.repository.core; package com.fanruan.fs.s3.repository.core;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSStaticCredentialsProvider; import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials; import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder; import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.CopyObjectRequest; import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
import com.amazonaws.services.s3.model.DeleteObjectRequest; import com.amazonaws.services.s3.model.DeleteObjectsRequest;
import com.amazonaws.services.s3.model.GetObjectRequest; import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
import com.amazonaws.services.s3.model.ListObjectsRequest; import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ObjectListing; import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.ObjectMetadata; import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.services.s3.model.PutObjectRequest; import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.S3Object; import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectSummary; import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.services.s3.model.UploadPartRequest;
import com.amazonaws.services.s3.model.UploadPartResult;
import com.amazonaws.util.IOUtils;
import com.fanruan.api.log.LogKit; import com.fanruan.api.log.LogKit;
import com.fanruan.api.util.StringKit; import com.fanruan.api.util.StringKit;
import com.fr.io.repository.FineFileEntry; import com.fr.io.repository.FineFileEntry;
import com.fr.io.repository.base.BaseResourceRepository; import com.fr.io.repository.base.BaseResourceRepository;
import com.fr.io.utils.ResourceIOUtils;
import com.fr.stable.Filter; import com.fr.stable.Filter;
import com.fr.stable.StringUtils;
import com.fr.third.org.apache.commons.io.output.NullOutputStream;
import com.fr.workspace.resource.ResourceIOException; import com.fr.workspace.resource.ResourceIOException;
import java.io.ByteArrayInputStream; import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream; import java.io.InputStream;
import java.net.URL; import java.net.URL;
import java.net.URLConnection; import java.net.URLConnection;
@ -35,7 +47,14 @@ import java.util.List;
*/ */
public class S3ResourceRepository extends BaseResourceRepository { public class S3ResourceRepository extends BaseResourceRepository {
private static final int PAGE_SIZE = 1000;
private static final int PART_SIZE = 5 * 1024 * 1024;
private static final int MULTIPART_UPLOAD_LIMIT = 20 * PART_SIZE;
private static final String DELIMITER = "/"; private static final String DELIMITER = "/";
public static final String HTTP = "http:";
private final AmazonS3 s3; private final AmazonS3 s3;
private final String bucket; private final String bucket;
@ -43,10 +62,25 @@ public class S3ResourceRepository extends BaseResourceRepository {
public S3ResourceRepository(String repoName, String workRoot, S3Config config) { public S3ResourceRepository(String repoName, String workRoot, S3Config config) {
super(repoName, workRoot); super(repoName, workRoot);
BasicAWSCredentials credentials = new BasicAWSCredentials(config.getAccessKeyId(), config.getAccessKeySecret()); BasicAWSCredentials credentials = new BasicAWSCredentials(config.getAccessKeyId(), config.getPassword());
this.s3 = AmazonS3ClientBuilder.standard() AmazonS3ClientBuilder amazonS3ClientBuilder = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(config.getEndPoint(), config.getRegion())) .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(config.getEndPoint(), config.getRegion()))
.withCredentials(new AWSStaticCredentialsProvider(credentials)).build(); .withCredentials(new AWSStaticCredentialsProvider(credentials)).disableChunkedEncoding();
if (config.isEnablePathStyleAccess()) {
amazonS3ClientBuilder = amazonS3ClientBuilder.enablePathStyleAccess();
}
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setMaxConnections(config.getMaxConnections());
LogKit.info("Max connections is {}!", clientConfiguration.getMaxConnections());
if (StringUtils.isNotEmpty(config.getSignerOverride())) {
clientConfiguration.setSignerOverride(config.getSignerOverride());
}
LogKit.debug("[S3] endpoint is {}", config.getEndPoint());
if (config.getEndPoint().startsWith(HTTP)) {
clientConfiguration.setProtocol(Protocol.HTTP);
}
amazonS3ClientBuilder = amazonS3ClientBuilder.withClientConfiguration(clientConfiguration);
this.s3 = amazonS3ClientBuilder.build();
this.bucket = config.getBucket(); this.bucket = config.getBucket();
} }
@ -57,14 +91,9 @@ public class S3ResourceRepository extends BaseResourceRepository {
@Override @Override
public FineFileEntry getEntry(String path) { public FineFileEntry getEntry(String path) {
GetObjectRequest request = new GetObjectRequest(bucket, path);
try { try {
S3Object s3Object = s3.getObject(request); ObjectMetadata metadata = s3.getObjectMetadata(bucket, path);
try { return s3Object2FileEntry(metadata, path);
return s3Object2FileEntry(s3Object, path);
} finally {
s3Object.close();
}
} catch (Exception e) { } catch (Exception e) {
LogKit.info("{} not exist!", path); LogKit.info("{} not exist!", path);
} }
@ -81,6 +110,15 @@ public class S3ResourceRepository extends BaseResourceRepository {
ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucket) ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucket)
.withPrefix(dir).withDelimiter(DELIMITER); .withPrefix(dir).withDelimiter(DELIMITER);
ObjectListing objectListing = s3.listObjects(listObjectsRequest); ObjectListing objectListing = s3.listObjects(listObjectsRequest);
collectFileEntry(dir, result, objectListing);
while (objectListing.isTruncated()) {
objectListing = s3.listNextBatchOfObjects(objectListing);
collectFileEntry(dir, result, objectListing);
}
return result.toArray(new FineFileEntry[0]);
}
private void collectFileEntry(String dir, List<FineFileEntry> result, ObjectListing objectListing) {
for (S3ObjectSummary summary : objectListing.getObjectSummaries()) { for (S3ObjectSummary summary : objectListing.getObjectSummaries()) {
String key = summary.getKey(); String key = summary.getKey();
if (!key.endsWith(DELIMITER)) { if (!key.endsWith(DELIMITER)) {
@ -88,28 +126,12 @@ public class S3ResourceRepository extends BaseResourceRepository {
} }
} }
for (String prefix : objectListing.getCommonPrefixes()) { for (String prefix : objectListing.getCommonPrefixes()) {
FineFileEntry entry = new FineFileEntry(prefix); if (StringUtils.isNotEmpty(prefix.substring(dir.length()).replaceAll(DELIMITER, StringUtils.EMPTY))) {
entry.setDirectory(true); FineFileEntry entry = new FineFileEntry(prefix);
result.add(entry); entry.setDirectory(true);
result.add(entry);
}
} }
return result.toArray(new FineFileEntry[0]);
}
private FineFileEntry s3Object2FileEntry(S3ObjectSummary s3Object, String path) {
FineFileEntry entry = new FineFileEntry(path);
entry.setDirectory(s3Object.getKey().endsWith("/"));
entry.setSize(s3Object.getSize());
entry.setTimestamp(s3Object.getLastModified().getTime());
return entry;
}
private FineFileEntry s3Object2FileEntry(S3Object s3Object, String path) {
FineFileEntry entry = new FineFileEntry(path);
entry.setDirectory(s3Object.getKey().endsWith("/"));
entry.setSize(s3Object.getObjectMetadata().getContentLength());
entry.setTimestamp(s3Object.getObjectMetadata().getLastModified().getTime());
return entry;
} }
@Override @Override
@ -123,33 +145,104 @@ public class S3ResourceRepository extends BaseResourceRepository {
try { try {
return s3.getObject(request).getObjectContent(); return s3.getObject(request).getObjectContent();
} catch (Exception e) { } catch (Exception e) {
LogKit.error("[S3] Failed to read file {}", filePath);
LogKit.error(e.getMessage(), e);
return new ByteArrayInputStream(new byte[0]); return new ByteArrayInputStream(new byte[0]);
} }
} }
@Override @Override
public void write(String path, byte[] data) { public void write(String path, byte[] data) {
ObjectMetadata metadata; int length = data.length;
try { if (length > MULTIPART_UPLOAD_LIMIT) {
metadata = s3.getObjectMetadata(bucket, path); multipartUpload(path, new ByteArrayInputStream(data));
} catch (Exception e) { } else {
metadata = new ObjectMetadata(); ObjectMetadata metadata;
String mimeType = URLConnection.guessContentTypeFromName(path); try {
if (mimeType != null) { metadata = s3.getObjectMetadata(bucket, path);
metadata.setContentType(mimeType); } catch (Exception e) {
metadata = new ObjectMetadata();
String mimeType = URLConnection.guessContentTypeFromName(path);
if (mimeType != null) {
metadata.setContentType(mimeType);
}
} }
metadata.setContentLength(data.length); if (metadata != null) {
metadata.setContentLength(length);
}
s3.putObject(bucket, path, new ByteArrayInputStream(data), metadata);
} }
s3.putObject(bucket, path, new ByteArrayInputStream(data), metadata);
} }
@Override
public void write(String path, InputStream inputStream) throws ResourceIOException {
long dataLength = 0;
try {
dataLength = inputStream.available();
} catch (IOException e) {
LogKit.error(e.getMessage(), e);
}
//超过一定大小才使用分片上传,小文件来说,网络传输时间可能较短,且上传失败的风险相对较低。
//在网络稳定的情况下,使用分片上传可能没有太多的优势,反而增加了额外开销和复杂性
if (dataLength > MULTIPART_UPLOAD_LIMIT) {
multipartUpload(path, inputStream);
} else {
super.write(path, inputStream);
}
}
private void multipartUpload(String path, InputStream inputStream) {
try {
// Step 1: 初始化分片上传
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucket, path);
InitiateMultipartUploadResult initResponse = s3.initiateMultipartUpload(initRequest);
String uploadId = initResponse.getUploadId();
// Step 2: 分片上传文件
List<PartETag> partETags = new ArrayList<>();
byte[] buffer = new byte[PART_SIZE];
int bytesRead;
int partNumber = 1;
while ((bytesRead = inputStream.read(buffer)) > 0) {
// 创建上传请求
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucket)
.withKey(path)
.withUploadId(uploadId)
.withPartNumber(partNumber)
.withInputStream(new ByteArrayInputStream(buffer, 0, bytesRead))
.withPartSize(bytesRead);
// 上传分片
UploadPartResult uploadResult = s3.uploadPart(uploadRequest);
partETags.add(uploadResult.getPartETag());
partNumber++;
}
// Step 3: 完成分片上传
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucket, path, uploadId, partETags);
s3.completeMultipartUpload(compRequest);
} catch (IOException e) {
throw new ResourceIOException(e);
} finally {
ResourceIOUtils.close(inputStream);
}
}
@Override @Override
public boolean createFile(String path) { public boolean createFile(String path) {
PutObjectRequest request = new PutObjectRequest(bucket, path, new ByteArrayInputStream(new byte[0]), new ObjectMetadata());
String parent = path.substring(0, path.lastIndexOf(DELIMITER) + 1);
if (!dirExist(parent)) {
createDirectory(parent);
}
PutObjectRequest req = new PutObjectRequest(bucket, path, new ByteArrayInputStream(new byte[0]), buildEmptyMetadata());
try { try {
s3.putObject(request); s3.putObject(req);
} catch (Exception e) { } catch (Exception e) {
LogKit.error("[S3] Failed to create file path {}", path);
return false; return false;
} }
return true; return true;
@ -157,10 +250,25 @@ public class S3ResourceRepository extends BaseResourceRepository {
@Override @Override
public boolean createDirectory(String path) { public boolean createDirectory(String path) {
PutObjectRequest request = new PutObjectRequest(bucket, path + "/", new ByteArrayInputStream(new byte[0]), new ObjectMetadata());
if (!path.endsWith(DELIMITER)) {
path += DELIMITER;
}
String temp = path.substring(0, path.length() - 1);
String parent = temp.substring(0, temp.lastIndexOf(DELIMITER) + 1);
if (StringUtils.isEmpty(parent)) {
return true;
}
if (!dirExist(parent)) {
if (!createDirectory(parent)) {
return false;
}
}
PutObjectRequest req = new PutObjectRequest(bucket, path, new ByteArrayInputStream(new byte[0]), buildEmptyMetadata());
try { try {
s3.putObject(request); s3.putObject(req);
} catch (Exception e) { } catch (Exception e) {
LogKit.error("[S3] Failed to create parent path {}", path);
return false; return false;
} }
return true; return true;
@ -169,16 +277,54 @@ public class S3ResourceRepository extends BaseResourceRepository {
@Override @Override
public boolean delete(String path) { public boolean delete(String path) {
try { try {
s3.deleteObject(bucket, path); if (isDirectory(path)) {
if (!path.endsWith(DELIMITER)) {
path += DELIMITER;
}
deleteDirectory(path);
} else {
deleteFile(path);
}
} catch (Exception e) { } catch (Exception e) {
LogKit.error(e.getMessage(), e); LogKit.error("[S3] delete {} failed, error message: {}", path, e.getMessage());
return false; return false;
} }
return true; return true;
} }
private void deleteFile(String path) throws Exception {
s3.deleteObject(bucket, path);
}
private void deleteDirectory(String path) throws Exception {
List<String> files = new ArrayList<>();
for (FineFileEntry fineFileEntry : listEntry(path)) {
if (fineFileEntry.isDirectory()) {
deleteDirectory(fineFileEntry.getPath());
} else {
files.add(fineFileEntry.getPath());
}
}
deleteFiles(files);
deleteFile(path);
}
private void deleteFiles(List<String> paths) throws Exception {
//最多只能同时指定1000个key
for (int i = 0; i < paths.size(); i = i + PAGE_SIZE) {
int toIndex = Math.min(i + PAGE_SIZE, paths.size());
DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(bucket)
.withKeys(paths.subList(i, toIndex).toArray(new String[0]));
s3.deleteObjects(deleteObjectsRequest);
}
}
@Override @Override
public boolean exist(String path) { public boolean exist(String path) {
return fileExist(path) || (!path.endsWith(DELIMITER) && dirExist(path)) || isParentPathAbsent(path);
}
private boolean fileExist(String path) {
try { try {
return s3.doesObjectExist(bucket, path); return s3.doesObjectExist(bucket, path);
} catch (Exception e) { } catch (Exception e) {
@ -186,6 +332,38 @@ public class S3ResourceRepository extends BaseResourceRepository {
} }
} }
private boolean dirExist(String path) {
if (StringUtils.equals(path, DELIMITER)) {
return true;
}
if (!path.endsWith(DELIMITER)) {
path += DELIMITER;
}
return fileExist(path);
}
/**
* 如果存在文件创建了但是其父目录没有创建的场景为其递归创建对象返回创建结果
*/
private boolean isParentPathAbsent(String path) {
if (path.startsWith(DELIMITER)) {
path = path.substring(1);
}
if (path.endsWith(DELIMITER)) {
path = path.substring(0, path.length() - 1);
}
ObjectListing objectListing = s3.listObjects(
new ListObjectsRequest()
.withBucketName(bucket)
.withPrefix(path));
for (S3ObjectSummary summary : objectListing.getObjectSummaries()) {
if (summary.getKey().startsWith(path + DELIMITER)) {
return createDirectory(path);
}
}
return false;
}
@Override @Override
public String[] list(String dir, final Filter<String> filter) { public String[] list(String dir, final Filter<String> filter) {
List<String> result = new ArrayList<>(); List<String> result = new ArrayList<>();
@ -195,45 +373,46 @@ public class S3ResourceRepository extends BaseResourceRepository {
ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucket) ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucket)
.withPrefix(dir).withDelimiter(DELIMITER); .withPrefix(dir).withDelimiter(DELIMITER);
ObjectListing objectListing = s3.listObjects(listObjectsRequest); ObjectListing objectListing = s3.listObjects(listObjectsRequest);
collectFileName(dir, result, objectListing);
while (objectListing.isTruncated()) {
objectListing = s3.listNextBatchOfObjects(objectListing);
collectFileName(dir, result, objectListing);
}
if (filter != null) {
return result.stream().filter(filter::accept).toArray(String[]::new);
}
return result.toArray(new String[0]);
}
private void collectFileName(String dir, List<String> result, ObjectListing objectListing) {
for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) { for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
String[] arr = objectSummary.getKey().split(DELIMITER); String key = objectSummary.getKey();
String name = arr[arr.length - 1]; if (StringKit.equals(key, dir)) {
if (filter == null) { continue;
result.add(name);
} else {
if (filter.accept(name)) {
result.add(name);
}
} }
result.add(key.substring(key.lastIndexOf(DELIMITER) + 1));
} }
for (String prefix : objectListing.getCommonPrefixes()) { for (String prefix : objectListing.getCommonPrefixes()) {
String[] arr = prefix.split(DELIMITER); if (StringUtils.isNotEmpty(prefix.substring(dir.length()).replaceAll(DELIMITER, StringUtils.EMPTY))) {
String name = arr[arr.length - 1] + DELIMITER; String[] arr = prefix.split(DELIMITER);
if (filter == null) { String name = arr[arr.length - 1] + DELIMITER;
result.add(name); result.add(name);
} else {
if (filter.accept(name)) {
result.add(name);
}
} }
} }
return result.toArray(new String[0]);
} }
@Override @Override
public boolean isDirectory(String path) { public boolean isDirectory(String path) {
if (path.endsWith(DELIMITER)) { if (path.endsWith(DELIMITER)) {
return true; return exist(path);
}
ObjectListing listing = s3.listObjects(bucket, path);
if (listing.getObjectSummaries().isEmpty()) {
return false;
}
if (listing.getObjectSummaries().size() > 1) {
return true;
} else { } else {
S3ObjectSummary summary = listing.getObjectSummaries().get(0); ObjectListing listing = s3.listObjects(bucket, path);
return !StringKit.equals(listing.getPrefix(), summary.getKey()); List<S3ObjectSummary> objectSummaries = listing.getObjectSummaries();
if (objectSummaries.isEmpty()) {
return false;
}
String dirFormat = path + DELIMITER;
return objectSummaries.stream().anyMatch(s3ObjectSummary -> StringUtils.equals(s3ObjectSummary.getKey(), dirFormat));
} }
} }
@ -243,6 +422,8 @@ public class S3ResourceRepository extends BaseResourceRepository {
S3Object s3Object = s3.getObject(bucket, path); S3Object s3Object = s3.getObject(bucket, path);
if (s3Object != null) { if (s3Object != null) {
try { try {
//s3Object要全部读完,否则会有警告
IOUtils.copy(s3Object.getObjectContent(), new NullOutputStream());
return s3Object.getObjectMetadata().getLastModified().getTime(); return s3Object.getObjectMetadata().getLastModified().getTime();
} finally { } finally {
s3Object.close(); s3Object.close();
@ -258,14 +439,8 @@ public class S3ResourceRepository extends BaseResourceRepository {
@Override @Override
public long length(String path) { public long length(String path) {
try { try {
S3Object s3Object = s3.getObject(bucket, path); ObjectMetadata metadata = s3.getObjectMetadata(bucket, path);
if (s3Object != null) { return metadata.getContentLength();
try {
return s3Object.getObjectMetadata().getContentLength();
} finally {
s3Object.close();
}
}
} catch (Exception e) { } catch (Exception e) {
LogKit.info("{} not exist!", path); LogKit.info("{} not exist!", path);
} }
@ -275,16 +450,21 @@ public class S3ResourceRepository extends BaseResourceRepository {
@Override @Override
public boolean rename(String path, String newPath) throws ResourceIOException { public boolean rename(String path, String newPath) throws ResourceIOException {
try {
CopyObjectRequest copyObjRequest = new CopyObjectRequest(bucket, if (copy(path, newPath) && delete(path)) {
path, bucket, newPath); if (LogKit.isDebugEnabled()) {
s3.copyObject(copyObjRequest); LogKit.debug("[S3] rename {} to {} success.", path, newPath);
s3.deleteObject(new DeleteObjectRequest(bucket, path)); }
} catch (Exception e) { return true;
LogKit.error(e.getMessage(), e);
return false;
} }
return true; LogKit.error("[S3] rename {} to {} failed.", path, newPath);
return false;
}
@Override
public boolean copyFile(String origPath, String desPath) throws ResourceIOException {
s3.copyObject(bucket, origPath, bucket, desPath);
return exist(desPath);
} }
@Override @Override
@ -297,4 +477,30 @@ public class S3ResourceRepository extends BaseResourceRepository {
return S3RepositoryFactory.IDENTITY; return S3RepositoryFactory.IDENTITY;
} }
/*--------------------------------------私有方法-----------------------------------------**/
private ObjectMetadata buildEmptyMetadata() {
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(0);
return metadata;
}
private FineFileEntry s3Object2FileEntry(S3ObjectSummary s3Object, String path) {
FineFileEntry entry = new FineFileEntry(path);
entry.setDirectory(s3Object.getKey().endsWith(DELIMITER));
entry.setSize(s3Object.getSize());
entry.setTimestamp(s3Object.getLastModified().getTime());
return entry;
}
private FineFileEntry s3Object2FileEntry(ObjectMetadata metadata, String path) {
FineFileEntry entry = new FineFileEntry(path);
entry.setDirectory(path.endsWith(DELIMITER));
entry.setSize(metadata.getContentLength());
entry.setTimestamp(metadata.getLastModified().getTime());
return entry;
}
} }

5
src/main/resources/com/fanruan/fs/s3/repository/locale/s3.properties

@ -3,4 +3,7 @@ Plugin-S3_End_Point=Endpoint
Plugin-S3_Region=Region Plugin-S3_Region=Region
Plugin-S3_Access_Key_Id=AccessKeyId Plugin-S3_Access_Key_Id=AccessKeyId
Plugin-S3_Access_Key_Secret=AccessKeySecret Plugin-S3_Access_Key_Secret=AccessKeySecret
Plugin-S3_Bucket=Bucket Plugin-S3_Bucket=Bucket
Dec-Error_Start_With_Slash_Or_End_Without_Slash=The path cannot start with "/", but must end with "/"
Plugin-S3_Other_Config=
Plugin-S3_EnablePathStyleAccess_Error_Tip=

11
src/main/resources/com/fanruan/fs/s3/repository/locale/s3_en_US.properties

@ -1,6 +1,9 @@
Plugin-S3_Input=Please Input Dec-Error_Start_With_Slash_Or_End_Without_Slash=The path cannot start with "/", but must end with "/"
Plugin-S3_End_Point=Endpoint
Plugin-S3_Region=Region
Plugin-S3_Access_Key_Id=AccessKeyId Plugin-S3_Access_Key_Id=AccessKeyId
Plugin-S3_Access_Key_Secret=AccessKeySecret Plugin-S3_Access_Key_Secret=AccessKeySecret
Plugin-S3_Bucket=Bucket Plugin-S3_Bucket=Bucket
Plugin-S3_End_Point=Endpoint
Plugin-S3_Input=Please input
Plugin-S3_Region=Region
Plugin-S3_Other_Config=
Plugin-S3_EnablePathStyleAccess_Error_Tip=

9
src/main/resources/com/fanruan/fs/s3/repository/locale/s3_ja_JP.properties

@ -0,0 +1,9 @@
Dec-Error_Start_With_Slash_Or_End_Without_Slash=\u30D1\u30B9\u306F\u300C/\u300D\u3067\u59CB\u307E\u308B\u3053\u3068\u304C\u3067\u304D\u306A\u3044\u3002\u300C/\u300D\u3067\u7D42\u308F\u308B\u5FC5\u8981\u304C\u3042\u308B\u3002
Plugin-S3_Access_Key_Id=AccessKeyId
Plugin-S3_Access_Key_Secret=AccessKeySecret
Plugin-S3_Bucket=Bucket
Plugin-S3_End_Point=Endpoint
Plugin-S3_Input=\u5165\u529B\u3057\u3066\u304F\u3060\u3055\u3044\u3002
Plugin-S3_Region=Region
Plugin-S3_Other_Config=
Plugin-S3_EnablePathStyleAccess_Error_Tip=

9
src/main/resources/com/fanruan/fs/s3/repository/locale/s3_ko_KR.properties

@ -0,0 +1,9 @@
Dec-Error_Start_With_Slash_Or_End_Without_Slash=
Plugin-S3_Access_Key_Id=
Plugin-S3_Access_Key_Secret=
Plugin-S3_Bucket=
Plugin-S3_End_Point=
Plugin-S3_Input=
Plugin-S3_Region=
Plugin-S3_Other_Config=
Plugin-S3_EnablePathStyleAccess_Error_Tip=

11
src/main/resources/com/fanruan/fs/s3/repository/locale/s3_zh_CN.properties

@ -1,6 +1,9 @@
Plugin-S3_Input=\u8BF7\u8F93\u5165 Dec-Error_Start_With_Slash_Or_End_Without_Slash=\u8DEF\u5F84\u5F00\u5934\u4E0D\u80FD\u52A0\u659C\u6760\uFF0C\u5FC5\u987B\u4EE5\u659C\u6760\u7ED3\u5C3E
Plugin-S3_End_Point=Endpoint
Plugin-S3_Region=Region
Plugin-S3_Access_Key_Id=AccessKeyId Plugin-S3_Access_Key_Id=AccessKeyId
Plugin-S3_Access_Key_Secret=AccessKeySecret Plugin-S3_Access_Key_Secret=AccessKeySecret
Plugin-S3_Bucket=Bucket Plugin-S3_Bucket=Bucket
Plugin-S3_End_Point=Endpoint
Plugin-S3_Input=\u8BF7\u8F93\u5165
Plugin-S3_Region=Region
Plugin-S3_Other_Config=\u66f4\u591a\u8bbe\u7f6e
Plugin-S3_EnablePathStyleAccess_Error_Tip=\u8bf7\u8f93\u5165true\u6216false

9
src/main/resources/com/fanruan/fs/s3/repository/locale/s3_zh_TW.properties

@ -0,0 +1,9 @@
Dec-Error_Start_With_Slash_Or_End_Without_Slash=\u8DEF\u5F91\u958B\u982D\u4E0D\u80FD\u52A0\u659C\u69D3\uFF0C\u5FC5\u9808\u4EE5\u659C\u69D3\u7D50\u5C3E
Plugin-S3_Access_Key_Id=AccessKeyId
Plugin-S3_Access_Key_Secret=AccessKeySecret
Plugin-S3_Bucket=Bucket
Plugin-S3_End_Point=Endpoint
Plugin-S3_Input=\u8ACB\u8F38\u5165
Plugin-S3_Region=Region
Plugin-S3_Other_Config=
Plugin-S3_EnablePathStyleAccess_Error_Tip=

192
src/main/resources/com/fanruan/fs/s3/repository/web/js/bundle.js

File diff suppressed because one or more lines are too long
Loading…
Cancel
Save