首页 > 其他分享 >SpringCloud实现浏览器端大文件分片上传

SpringCloud实现浏览器端大文件分片上传

时间:2023-07-12 19:00:37浏览次数:37  
标签:UUID 端大 SpringCloud request process 源码 分片 UploadServletParameter fileUploaderHel

 1,项目调研

 

因为需要研究下断点上传的问题。找了很久终于找到一个比较好的项目。

 

在GoogleCode上面,代码弄下来超级不方便,还是配置hosts才好,把代码重新上传到了github上面。

 

https://github.com/freewebsys/java-large-file-uploader-demo

 

效果:

 

上传中,显示进度,时间,百分比。

 

编辑

点击【Pause】暂停,点击【Resume】继续。

 

编辑

2,代码分析

原始项目:

 

https://code.google.com/p/java-large-file-uploader/

 

这个项目最后更新的时间是 2012 年,项目进行了封装使用最简单的方法实现了http的断点上传。

 

因为html5 里面有读取文件分割文件的类库,所以才可以支持断点上传,所以这个只能在html5 支持的浏览器上面展示。

 

同时,在js 和 java 同时使用 cr32 进行文件块的校验,保证数据上传正确。

 

代码在使用了最新的servlet 3.0 的api,使用了异步执行,监听等方法。

 

上传类UploadServlet

@Component("javaLargeFileUploaderServlet")

@WebServlet(name = "javaLargeFileUploaderServlet", urlPatterns = { "/javaLargeFileUploaderServlet" })

public class UploadServlet extends HttpRequestHandlerServlet

        implements HttpRequestHandler {

  

    private static final Logger log = LoggerFactory.getLogger(UploadServlet.class);

  

    @Autowired

    UploadProcessor uploadProcessor;

  

    @Autowired

    FileUploaderHelper fileUploaderHelper;

  

    @Autowired

    ExceptionCodeMappingHelper exceptionCodeMappingHelper;

  

    @Autowired

    Authorizer authorizer;

  

    @Autowired

    StaticStateIdentifierManager staticStateIdentifierManager;

  

  

  

    @Override

    public void handleRequest(HttpServletRequest request, HttpServletResponse response)

            throws IOException {

        log.trace("Handling request");

  

        Serializable jsonObject = null;

        try {

            // extract the action from the request

            UploadServletAction actionByParameterName =

                    UploadServletAction.valueOf(fileUploaderHelper.getParameterValue(request, UploadServletParameter.action));

  

            // check authorization

            checkAuthorization(request, actionByParameterName);

  

            // then process the asked action

            jsonObject = processAction(actionByParameterName, request);

  

  

            // if something has to be written to the response

            if (jsonObject != null) {

                fileUploaderHelper.writeToResponse(jsonObject, response);

            }

  

        }

        // If exception, write it

        catch (Exception e) {

            exceptionCodeMappingHelper.processException(e, response);

        }

  

    }

  

  

    private void checkAuthorization(HttpServletRequest request, UploadServletAction actionByParameterName)

            throws MissingParameterException, AuthorizationException {

  

        // check authorization

        // if its not get progress (because we do not really care about authorization for get

        // progress and it uses an array of file ids)

        if (!actionByParameterName.equals(UploadServletAction.getProgress)) {

  

            // extract uuid

            final String fileIdFieldValue = fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId, false);

  

            // if this is init, the identifier is the one in parameter

            UUID clientOrJobId;

            String parameter = fileUploaderHelper.getParameterValue(request, UploadServletParameter.clientId, false);

            if (actionByParameterName.equals(UploadServletAction.getConfig) && parameter != null) {

                clientOrJobId = UUID.fromString(parameter);

            }

            // if not, get it from manager

            else {

                clientOrJobId = staticStateIdentifierManager.getIdentifier();

            }

  

              

            // call authorizer

            authorizer.getAuthorization(

                    request,

                    actionByParameterName,

                    clientOrJobId,

                    fileIdFieldValue != null ? getFileIdsFromString(fileIdFieldValue).toArray(new UUID[] {}) : null);

  

        }

    }

  

  

    private Serializable processAction(UploadServletAction actionByParameterName, HttpServletRequest request)

            throws Exception {

        log.debug("Processing action " + actionByParameterName.name());

  

        Serializable returnObject = null;

        switch (actionByParameterName) {

            case getConfig:

                String parameterValue = fileUploaderHelper.getParameterValue(request, UploadServletParameter.clientId, false);

                returnObject =

                        uploadProcessor.getConfig(

                                parameterValue != null ? UUID.fromString(parameterValue) : null);

                break;

            case verifyCrcOfUncheckedPart:

                returnObject = verifyCrcOfUncheckedPart(request);

                break;

            case prepareUpload:

                returnObject = prepareUpload(request);

                break;

            case clearFile:

                uploadProcessor.clearFile(UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId)));

                break;

            case clearAll:

                uploadProcessor.clearAll();

                break;

            case pauseFile:

                List<UUID> uuids = getFileIdsFromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId));

                uploadProcessor.pauseFile(uuids);

                break;

            case resumeFile:

                returnObject =

                        uploadProcessor.resumeFile(UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId)));

                break;

            case setRate:

                uploadProcessor.setUploadRate(UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId)),

                        Long.valueOf(fileUploaderHelper.getParameterValue(request, UploadServletParameter.rate)));

                break;

            case getProgress:

                returnObject = getProgress(request);

                break;

        }

        return returnObject;

    }

  

  

    List<UUID> getFileIdsFromString(String fileIds) {

        String[] splittedFileIds = fileIds.split(",");

        List<UUID> uuids = Lists.newArrayList();

        for (int i = 0; i < splittedFileIds.length; i++) {

            uuids.add(UUID.fromString(splittedFileIds[i]));

        } 

        return uuids;

    }

  

  

    private Serializable getProgress(HttpServletRequest request)

            throws MissingParameterException {

        Serializable returnObject;

        String[] ids =

                new Gson()

                        .fromJson(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId), String[].class);

        Collection<UUID> uuids = Collections2.transform(Arrays.asList(ids), new Function<String, UUID>() {

  

            @Override

            public UUID apply(String input) {

                return UUID.fromString(input);

            }

  

        });

        returnObject = Maps.newHashMap();

        for (UUID fileId : uuids) {

            try {

                ProgressJson progress = uploadProcessor.getProgress(fileId);

                ((HashMap<String, ProgressJson>) returnObject).put(fileId.toString(), progress);

            }

            catch (FileNotFoundException e) {

                log.debug("No progress will be retrieved for " + fileId + " because " + e.getMessage());

            }

        }

        return returnObject;

    }

  

  

    private Serializable prepareUpload(HttpServletRequest request)

            throws MissingParameterException, IOException {

  

        // extract file information

        PrepareUploadJson[] fromJson =

                new Gson()

                        .fromJson(fileUploaderHelper.getParameterValue(request, UploadServletParameter.newFiles), PrepareUploadJson[].class);

  

        // prepare them

        final HashMap<String, UUID> prepareUpload = uploadProcessor.prepareUpload(fromJson);

  

        // return them

        return Maps.newHashMap(Maps.transformValues(prepareUpload, new Function<UUID, String>() {

  

            public String apply(UUID input) {

                return input.toString();

            };

        }));

    }

  

  

    private Boolean verifyCrcOfUncheckedPart(HttpServletRequest request)

            throws IOException, MissingParameterException, FileCorruptedException, FileStillProcessingException {

        UUID fileId = UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId));

        try {

            uploadProcessor.verifyCrcOfUncheckedPart(fileId,

                    fileUploaderHelper.getParameterValue(request, UploadServletParameter.crc));

        }

        catch (InvalidCrcException e) {

            // no need to log this exception, a fallback behaviour is defined in the

            // throwing method.

            // but we need to return something!

            return Boolean.FALSE;

        }

        return Boolean.TRUE;

    }

}

 

异步上传UploadServletAsync

 

@Component("javaLargeFileUploaderAsyncServlet")

@WebServlet(name = "javaLargeFileUploaderAsyncServlet", urlPatterns = { "/javaLargeFileUploaderAsyncServlet" }, asyncSupported = true)

public class UploadServletAsync extends HttpRequestHandlerServlet

        implements HttpRequestHandler {

  

    private static final Logger log = LoggerFactory.getLogger(UploadServletAsync.class);

  

    @Autowired

    ExceptionCodeMappingHelper exceptionCodeMappingHelper;

  

    @Autowired

    UploadServletAsyncProcessor uploadServletAsyncProcessor;

      

    @Autowired

    StaticStateIdentifierManager staticStateIdentifierManager;

  

    @Autowired

    StaticStateManager<StaticStatePersistedOnFileSystemEntity> staticStateManager;

   

    @Autowired

    FileUploaderHelper fileUploaderHelper;

  

    @Autowired

    Authorizer authorizer;

  

    /**

     * Maximum time that a streaming request can take.<br>

     */

    private long taskTimeOut = DateUtils.MILLIS_PER_HOUR;

  

  

    @Override

    public void handleRequest(final HttpServletRequest request, final HttpServletResponse response)

            throws ServletException, IOException {

  

        // process the request

        try {

  

            //check if uploads are allowed

            if (!uploadServletAsyncProcessor.isEnabled()) {

                throw new UploadIsCurrentlyDisabled();

            }

              

            // extract stuff from request

            final FileUploadConfiguration process = fileUploaderHelper.extractFileUploadConfiguration(request);

  

            log.debug("received upload request with config: "+process);

  

            // verify authorization

            final UUID clientId = staticStateIdentifierManager.getIdentifier();

            authorizer.getAuthorization(request, UploadServletAction.upload, clientId, process.getFileId());

  

            //check if that file is not paused

            if (uploadServletAsyncProcessor.isFilePaused(process.getFileId())) {

                log.debug("file "+process.getFileId()+" is paused, ignoring async request.");

                return;

            }

              

            // get the model

            StaticFileState fileState = staticStateManager.getEntityIfPresent().getFileStates().get(process.getFileId());

            if (fileState == null) {

                throw new FileNotFoundException("File with id " + process.getFileId() + " not found");

            }

  

            // process the request asynchronously

            final AsyncContext asyncContext = request.startAsync();

            asyncContext.setTimeout(taskTimeOut);

  

  

            // add a listener to clear bucket and close inputstream when process is complete or

            // with

            // error

            asyncContext.addListener(new UploadServletAsyncListenerAdapter(process.getFileId()) {

  

                @Override

                void clean() {

                    log.debug("request " + request + " completed.");

                    // we do not need to clear the inputstream here.

                    // and tell processor to clean its shit!

                    uploadServletAsyncProcessor.clean(clientId, process.getFileId());

                }

            });

  

            // then process

            uploadServletAsyncProcessor.process(fileState, process.getFileId(), process.getCrc(), process.getInputStream(),

                    new WriteChunkCompletionListener() {

   

                        @Override

                        public void success() {

                            asyncContext.complete();

                        }

  

  

                        @Override

                        public void error(Exception exception) {

                            // handles a stream ended unexpectedly , it just means the user has

                            // stopped the

                            // stream

                            if (exception.getMessage() != null) {

                                if (exception.getMessage().equals("Stream ended unexpectedly")) {

                                    log.warn("User has stopped streaming for file " + process.getFileId());

                                }

                                else if (exception.getMessage().equals("User cancellation")) {

                                    log.warn("User has cancelled streaming for file id " + process.getFileId());

                                    // do nothing

                                }

                                else {

                                    exceptionCodeMappingHelper.processException(exception, response);

                                }

                            }

                            else {

                                exceptionCodeMappingHelper.processException(exception, response);

                            }

  

                            asyncContext.complete();

                        }

  

                    });

        }

        catch (Exception e) {

            exceptionCodeMappingHelper.processException(e, response);

        }

  

    }

  

}

 

 

3,请求流程图:

 

主要思路就是将文件切分,然后分块上传。

编辑

 

效果展示:

​编辑

​编辑

​编辑

​编辑

 

视频演示:

 

windows控件安装,,linux-deb控件包安装,linux-rpm控件包安装,php7测试,php5测试,vue-cli-测试,asp.net-IIS测试,asp.net-阿里云(oss)测试,asp.net-华为云(obs)测试,jsp-springboot测试,ActiveX(x86)源码编译,ActiveX(x64)源码编译,Windows(npapi)源码编译,macOS源码编译,Linux(x86_64)源码编译,Linux(arm)源码编译,Linux(mips-uos)源码编译,Linux(mips-kylin-涉密环境)源码编译,sm4加密传输,压缩传输,

示例下载地址

源代码文档

asp.net源码下载jsp-springboot源码下载jsp-eclipse源码下载jsp-myeclipse源码下载php源码下载csharp-winform源码下载vue-cli源码下载c++源码下载

详细配置信息及思路

标签:UUID,端大,SpringCloud,request,process,源码,分片,UploadServletParameter,fileUploaderHel
From: https://www.cnblogs.com/songsu/p/17548560.html

相关文章

  • jsp实现浏览器端大文件分片上传
    ​ 在web项目中上传文件夹现在已经成为了一个主流的需求。在OA,或者企业ERP系统中都有类似的需求。上传文件夹并且保留层级结构能够对用户行成很好的引导,用户使用起来也更方便。能够提供更高级的应用支撑。数据表结构文件​编辑文件夹数据表结构​编辑文件数据表结构......
  • Java实现浏览器端大文件分片上传解决方案
    ​ 上周遇到这样一个问题,客户上传高清视频(1G以上)的时候上传失败。一开始以为是session过期或者文件大小受系统限制,导致的错误。查看了系统的配置文件没有看到文件大小限制,web.xml中seesiontimeout是30,我把它改成了120。但还是不行,有时候10分钟就崩了。同事说,可能是客户这里......
  • Java实现浏览器端大文件分片上传功能
    ​ 第一点:Java代码实现文件上传FormFilefile=manform.getFile();StringnewfileName= null;Stringnewpathname= null;StringfileAddre= "/numUp";try{    InputStreamstream=file.getInputStream();// 把文件读入    StringfilePath=request.......
  • Java实现浏览器端大文件分片上传方案
    ​ 这里只写后端的代码,基本的思想就是,前端将文件分片,然后每次访问上传接口的时候,向后端传入参数:当前为第几块文件,和分片总数下面直接贴代码吧,一些难懂的我大部分都加上注释了:上传文件实体类:看得出来,实体类中已经有很多我们需要的功能了,还有实用的属性。如MD5秒传的信息。pub......
  • Java实现浏览器端大文件分片上传技术
    ​ javaweb上传文件上传文件的jsp中的部分上传文件同样可以使用form表单向后端发请求,也可以使用ajax向后端发请求    1.通过form表单向后端发送请求         <formid="postForm"action="${pageContext.request.contextPath}/UploadServlet"method="post"e......
  • Java实现浏览器端大文件分片上传实例解析
    ​ 我们平时经常做的是上传文件,上传文件夹与上传文件类似,但也有一些不同之处,这次做了上传文件夹就记录下以备后用。首先我们需要了解的是上传文件三要素:1.表单提交方式:post(get方式提交有大小限制,post没有)2.表单的enctype属性:必须设置为multipart/form-data.3.表单必须......
  • springcloud -分布式事务解决方案 seata 分布式id生成方案
     使用三个服务来进行演示三个服务的共同部分 pom相关依赖 <!--nacos--> <dependency>   <groupId>com.alibaba.cloud</groupId>   <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId> </dependency> <!--seata-->......
  • springcloud alibaba -sentinel 配置持久化(datasource -nacos)
    当我们对sentinel进行规则配置之后如果关闭服务在重新启动会发现配置的服务消失了,这样很不方便的,我们需要将它持久化,使用nacos对其进行持久化1.导入依赖让sentinel和nacos产生关系<!--SpringCloudailibabasentinel-datasource-nacos--><dependency><groupId>com.alib......
  • springcloud -sentinel 用户自定义限流错误处理(仅限限流异常,其他异常请使用fallback属
    pom依赖<!--SpringCloudailibabanacos--><dependency><groupId>com.alibaba.cloud</groupId><artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId></dependency>......
  • Java实现浏览器端大文件分片上传实例
    ​ 上周遇到这样一个问题,客户上传高清视频(1G以上)的时候上传失败。一开始以为是session过期或者文件大小受系统限制,导致的错误。查看了系统的配置文件没有看到文件大小限制,web.xml中seesiontimeout是30,我把它改成了120。但还是不行,有时候10分钟就崩了。同事说,可能是客户这里......