feat(dataset): 原生支持 OFD 文档解析与解析失败诊断错误码 - #7806
sunjiaqi52777 wants to merge 3 commits into
Conversation
|
建议这里超过 2000 页时直接拒绝解析,而不是 |
|
补充几个需要合并前处理的风险点:
|
|
关于 ZIP/XML 资源风险,建议按仓库现有 Office 解析器的方式落地,参考
这样可以同时防 ZIP bomb、伪造解压大小和超大页面 XML;现有的 2000 页限制则继续单独处理为直接拒绝。 |
- add .ofd to the built-in document extension whitelist - add worker OFD parser (zip+xml text extraction with layout heuristics: Y grouping, title detection, page-number filtering); match elements by localName for vendor namespace compatibility; reject encrypted files; unknown failures fall back to the shared parse-failure diagnosis code Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- register 6 parse failure codes in CommonErrEnum with 4-locale i18n - map Sangfor provider failures to UserError statusText (raw details log-only) and anydoc ConvertErrorCode to diagnosis codes; unknown codes fall back to pdfParseFailed instead of leaking raw errors - map docx mammoth failures to docxConversionFailed - unify unknown local pipeline failures to UserError(pdfParseFailed); worker-mapped diagnosis errors pass through untouched Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Replace JSZip full-materialize with yauzl streaming and enforce entry count / per-XML / cumulative XML size limits (align with parseOffice) - Reject when declared pages exceed limit instead of silently truncating - Map missing page resources and malformed XML to invalidParseFile - Only demote base font size when page-number evidence exists, so titles in no-page-number documents are recognized - Keep mapped diagnosis codes across the worker boundary: worker errors lose the Error subclass and its `name`, so match the preserved `message` against the accepted status-text set instead of checking `error.name` - Drop the docx-specific failure code so docx keeps its original "convert to PDF" hint - Align sangfor provider tests with parse diagnosis codes - Dedupe imports in read/utils.ts Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1024d56 to
16b8276
Compare
|
感谢贡献!关于解析错误处理有两点小建议:
|
|
@c121914yu 余总,docxParseInvalid/docxConversionFailed 这两个码是 sangfor外部解析服务契约,外部服务解析 docx 失败时可能返回。删除后服务返回这两个码会兜底成 pdfParseFailed,麻烦看下能否保留 |
那加个注释 |
.ofd加入内置文档扩展名白名单,前端上传与后端校验自动放行