v3.0.0 #1299
kba
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Changed:
ocrd) level fromINFOtoWARNINGinitLogging: do not remove any previous handlers/levels, unlessforce_reinitdisableLogging: remove all handlers, reset all levels - instead of being selectiveweakrefwith__del__to triggershutdownOCRD_MAX_PARALLEL_PAGES>1: log viaQueueHandlerin subprocess,QueueListenerin mainocrd_utils.initLogging: also add handler to root logger (as in file config),but disable message propagation to avoid duplication
ocrd_networkinsrc/ocrd/decorators/__init__.pyonce neededProcessor.process_page_file: skip computingprocess_page_pcgtsif output already exists,but
OCRD_EXISTING_OUTPUT!=OVERWRITEOCRD_MAX_PARALLEL_PAGES>1: switch from multithreading to multiprocessing, depend onlokyinstead of stdlibconcurrent.futuresOCRD_PROCESSING_PAGE_TIMEOUT>0: actually enforce timeout within workerOCRD_MAX_MISSING_OUTPUTS>0: abort early if too many failures already, prospectivelyProcessor.process_workspace: split up into overridable sub-methods:process_workspace_submit_tasks(iterate input file group and schedule page tasks)process_workspace_submit_page_task(download input files and submit single page task)process_workspace_handle_tasks(monitor page tasks and aggregate results)process_workspace_handle_page_task(await single page task and handle errors)Processor/Workspace.add_file: alwaysforceifOCRD_EXISTING_OUTPUT==OVERWRITEProcessor.verify: revert 3.0.0b1 enforcing cardinality checks (stay backwards compatible)Processor.verify: check output fileGrps, too(must not exist unless
OCRD_EXISTING_OUTPUT=OVERWRITE|SKIPor disjoint--page-idrange)input-files: do not try to validate tasks here (now covered byProcessor.verify())run_processor: be robust ifocrd_toolis missingstepsPcGtsType.PageType.idviamake_xml_id: replace/with_ocrd_utils,ocrd_models,ocrd_modelfactory,ocrd_validatorsandocrd_networkare not publishedas separate packages anymore, everything is contained in
ocrd- you should adapt yourrequirements.txtaccordinglyProcessor.parameternow a property (attribute always exists, butNonefor non-processing contexts)Processor.parameteris now afrozendict(contents immutable)Processor.parametervalidate when(ever) set instead of (just) the constructorProcessor.parameterwill also trigger (Processor.shutdown() and) Processor.setup()get_processor(... instance_caching=True): usemin(max_instances, OCRD_MAX_PROCESSOR_CACHE)Processor.verifyalways validates fileGrp cardinalities (because we haveocrd-tool.jsondefaults now)OcrdMets.add_agentwithout positional argumentsocrd bashlib input-filesnow uses normal Processor decorator, and gets passed actualocrd-tool.jsonand tool namefrom bashlib's
ocrd__wrapOcrdPageas proxy ofPcGtsTypeinstead of alias; also containsetreeandmappingnowpage_from_file: removed kwargwith_tree- useOcrdPage.etreeandOcrdPage.mappinginsteadProcessor.zip_input_filesnow can throwocrd.NonUniqueInputFileandocrd.MissingInputFile(the latter only if
OCRD_MISSING_INPUT=ABORT)Processor.zip_input_filesdoes not by default userequire_firstanymore(so the first file in any input file tuple per page can be
Noneas well)Workspace.overwrite_mode, merely delegate toOCRD_EXISTING_OUTPUT=OVERWRITEocrd_utils.configProcessor.processocrd-tool.jsonProcessorconstructor, add as members(i.e.
show_help,dump_json,dump_module_dir,list_resources,show_resource,resolve_resource)Processorconstructor(i.e.
workspace,page_id,input_file_grp,output_file_grp; now all set byrun_processor)ocrd-tool.jsonmetadata toProcessorconstructorocrd.processor: Handle loading of bundledocrd-tool.jsongenericallyFixed:
ocrd --helpoutput was broken for multiline config options, fix help output for multi-line config option descriptions bertsky/core#25initLoggingbefore instantiating processors inocrd_cli_wrap_processor, ocrd_cli_wrap_processor: always do initLogging bertsky/core#24, ocrd_cli_wrap_processor: always do initLogging #1296initLogging: only add root handler instead of multiple redundant handlers withpropagate=falsesetOverrideLogLevel: override all currently active loggers' levelOcrdMets.get_physical_pages: coverreturn_divsw/ofor_fileIdsandfor_pageIdsocrd_utils.configgets reset whenever changing it globallyOcrdMetsServer.add_file: pass onforcekwargocrd.cli.workspace: consistently pass on--mets-server-urland--backupocrd.cli.validate "tasks": pass on--mets-server-urlocrd.cli.bashlib "input-files": pass on--mets-server-urllib.bash input-files: pass on--mets-server-url,--overwrite, and parameterslib.bash: fixerrexithandlingocrd.cli.ocrd-tool "resolve-resource": forgot to actually print resultProcessor.metadata_location:srcworkaround respects namespace packages, OCR-D v3 API: fixes qurator-spk/eynollah#134Workspace.reload_mets: handle ClientSideOcrdMets as welldisableLogging: also re-instate root logger to Python defaults--log-filename, and show in--helpocrd workspace clone: do pass on--file-grp(for download filtering)Added:
ocrd-filterprocessor to remove segments based on XPath expressions, v3 API: general XPath 2.0 mechanism, generateDS true reverse mapping, ocrd-filter bertsky/core#21pc:pixelareafor the number of pixels of the bounding box (or sum area on node sets), v3 API: general XPath 2.0 mechanism, generateDS true reverse mapping, ocrd-filter bertsky/core#21pc:textequivfor the first TextEquiv unicode string (or concatenated string on node sets), v3 API: general XPath 2.0 mechanism, generateDS true reverse mapping, ocrd-filter bertsky/core#21OcrdPage: newPageType.get_ReadingOrderGroups()to retrieve recursive RO as dictserver: add subcommandsreloadandsavephysical_pages--resolve-resource, tooProcessor.process_page_file/OcrdPageResultImage: allowNonebesidesAlternativeImageTypeOcrdConfig.reset_defaultsto reset config variables to their defaultsProcessor.max_workers: class attribute to control per-page parallelism of this implementationProcessor.max_page_seconds: class attribute to control per-page timeout of this implementationOCRD_MAX_PARALLEL_PAGESfor whether and how many workers should process pages in parallelOCRD_PROCESSING_PAGE_TIMEOUTfor whether and how long processors should wait for single pagesOCRD_MAX_MISSING_OUTPUTSfor maximum rate (fraction) of pages before makingOCRD_MISSING_OUTPUT=abortProcessor.metadata_filename: expose to make local path ofocrd-tool.jsonin Python distribution reusable+overridableProcessor.metadata_location: expose to make absolute path ofocrd-tool.jsonreusable+overridableProcessor.metadata_rawdict: expose to make in-memory contents ofocrd-tool.jsonreusable+overridableProcessor.metadata: expose to make validated and default-expanded contents ofocrd-tool.jsonreusable+overridableProcessor.shutdown: to shut down processor after processing, optionalProcessor.max_instances: class attribute to control instance caching of this implementationOCRD_DOWNLOAD_INPUTfor whether input files should be downloaded before processingOCRD_MISSING_INPUTfor how to handle missing input files (SKIPorABORT)OCRD_MISSING_OUTPUTfor how to handle processing failures (SKIPorABORTorCOPY)the latter behaves like ocrd-dummy for the failed page(s)
OCRD_EXISTING_OUTPUTfor how to handle existing output files (SKIPorABORTorOVERWRITE)--debugas short-hand forABORTchoices aboveProcessor.loggerset up by constructor already (for re-use by processor implementors)default-expand and validateocrd_tool.jsoninProcessorconstructor, log invaliditiesdeprecationinocrd_tool.jsonby reporting warningsProcessor.process_workspace: process a complete workspace, with default implementationProcessor.process_page_file: process an OcrdFile, with default implementationProcessor.process_page_pcgts: process a single OcrdPage, produce a single OcrdPage, required to implementProcessor.verify: handle fileGrp cardinality verification, with default implementationProcessor.setup: to set up processor before processing, optionalThis discussion was created from the release v3.0.0.
Beta Was this translation helpful? Give feedback.
All reactions