MetadataWorkflow
OpenMetadata Ingestion Framework definition.Properties
id: Unique identifier that identifies this pipeline. Refer to ../type/basic.json#/definitions/uuid.name: Name that identifies this pipeline instance uniquely. Refer to ../type/basic.json#/definitions/entityName.openMetadataWorkflowConfig(object): OpenMetadata Ingestion Workflow Config. Cannot contain additional properties.source: Refer to #/definitions/source.processor: Refer to #/definitions/processor.sink: Refer to #/definitions/sink.stage: Refer to #/definitions/stage.bulkSink: Refer to #/definitions/bulkSink.workflowConfig: Refer to #/definitions/workflowConfig.ingestionPipelineFQN(string): Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline.pipelineRunId: Unique identifier of pipeline run, used to identify the current pipeline run. Refer to ../type/basic.json#/definitions/uuid.
Definitions
sourceConfig(object): Additional connection configuration. Cannot contain additional properties.config- One of
- : Refer to databaseServiceMetadataPipeline.json.
- : Refer to databaseServiceQueryUsagePipeline.json.
- : Refer to databaseServiceQueryLineagePipeline.json.
- : Refer to dashboardServiceMetadataPipeline.json.
- : Refer to messagingServiceMetadataPipeline.json.
- : Refer to databaseServiceProfilerPipeline.json.
- : Refer to databaseServiceAutoClassificationPipeline.json.
- : Refer to pipelineServiceMetadataPipeline.json.
- : Refer to mlmodelServiceMetadataPipeline.json.
- : Refer to storageServiceMetadataPipeline.json.
- : Refer to searchServiceMetadataPipeline.json.
- : Refer to testSuitePipeline.json.
- : Refer to metadataToElasticSearchPipeline.json.
- : Refer to dataInsightPipeline.json.
- : Refer to dbtPipeline.json.
- : Refer to applicationPipeline.json.
- : Refer to apiServiceMetadataPipeline.json.
- One of
source(object): Configuration for Source component in OpenMetadata Ingestion Framework. Cannot contain additional properties.type(string, required): Type of the source connector ex: mysql, snowflake, tableau etc..serviceName(string): Type of the source connector ex: mysql, snowflake, tableau etc..serviceConnection: Connection configuration for the source. ex: mysql , tableau connection. Refer to ../entity/services/connections/serviceConnection.json#/definitions/serviceConnection.sourceConfig: Refer to #/definitions/sourceConfig.
processor(object): Configuration for Processor Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.type(string, required): Type of processor component ex: pii-processor.config: Refer to ../type/basic.json#/definitions/componentConfig.
stage(object): Configuration for Stage Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.type(string, required): Type of stage component ex: table-usage.config: Refer to ../type/basic.json#/definitions/componentConfig.
sink(object): Configuration for Sink Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.type(string, required): Type of sink component ex: metadata.config: Refer to ../type/basic.json#/definitions/componentConfig.
bulkSink(object): Configuration for BulkSink Component in the OpenMetadata Ingestion Framework. Cannot contain additional properties.type(string, required): Type of BulkSink component ex: metadata-usage.config: Refer to ../type/basic.json#/definitions/componentConfig.
logLevels(string): Supported logging levels. Must be one of:["DEBUG", "INFO", "WARN", "ERROR"]. Default:"INFO".workflowConfig(object): Configuration for the entire Ingestion Workflow. Cannot contain additional properties.loggerLevel: Refer to #/definitions/logLevels. Default:"INFO".successThreshold(integer): The percentage of successfully processed records that must be achieved for the pipeline to be considered successful. Otherwise, the pipeline will be marked as failed. Minimum:0. Maximum:100. Default:90.openMetadataServerConfig: Refer to ../entity/services/connections/metadata/openMetadataConnection.json.config: Refer to ../type/basic.json#/definitions/componentConfig.