Class DecoderLecoGithubDataIngester

Ingestion des données csv provenant du repsitory Github https://github.com/decoderleco/deces_europe

Download the CSV dataset file,
Sends the CSV dataset file to kafka (publish to kafka topic)
Persists the CSV dataset file to an S3 bucket (using the local filesystem in much too uncomfortable, using an S3 bucket so much better)
the S3 bucket (containing the CSV dataset file) is added on a git branch in LakeFS

Index

Constructors

Properties

dataWorkDir? filePathInRepo gitVersion baseUrl

Methods

createDir download getIngestedDataFileFolderPath run

Constructors

constructor

new DecoderLecoGithubDataIngester(gitVersion, filePathInRepo, dataWorkDir?): DecoderLecoGithubDataIngester
Parameters
- gitVersion: string
- filePathInRepo: string
- Optional dataWorkDir: string
Returns DecoderLecoGithubDataIngester
- Defined in ingesters/DecoderLecoGithubDataIngester.ts:20

Properties

`Protected` `Optional` dataWorkDir

dataWorkDir?: string

`Protected` filePathInRepo

filePathInRepo: string

`Protected` gitVersion

gitVersion: string

`Static` baseUrl

baseUrl: string = "https://raw.githubusercontent.com/decoderleco/deces_europe"

Methods

createDir

createDir(): void
Returns void
- Defined in ingesters/DecoderLecoGithubDataIngester.ts:46

download

download(): Promise<string>
Returns Promise<string>
- Defined in ingesters/DecoderLecoGithubDataIngester.ts:67

getIngestedDataFileFolderPath

getIngestedDataFileFolderPath(): string
Returns string
- Defined in ingesters/DecoderLecoGithubDataIngester.ts:41

run

run(): Promise<void>
Returns Promise<void>
- Defined in ingesters/DecoderLecoGithubDataIngester.ts:36

Generated using TypeDoc