What is CodeFeedr?
CodeFeedr is a software analytics infrastructure designed to simplify setting up complex stream processing architectures. It is built on top of Apache Flink and Apache Kafka and has the following features:
- (Inter)connecting Apache Flink streaming jobs using Kafka as message broker.
- Orchestration tools to run interconnected Flink setup.
- Distributed API key management.
- Plugin system to share re-usable streaming components.
In order to get a grasp of the architecture of CodeFeedr have a look at the architecture page. In general CodeFeedr can be split into three parts:
- Core: All the implementations for stages, pipelines and utilities.
- Plugins: Set of plugins, which contain context related stages.
- Orchestration: Tools for running your pipeline on a cluster.
You can contribute to CodeFeedr by either improving and extending the core and already existing plugins or by writing your own plugin.
In this section some terminology from the CodeFeedr framework will be discussed:
- Stage: A Flink job. There are three types; InputStage, TransformStage and OutputStage.
- Buffer: A message broker in between stages to flow data from one stage to another (like Apache Kafka).
- Pipeline: A set of stages linked with buffers.
- PipelineBuilder: API interface to build a pipeline.
- Plugin: A set of stages in a certain context which can be used to build a pipeline.
CodeFeedr is currently maintained and improved by: