This page will get you started with the CodeFeedr framework. It will give a brief explanation on CodeFeedr and some leads to continue digging into the framework.
Edit me

What is CodeFeedr?

CodeFeedr is a software analytics infrastructure designed to simplify setting up complex stream processing architectures. It is built on top of Apache Flink and Apache Kafka and has the following features:

  • (Inter)connecting Apache Flink streaming jobs using Kafka as message broker.
  • Orchestration tools to run interconnected Flink setup.
  • Distributed API key management.
  • Plugin system to share re-usable streaming components.

Documentation

Structure

In order to get a grasp of the architecture of CodeFeedr have a look at the architecture page. In general CodeFeedr can be split into three parts:

  • Core: All the implementations for stages, pipelines and utilities.
  • Plugins: Set of plugins, which contain context related stages.
  • Orchestration: Tools for running your pipeline on a cluster.

You can contribute to CodeFeedr by either improving and extending the core and already existing plugins or by writing your own plugin.

Terminology

In this section some terminology from the CodeFeedr framework will be discussed:

  • Stage: A Flink job. There are three types; InputStage, TransformStage and OutputStage.
  • Buffer: A message broker in between stages to flow data from one stage to another (like Apache Kafka).
  • Pipeline: A set of stages linked with buffers.
  • PipelineBuilder: API interface to build a pipeline.
  • Plugin: A set of stages in a certain context which can be used to build a pipeline.

Contributors

CodeFeedr is currently maintained and improved by:

Former contributors: