workflow & process engine

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.

• Compatible with any version of Hadoop
• Easy to use web UI
• Simple web and http workflow uploads
• Project workspaces
• Scheduling of workflows
• Modular and pluginable
• Authentication and Authorization
• Tracking of user actions
• Email alerts on failure and successes
• SLA alerting and auto killing
• Retrying of failed jobs

Apche Nifi An easy to use, powerful, and reliable system to process and distribute data.
Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:

Web-based user interface

• Seamless experience between design, control, feedback, and monitoring
  Highly configurable

• Loss tolerant vs guaranteed delivery

• Low latency vs high throughput
• Dynamic prioritization
• Flow can be modified at runtime
• Back pressure

Data Provenance

• Track dataflow from beginning to end

Designed for extension

• Build your own processors and more
• Enables rapid development and effective testing

Secure

• SSL, SSH, HTTPS, encrypted content, etc…
• Multi-tenant authorization and internal authorization/policy management