As part of research and innovation challenge, some months ago We started evaluating new collectors tools, so, Fluentd came to the game, here is short description about it, hope help you to improve your Big Data Platform.
Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
Fluentd is Apache 2.0 Licensed, fully open source software. That means your imagination, not license restrictions, is the limit of what you can achieve with Fluentd. The source code is available on GitHub.
Fluentd decouples data sources from backend systems by providing a unified logging layer in between.
This layer allows developers and data analysts to utilize many types of logs as they are generated. Just as importantly, it mitigates the risk of “bad data” slowing down and misinforming your organization.
A unified logging layer lets you and your organization make better use of data and iterate more quickly on your software.
Unified Logging with JSON
Fluentd tries to structure data as JSON as much as possible: this allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations (Unified Logging Layer). The downstream data processing is much easier with JSON, since it has enough structure to be accessible while retaining flexible schemas.
Fluentd has a flexible plugin system that allows the community to extend its functionality. Our 500+ community-contributed plugins connect dozens of data sources and data outputs. By leveraging the plugins, you can start making better use of your logs right away.
Fluentd supports memory- and file-based buffering to prevent inter-node data loss. Fluentd also support robust failover and can be set up for high availability. 2,000+ data-driven companies rely on Fluentd to differentiate their products and services through a better use and understanding of their log data.
Who is using Fluentd?