什么是流批一体?

批流一体中的“批流”是指批处理与流处理:中国数字化转型网www.szhzxw.cn

计算一体

同一套计算逻辑可以同时应用于流处理与批处理两种模式,且在最终结果上一致。

存储一体

流处理与批处理过程中全程数据存储在同一介质,且同一数据只需要存储一遍。

流批一体就是同时支持离线和实时计算,我们假设数据湖是一个堰塞湖,堰塞湖的水顺着山体不断向下流淌,山体有各种洼沟,水流向第一层洼沟后,水会先横向渗透洼沟,这个过程可理解为批处理过程

当第一层漕沟填满水后会流向下一层,同样下一层横向渗透洼沟的批处理过程。就这样水流一层一层往下传递(类似ODS, DWD,ADS各层)。中国数字化转型网www.szhzxw.cn

当各层洼沟都溢满水后,如果上游水流持续流淌,那么就看到各层也实时有水垂直到下一层(实时),同时也会横向洼沟波动(批处理),效果上看就是实时处理和同步处理同时处理,融合为一个体系。

翻译:

What is flow in one?

“Batch stream” in batch stream integration refers to batch processing and stream processing: China Digital Transformation network www.szhzxw.cn中国数字化转型网www.szhzxw.cn

Computing in one

The same set of computational logic can be applied to both stream and batch processing modes, and the final result is the same.

Memory integration

In the process of stream processing and batch processing, the whole data is stored in the same medium, and the same data needs to be stored only once.中国数字化转型网www.szhzxw.cn

Flow batch integration is to support both offline and real-time computing, we assume that the data lake is a barrier lake, the water of the barrier lake continues to flow down the mountain, the mountain has a variety of gullies, after the water flows to the first layer of the gullies, the water will first transverse penetration of the gullies, this process can be understood as a batch process.

When the first trench is filled with water, it will flow to the next layer, and the next layer will also penetrate the trench horizontally. In this way, the water flow is passed down layer by layer (similar to ODS, DWD,ADS layers). China Digital Transformation network www.szhzxw.cn

When each layer of gully is full of water, if the upstream water continues to flow, then you can see that each layer also has water vertical to the next layer (real-time), but also lateral gully fluctuations (batch processing), the effect is real-time processing and synchronous processing at the same time, integrated into a system.中国数字化转型网www.szhzxw.cn

中国数字化转型网www.szhzxw.cn
免责声明: 本网站(http://www.szhzxw.cn/)内容主要来自原创、合作媒体供稿和第三方投稿,凡在本网站出现的信息,均仅供参考。本网站将尽力确保所提供信息的准确性及可靠性,但不保证有关资料的准确性及可靠性,读者在使用前请进一步核实,并对任何自主决定的行为负责。本网站对有关资料所引致的错误、不确或遗漏,概不负任何法律责任。 本网站刊载的所有内容(包括但不仅限文字、图片、LOGO、音频、视频、软件、程序等) 版权归原作者所有。任何单位或个人认为本网站中的内容可能涉嫌侵犯其知识产权或存在不实内容时,请及时通知本站,予以删除。http://www.szhzxw.cn/21337.html

什么是数据资产?

什么是机器学习?

联系我们

联系我们

17717556551

邮箱: editor@cxounion.org

关注微信
微信扫一扫关注我们

微信扫一扫关注我们

关注微博
返回顶部