An airplane has 50,000 sensors and generates 2.5 TB of data per day. Flying to a destination, downloading the data, and processing it, is heavy duty. But if processing it in real-time, that is 0.12 MB/s, which is possible to do on a Raspberry Pi. By doing streaming analytics you can process the data in memory without having to read and write data to disk (though there might be other reasons to store data). This saves a lot of computing cycles. Cycles that can be used to do more at the edge. The processing power is there, so why not use it?
The general perception is that you need a lot of data to do AI, which is true. But, you know what, processing 2.5 TB of data in real-time per day at the edge equals the same amount of big data as if you store, transfer and process 912.5 TB in the cloud over a year. I am sure you have already had your first encounter with the CFO when he or she realized that the cloud cost that was so compelling in the beginning has skyrocketed through the roof.
The difference with using an incremental streaming approach is that you get instant insights that get better and better over time. You can adjust what you look for based on early conclusions. And you get the adequate insights when the model reaches the level of training rather than when you think you have sufficient data. And you don’t stop learning after a year, but continue to analyze the streaming data, keeping the model up to date by accommodating changes to the surrounding environment, a new sensor, and seasonal changes by comparing year two and three.
Not only will you need to send less data but you provide better and actionable data to the cloud. So, instead of rivers pouring loads of dirty water into the data lake, you clean it upstream and create a smaller lake with crystal clear water. We know from experience that the best place to learn individually is close to the source where you get the high resolution, real-time data, i.e. at the edge. Then you can aggregate in steps to compare insights with peers, compare to larger groups, to eventually look at an aggregate view of all your operational products.
And make no mistake, learning takes time. Just like when we send our kids off to school, they do not come home with all the knowledge on day one. It takes time, and today’s youngsters, just like your products, will have to do lifelong learning to keep up with the development. The sooner you start learning and getting these insights, the bigger the advantage you get over the competition, which will be hard for them to make up – or the opposite if they beat you to it… So, don’t wait. Start small. Don’t boil the ocean. But do something. And you can’t look at AI for your products as a project that will get done. Once you start, it is the beginning of a journey. You will constantly add things that you want to know. This is why we designed Ekkono as a tool (SDK) that enables you to constantly evolve and develop new features. All aboard! It’s time for departure!