Falra.jl : Distributed Computing with AI Source Code Generation

Falra.jl in Julia provides a straightforward approach to implementing distributed computing, equipped with an AI-assisted feature for generating source code. This addition facilitates more efficient big data transformations. Tasks such as preprocessing 16TB of IoT data can be done in 1/100 of the original time. Developers are now able to generate Julia source code more easily with the aid of AI, further aiding in distributed computing tasks.

This is a real development scenario that we encountered to preprocess 6-year, 16TB historical IoT raw datasets for data cleaning and transformation. It takes 100 days to complete processing in a single-machine environment, which is time-consuming. So, the Falra.jl was developed to allow us to divide the data cleaning and transformation tasks that we need to perform into smaller tasks. Falra.jl then automatically distributes these tasks for distributed processing. This architecture saves a lot of computing time and development costs. Through Falra.jl, we were able to complete all IoT data transformations in 1/100 of the time.

Compared to the native Julia distributed module, the advantage of Falra.jl is that developers do not need to learn how to develop the Julia distributed programming syntax. They can just use their single-machine programs as they used to do. In addition, Falra.jl can be deployed on any network that can be called via HTTPS. There is no need to deal with TCP or other network or firewall issues.

Moreover, we've enhanced our approach by integrating AI-assisted Julia source code auto-generation. This novel feature allows developers to efficiently create Julia code using artificial intelligence. Rather than manually crafting each line of code, the AI can generate source code based on the developer's requirements, thus accelerating the development process. It makes it feasible for developers, even those unfamiliar with Julia, to quickly produce distributed programs. This AI-driven tool not only simplifies code creation but also enables the rapid adaptation and extension of the applications under the Falra.jl . The fusion of distributed computing and AI-assisted auto-generation of Julia source code significantly boosts productivity.

Currently, we have released the Falra.jl on Github (https://github.com/bohachu/Falra.jl) for everyone to use.