Coiled is trying to meet data scientists where they are - in the Python Data Science Ecosystem. Dask and Coiled in this Landscapeĭask is only one component of this complex system and Coiled is building products in the scalable computing axis of this landscape, an axis that is becoming increasingly important. To maximize investment in the data science teams, you’ll need to provide a holistic solution, simultaneously solving for all the axes that are important to your business problems. Each of these axes (and more that we haven’t even discussed) have specialized software and companies behind them. In short, it’s supremely complex.Īdditionally, you have the build-vs-buy choice for most of these elements - further complicating matters. In many cases, this whole chart is duplicated for production with different ACLs on different pieces. Infrastructure also needs to be considered for every element in the above image: Are you on-prem, on the cloud (AWS, GCP, Azure, etc.), both? And, there’s also the consideration of how your workflows for development and production break down. There are also more niche problems like model management, packaging, CI/CD, etc. In enterprise data science, there are roughly 5 axes - data, development environment, scalable computing, workflow managers, and dashboarding. These questions answered, the interesting topic now is the current scalable data science landscape and the infrastructure around it. In fact, the data science space has a diverse range of people, some of whom may not have a software engineering background to work with Java and Spark.ĭask didn’t have many companies backing it in 2019, which led to people continuing to choose Spark. Most data scientists clearly prefer Pythonic frameworks over Java-based Spark. The interesting question here is: Why are enterprises still choosing Spark? There are lots of people doing lots of things with it and selling lots of products that are powered by it.” “Of course Spark is still relevant, because it’s everywhere. You can check out the livestream replay here: How do Dask and Coiled fit in this landscape?.What does today’s scalable data science landscape look like?.How scalable computing relates to the business world?.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |