In this blog, we want to explore the data size limit for CSV format files on AWS S3 for the combination of AWS Wrangler and Pandas on AWS Lambda. The data set used for the benchmarking was downloaded from the geographical Society. The results of the performance test may vary…

AWS Lambda supports Docker Container as a Function(CaaF). This adds other dimensions to AWS Lambda, portability, ,dependency handling, and environment setup to the serverless architecture. The CaaF supports use cases where standard environment setup is required, portability, and handling unsupported languages like PHP and hybrid cloud environments. There exists a…

AWS SAM(Serverless Application Model)is an open source framework that enables AWS users to build, test locally and deploy AWS Lambda to the Cloud. There are many tools to simplify the development of Serverless functions: cross-vendor Serverless framework, AWS-specific Serverless Application Model (SAM), and others. The AWS SAM is widely used…

Using UDFs (User Defined Functions) in spark is probably the last resort for building column-based data processing logic. The Spark UDF is an expensive operation and is used only to extend or fill in missing functionality of Spark methods or libraries or frameworks that do not have a Python wrapper…

SerDe(serialization and deserialization) is a process in the programming world where convert object into byte stream format, so that reusability of the object in the same or across different script or environment. …

Hi, I am John Cherian working as a Data engineering architect consulting for various clients in the DC area. My key area of focus is designing big data and low latency ETL architecture on Cloud platforms like AWS, Azure, and GCP. Recently noticed that a lot of firms are moving…

Have you heard comments like “throw the files into a filesystem and big data framework will take care of the rest”, “Spark is 100x faster so it can handle any volume and any datasets”, “ Why should I worry about partition, sorting in a Big data ecosystem? It is an…