Amazon Bedrock emerges as a fully managed service, designed to simplify the development and deployment of generative AI applications. Drawing parallels with AWS Lambda’s impact on serverless computing, Bedrock aims to be the go-to platform for generative AI. Overview Amazon Bedrock offers a range of foundation models (FMs) from prominent…

Introduction GPTJ-6B is a powerful transformer model with 6 billion trainable parameters, developed using Ben Wang’s Mesh Transformer JAX. As large language models (LLMs) like GPTJ-6B gain prominence in various industries, hosting them on scalable and efficient platforms like AWS SageMaker becomes crucial. …

“Outpacing Spark with Pen and Paper: A Data Processing Dilemma." Have you heard customers claim that manually inserting 10 rows of data is faster than letting Apache Spark work its magic? — Do you have the same experience waiting for your small files to process using Apache Spark? Well, guess what? Apache Spark may not be your solution. You see, unlike Pandas, Spark wasn’t designed with small files (under 400 MB) in mind. …

In this blog, we want to explore the data size limit for CSV format files on AWS S3 for the combination of AWS Wrangler and Pandas on AWS Lambda. The data set used for the benchmarking was downloaded from the geographical Society. The results of the performance test may vary…

AWS Lambda supports Docker Container as a Function(CaaF). This adds other dimensions to AWS Lambda, portability, ,dependency handling, and environment setup to the serverless architecture. The CaaF supports use cases where standard environment setup is required, portability, and handling unsupported languages like PHP and hybrid cloud environments. There exists a…

AWS SAM(Serverless Application Model)is an open source framework that enables AWS users to build, test locally and deploy AWS Lambda to the Cloud. There are many tools to simplify the development of Serverless functions: cross-vendor Serverless framework, AWS-specific Serverless Application Model (SAM), and others. The AWS SAM is widely used…

Using UDFs (User Defined Functions) in spark is probably the last resort for building column-based data processing logic. The Spark UDF is an expensive operation and is used only to extend or fill in missing functionality of Spark methods or libraries or frameworks that do not have a Python wrapper…

SerDe(serialization and deserialization) is a process in the programming world where convert object into byte stream format, so that reusability of the object in the same or across different script or environment. …

Hi, I am John Cherian working as a Data engineering architect consulting for various clients in the DC area. My key area of focus is designing big data and low latency ETL architecture on Cloud platforms like AWS, Azure, and GCP. Recently noticed that a lot of firms are moving…