#data-engineer-interview

#data-engineer-interview

Apache Kafka has cemented itself as the de facto platform for event streaming, often referred to as the 'universal data substrate' due to its extensive ecosystem that enables connectivity and processing capabilities.

"This is more likely to complement existing SIEMs than replace them. Early adoption will come from large enterprises already committed to Databricks, especially those seeking flexibility or cost control."

Uber's engineering team has transformed its data replication platform to move petabytes of data daily across hybrid cloud and on-premise data lakes, addressing scaling challenges caused by rapidly growing workloads. Built on Hadoop's open-source Distcp framework, the platform now handles over one petabyte of daily replication and hundreds of thousands of jobs with improved speed, reliability, and observability.

Snowflake offers a fully managed data platform, but Sumo Logic users often lack insight into performance, login activity, and operational health. The Sumo Logic Snowflake Logs App analyzes login and access activity to identify anomalies or suspicious behavior. It also optimizes data pipelines with insights into long-running or failing queries. Teams can centralize log data to facilitate correlation across applications, cloud services, and data platforms.

Developers have spent the past decade trying to forget databases exist. Not literally, of course. We still store petabytes. But for the average developer, the database became an implementation detail; an essential but staid utility layer we worked hard not to think about. We abstracted it behind object-relational mappers (ORM). We wrapped it in APIs. We stuffed semi-structured objects into columns and told ourselves it was flexible.

Most beginner data portfolios look similar. They include: A few cleaned datasets Some charts or dashboards A notebook with code and commentary Again, nothing here is wrong. But hiring teams don't review portfolios to check whether you can follow instructions. They review them to see whether you can think like a data analyst. When projects feel generic, reviewers are left guessing:

Databricks today announced the general availability of Lakebase on AWS, a new database architecture that separates compute and storage. The managed serverless Postgres service is designed to help organizations build faster without worrying about infrastructure management. When databases link compute and storage, every query must use the same CPU and memory resources. This can cause a single heavy query to affect all other operations. By separating compute and storage, resources automatically scale with the actual load.

I've learned which skill sets software engineers need to land a job offer in the AI era. Tech companies agree that AI makes engineers more productive, so engineers are expected to use it to build things more quickly and reliably. I personally make heavy use of AI to help me with boilerplate stuff so that I can concentrate on the hard stuff, like system design and complex business logic.

The title "data scientist" is quietly disappearing from job postings, internal org charts, and LinkedIn headlines. In its place, roles like "AI engineer," "applied AI engineer," and "machine learning engineer" are becoming the norm. This Data Scientist vs AI Engineer shift raises an important question for practitioners and leaders alike: what actually changes when a data scientist becomes an AI engineer, and what stays the same? More importantly, what skills matter if you want to make this transition intentionally rather than by accident?

Knowing how to query ChatGPT, Gemini or Claude doesn't make you an expert on genAI and AI tools. There's a lot more to learn about in this fast-evolving area of tech. Even as thousands of workers lose their jobs to AI, the number of job openings seeking "AI skills" continues to grow. Mentions of AI skills in job postings rose 5% year over year, according to data released in December by staffing firm Manpower Group.

SHAP for feature attribution SHAP quantifies each feature's contribution to a model prediction, enabling: LIME for local interpretability LIME builds simple local models around a prediction to show how small changes influence outcomes. It answers questions like: "Would correcting age change the anomaly score?" "Would adjusting the ZIP code affect classification?" Explainability makes AI-based data remediation acceptable in regulated industries.

For the past few years, artificial intelligence has been discussed almost exclusively in terms of models. Bigger models, faster models, smarter models. More recently, the focus shifted to agents, systems capable of planning, reasoning, and acting autonomously. Yet the real leap in usefulness does not happen at the model level, nor at the agent level. It happens one layer above, at the level of Skills.