• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI
    Reaction

    Matei Zaharia's path from Berkeley's Apache Spark project to AI-agent research

    A post traces Spark's 2009 start at Berkeley through Zaharia's role in building Databricks and his return as a professor.

    JF
    1 Source, ,

    TLDR

    A post on Berkeley's technology impact highlights Zaharia's work on Apache Spark as a PhD student and his role in founding Databricks with fellow researchers. It says he remains Databricks' co-founder and CTO while researching reliable AI agents as a Berkeley associate professor, including work on the open-source projects DSPy and GEPA.

    Combined views

    1.7K

    1 Source, first seen 11h ago

    Combined views

    1.7K

    1 Source, first seen 11h ago

    42 likes
    11h ago
    first seen 11h ago
    42 likes
    4 comments
    1 saves
    8 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Featured Source
    4 comments
    1 saves
    8 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    1 Source

    @jfianceIt would be hard to do a weekly series on Berkeley's impact on technology without getting to @matei_zaharia. His Berkeley story starts with Apache Spark, which began as a research project on campus in 2009, when Matei was a PhD student in the lab that became AMPLab, working on ways to make large-scale data processing faster and more flexible. Spark eventually became one of the most widely used open-source systems for processing enormous datasets. Then the researchers behind it set the foundations for @databricks. Matei and a group of fellow Berkeley researchers turned the technology they had been developing on campus into a company that has since become one of the defining data and AI platforms of its generation. It's now valued at $190B! The story didn't end with Spark, either. Matei has continued working on widely used open-source data and AI projects including MLflow and Delta Lake, while Databricks has grown from a Berkeley research spinout into a company serving organizations around the world. And now there's another Berkeley chapter. Matei is still Databricks' Co-Founder/CTO, but he's also back at Berkeley as an Associate Professor in EECS, conducting research and working alongside another generation of students who may eventually spark pioneering research and/or companies of their own. Today, his research focuses on one of the biggest open questions in AI: how to build and scale reliable agents. He's a co-author on recent open-source projects including DSPy and GEPA, which automatically optimize prompts and models so agents get better at specific tasks. This April, ACM awarded him the 2025 ACM Prize in Computing for the distributed data systems that made large-scale analytics, machine learning, and AI possible. A fitting bookend: his Spark dissertation won ACM's Doctoral Dissertation Award back in 2014. I talk a lot about Berkeley's startup flywheel because Matei's career is a pretty clean example of how it works. A PhD student invents important technology on campus. The work is released openly and gets adopted around the world. A company grows out of it and becomes a category leader. The people responsible stay connected to Berkeley, and the next generation gets to learn and build alongside them. Nearly two decades after Spark got its start on campus, that cycle is still going.