Three-part Medium.com Article on Option, Try & Either

I’ve been working for the past couple weeks on writing a three-part series based on last month’s Scala Beginner’s presentation on using Option, Try and Either. I just finished them and published them on Medium.com.

Scala for Beginners: Option, Try and Either

Beginning-Scala-Options-Either-Try

Use this link to Download the above PDF. The video below can also be accessed via this link.

Meetup lecture video recording.

I have been programming with Scala now for about five years, and I’ve got a confession: until really really recently, there’s a bunch of stuff I never really understood. Now that I look back, I guess it’s understandable. But to be honest, I’m a bit embarrassed.

For example, when I was first learning about Scala, I read that it was a great language because—if you used the right tools—you could write fault-tolerant code that could avoid things like NullPointerExceptions.

I’m talking about Option.

So at times I would use Option if there were scenarios where I might or might not have a value, and I would write inelegant code that would first test whether the item was defined or not, and then would try to extract the value and do something with it. I thought I was just properly following the Scala principles, but things were clunky.

And to be honest, just two weeks ago I was writing some code that was testing and validating a number of scenarios before getting to the core functionality. It was the inner framework to a toolkit that I wanted developers to use, and if a problem happened, it would be really important to catch it and get the appropriate feedback back to the developer. (If I didn’t do that, nobody would ever use my application, or worse: my ears might start burning as they cursed the day they ever decided to give my product a chance!)

I started writing nested code that kept unboxing Try and Option and Either containers, and what felt like it should take take five or six simple lines just looked messy. And I hate messy code!

Long story short, I thought about what I was doing, and rewrote everything so that it involved some nice, clean looking for comprehensions. And then I thought about how I’d first tried using these Option and Try typeclasses. I wished that someone could have sat me down and shown me how to use them properly, way back when about five years ago.

This month, I’ve volunteered to give a presentation in our local Austin Scala Enthusiasts Meetup, and I decided to do another one of my “beginners learning series” with the goal of showing how to use these tools in a way I’d wished someone could have taught it to me.

The code sample for the talk can be found in my Scala Tutorials Github repository (under the “monads” subdirectory)

Advanced Scala: Implicits

First draft of my Implicits lecture.

For about the last 3-4 years I’ve been on a journey that wouldn’t have started if it were for a series of presentations in my local Austin “Scala Enthusiasts” meetup, starting with one where I was first introduced to the concept of “implicits”.

Now, years later, and after reading several books and scratching my head and butting it into walls a whole bunch, I’ve been feeling the desire to create a lecture aimed for the person I was at the very beginning of the journey. You see, I feel like a lot of time I knew I wanted to understand these interesting things… monads and monoids and typeclasses, but I lacked some sort of roadmap that put everything together. I would be grasping at drips and drabs of concepts, but I felt like I was lacking a sort of map that would help me understand where I was headed and where some of the traps and esoteric syntax were all about.

The above video weighs in at 1 hour 25 minutes, and it’s my first attempt to deliver the first of a planned two lectures. I’m planning on re-recording this (having stumbled and mumbled through the first dry run here) and I might break it into a few smaller parts. But rather than letting the “perfect be the enemy of the good” I’m just going to post this now.

Also, here are the slides in PDF form:

Advanced_Scala_Implicits_reduced

Spark 101 for Scala Users

A quick hands-on intro into Spark for Scala users.

I’ll format this into a more detailed presentation later (so feel free to check back and bug me if I’m not getting around to it) but here are some immediate things you may be interested in if you saw my Austin Scala Enthusiasts Meetup presentation…

Here’s a link to the PDF of the slides I talked to.

Running Zeppelin via a Docker container

docker run --name zeppelin -p 8080:8080 -p 4040:4040 -v $HOME/spark/data:/data -v $HOME/spark/logs:/logs -v $HOME/spark/notebook:/notebook -e ZEPPELIN_NOTEBOOK_DIR='/notebook' -e ZEPPELIN_LOG_DIR='/logs' -e ZEPPELIN_INT_JAVA_OPTS="-Dspark.driver.memory=4G" -e ZEPPELIN_INTP_MEM="-Xmx4g" -d apache/zeppelin:0.9.0 /zeppelin/bin/zeppelin.sh

Running Spark via a Docker container

docker run --name spark -v $HOME/spark/data:/data -p 4040:4040 -it mesosphere/spark bin/spark-shell

For a basic Spark SBT project

build.sbt:

import Dependencies._

ThisBuild / scalaVersion     := "2.12.11"
ThisBuild / version          := "0.1.0-SNAPSHOT"
ThisBuild / organization     := "com.example"
ThisBuild / organizationName := "Meetup Spark Example"
ThisBuild / scalacOptions ++= Seq("-language:higherKinds")

lazy val root = (project in file("."))
  .settings(
    name := "SparkCatScratch",
    libraryDependencies ++= Seq( scalaTest % Test, sparkCore, sparkSQL, catsCore, catsFree, catsMTL)
  )

initialCommands in console :=
  s"""
    |import cats._, cats.data._, cats.implicits._, org.apache.spark.sql.SparkSession
    |val spark = SparkSession.builder().master("local").getOrCreate
    |""".stripMargin

cleanupCommands in console := "spark.close"

project/Dependencies.scala:

import sbt._

object Dependencies {

  val sparkVersion = "2.4.5"
  val catsVersion = "2.0.0"

  lazy val scalaTest = "org.scalatest" %% "scalatest" % "3.0.8"
  lazy val sparkCore = "org.apache.spark" %% "spark-core" % sparkVersion
  lazy val sparkSQL = "org.apache.spark" %% "spark-sql" % sparkVersion
  lazy val catsCore = "org.typelevel" %% "cats-core" % catsVersion
  lazy val catsFree = "org.typelevel" %% "cats-free" % catsVersion
  lazy val catsMTL = "org.typelevel" %% "cats-mtl-core" % "0.7.0"
}

Starting Spark in the SBT console:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().master(?local").getOrCreate
val sc = spark.SparkContext