What is the output of h2o.clusterStatus(), right after calling h2o.init()?
You may need to explicitly limit the memory that h2o consumes, as I don't believe that it properly detects the cgroup limit and instead looks at the host machines total memory and attempts to start a JVM with too high a maximum memory.
That said, I still can't get the Flow interface to properly launch, fiddling with it...