Cómo obtener el promedio con orderBy Desc en Laravel 5

Puede usar un withCount() modificado :

$books = Book::withCount(['ratings as average_rating' => function($query) {
    $query->select(DB::raw('coalesce(avg(rating),0)'));
}])->orderByDesc('average_rating')->get();