ORA-01799:una columna no se puede unir externamente a una subconsulta

puede reescribir eso presionando la subconsulta para que no esté unida externamente:

select Count(C.setid)
  from mycustomer C
       left outer join (select *
                          from mycustoption co
                         where co.effdt <= (select Max(COI.effdt)
                                              from mycustoption COI
                                             where COI.setid = co.setid
                                               and COI.cust_id = co.cust_id
                                               and COI.effdt <= sysdate)) co
                    on ( C.setid = CO.setid
                         and C.cust_id = CO.cust_id )