¿Cómo obtener el promedio de los valores 'medios' en un grupo?

Otra opción usando funciones analíticas;

SELECT group_id,
       avg( test_value )
FROM (
  select t.*,
         row_number() over (partition by group_id order by test_value ) rn,
         count(*) over (partition by group_id  ) cnt
  from test t
) alias 
where 
   cnt <= 3
   or 
   rn between floor( cnt / 2 )-1 and ceil( cnt/ 2 ) +1
group by group_id
;

Demostración --> http://www.sqlfiddle.com/#!11/af5e0 /59