Many fishing methods result in significant catches of non-target species or individuals that are too small to be retained for economic values or within quota regulations. In the Baltic Sea trap fishery, the major problem is the bycatch of juvenile and non-marketable whitefish (Coregonus maraena) which constitutes a threat to the sustainability of the fishery and a time-consuming problem for the fishers. This study evaluates the effectiveness of a modified fish trap in reducing such bycatch. An encircling selection panel consisting of 50 × 50 mm square mesh netting was installed in a pontoon fish chamber of a salmon/whitefish trap. Comparative fishing was conducted against a control trap without selection panel in the inshore waters of the Bothnian Sea. Comparisons of catch compositions between the traps showed that there was a 72% reduction in juvenile whitefish catch in the experimental trap. The length at 50% retention (30.1 cm) corresponded well to the minimum market size (>30 cm) of whitefish that are desired for the local market. Fishers are advised to use a netting panel of 50 × 50 mm square mesh for their traps to reduce undersized whitefish. The results are important for the sustainability of whitefish stocks in the Baltic Sea.