Well, dust just proves the theory of conservation of matter and energy! And I'd be concerned about lint from the damp paper towels. Now I don't use sheet film in my Century (!), but I've had problems with dust using 120 roll film. Based on my experience as well as the relevant post above, my guess is that the dark slide light trap is the culprit---though I don't discount the possibility that it's also a processing thing (all my film is developed by a lab).

All my lint is pretty big-) Plus the towel is white so any lint against the black holder would really stand out. The idea of blowing dust around the darkroom doesn't appeal to me.

You are right to be concerned about blowing dust around the darkroom. I take the holders out to the garage and blast those suckers with the air hose, seal them in plastic bags and put 'em away until needed.

Years ago, a columnist in one of the photo mags suggested vacuuming the film holders to remove any dust before loading them.