Creating a bitmap, even a low resolution monochrome bitmap, is likely to be larger than a vector-based description language.
Consider:
(Hello World) Tj
That's 16 bytes in a PDF file, and it doens't change if you change the font size. If you turn it into a bitmap, even at low resolution and compressed, it probably exceeds that size.
That's why rendering a page description language to a bitmap produces larger files and is one of the reasons for using a page description language for printing, instead of sending large bitmaps around.
The tiffg3 and tiffg4 devices in Ghostscript only produce monochrome output, because that's all you can encode with G3 and G4 encoding. TIFF G3 is already compressed using the Fax CCITT group 3 compression scheme (Group 3 = g3). If you try to compress that using some other scheme, then your fax software wont be able to read it.
You could try using CCITT Group 4 fax compression instead (the tiffg4 device) but if that doesn't help then basically that's what you get. Your only other option is to create the TIFF at a lower resolution. You don't say what resolution you are currently using. Fax normally supports 3 resolutions; 408x391, 204x196 and 204x98. If you are using superfine (408x391) then you could switch to a lower resolution.
I'm at a loss to see why this is a problem since you are sending the files by fax anyway, why do you care how large an intermediate TIFF file you get ?