Git operates on "commit objects", not files / directories. Each commit object has an unique commit ID which is a hash of its previous commit id and the changes of that commit.
Which means that ignoring some files / directories would create a different commit ID - so it's basically equivalent to rebasing all the upstream changes, then creating a new commit which deletes all the files that you don't like.
You could probably do that locally with some scripts, but if you do that, every single commit in your local repository will be different from the version on the server, so you'd have to merge each time you pull new upstream changes / push something upstream.
Maybe you could convince the maintainers of the upstream project to split it into smaller submodules ?