An elaboration of Tass' comment and Mark's answer (for which +1):
You can insert cell values efficiently if you use what wikipedia calls Dictionary Of Keys or DOK (which is essentially Jens' answer), but as you rightly comment, getByRow and getByColumn will be fairly slow.
A better option would be what wikipedia calls Coordinate List or COO: just a set of triples (rowindex, columnindex, value). You'd probably actually store this as three arrays. In order to make insertion fast, keep a set of sorted and unsorted entries, and insert into the unsorted set; whenever the number of unsorted entries goes over a threshold T (which might depend on the total number of nonempty cells K), sort them into the sorted set.
You'll want to sort them all by, say, row index, and keep another array with indices into the arrays to give the version that is sorted by column index.
For getByRow you would take the correct section of the arrays sorted by row index, and additionally search through the unsorted set.
All of this assumes that you do have enough memory to store a couple of words for every nonempty entry in the matrix. If not, you'll need to combine this with some sort of external memory approach.