I would recommend to save changes to database if you are doing or can do postback on add/remove. As not saving data along change might cause loss of information. You can use ajax call for add/remove to make it more elegant and fast without reloading whole page.
or
If you want to do all add/remove on client side and finally saving to database, you can uses client storage to keep the changes temporarily and finally sending the changes back to server and commit them to database. Article about client storage here but it might not be supported by old browsers
or
You can put change in hidden fields e.g for deleted records you have one hidden field which store deleted record ids, on hidden field with added records with set pattern and update the database on postback using these hidden fields again there is hazard for losing information with this method