One solution could be to create the 6000 simple products with color and size as attributes.
You would then observe on the add to cart event, or create a custom controller action, that would create a new product on the fly and populate the color and size attributes based on post values from the product view page. Then you would have to create a custom flat table to manage the inventory of each product-color-size combination. The custom inventory table can be referenced to set the stock item for the generated product and in the product grid/view to check the availability.
This would create a new product for each purchase, but you may be able to get away with deleting the product after the sale is made. Also you may have to customize the reordering process, if you need that feature as well.
This may not be the most straight forward solution, but it should work, in theory.