I'm reading a 500MB+ log file, I was wondering which would be faster in terms of speed.
I want to be able to scroll through the ~1mil entries, but shouldn't be slow because of the virtual mode(hopefully). It loads successfully, however scrolling is a bit lagged.
Currently I'm using a listview in virtual mode.
QUESTION - virtual-retrieval-item function -
- list storing each entry of log info, and the retrieve item call to display the specific indexes displayed of the list such as list[e.getindex]
- store the beginning of each log info (position in the log file) to an list, then calling a read function to read from the position to the escape character (gets one log entry). For example first entry would read 0 - escape character, say the second entry would read 16-43, third, 43-60 (the size of the log entries all vary)
There are pros and cons of both, but I am curious to see what others think in terms of speed.
On one hand, (1) reads all the data of a 1mil entries into a list, then reads them from memory because virtual mode helps display only the items that is viewable at the time (approx 10). However the overhead is that all the data is in memory
With (2) there is no storage of actual log entries into memory, however it has to make a call to file to scan through the file, and start reading at a specific line. It has to make this call for each item.
Is there an alternate? These are the fastest ways in which were researched.