Improved eval monitor sql queries

Review Request #33 - Created Nov. 17, 2008 and submitted

Information
Bogdan-Cristian Tătăroiu
infoarena
Reviewers
hackers
I can't attach diff to valentin's review, so here's a new one.

Optimized monitor queries when there are only filters related to `ia_job` table.

Changed ia_job indexes from (*, submit_time) to (*, id).

When using a large offset mysql queries are very slow even when using index if there are a lot of other fields in the SELECT. When filters are only related to `ia_job` we do a SELECT `id` FROM `ia_job` query which uses the available indexes to return the job ids fast and ordered properly and then do the full query with a WHERE `ID` in (...).

There is a great performance increase for all queries for which we had ia_job indexes but were slowed down by mysql's poor limit handling.

Benchmarks: http://hackers.devnet.ro/wiki/Benchmarks

The monitor?round=arhiva&first_entry=100000 is still slow, but I think this is because round is a varchar(64) and thus the index_key is pretty big. If we add numeric ids to round this should work a lot better too (but that's for another patch).

Bogdan-Cristian Tătăroiu
Review request changed

Change Summary:

Fixed bug.

Description:

   

I can't attach diff to valentin's review, so here's a new one.

   
   

Optimized monitor queries when there are only filters related to ia_job table.

   
~  

Changed ia_job indexes from (, submit_time) to (, id). When using a large offset mysql queries are very slow even when using index if not all selected data is in the index.

  ~

Changed ia_job indexes from (, submit_time) to (, id).

   
~  

When filters are only related to ia_job we do a SELECT id FROM ia_job query which uses the available indexes to return the job ids fast and ordered properly and then do the full query with a WHERE ID in (...).

  ~

When using a large offset mysql queries are very slow even when using index if there are a lot of other fields in the SELECT. When filters are only related to ia_job we do a SELECT id FROM ia_job query which uses the available indexes to return the job ids fast and ordered properly and then do the full query with a WHERE ID in (...).

   
   

There is a great performance increase for all queries for which we had ia_job indexes but were slowed down by mysql's poor limit handling.

   
~  

(Valentin: what you were proposing was much more complicated to code and didn't actually work ... when selecting with where clauses (for example compiler_id), results aren't always sorted by id, so reversing the results doesn't work)

  ~

Benchmarks: http://hackers.devnet.ro/wiki/Benchmarks

  +
  +

The monitor?round=arhiva&first_entry=100000 is still slow, but I think this is because round is a varchar(64) and thus the index_key is pretty big. If we add numeric ids to round this should work a lot better too (but that's for another patch).

Diff:

Revision 3 (+151 -81)

Show changes

Valentin Stanciu
We could make round index only for the first 10 characters and it should be enough while reducing memory at the same time.
  1. I meant:
    We could make round index only for the first 10 characters and it should reduce the size of the index and increase query performance because of the memory reduction.
  2. I tried that now, but if the index is not on full length, EXPLAIN tells me that mysql doesn't it for the query. It's much slower that way.
    
Mircea Pasoi

Loading...