… I’ve found. I’m using PHP 5.1.6, MySQL 5.0.24a default installs from Ubuntu 6.10, so… instead it remains locked. Another PHP script attempting to insert takes 50 seconds to time-out with a “1205 – Lock wait timeout exceeded; try restarting transaction“. The 1205 error won’t go away until I restart Apache. For whatever reason, the resources…

The post Comment: Pitfalls of converting to InnoDB appeared first on MySQL Performance Blog.

… ? I just tried it with PS 5.5 and default (REPEATABLE READ) isolation mode and the insert is failing mysql> START TRANSACTION; Query OK, 0 rows affected (0.00 sec) mysql> INSERT INTO t VALUES(26); ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction mysql> show…

The post Comment: InnoDB's gap locks appeared first on MySQL Performance Blog.

…here – but not – just waiting… In 30 sec Session1: ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction So as we can …waiting… Session2: mysql> update test2 set names=’mysql2′ where id=1; ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction…

The post Post: Test Drive of Solid appeared first on MySQL Performance Blog.

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some tables, etc.

user1> CREATE TABLE t1 (a INT);
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the MDL created by mysqldump
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this blocks on a pending MDL request by user1
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

user1> CREATE TABLE t1 (a INT);
mysqldump> LOCK TABLES FOR BACKUP;
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the backup MDL lock
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this one is not blocked

# copy InnoDB data
FLUSH TABLES WITH READ LOCK;
# copy .frm, MyISAM, etc.
# get the binary log coordinates
# finalize the background copy of REDO log
UNLOCK TABLES;

# copy InnoDB data
LOCK TABLES FOR BACKUP;
# copy .frm, MyISAM, etc
LOCK BINLOG FOR BACKUP;
# finalize the background copy of REDO log
UNLOCK TABLES;
# get the binary log coordinates
UNLOCK BINLOG;

The post Introducing backup locks in Percona Server appeared first on MySQL Performance Blog.

Not so long ago Solid released solidDB for MySQL Beta 3 so I decided now is time to take a bit closer look on new transactional engine for MySQL. While my far goal is the performance and scalability testing before I wanted to look at basic transactional properties such as deadlock detection, select for update handling and phantom reads in the repeatable read isolation level.

Solid has OPTMISTIC (default) and PESSIMISTIC concurrency control, so it was interesting to test both.

We used default isolation mode which is REPEATABLE-READ for this test.

Test 1: Solid, deadlock detection, default (OPTIMISTIC) concurrency control.

CREATE TABLE `test2` (
  `id` int(11) NOT NULL,
  `names` varchar(255) default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=solidDB DEFAULT CHARSET=latin1
insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
mysql> begin;
mysql> update test2 set names='mysql' where id=1;
Session 2:
mysql> begin;
mysql> update test2 set names='Solid' where id=2;
Session 1:
mysql> update test2 set names='solid1' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So instead of waiting on row with id=2 to be unlocked by transaction in session 1 with OPTIMISTIC concurrency Solid simply reports deadlock due to conflicting update. For applications with rare update conflicts this actually may be good because it is simplier – you do not have to deal with complicated lock graphs or rely on timeout for deadlock resolution. For workloads with frequent row waits this however would result in increased number of deadlocks.

Test 2: SELECT FOR UPDATE:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id =2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session 2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

As we can see deadlock happens in this case even for transaction which runs plain select. Behavior shown in this case does not match the one you would expect from multi versining engine – all the meaning of multi versioning is to allow you to perform consistent read even if rows are locked or even modified unless you require locking read, in which case SELECT FOR UPDATE or SELECT … LOCK IN SHARE MODE can be used. Quite possibly it is a bug which will be fixed in the future, if not it will be serious handycap.

Too bas Solid does not yet have lock investigation tool as SHOW INNODB STATUS (at least) – so it is hard to tell what is really happening.

Test 3: Phantom rows:

Table before test:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

So there are no phantom rows which is what we would expect from repeatable-read isolation mode, at least in its interpretation by Innodb.

Let’s look how PESIMISTIC works (mysqld started with –soliddb_pessimistic).

Test 1: Deadlock detection:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id = 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session1:
mysql> update test2 set names='solid1' where id = 2;
...waiting...
Session2:
mysql> update test2 set names='mysql1' where id=1;
I would expect deadlock here - but not - just waiting...
In 30 sec Session1:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

So as we can see Solid does not have instant deadlock detection as Innodb and have to rely on timeouts instead. This may be serious problem for some applications, especially interactive ones (ie web) – instead of getting quick deadlock and retrying transaction or giving error to the user you’re forced to wait. Setting lock wait timeout to some small value is not optimal solution ether as it would make some transactions to be terminated even though they just had normal lock wait not a deadlock.

Test 2: SELECT FOR UPDATE:

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
..waiting..
Session1:
mysql> commit;
Session2:
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (6.11 sec)

Result well matches one which we got with OPTIMISTIC locking – just instead of simply balling out on first row lock conflict, query waits for rows to be unlocked. But wait why is it waiting if it suppose to be multi versioning system ?

Test 3: Phantom rows:

before test:
mysql> select * from test2;
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
+----+-----------+
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
...waiting...
Session1:
commit;
Session2:
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
|  4 | HEAP      | <- phantom
+----+-----------+
4 rows in set (4.25 sec)
show variables like 'tx%';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| tx_isolation  | REPEATABLE-READ |
+---------------+-----------------+
1 row in set (0.00 sec)

In this case we get phantom row, which looks like double bug to us. First we should not get phantom row in repeatable-read isolation mode, second query results should not be different in OPTIMISTIC vs PESSIMISTIC concurrency mode. We can get deadlocks differently but query results should be the same.

Test 4: Update Handling:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> update test2 set names='SolidDB' where id=2;
... waiting...
in 30 sec:
ERROR 1031 (HY000): Table storage engine for 'test2' doesn't have this option

Strange error message. This is probably one more bug which hopefully be resolved in further versions.

Test 5: UPDATE / SELECT FOR UPDATE in OPTIMISTIC mode

Solid optimistic:
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
Session1:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
mysql> update test2 set names='SolidDB2' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.01 sec
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Please note transaction 1 was commited.
Session2:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
In this case we can se Multi-versioning in action, so it seems to work in some cases.  However behavior is different from Innodb which will do READ-COMMITED for locking reads (SELECT FOR UPDATE/LOCK IN SHARE MODE) as only actual row versions can be locked.   With Innodb we would get "SolidDB2" as result.
mysql> update test2 set names='SolidDB3' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Error message is obviously wrong in this case. It is not deadlock but update conflict – other transaction has already modified row and commited so update could not proceed.

Also in this case we can see SELECT FOR UPDATE does not play nicely with OPTIMISTIC concurrency. Generally you would use this statement to ensure rows are locked at once so transaction could proceed modifying them without causing deadlocks.

For comparion here are InnoDB sessions:

Test 1: Deadlock detection (deadlock is correctly detected):

insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1  Changed: 0  Warnings: 0
Session1:
mysql>  update test2 set names='solid1' where id=2;
...waiting...
Session2:
mysql> update test2 set names='mysql2' where id=1;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Session1:
Query OK, 1 row affected (9.65 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Test 2: SELECT FOR UPDATE (second session is not blocked):

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Test 3: Phantom rows (no phantom rows in repeatable-reads):

Session1:
mysql> begin;
Sessino2:
mysql> begin;
Session1:
mysql> insert into test2 values (4,'FEDERATED');
Session2:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Conclusion: Current Beta version seems to have number of bugs when it comes to concurrency control, so it is hard to say what we’ll see in final version. Current behavior is however very different from Innodb in many cases which may make porting applications to use SolidDB instead of Innodb complicated.

P.S.

After previous experiments I tried SolidDB under sysbench and got next (1 client):

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> UPDATE sbtests set k=k+1 where id=401692;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> UPDATE sbtests set c='773425598-640282370-898784553-626619839-555294498-279281971-788986238-202873246-188128003-162592967' where id=496201;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So we can’t run SysBench benchmark with current version. Hopefully next beta will be solid enough to run benchmark.

The post Test Drive of Solid appeared first on MySQL Performance Blog.

One of the most important features of InnoDB is the row level locking. This feature provides better concurrency under heavy write load but needs additional precautions to avoid phantom reads and to get a consistent Statement based replication. To accomplish that, row level locking databases also acquire gap locks.

What is a Phantom Read

A Phantom Read happens when in a running transaction, two identical statements get different values, because some other transaction has modified the table’s rows. For example:


transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+


transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;

transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+

Phantom reads do not occur if you’re simply doing a SELECT. They only occur if you do UPDATE or DELETE or SELECT FOR UPDATE. InnoDB provides REPEATABLE READ for read-only SELECT, but it behaves as if you use READ COMMITTED for all write queries, in spite of your chosen transaction isolation level (considering only the two most common isolation levels, REPEATABLE READ and READ COMMITTED).

What is a gap lock?

A gap lock is a lock on the gap between index records. Thanks to this gap lock, when you run the same query twice, you get the same result, regardless other session modifications on that table. This makes reads consistent and therefore makes the replication between servers consistent. If you execute SELECT * FROM id > 1000 FOR UPDATE twice, you expect to get the same value twice. To accomplish that, InnoDB locks all index records found by the WHERE clause with an exclusive lock and the gaps between them with a shared gap lock.

This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:

transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+

Start a transaction and delete the record 25:

transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;

At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:

transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)

After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.

How to troubleshoot gap locks?

Is possible to detect those gap locks using SHOW ENGINE INNODB STATUS:

---TRANSACTION 72C, ACTIVE 755 sec
4 lock struct(s), heap size 1248, 3 row lock(s), undo log entries 1
MySQL thread id 3, OS thread handle 0x7f84a78ba700, query id 163 localhost msandbox
TABLE LOCK table test.t trx id 72C lock mode IX
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X
RECORD LOCKS space id 19 page no 3 n bits 80 index GEN_CLUST_INDEX of table test.t trx id 72C lock_mode X locks rec but not gap
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X locks gap before rec

If you have lot of gaps locks in your transactions affecting the concurrency and the performance you can disable them in two different ways:

1- Change the ISOLATION level to READ COMMITTED. In this isolation level, it is normal and expected that query results can change during a transaction, so there is no need to create locks to prevent that from happening.
2- innodb_locks_unsafe_for_binlog = 1. Disables the gap locks except for foreign-key constraint checking or duplicate-key checking.

The most important difference between these two options is that the second one is a global variable that affects all sessions and needs a server restart to change its value. Both options cause phantom reads (non repeatable reads) so in order to prevent problems with the replication you should change the binary log format to “row”.

Depending on the statement, the behavior of these locks can be different. In the following link there is a good source of information:

http://dev.mysql.com/doc/refman/5.1/en/innodb-locks-set.html.

Conclusion

MySQL uses REPEATABLE READ as the default isolation level so it needs to lock the index records and the gaps to avoid phantom reads and to get a consistent Statement based replication. If your application can deal with phantom reads and your binary log is in row format, changing the ISOLATION to READ COMMITTED will help you to avoid all those extra locks. As a final advice, keep your transactions short

The post InnoDB’s gap locks appeared first on MySQL Performance Blog.

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some tables, etc.

user1> CREATE TABLE t1 (a INT);
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the MDL created by mysqldump
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this blocks on a pending MDL request by user1
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

user1> CREATE TABLE t1 (a INT);
mysqldump> LOCK TABLES FOR BACKUP;
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the backup MDL lock
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this one is not blocked

# copy InnoDB data
FLUSH TABLES WITH READ LOCK;
# copy .frm, MyISAM, etc.
# get the binary log coordinates
# finalize the background copy of REDO log
UNLOCK TABLES;

# copy InnoDB data
LOCK TABLES FOR BACKUP;
# copy .frm, MyISAM, etc
LOCK BINLOG FOR BACKUP;
# finalize the background copy of REDO log
UNLOCK TABLES;
# get the binary log coordinates
UNLOCK BINLOG;

The post Introducing backup locks in Percona Server appeared first on MySQL Performance Blog.

Not so long ago Solid released solidDB for MySQL Beta 3 so I decided now is time to take a bit closer look on new transactional engine for MySQL. While my far goal is the performance and scalability testing before I wanted to look at basic transactional properties such as deadlock detection, select for update handling and phantom reads in the repeatable read isolation level.

Solid has OPTMISTIC (default) and PESSIMISTIC concurrency control, so it was interesting to test both.

We used default isolation mode which is REPEATABLE-READ for this test.

Test 1: Solid, deadlock detection, default (OPTIMISTIC) concurrency control.

CREATE TABLE `test2` (
  `id` int(11) NOT NULL,
  `names` varchar(255) default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=solidDB DEFAULT CHARSET=latin1
insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
mysql> begin;
mysql> update test2 set names='mysql' where id=1;
Session 2:
mysql> begin;
mysql> update test2 set names='Solid' where id=2;
Session 1:
mysql> update test2 set names='solid1' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So instead of waiting on row with id=2 to be unlocked by transaction in session 1 with OPTIMISTIC concurrency Solid simply reports deadlock due to conflicting update. For applications with rare update conflicts this actually may be good because it is simplier – you do not have to deal with complicated lock graphs or rely on timeout for deadlock resolution. For workloads with frequent row waits this however would result in increased number of deadlocks.

Test 2: SELECT FOR UPDATE:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id =2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session 2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

As we can see deadlock happens in this case even for transaction which runs plain select. Behavior shown in this case does not match the one you would expect from multi versining engine – all the meaning of multi versioning is to allow you to perform consistent read even if rows are locked or even modified unless you require locking read, in which case SELECT FOR UPDATE or SELECT … LOCK IN SHARE MODE can be used. Quite possibly it is a bug which will be fixed in the future, if not it will be serious handycap.

Too bas Solid does not yet have lock investigation tool as SHOW INNODB STATUS (at least) – so it is hard to tell what is really happening.

Test 3: Phantom rows:

Table before test:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

So there are no phantom rows which is what we would expect from repeatable-read isolation mode, at least in its interpretation by Innodb.

Let’s look how PESIMISTIC works (mysqld started with –soliddb_pessimistic).

Test 1: Deadlock detection:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id = 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session1:
mysql> update test2 set names='solid1' where id = 2;
...waiting...
Session2:
mysql> update test2 set names='mysql1' where id=1;
I would expect deadlock here - but not - just waiting...
In 30 sec Session1:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

So as we can see Solid does not have instant deadlock detection as Innodb and have to rely on timeouts instead. This may be serious problem for some applications, especially interactive ones (ie web) – instead of getting quick deadlock and retrying transaction or giving error to the user you’re forced to wait. Setting lock wait timeout to some small value is not optimal solution ether as it would make some transactions to be terminated even though they just had normal lock wait not a deadlock.

Test 2: SELECT FOR UPDATE:

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
..waiting..
Session1:
mysql> commit;
Session2:
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (6.11 sec)

Result well matches one which we got with OPTIMISTIC locking – just instead of simply balling out on first row lock conflict, query waits for rows to be unlocked. But wait why is it waiting if it suppose to be multi versioning system ?

Test 3: Phantom rows:

before test:
mysql> select * from test2;
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
+----+-----------+
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
...waiting...
Session1:
commit;
Session2:
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
|  4 | HEAP      | <- phantom
+----+-----------+
4 rows in set (4.25 sec)
show variables like 'tx%';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| tx_isolation  | REPEATABLE-READ |
+---------------+-----------------+
1 row in set (0.00 sec)

In this case we get phantom row, which looks like double bug to us. First we should not get phantom row in repeatable-read isolation mode, second query results should not be different in OPTIMISTIC vs PESSIMISTIC concurrency mode. We can get deadlocks differently but query results should be the same.

Test 4: Update Handling:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> update test2 set names='SolidDB' where id=2;
... waiting...
in 30 sec:
ERROR 1031 (HY000): Table storage engine for 'test2' doesn't have this option

Strange error message. This is probably one more bug which hopefully be resolved in further versions.

Test 5: UPDATE / SELECT FOR UPDATE in OPTIMISTIC mode

Solid optimistic:
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
Session1:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
mysql> update test2 set names='SolidDB2' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.01 sec
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Please note transaction 1 was commited.
Session2:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
In this case we can se Multi-versioning in action, so it seems to work in some cases.  However behavior is different from Innodb which will do READ-COMMITED for locking reads (SELECT FOR UPDATE/LOCK IN SHARE MODE) as only actual row versions can be locked.   With Innodb we would get "SolidDB2" as result.
mysql> update test2 set names='SolidDB3' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Error message is obviously wrong in this case. It is not deadlock but update conflict – other transaction has already modified row and commited so update could not proceed.

Also in this case we can see SELECT FOR UPDATE does not play nicely with OPTIMISTIC concurrency. Generally you would use this statement to ensure rows are locked at once so transaction could proceed modifying them without causing deadlocks.

For comparion here are InnoDB sessions:

Test 1: Deadlock detection (deadlock is correctly detected):

insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1  Changed: 0  Warnings: 0
Session1:
mysql>  update test2 set names='solid1' where id=2;
...waiting...
Session2:
mysql> update test2 set names='mysql2' where id=1;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Session1:
Query OK, 1 row affected (9.65 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Test 2: SELECT FOR UPDATE (second session is not blocked):

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Test 3: Phantom rows (no phantom rows in repeatable-reads):

Session1:
mysql> begin;
Sessino2:
mysql> begin;
Session1:
mysql> insert into test2 values (4,'FEDERATED');
Session2:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Conclusion: Current Beta version seems to have number of bugs when it comes to concurrency control, so it is hard to say what we’ll see in final version. Current behavior is however very different from Innodb in many cases which may make porting applications to use SolidDB instead of Innodb complicated.

P.S.

After previous experiments I tried SolidDB under sysbench and got next (1 client):

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> UPDATE sbtests set k=k+1 where id=401692;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> UPDATE sbtests set c='773425598-640282370-898784553-626619839-555294498-279281971-788986238-202873246-188128003-162592967' where id=496201;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So we can’t run SysBench benchmark with current version. Hopefully next beta will be solid enough to run benchmark.

The post Test Drive of Solid appeared first on MySQL Performance Blog.

One of the most important features of InnoDB is the row level locking. This feature provides better concurrency under heavy write load but needs additional precautions to avoid phantom reads and to get a consistent Statement based replication. To accomplish that, row level locking databases also acquire gap locks.

What is a Phantom Read

A Phantom Read happens when in a running transaction, two identical statements get different values, because some other transaction has modified the table’s rows. For example:


transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+


transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;

transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+

Phantom reads do not occur if you’re simply doing a SELECT. They only occur if you do UPDATE or DELETE or SELECT FOR UPDATE. InnoDB provides REPEATABLE READ for read-only SELECT, but it behaves as if you use READ COMMITTED for all write queries, in spite of your chosen transaction isolation level (considering only the two most common isolation levels, REPEATABLE READ and READ COMMITTED).

What is a gap lock?

A gap lock is a lock on the gap between index records. Thanks to this gap lock, when you run the same query twice, you get the same result, regardless other session modifications on that table. This makes reads consistent and therefore makes the replication between servers consistent. If you execute SELECT * FROM id > 1000 FOR UPDATE twice, you expect to get the same value twice. To accomplish that, InnoDB locks all index records found by the WHERE clause with an exclusive lock and the gaps between them with a shared gap lock.

This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:

transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+

Start a transaction and delete the record 25:

transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;

At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:

transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)

After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.

How to troubleshoot gap locks?

Is possible to detect those gap locks using SHOW ENGINE INNODB STATUS:

---TRANSACTION 72C, ACTIVE 755 sec
4 lock struct(s), heap size 1248, 3 row lock(s), undo log entries 1
MySQL thread id 3, OS thread handle 0x7f84a78ba700, query id 163 localhost msandbox
TABLE LOCK table test.t trx id 72C lock mode IX
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X
RECORD LOCKS space id 19 page no 3 n bits 80 index GEN_CLUST_INDEX of table test.t trx id 72C lock_mode X locks rec but not gap
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X locks gap before rec

If you have lot of gaps locks in your transactions affecting the concurrency and the performance you can disable them in two different ways:

1- Change the ISOLATION level to READ COMMITTED. In this isolation level, it is normal and expected that query results can change during a transaction, so there is no need to create locks to prevent that from happening.
2- innodb_locks_unsafe_for_binlog = 1. Disables the gap locks except for foreign-key constraint checking or duplicate-key checking.

The most important difference between these two options is that the second one is a global variable that affects all sessions and needs a server restart to change its value. Both options cause phantom reads (non repeatable reads) so in order to prevent problems with the replication you should change the binary log format to “row”.

Depending on the statement, the behavior of these locks can be different. In the following link there is a good source of information:

http://dev.mysql.com/doc/refman/5.1/en/innodb-locks-set.html.

Conclusion

MySQL uses REPEATABLE READ as the default isolation level so it needs to lock the index records and the gaps to avoid phantom reads and to get a consistent Statement based replication. If your application can deal with phantom reads and your binary log is in row format, changing the ISOLATION to READ COMMITTED will help you to avoid all those extra locks. As a final advice, keep your transactions short

The post InnoDB’s gap locks appeared first on MySQL Performance Blog.

I am developing a concurrent application that uses TokuDB to store its database. Sometimes, one of my SQL statements returns with a ‘lock wait timeout exceeded’ error. How do I identify the cause of this error? First, I need to understand a little bit about how TokuDB transactions use locks. Then, I need to understand how to use the MySQL information schema to look at the current state of the locks.

Transactions and Locks

TokuDB uses key range locks to implement serializable transactions. These locks are acquired as the transaction progresses. The locks are released when the transaction commits or aborts.

TokuDB stores these locks in a data structure called the lock tree. The lock tree stores the set of range locks granted to each transaction. In addition, the lock tree stores the set of locks that are not granted due to a conflict with locks granted to some other transaction.

TokuDB 7.1 provides access to the set of locks taken and the set of locks that are blocked via tables in the MySQL information schema. In addition, TokuDB stores the last lock conflict information in a session variable that can be retrieved by the application.

Example

Here is an example debug session.

I create a table with a single column that is the primary key.

mysql> show create table t;
Create Table: CREATE TABLE `t` (
`id` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1;

I have 2 MySQL clients with ID’s 1 and 2 respectively. MySQL client 1 inserts some values into the table ‘t’. TokuDB transaction 51 is created for the insert statement. Since autocommit is off, transaction 51 is still live after the insert statement completes, and I can query the ‘tokudb_locks’ information schema table to see the locks that are held by the transaction.

mysql 1> set autocommit=off;
mysql 1> insert into t values (1),(10),(100);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql 1> select * from information_schema.tokudb_locks;
+--------------+-----------------------+---------------+----------------+-----------------+
| locks_trx_id | locks_mysql_thread_id | locks_dname   | locks_key_left | locks_key_right |
+--------------+-----------------------+---------------+----------------+-----------------+
| 51           | 1                     | ./test/t-main | 0001000000     | 0001000000      |
| 51           | 1                     | ./test/t-main | 000a000000     | 000a000000      |
| 51           | 1                     | ./test/t-main | 0064000000     | 0064000000      |
+--------------+-----------------------+---------------+----------------+-----------------+
mysql 1> select * from information_schema.tokudb_lock_waits;
Empty set (0.00 sec)

The keys are currently hex dumped. Someday we will pretty print the keys.
Now we switch to the other MySQL client with ID 2.

mysql 2> insert into t values (2),(20),(100);

The insert gets blocked since there is a conflict on the primary key with value 100.
The granted TokuDB locks are:

mysql 1> select * from information_schema.tokudb_locks;
+--------------+-----------------------+---------------+----------------+-----------------+
| locks_trx_id | locks_mysql_thread_id | locks_dname   | locks_key_left | locks_key_right |
+--------------+-----------------------+---------------+----------------+-----------------+
| 51           | 1                     | ./test/t-main | 0001000000     | 0001000000      |
| 51           | 1                     | ./test/t-main | 000a000000     | 000a000000      |
| 51           | 1                     | ./test/t-main | 0064000000     | 0064000000      |
| 62           | 2                     | ./test/t-main | 0002000000     | 0002000000      |
| 62           | 2                     | ./test/t-main | 0014000000     | 0014000000      |
+--------------+-----------------------+---------------+----------------+-----------------+

The locks that are pending due to a conflict are:

mysql 1> select * from information_schema.tokudb_lock_waits;
+-------------------+-----------------+------------------+---------------------+----------------------+-----------------------+
| requesting_trx_id | blocking_trx_id | lock_waits_dname | lock_waits_key_left | lock_waits_key_right | lock_waits_start_time |
+-------------------+-----------------+------------------+---------------------+----------------------+-----------------------+
| 62                | 51              | ./test/t-main    | 0064000000          | 0064000000           | 1380656990910         |
+-------------------+-----------------+------------------+---------------------+----------------------+-----------------------+

Eventually, the lock for client 2 times out, and we can retrieve a JSON coded document that describes the conflict.

mysql 2 insert continued>
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql 2> select @@tokudb_last_lock_timeout;
+-------------------------------------------------------------------------------------------------------------+
| @@tokudb_last_lock_timeout |
+-------------------------------------------------------------------------------------------------------------+
| {"mysql_thread_id":2,"dbname":"./test/t-main","requesting_txnid":62,"blocking_txnid":51,"key":"0064000000"} |
+-------------------------------------------------------------------------------------------------------------+

The JSON document encodes the database, table and index where the conflict occurred in the ‘dname’ key. The ‘key’ encodes the conflicting key. The ‘blocking_txnid’ encodes the transaction identifier that is blocking my transaction. I can find the MySQL client that created the blocking transaction via joins of information schema tables.

mysql> select * from information_schema.tokudb_trx,information_schema.processlist
where trx_id = 51 and trx_mysql_thread_id = id;
+--------+---------------------+----+----------+-----------+------+---------+------+-------+------+
| trx_id | trx_mysql_thread_id | ID | USER     | HOST      | DB   | COMMAND | TIME | STATE | INFO |
+--------+---------------------+----+----------+-----------+------+---------+------+-------+------+
| 51     | 1                   | 1  | prohaska | localhost | test | Sleep   | 151  |       | NULL |
+--------+---------------------+----+----------+-----------+------+---------+------+-------+------+

Now I have information about the two client operations that are conflicting and can hopefully change my application to avoid this conflict.

The post Interactive Debugging of Transaction Conflicts with TokuDB appeared first on MySQL Performance Blog.

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some tables, etc.

user1> CREATE TABLE t1 (a INT);
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the MDL created by mysqldump
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this blocks on a pending MDL request by user1
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

user1> CREATE TABLE t1 (a INT);
mysqldump> LOCK TABLES FOR BACKUP;
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the backup MDL lock
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this one is not blocked

# copy InnoDB data
FLUSH TABLES WITH READ LOCK;
# copy .frm, MyISAM, etc.
# get the binary log coordinates
# finalize the background copy of REDO log
UNLOCK TABLES;

# copy InnoDB data
LOCK TABLES FOR BACKUP;
# copy .frm, MyISAM, etc
LOCK BINLOG FOR BACKUP;
# finalize the background copy of REDO log
UNLOCK TABLES;
# get the binary log coordinates
UNLOCK BINLOG;

The post Introducing backup locks in Percona Server appeared first on MySQL Performance Blog.