Quantcast
Channel: MySQL Performance Blog » Search Results » mysql lock wait timeout exceeded; try restarting transaction
Viewing all 11 articles
Browse latest View live

Comment: Pitfalls of converting to InnoDB

$
0
0

… I’ve found. I’m using PHP 5.1.6, MySQL 5.0.24a default installs from Ubuntu 6.10, so… instead it remains locked. Another PHP script attempting to insert takes 50 seconds to time-out with a “1205 – Lock wait timeout exceeded; try restarting transaction“. The 1205 error won’t go away until I restart Apache. For whatever reason, the resources…

The post Comment: Pitfalls of converting to InnoDB appeared first on MySQL Performance Blog.


Comment: InnoDB's gap locks

$
0
0

… ? I just tried it with PS 5.5 and default (REPEATABLE READ) isolation mode and the insert is failing mysql> START TRANSACTION; Query OK, 0 rows affected (0.00 sec) mysql> INSERT INTO t VALUES(26); ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction mysql> show…

The post Comment: InnoDB's gap locks appeared first on MySQL Performance Blog.

Post: Test Drive of Solid

$
0
0

…here – but not – just waiting… In 30 sec Session1: ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction So as we can …waiting… Session2: mysql> update test2 set names=’mysql2′ where id=1; ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

The post Post: Test Drive of Solid appeared first on MySQL Performance Blog.

Introducing backup locks in Percona Server

$
0
0

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

In the beginning…

In the beginning there was FLUSH TABLES, and users messed with their MyISAM tables under a live server and were not ashamed. Users could do nice things like:

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some tables, etc.

And users were happy until someone realized that tables must be protected against concurrent access by queries in other connections. So Monty gave them FLUSH TABLES WITH READ LOCK, and users were enlightened.

Online backups

Users then started dreaming about online backups, i.e. creating consistent snapshots of a live MySQL server. mysqldump --lock-all-tables had been a viable option for a while. To provide consistency it used FLUSH TABLES WITH READ LOCK which was not quite the right tool for the job, but was “good enough”. Who cares if a mom-and-pop shop becomes unavailable for a few seconds required to dump ~100 MB of data, right?

With InnoDB gaining popularity users had realized that one could employ MVCC to guarantee consistency and FLUSH TABLES WITH READ LOCK doesn’t make much sense for InnoDB tables anyway (you cannot modify InnoDB tables under a live server even if the server is read-only). So Peter gave mysqldump the --single-transaction option, and users were enlightened. mysqldump --single-transaction allowed to avoid FTWRL, but there was a few catches:

  • one cannot perform any schema modifications or updates to non-InnoDB tables while mysqldump --single-transaction is in progress, because those operations are not transactional and thus would ignore the data snapshot created by --single-transaction;
  • one cannot get binary log coordinates with --master-data or
    --dump-slave, because in that case FTWRL would still be used to ensure that the binary log coordinates are consistent with the data dump;

Which makes --single-transaction similar to the --no-lock option in Percona XtraBackup: it shifts the responsibility for backup consistency to the user. Any change in the workload violating the prerequisites for those options may result in a broken backup without any signs for the user to take action.

Present

Fast forward to present day. MySQL is capable of handling over a million queries per second, MyISAM is certainly not a popular choice to store data, and there are many backup solutions to choose from. Yet all of them still rely on FLUSH TABLES WITH READ LOCK in one way or another to guarantee consistency of .frm files, non-transactional tables and binary log coordinates.

To some extent, the problem with concurrent DDL + mysqldump --single-transaction has been alleviated with metadata locks in MySQL 5.5, which however made some users unhappy, and that behavior was partially reverted in MySQL 5.6.16 with the fix for bug #71017. But the fundamental problem is still there: mysqldump --single-transaction does not guarantee consistency with concurrent DDL statements and updates to non-transactional tables.

So the fact that FTWRL is an overkill for backups has been increasingly obvious for the reasons described below.

What’s the problem with FTWRL anyway?

A lot has been written on what FLUSH TABLES WITH READ LOCK really does. Here’s yet another walk-through in a bit more detail than described elsewhere:

  1. It first invalidates the Query Cache.
  2. It then waits for all in-flight updates to complete and at the same time it blocks all incoming updates. This is one problem for busy servers.
  3. It then closes all open tables (the FLUSH part) and expels them from the table cache. This is also when FTWRL has to wait for all SELECT queries to complete. And this is another, even bigger problem for busy servers, because that wait happens to occur with all updates blocked. What’s even worse, the server at this stage is essentially offline, because even incoming SELECT queries will get blocked.
  4. Finally, it blocks COMMITs.

Action #4 is not required for the original purpose of FTWRL, but is rather a kludge implemented due to the fact that FTWRL is (mis)used by backup utilities.

Actions #1-3 make perfect sense for the original reasons why FTWRL has been implemented. If we are going to access and possibly modify tables outside of the server, we want the server to forget everything it knows about both schema and data for all tables, and flush all in-memory buffers to make the on-disk data representation consistent.

And that’s what makes it an overkill for backup utilities: they don’t require #1, because they never modify data. #2 is only required for non-InnoDB tables, because InnoDB provides other ways to ensure consistency for both logical and physical backups. And #3 is certainly not a problem for logical backup utilities like mysqldump or mydumper, because they don’t even access on-disk data directly. As we will see, it is not a big problem for physical backup solutions either.

To FLUSH or not to FLUSH?

So what exactly is flushed by FLUSH TABLES WITH READ LOCK?

Nothing for InnoDB tables, and no physical backup solution require it to flush anything.

For MyISAM it is more complicated. MyISAM key caches are normally write-through, i.e. by the time each update to a MyISAM table completes, all index updates are written to disk. The only exception is delayed key writing feature which you should not be using anyway, if you care about your data. MyISAM may also do data buffering for bulk inserts, e.g. while executing multi-row INSERTs or LOAD DATA statements. Those buffers, however, are flushed between statements, so have no effect on physical backups as long as we block all statements updating MyISAM tables.

The point is that without flushing each storage engine is not any less backup-safe as it is crash-safe, with the only difference that backups are guaranteed to wait for all currently executing INSERT/REPLACE/DELETE/UPDATE statements to complete.

Backup locks

Enter the backup locks feature. The following 3 new SQL statements have been introduced in Percona Server:

  • LOCK TABLES FOR BACKUP
  • LOCK BINLOG FOR BACKUP
  • UNLOCK BINLOG

LOCK TABLES FOR BACKUP

Quoting the documentation page from the manual:

LOCK TABLES FOR BACKUP uses a new MDL lock type to block updates to non-transactional tables and DDL statements for all tables. More specifically, if there’s an active LOCK TABLES FOR BACKUP lock, all DDL statements and updates to MyISAM, CSV, MEMORY and ARCHIVE tables will be blocked in the “Waiting for backup lock” status as visible in PERFORMANCE_SCHEMA or PROCESSLIST. SELECT queries for all tables and INSERT/REPLACE/UPDATE/DELETE against InnoDB, Blackhole and Federated tables are not affected by LOCK TABLES FOR BACKUP. Blackhole tables obviously have no relevance for backups, and Federated tables are ignored by both logical and physical backup tools.

Like FTWRL, the LOCK TABLES FOR BACKUP statement:

  • blocks updates to MyISAM, MEMORY, CSV and ARCHIVE tables;
  • blocks DDL against any tables;
  • does not block updates to temporary and log tables.

Unlike FTWRL, the LOCK TABLES FOR BACKUP statement:

  • does not invalidate the Query Cache;
  • never waits for SELECT queries to complete regardless of the storage engines involved;
  • never blocks SELECTs, or updates to InnoDB, Blackhole and Federated tables.

In other words, it does exactly what backup utilities need: block non-transactional changes that are included into the backup, and leave everything else to InnoDB MVCC and crash recovery.

With the only exception of binary log coordinates obtained with SHOW MASTER STATUS and SHOW SLAVE STATUS.

LOCK BINLOG FOR BACKUP

This is when LOCK BINLOG FOR BACKUP comes in handy. It blocks all updates to binary log coordinates as reported by SHOW MASTER/SLAVE STATUS and used by backup utilities. It has no effect when all of the following conditions apply:

  • when the binary log is disabled. If it is disabled globally, then all connections will not be affected by LOCK BINLOG FOR BACKUP. If it is enabled globally, but disabled for specific connections via sql_log_bin, only those connections are allowed to commit;
  • the server is not a replication slave;

Even if binary logging is used, LOCK BINLOG FOR BACKUP will allow DDL and updates to any tables to proceed until they will be written to binlog (i.e. commit), and/or advance Exec_Master_Log_* / Exec_Gtid_Set when executed by a replication thread, provided that no other global locks are acquired.

To release the lock acquired by LOCK TABLES FOR BACKUP there’s already UNLOCK TABLES. And the LOCK BINLOG FOR BACKUP lock is released with UNLOCK BINLOG.

Let’s look how these statements can be used by backup utilities.

mysqldump

mysqldump got a new option, --lock-for-backup which along with --single-transaction essentially obsoletes --lock-all-tables (i.e. FLUSH TABLES WITH READ LOCK). It makes mysqldump use LOCK TABLES FOR BACKUP before it starts dumping tables to block all “unsafe” statement that might otherwise interfere with backup consistency.

Of course, that requires backup locks support by the target server, so mysqldump checks if they are indeed supported and fails with an error if they are not.

However, at the moment if binary lock coordinates are requested with --master-data, FTWRL is still used even if --lock-for-backup is specified. mysqldump could use LOCK BINLOG FOR BACKUP, but there’s a better solution for logical backups implemented in MariaDB, which has already been ported to Percona Server and queued for the next release.

There is also another important difference between just mysqldump --single-transaction and mysqldump --lock-for-backup --single-transaction. As of MySQL 5.5 mysqldump --single-transaction acquires shared metadata locks on all tables processed within the transaction. Which will also block DDL statements on those tables when they will try to acquire an exclusive lock. So far, so good. The problems start when there’s an incoming SELECT query against a table that already has a pending DDL statement. It will also be blocked on a pending exclusive MDL request for no apparent reasons. Which was one of the complaints in bug #71017.

It’s better illustrated with an example. Suppose there are 3 sessions: one created by mysqldump, and 2 user sessions.

user1> CREATE TABLE t1 (a INT);
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the MDL created by mysqldump
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this blocks on a pending MDL request by user1
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

This is what would happen with mysqldump --lock-for-backup --single-transaction:

user1> CREATE TABLE t1 (a INT);
mysqldump> LOCK TABLES FOR BACKUP;
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the backup MDL lock
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this one is not blocked

This immediate problem was partially fixed in MySQL 5.6.16 by releasing metadata locks after processing each table with the help of savepoints. There is a couple of issues with this approach:

  • there is still a table metadata lock for the duration of SELECT executed by mysqldump. Which, as before, blocks DDL. So there is still a chance that mysqldump --single-transaction may eventually block SELECT queries.
  • after the table is processed and the metadata lock is released, there is now an opportunity for RENAME to break the backup, see bug #71214.

Both issues above along with bug #71215 and bug #71216 do not exist with mysqldump --lock-for-backup --single-transaction as all kinds of DDL statements are properly isolated by backup locks, which do not block SELECT queries at the same time.

Percona XtraBackup

Percona XtraBackup 2.2 will support backup locks and use them automatically if supported by the server being backed up.

The current locking used by XtraBackup is:

# copy InnoDB data
FLUSH TABLES WITH READ LOCK;
# copy .frm, MyISAM, etc.
# get the binary log coordinates
# finalize the background copy of REDO log
UNLOCK TABLES;

With backup locks it becomes:

# copy InnoDB data
LOCK TABLES FOR BACKUP;
# copy .frm, MyISAM, etc
LOCK BINLOG FOR BACKUP;
# finalize the background copy of REDO log
UNLOCK TABLES;
# get the binary log coordinates
UNLOCK BINLOG;

Note that under the following conditions, no blocking occurs at any stage in the server:

  • no updates to non-transactional tables;
  • no DDL;
  • binary log is disabled;

They may look familiar, because they are essentially prerequisites for the --no-lock option. Except that with backup locks, you don’t have to take chances and take responsibility for backup consistency. All the locking will be handled automatically by the server, if and when it is necessary.

mylvmbackup

mylvmbackup takes the server read-only with FLUSH TABLES WITH READ LOCK while the snapshot is being created for two reasons:

  • flush non-transactional tables
  • ensure consistency with the binary log coordinates

For exactly the same reasons as with XtraBackup, it can use backup locks instead of FTWRL.

mydumper

mydumper developers may want to add support for backup locks as well. mydumper relies on START TRANSACTION WITH CONSISTENT SNAPSHOT to ensure InnoDB consistency, but has to resort to FLUSH TABLES WITH READ LOCK to ensure consistency of non-InnoDB tables and binary log coordinates.

Another problem is that START TRANSACTION WITH CONSISTENT SNAPSHOT is not supposed to be used by multi-threaded logical backup utilities. But that is an opportunity for another server-side improvement and probably a separate blog post.

The post Introducing backup locks in Percona Server appeared first on MySQL Performance Blog.

Test Drive of Solid

$
0
0

Not so long ago Solid released solidDB for MySQL Beta 3 so I decided now is time to take a bit closer look on new transactional engine for MySQL. While my far goal is the performance and scalability testing before I wanted to look at basic transactional properties such as deadlock detection, select for update handling and phantom reads in the repeatable read isolation level.

Solid has OPTMISTIC (default) and PESSIMISTIC concurrency control, so it was interesting to test both.

We used default isolation mode which is REPEATABLE-READ for this test.

Test 1: Solid, deadlock detection, default (OPTIMISTIC) concurrency control.

CREATE TABLE `test2` (
  `id` int(11) NOT NULL,
  `names` varchar(255) default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=solidDB DEFAULT CHARSET=latin1
insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
mysql> begin;
mysql> update test2 set names='mysql' where id=1;
Session 2:
mysql> begin;
mysql> update test2 set names='Solid' where id=2;
Session 1:
mysql> update test2 set names='solid1' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So instead of waiting on row with id=2 to be unlocked by transaction in session 1 with OPTIMISTIC concurrency Solid simply reports deadlock due to conflicting update. For applications with rare update conflicts this actually may be good because it is simplier – you do not have to deal with complicated lock graphs or rely on timeout for deadlock resolution. For workloads with frequent row waits this however would result in increased number of deadlocks.

Test 2: SELECT FOR UPDATE:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id =2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session 2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

As we can see deadlock happens in this case even for transaction which runs plain select. Behavior shown in this case does not match the one you would expect from multi versining engine – all the meaning of multi versioning is to allow you to perform consistent read even if rows are locked or even modified unless you require locking read, in which case SELECT FOR UPDATE or SELECT … LOCK IN SHARE MODE can be used. Quite possibly it is a bug which will be fixed in the future, if not it will be serious handycap.

Too bas Solid does not yet have lock investigation tool as SHOW INNODB STATUS (at least) – so it is hard to tell what is really happening.

Test 3: Phantom rows:

Table before test:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

So there are no phantom rows which is what we would expect from repeatable-read isolation mode, at least in its interpretation by Innodb.

Let’s look how PESIMISTIC works (mysqld started with –soliddb_pessimistic).

Test 1: Deadlock detection:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id = 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session1:
mysql> update test2 set names='solid1' where id = 2;
...waiting...
Session2:
mysql> update test2 set names='mysql1' where id=1;
I would expect deadlock here - but not - just waiting...
In 30 sec Session1:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

So as we can see Solid does not have instant deadlock detection as Innodb and have to rely on timeouts instead. This may be serious problem for some applications, especially interactive ones (ie web) – instead of getting quick deadlock and retrying transaction or giving error to the user you’re forced to wait. Setting lock wait timeout to some small value is not optimal solution ether as it would make some transactions to be terminated even though they just had normal lock wait not a deadlock.

Test 2: SELECT FOR UPDATE:

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
..waiting..
Session1:
mysql> commit;
Session2:
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (6.11 sec)

Result well matches one which we got with OPTIMISTIC locking – just instead of simply balling out on first row lock conflict, query waits for rows to be unlocked. But wait why is it waiting if it suppose to be multi versioning system ?

Test 3: Phantom rows:

before test:
mysql> select * from test2;
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
+----+-----------+
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
...waiting...
Session1:
commit;
Session2:
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
|  4 | HEAP      | <- phantom
+----+-----------+
4 rows in set (4.25 sec)
show variables like 'tx%';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| tx_isolation  | REPEATABLE-READ |
+---------------+-----------------+
1 row in set (0.00 sec)

In this case we get phantom row, which looks like double bug to us. First we should not get phantom row in repeatable-read isolation mode, second query results should not be different in OPTIMISTIC vs PESSIMISTIC concurrency mode. We can get deadlocks differently but query results should be the same.

Test 4: Update Handling:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> update test2 set names='SolidDB' where id=2;
... waiting...
in 30 sec:
ERROR 1031 (HY000): Table storage engine for 'test2' doesn't have this option

Strange error message. This is probably one more bug which hopefully be resolved in further versions.

Test 5: UPDATE / SELECT FOR UPDATE in OPTIMISTIC mode

Solid optimistic:
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
Session1:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
mysql> update test2 set names='SolidDB2' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.01 sec
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Please note transaction 1 was commited.
Session2:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
In this case we can se Multi-versioning in action, so it seems to work in some cases.  However behavior is different from Innodb which will do READ-COMMITED for locking reads (SELECT FOR UPDATE/LOCK IN SHARE MODE) as only actual row versions can be locked.   With Innodb we would get "SolidDB2" as result.
mysql> update test2 set names='SolidDB3' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Error message is obviously wrong in this case. It is not deadlock but update conflict – other transaction has already modified row and commited so update could not proceed.

Also in this case we can see SELECT FOR UPDATE does not play nicely with OPTIMISTIC concurrency. Generally you would use this statement to ensure rows are locked at once so transaction could proceed modifying them without causing deadlocks.

For comparion here are InnoDB sessions:

Test 1: Deadlock detection (deadlock is correctly detected):

insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1  Changed: 0  Warnings: 0
Session1:
mysql>  update test2 set names='solid1' where id=2;
...waiting...
Session2:
mysql> update test2 set names='mysql2' where id=1;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Session1:
Query OK, 1 row affected (9.65 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Test 2: SELECT FOR UPDATE (second session is not blocked):

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Test 3: Phantom rows (no phantom rows in repeatable-reads):

Session1:
mysql> begin;
Sessino2:
mysql> begin;
Session1:
mysql> insert into test2 values (4,'FEDERATED');
Session2:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Conclusion: Current Beta version seems to have number of bugs when it comes to concurrency control, so it is hard to say what we’ll see in final version. Current behavior is however very different from Innodb in many cases which may make porting applications to use SolidDB instead of Innodb complicated.

P.S.

After previous experiments I tried SolidDB under sysbench and got next (1 client):

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> UPDATE sbtests set k=k+1 where id=401692;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> UPDATE sbtests set c='773425598-640282370-898784553-626619839-555294498-279281971-788986238-202873246-188128003-162592967' where id=496201;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So we can’t run SysBench benchmark with current version. Hopefully next beta will be solid enough to run benchmark.

The post Test Drive of Solid appeared first on MySQL Performance Blog.

InnoDB’s gap locks

$
0
0

One of the most important features of InnoDB is the row level locking. This feature provides better concurrency under heavy write load but needs additional precautions to avoid phantom reads and to get a consistent Statement based replication. To accomplish that, row level locking databases also acquire gap locks.

What is a Phantom Read

A Phantom Read happens when in a running transaction, two identical statements get different values, because some other transaction has modified the table’s rows. For example:


transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+


transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;

transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+

Phantom reads do not occur if you’re simply doing a SELECT. They only occur if you do UPDATE or DELETE or SELECT FOR UPDATE. InnoDB provides REPEATABLE READ for read-only SELECT, but it behaves as if you use READ COMMITTED for all write queries, in spite of your chosen transaction isolation level (considering only the two most common isolation levels, REPEATABLE READ and READ COMMITTED).

What is a gap lock?

A gap lock is a lock on the gap between index records. Thanks to this gap lock, when you run the same query twice, you get the same result, regardless other session modifications on that table. This makes reads consistent and therefore makes the replication between servers consistent. If you execute SELECT * FROM id > 1000 FOR UPDATE twice, you expect to get the same value twice. To accomplish that, InnoDB locks all index records found by the WHERE clause with an exclusive lock and the gaps between them with a shared gap lock.

This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:

transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+

Start a transaction and delete the record 25:

transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;

At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:

transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)

After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.

How to troubleshoot gap locks?

Is possible to detect those gap locks using SHOW ENGINE INNODB STATUS:

---TRANSACTION 72C, ACTIVE 755 sec
4 lock struct(s), heap size 1248, 3 row lock(s), undo log entries 1
MySQL thread id 3, OS thread handle 0x7f84a78ba700, query id 163 localhost msandbox
TABLE LOCK table test.t trx id 72C lock mode IX
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X
RECORD LOCKS space id 19 page no 3 n bits 80 index GEN_CLUST_INDEX of table test.t trx id 72C lock_mode X locks rec but not gap
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X locks gap before rec

If you have lot of gaps locks in your transactions affecting the concurrency and the performance you can disable them in two different ways:

1- Change the ISOLATION level to READ COMMITTED. In this isolation level, it is normal and expected that query results can change during a transaction, so there is no need to create locks to prevent that from happening.
2- innodb_locks_unsafe_for_binlog = 1. Disables the gap locks except for foreign-key constraint checking or duplicate-key checking.

The most important difference between these two options is that the second one is a global variable that affects all sessions and needs a server restart to change its value. Both options cause phantom reads (non repeatable reads) so in order to prevent problems with the replication you should change the binary log format to “row”.

Depending on the statement, the behavior of these locks can be different. In the following link there is a good source of information:

http://dev.mysql.com/doc/refman/5.1/en/innodb-locks-set.html.

Conclusion

MySQL uses REPEATABLE READ as the default isolation level so it needs to lock the index records and the gaps to avoid phantom reads and to get a consistent Statement based replication. If your application can deal with phantom reads and your binary log is in row format, changing the ISOLATION to READ COMMITTED will help you to avoid all those extra locks. As a final advice, keep your transactions short :)

The post InnoDB’s gap locks appeared first on MySQL Performance Blog.

Introducing backup locks in Percona Server

$
0
0

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

In the beginning…

In the beginning there was FLUSH TABLES, and users messed with their MyISAM tables under a live server and were not ashamed. Users could do nice things like:

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some tables, etc.

And users were happy until someone realized that tables must be protected against concurrent access by queries in other connections. So Monty gave them FLUSH TABLES WITH READ LOCK, and users were enlightened.

Online backups

Users then started dreaming about online backups, i.e. creating consistent snapshots of a live MySQL server. mysqldump --lock-all-tables had been a viable option for a while. To provide consistency it used FLUSH TABLES WITH READ LOCK which was not quite the right tool for the job, but was “good enough”. Who cares if a mom-and-pop shop becomes unavailable for a few seconds required to dump ~100 MB of data, right?

With InnoDB gaining popularity users had realized that one could employ MVCC to guarantee consistency and FLUSH TABLES WITH READ LOCK doesn’t make much sense for InnoDB tables anyway (you cannot modify InnoDB tables under a live server even if the server is read-only). So Peter gave mysqldump the --single-transaction option, and users were enlightened. mysqldump --single-transaction allowed to avoid FTWRL, but there was a few catches:

  • one cannot perform any schema modifications or updates to non-InnoDB tables while mysqldump --single-transaction is in progress, because those operations are not transactional and thus would ignore the data snapshot created by --single-transaction;
  • one cannot get binary log coordinates with --master-data or
    --dump-slave, because in that case FTWRL would still be used to ensure that the binary log coordinates are consistent with the data dump;

Which makes --single-transaction similar to the --no-lock option in Percona XtraBackup: it shifts the responsibility for backup consistency to the user. Any change in the workload violating the prerequisites for those options may result in a broken backup without any signs for the user to take action.

Present

Fast forward to present day. MySQL is capable of handling over a million queries per second, MyISAM is certainly not a popular choice to store data, and there are many backup solutions to choose from. Yet all of them still rely on FLUSH TABLES WITH READ LOCK in one way or another to guarantee consistency of .frm files, non-transactional tables and binary log coordinates.

To some extent, the problem with concurrent DDL + mysqldump --single-transaction has been alleviated with metadata locks in MySQL 5.5, which however made some users unhappy, and that behavior was partially reverted in MySQL 5.6.16 with the fix for bug #71017. But the fundamental problem is still there: mysqldump --single-transaction does not guarantee consistency with concurrent DDL statements and updates to non-transactional tables.

So the fact that FTWRL is an overkill for backups has been increasingly obvious for the reasons described below.

What’s the problem with FTWRL anyway?

A lot has been written on what FLUSH TABLES WITH READ LOCK really does. Here’s yet another walk-through in a bit more detail than described elsewhere:

  1. It first invalidates the Query Cache.
  2. It then waits for all in-flight updates to complete and at the same time it blocks all incoming updates. This is one problem for busy servers.
  3. It then closes all open tables (the FLUSH part) and expels them from the table cache. This is also when FTWRL has to wait for all SELECT queries to complete. And this is another, even bigger problem for busy servers, because that wait happens to occur with all updates blocked. What’s even worse, the server at this stage is essentially offline, because even incoming SELECT queries will get blocked.
  4. Finally, it blocks COMMITs.

Action #4 is not required for the original purpose of FTWRL, but is rather a kludge implemented due to the fact that FTWRL is (mis)used by backup utilities.

Actions #1-3 make perfect sense for the original reasons why FTWRL has been implemented. If we are going to access and possibly modify tables outside of the server, we want the server to forget everything it knows about both schema and data for all tables, and flush all in-memory buffers to make the on-disk data representation consistent.

And that’s what makes it an overkill for MySQL database backup utilities: they don’t require #1, because they never modify data. #2 is only required for non-InnoDB tables, because InnoDB provides other ways to ensure consistency for both logical and physical backups. And #3 is certainly not a problem for logical backup utilities like mysqldump or mydumper, because they don’t even access on-disk data directly. As we will see, it is not a big problem for physical backup solutions either.

To FLUSH or not to FLUSH?

So what exactly is flushed by FLUSH TABLES WITH READ LOCK?

Nothing for InnoDB tables, and no physical backup solution require it to flush anything.

For MyISAM it is more complicated. MyISAM key caches are normally write-through, i.e. by the time each update to a MyISAM table completes, all index updates are written to disk. The only exception is delayed key writing feature which you should not be using anyway, if you care about your data. MyISAM may also do data buffering for bulk inserts, e.g. while executing multi-row INSERTs or LOAD DATA statements. Those buffers, however, are flushed between statements, so have no effect on physical backups as long as we block all statements updating MyISAM tables.

The point is that without flushing each storage engine is not any less backup-safe as it is crash-safe, with the only difference that backups are guaranteed to wait for all currently executing INSERT/REPLACE/DELETE/UPDATE statements to complete.

Backup locks

Enter the backup locks feature. The following 3 new SQL statements have been introduced in Percona Server:

  • LOCK TABLES FOR BACKUP
  • LOCK BINLOG FOR BACKUP
  • UNLOCK BINLOG

LOCK TABLES FOR BACKUP

Quoting the documentation page from the manual:

LOCK TABLES FOR BACKUP uses a new MDL lock type to block updates to non-transactional tables and DDL statements for all tables. More specifically, if there’s an active LOCK TABLES FOR BACKUP lock, all DDL statements and updates to MyISAM, CSV, MEMORY and ARCHIVE tables will be blocked in the “Waiting for backup lock” status as visible in PERFORMANCE_SCHEMA or PROCESSLIST. SELECT queries for all tables and INSERT/REPLACE/UPDATE/DELETE against InnoDB, Blackhole and Federated tables are not affected by LOCK TABLES FOR BACKUP. Blackhole tables obviously have no relevance for backups, and Federated tables are ignored by both logical and physical backup tools.

Like FTWRL, the LOCK TABLES FOR BACKUP statement:

  • blocks updates to MyISAM, MEMORY, CSV and ARCHIVE tables;
  • blocks DDL against any tables;
  • does not block updates to temporary and log tables.

Unlike FTWRL, the LOCK TABLES FOR BACKUP statement:

  • does not invalidate the Query Cache;
  • never waits for SELECT queries to complete regardless of the storage engines involved;
  • never blocks SELECTs, or updates to InnoDB, Blackhole and Federated tables.

In other words, it does exactly what backup utilities need: block non-transactional changes that are included into the backup, and leave everything else to InnoDB MVCC and crash recovery.

With the only exception of binary log coordinates obtained with SHOW MASTER STATUS and SHOW SLAVE STATUS.

LOCK BINLOG FOR BACKUP

This is when LOCK BINLOG FOR BACKUP comes in handy. It blocks all updates to binary log coordinates as reported by SHOW MASTER/SLAVE STATUS and used by backup utilities. It has no effect when all of the following conditions apply:

  • when the binary log is disabled. If it is disabled globally, then all connections will not be affected by LOCK BINLOG FOR BACKUP. If it is enabled globally, but disabled for specific connections via sql_log_bin, only those connections are allowed to commit;
  • the server is not a replication slave;

Even if binary logging is used, LOCK BINLOG FOR BACKUP will allow DDL and updates to any tables to proceed until they will be written to binlog (i.e. commit), and/or advance Exec_Master_Log_* / Exec_Gtid_Set when executed by a replication thread, provided that no other global locks are acquired.

To release the lock acquired by LOCK TABLES FOR BACKUP there’s already UNLOCK TABLES. And the LOCK BINLOG FOR BACKUP lock is released with UNLOCK BINLOG.

Let’s look how these statements can be used by MySQL backup utilities.

mysqldump

mysqldump got a new option, --lock-for-backup which along with --single-transaction essentially obsoletes --lock-all-tables (i.e. FLUSH TABLES WITH READ LOCK). It makes mysqldump use LOCK TABLES FOR BACKUP before it starts dumping tables to block all “unsafe” statement that might otherwise interfere with backup consistency.

Of course, that requires backup locks support by the target server, so mysqldump checks if they are indeed supported and fails with an error if they are not.

However, at the moment if binary lock coordinates are requested with --master-data, FTWRL is still used even if --lock-for-backup is specified. mysqldump could use LOCK BINLOG FOR BACKUP, but there’s a better solution for logical backups implemented in MariaDB, which has already been ported to Percona Server and queued for the next release.

There is also another important difference between just mysqldump --single-transaction and mysqldump --lock-for-backup --single-transaction. As of MySQL 5.5 mysqldump --single-transaction acquires shared metadata locks on all tables processed within the transaction. Which will also block DDL statements on those tables when they will try to acquire an exclusive lock. So far, so good. The problems start when there’s an incoming SELECT query against a table that already has a pending DDL statement. It will also be blocked on a pending exclusive MDL request for no apparent reasons. Which was one of the complaints in bug #71017.

It’s better illustrated with an example. Suppose there are 3 sessions: one created by mysqldump, and 2 user sessions.

user1> CREATE TABLE t1 (a INT);
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the MDL created by mysqldump
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this blocks on a pending MDL request by user1
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

This is what would happen with mysqldump --lock-for-backup --single-transaction:

user1> CREATE TABLE t1 (a INT);
mysqldump> LOCK TABLES FOR BACKUP;
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the backup MDL lock
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this one is not blocked

This immediate problem was partially fixed in MySQL 5.6.16 by releasing metadata locks after processing each table with the help of savepoints. There is a couple of issues with this approach:

  • there is still a table metadata lock for the duration of SELECT executed by mysqldump. Which, as before, blocks DDL. So there is still a chance that mysqldump --single-transaction may eventually block SELECT queries.
  • after the table is processed and the metadata lock is released, there is now an opportunity for RENAME to break the backup, see bug #71214.

Both issues above along with bug #71215 and bug #71216 do not exist with mysqldump --lock-for-backup --single-transaction as all kinds of DDL statements are properly isolated by backup locks, which do not block SELECT queries at the same time.

Percona XtraBackup

Percona XtraBackup 2.2 will support backup locks and use them automatically if supported by the server being backed up.

The current locking used by XtraBackup is:

# copy InnoDB data
FLUSH TABLES WITH READ LOCK;
# copy .frm, MyISAM, etc.
# get the binary log coordinates
# finalize the background copy of REDO log
UNLOCK TABLES;

With backup locks it becomes:

# copy InnoDB data
LOCK TABLES FOR BACKUP;
# copy .frm, MyISAM, etc
LOCK BINLOG FOR BACKUP;
# finalize the background copy of REDO log
UNLOCK TABLES;
# get the binary log coordinates
UNLOCK BINLOG;

Note that under the following conditions, no blocking occurs at any stage in the server:

  • no updates to non-transactional tables;
  • no DDL;
  • binary log is disabled;

They may look familiar, because they are essentially prerequisites for the --no-lock option. Except that with backup locks, you don’t have to take chances and take responsibility for backup consistency. All the locking will be handled automatically by the server, if and when it is necessary.

mylvmbackup

mylvmbackup takes the server read-only with FLUSH TABLES WITH READ LOCK while the snapshot is being created for two reasons:

  • flush non-transactional tables
  • ensure consistency with the binary log coordinates

For exactly the same reasons as with XtraBackup, it can use backup locks instead of FTWRL.

mydumper

mydumper developers may want to add support for backup locks as well. mydumper relies on START TRANSACTION WITH CONSISTENT SNAPSHOT to ensure InnoDB consistency, but has to resort to FLUSH TABLES WITH READ LOCK to ensure consistency of non-InnoDB tables and binary log coordinates.

Another problem is that START TRANSACTION WITH CONSISTENT SNAPSHOT is not supposed to be used by multi-threaded logical backup utilities. But that is an opportunity for another server-side improvement and probably a separate blog post.

The post Introducing backup locks in Percona Server appeared first on MySQL Performance Blog.

Test Drive of Solid

$
0
0

Not so long ago Solid released solidDB for MySQL Beta 3 so I decided now is time to take a bit closer look on new transactional engine for MySQL. While my far goal is the performance and scalability testing before I wanted to look at basic transactional properties such as deadlock detection, select for update handling and phantom reads in the repeatable read isolation level.

Solid has OPTMISTIC (default) and PESSIMISTIC concurrency control, so it was interesting to test both.

We used default isolation mode which is REPEATABLE-READ for this test.

Test 1: Solid, deadlock detection, default (OPTIMISTIC) concurrency control.

CREATE TABLE `test2` (
  `id` int(11) NOT NULL,
  `names` varchar(255) default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=solidDB DEFAULT CHARSET=latin1
insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
mysql> begin;
mysql> update test2 set names='mysql' where id=1;
Session 2:
mysql> begin;
mysql> update test2 set names='Solid' where id=2;
Session 1:
mysql> update test2 set names='solid1' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So instead of waiting on row with id=2 to be unlocked by transaction in session 1 with OPTIMISTIC concurrency Solid simply reports deadlock due to conflicting update. For applications with rare update conflicts this actually may be good because it is simplier – you do not have to deal with complicated lock graphs or rely on timeout for deadlock resolution. For workloads with frequent row waits this however would result in increased number of deadlocks.

Test 2: SELECT FOR UPDATE:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id =2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session 2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

As we can see deadlock happens in this case even for transaction which runs plain select. Behavior shown in this case does not match the one you would expect from multi versining engine – all the meaning of multi versioning is to allow you to perform consistent read even if rows are locked or even modified unless you require locking read, in which case SELECT FOR UPDATE or SELECT … LOCK IN SHARE MODE can be used. Quite possibly it is a bug which will be fixed in the future, if not it will be serious handycap.

Too bas Solid does not yet have lock investigation tool as SHOW INNODB STATUS (at least) – so it is hard to tell what is really happening.

Test 3: Phantom rows:

Table before test:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

So there are no phantom rows which is what we would expect from repeatable-read isolation mode, at least in its interpretation by Innodb.

Let’s look how PESIMISTIC works (mysqld started with –soliddb_pessimistic).

Test 1: Deadlock detection:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id = 1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session1:
mysql> update test2 set names='solid1' where id = 2;
...waiting...
Session2:
mysql> update test2 set names='mysql1' where id=1;
I would expect deadlock here - but not - just waiting...
In 30 sec Session1:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

So as we can see Solid does not have instant deadlock detection as Innodb and have to rely on timeouts instead. This may be serious problem for some applications, especially interactive ones (ie web) – instead of getting quick deadlock and retrying transaction or giving error to the user you’re forced to wait. Setting lock wait timeout to some small value is not optimal solution ether as it would make some transactions to be terminated even though they just had normal lock wait not a deadlock.

Test 2: SELECT FOR UPDATE:

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
..waiting..
Session1:
mysql> commit;
Session2:
+----+---------+
| id | names   |
+----+---------+
|  2 | SolidDB |
+----+---------+
1 row in set (6.11 sec)

Result well matches one which we got with OPTIMISTIC locking – just instead of simply balling out on first row lock conflict, query waits for rows to be unlocked. But wait why is it waiting if it suppose to be multi versioning system ?

Test 3: Phantom rows:

before test:
mysql> select * from test2;
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
+----+-----------+
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> insert into test2 values (4,'HEAP');
Query OK, 1 row affected (0.00 sec)
Session2:
mysql> select * from test2;
...waiting...
Session1:
commit;
Session2:
+----+-----------+
| id | names     |
+----+-----------+
|  1 | Mysql     |
|  2 | Solid     |
|  3 | MyISAM    |
|  4 | HEAP      | <- phantom
+----+-----------+
4 rows in set (4.25 sec)
show variables like 'tx%';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| tx_isolation  | REPEATABLE-READ |
+---------------+-----------------+
1 row in set (0.00 sec)

In this case we get phantom row, which looks like double bug to us. First we should not get phantom row in repeatable-read isolation mode, second query results should not be different in OPTIMISTIC vs PESSIMISTIC concurrency mode. We can get deadlocks differently but query results should be the same.

Test 4: Update Handling:

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> update test2 set names='SolidDB' where id=2;
... waiting...
in 30 sec:
ERROR 1031 (HY000): Table storage engine for 'test2' doesn't have this option

Strange error message. This is probably one more bug which hopefully be resolved in further versions.

Test 5: UPDATE / SELECT FOR UPDATE in OPTIMISTIC mode

Solid optimistic:
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
Session1:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
mysql> update test2 set names='SolidDB2' where id=2;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> select * from test2 where id=2;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.01 sec
Session1:
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
Please note transaction 1 was commited.
Session2:
mysql> select * from test2 where id=2 for update;
+----+----------+
| id | names    |
+----+----------+
|  2 | SolidDB1 |
+----+----------+
1 row in set (0.00 sec)
In this case we can se Multi-versioning in action, so it seems to work in some cases.  However behavior is different from Innodb which will do READ-COMMITED for locking reads (SELECT FOR UPDATE/LOCK IN SHARE MODE) as only actual row versions can be locked.   With Innodb we would get "SolidDB2" as result.
mysql> update test2 set names='SolidDB3' where id=2;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Error message is obviously wrong in this case. It is not deadlock but update conflict – other transaction has already modified row and commited so update could not proceed.

Also in this case we can see SELECT FOR UPDATE does not play nicely with OPTIMISTIC concurrency. Generally you would use this statement to ensure rows are locked at once so transaction could proceed modifying them without causing deadlocks.

For comparion here are InnoDB sessions:

Test 1: Deadlock detection (deadlock is correctly detected):

insert into test2 values (1,'Mysql'),(2,'Solid'),(3,'MyISAM');
Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='mysql' where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
Session2:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> update test2 set names='Solid' where id=2;
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1  Changed: 0  Warnings: 0
Session1:
mysql>  update test2 set names='solid1' where id=2;
...waiting...
Session2:
mysql> update test2 set names='mysql2' where id=1;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
Session1:
Query OK, 1 row affected (9.65 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Test 2: SELECT FOR UPDATE (second session is not blocked):

Session1:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from test2 where id=2 for update;
+----+-------+
| id | names |
+----+-------+
|  2 | Solid |
+----+-------+
1 row in set (0.00 sec)
Session2:
mysql> begin;
Query OK, 0 rows affected (0.01 sec)
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Test 3: Phantom rows (no phantom rows in repeatable-reads):

Session1:
mysql> begin;
Sessino2:
mysql> begin;
Session1:
mysql> insert into test2 values (4,'FEDERATED');
Session2:
select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)
Session1:
mysql> commit;
Session2:
mysql> select * from test2;
+----+--------+
| id | names  |
+----+--------+
|  1 | Mysql  |
|  2 | Solid  |
|  3 | MyISAM |
+----+--------+
3 rows in set (0.00 sec)

Conclusion: Current Beta version seems to have number of bugs when it comes to concurrency control, so it is hard to say what we’ll see in final version. Current behavior is however very different from Innodb in many cases which may make porting applications to use SolidDB instead of Innodb complicated.

P.S.

After previous experiments I tried SolidDB under sysbench and got next (1 client):

mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> UPDATE sbtests set k=k+1 where id=401692;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> UPDATE sbtests set c='773425598-640282370-898784553-626619839-555294498-279281971-788986238-202873246-188128003-162592967' where id=496201;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

So we can’t run SysBench benchmark with current version. Hopefully next beta will be solid enough to run benchmark.

The post Test Drive of Solid appeared first on MySQL Performance Blog.


InnoDB’s gap locks

$
0
0

One of the most important features of InnoDB is the row level locking. This feature provides better concurrency under heavy write load but needs additional precautions to avoid phantom reads and to get a consistent Statement based replication. To accomplish that, row level locking databases also acquire gap locks.

What is a Phantom Read

A Phantom Read happens when in a running transaction, two identical statements get different values, because some other transaction has modified the table’s rows. For example:


transaction1> START TRANSACTION;
transaction1> SELECT * FROM t WHERE i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 30 |
+------+


transaction2> START TRANSACTION;
transaction2> INSERT INTO t VALUES(26);
transaction2> COMMIT;

transaction1> select * from t where i > 20 FOR UPDATE;
+------+
| i |
+------+
| 21 |
| 25 |
| 26 |
| 30 |
+------+

Phantom reads do not occur if you’re simply doing a SELECT. They only occur if you do UPDATE or DELETE or SELECT FOR UPDATE. InnoDB provides REPEATABLE READ for read-only SELECT, but it behaves as if you use READ COMMITTED for all write queries, in spite of your chosen transaction isolation level (considering only the two most common isolation levels, REPEATABLE READ and READ COMMITTED).

What is a gap lock?

A gap lock is a lock on the gap between index records. Thanks to this gap lock, when you run the same query twice, you get the same result, regardless other session modifications on that table. This makes reads consistent and therefore makes the replication between servers consistent. If you execute SELECT * FROM id > 1000 FOR UPDATE twice, you expect to get the same value twice. To accomplish that, InnoDB locks all index records found by the WHERE clause with an exclusive lock and the gaps between them with a shared gap lock.

This lock doesn’t only affect to SELECT … FOR UPDATE. This is an example with a DELETE statement:

transaction1 > SELECT * FROM t;
+------+
| age |
+------+
| 21 |
| 25 |
| 30 |
+------+

Start a transaction and delete the record 25:

transaction1 > START TRANSACTION;
transaction1 > DELETE FROM t WHERE age=25;

At this point we suppose that only the record 25 is locked. Then, we try to insert another value on the second session:

transaction2 > START TRANSACTION;
transaction2 > INSERT INTO t VALUES(26);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(29);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(23);
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
transaction2 > INSERT INTO t VALUES(31);
Query OK, 1 row affected (0.00 sec)

After running the delete statement on the first session, not only the affected index record has been locked but also the gap before and after that record with a shared gap lock preventing the insertion of data to other sessions.

How to troubleshoot gap locks?

Is possible to detect those gap locks using SHOW ENGINE INNODB STATUS:

---TRANSACTION 72C, ACTIVE 755 sec
4 lock struct(s), heap size 1248, 3 row lock(s), undo log entries 1
MySQL thread id 3, OS thread handle 0x7f84a78ba700, query id 163 localhost msandbox
TABLE LOCK table test.t trx id 72C lock mode IX
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X
RECORD LOCKS space id 19 page no 3 n bits 80 index GEN_CLUST_INDEX of table test.t trx id 72C lock_mode X locks rec but not gap
RECORD LOCKS space id 19 page no 4 n bits 80 index age of table test.t trx id 72C lock_mode X locks gap before rec

If you have lot of gaps locks in your transactions affecting the concurrency and the performance you can disable them in two different ways:

1- Change the ISOLATION level to READ COMMITTED. In this isolation level, it is normal and expected that query results can change during a transaction, so there is no need to create locks to prevent that from happening.
2- innodb_locks_unsafe_for_binlog = 1. Disables the gap locks except for foreign-key constraint checking or duplicate-key checking.

The most important difference between these two options is that the second one is a global variable that affects all sessions and needs a server restart to change its value. Both options cause phantom reads (non repeatable reads) so in order to prevent problems with the replication you should change the binary log format to “row”.

Depending on the statement, the behavior of these locks can be different. In the following link there is a good source of information:

http://dev.mysql.com/doc/refman/5.1/en/innodb-locks-set.html.

Conclusion

MySQL uses REPEATABLE READ as the default isolation level so it needs to lock the index records and the gaps to avoid phantom reads and to get a consistent Statement based replication. If your application can deal with phantom reads and your binary log is in row format, changing the ISOLATION to READ COMMITTED will help you to avoid all those extra locks. As a final advice, keep your transactions short :)

The post InnoDB’s gap locks appeared first on MySQL Performance Blog.

Interactive Debugging of Transaction Conflicts with TokuDB

$
0
0

I am developing a concurrent application that uses TokuDB to store its database. Sometimes, one of my SQL statements returns with a ‘lock wait timeout exceeded’ error. How do I identify the cause of this error? First, I need to understand a little bit about how TokuDB transactions use locks. Then, I need to understand how to use the MySQL information schema to look at the current state of the locks.

Transactions and Locks

TokuDB uses key range locks to implement serializable transactions. These locks are acquired as the transaction progresses. The locks are released when the transaction commits or aborts.

TokuDB stores these locks in a data structure called the lock tree. The lock tree stores the set of range locks granted to each transaction. In addition, the lock tree stores the set of locks that are not granted due to a conflict with locks granted to some other transaction.

TokuDB 7.1 provides access to the set of locks taken and the set of locks that are blocked via tables in the MySQL information schema. In addition, TokuDB stores the last lock conflict information in a session variable that can be retrieved by the application.

Example

Here is an example debug session.

I create a table with a single column that is the primary key.

mysql> show create table t;
Create Table: CREATE TABLE `t` (
`id` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=TokuDB DEFAULT CHARSET=latin1;

I have 2 MySQL clients with ID’s 1 and 2 respectively. MySQL client 1 inserts some values into the table ‘t’. TokuDB transaction 51 is created for the insert statement. Since autocommit is off, transaction 51 is still live after the insert statement completes, and I can query the ‘tokudb_locks’ information schema table to see the locks that are held by the transaction.

mysql 1> set autocommit=off;
mysql 1> insert into t values (1),(10),(100);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql 1> select * from information_schema.tokudb_locks;
+--------------+-----------------------+---------------+----------------+-----------------+
| locks_trx_id | locks_mysql_thread_id | locks_dname   | locks_key_left | locks_key_right |
+--------------+-----------------------+---------------+----------------+-----------------+
| 51           | 1                     | ./test/t-main | 0001000000     | 0001000000      |
| 51           | 1                     | ./test/t-main | 000a000000     | 000a000000      |
| 51           | 1                     | ./test/t-main | 0064000000     | 0064000000      |
+--------------+-----------------------+---------------+----------------+-----------------+
mysql 1> select * from information_schema.tokudb_lock_waits;
Empty set (0.00 sec)

The keys are currently hex dumped. Someday we will pretty print the keys.
Now we switch to the other MySQL client with ID 2.

mysql 2> insert into t values (2),(20),(100);

The insert gets blocked since there is a conflict on the primary key with value 100.
The granted TokuDB locks are:

mysql 1> select * from information_schema.tokudb_locks;
+--------------+-----------------------+---------------+----------------+-----------------+
| locks_trx_id | locks_mysql_thread_id | locks_dname   | locks_key_left | locks_key_right |
+--------------+-----------------------+---------------+----------------+-----------------+
| 51           | 1                     | ./test/t-main | 0001000000     | 0001000000      |
| 51           | 1                     | ./test/t-main | 000a000000     | 000a000000      |
| 51           | 1                     | ./test/t-main | 0064000000     | 0064000000      |
| 62           | 2                     | ./test/t-main | 0002000000     | 0002000000      |
| 62           | 2                     | ./test/t-main | 0014000000     | 0014000000      |
+--------------+-----------------------+---------------+----------------+-----------------+

The locks that are pending due to a conflict are:

mysql 1> select * from information_schema.tokudb_lock_waits;
+-------------------+-----------------+------------------+---------------------+----------------------+-----------------------+
| requesting_trx_id | blocking_trx_id | lock_waits_dname | lock_waits_key_left | lock_waits_key_right | lock_waits_start_time |
+-------------------+-----------------+------------------+---------------------+----------------------+-----------------------+
| 62                | 51              | ./test/t-main    | 0064000000          | 0064000000           | 1380656990910         |
+-------------------+-----------------+------------------+---------------------+----------------------+-----------------------+

Eventually, the lock for client 2 times out, and we can retrieve a JSON coded document that describes the conflict.

mysql 2 insert continued>
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql 2> select @@tokudb_last_lock_timeout;
+-------------------------------------------------------------------------------------------------------------+
| @@tokudb_last_lock_timeout |
+-------------------------------------------------------------------------------------------------------------+
| {"mysql_thread_id":2,"dbname":"./test/t-main","requesting_txnid":62,"blocking_txnid":51,"key":"0064000000"} |
+-------------------------------------------------------------------------------------------------------------+

The JSON document encodes the database, table and index where the conflict occurred in the ‘dname’ key. The ‘key’ encodes the conflicting key. The ‘blocking_txnid’ encodes the transaction identifier that is blocking my transaction. I can find the MySQL client that created the blocking transaction via joins of information schema tables.

mysql> select * from information_schema.tokudb_trx,information_schema.processlist
where trx_id = 51 and trx_mysql_thread_id = id;
+--------+---------------------+----+----------+-----------+------+---------+------+-------+------+
| trx_id | trx_mysql_thread_id | ID | USER     | HOST      | DB   | COMMAND | TIME | STATE | INFO |
+--------+---------------------+----+----------+-----------+------+---------+------+-------+------+
| 51     | 1                   | 1  | prohaska | localhost | test | Sleep   | 151  |       | NULL |
+--------+---------------------+----+----------+-----------+------+---------+------+-------+------+

Now I have information about the two client operations that are conflicting and can hopefully change my application to avoid this conflict.

The post Interactive Debugging of Transaction Conflicts with TokuDB appeared first on MySQL Performance Blog.

Introducing backup locks in Percona Server

$
0
0

TL;DR version: The backup locks feature introduced in Percona Server 5.6.16-64.0 is a lightweight alternative to FLUSH TABLES WITH READ LOCK and can be used to take both physical and logical backups with less downtime on busy servers. To employ the feature with mysqldump, use mysqldump --lock-for-backup --single-transaction. The next release of Percona XtraBackup will also be using backup locks automatically if the target server supports the feature.

Now on to the gory details, but let’s start with some history.

In the beginning…

In the beginning there was FLUSH TABLES, and users messed with their MyISAM tables under a live server and were not ashamed. Users could do nice things like:

mysql> FLUSH TABLES;
# execute myisamchk, myisampack, backup / restore some tables, etc.

And users were happy until someone realized that tables must be protected against concurrent access by queries in other connections. So Monty gave them FLUSH TABLES WITH READ LOCK, and users were enlightened.

Online backups

Users then started dreaming about online backups, i.e. creating consistent snapshots of a live MySQL server. mysqldump --lock-all-tables had been a viable option for a while. To provide consistency it used FLUSH TABLES WITH READ LOCK which was not quite the right tool for the job, but was “good enough”. Who cares if a mom-and-pop shop becomes unavailable for a few seconds required to dump ~100 MB of data, right?

With InnoDB gaining popularity users had realized that one could employ MVCC to guarantee consistency and FLUSH TABLES WITH READ LOCK doesn’t make much sense for InnoDB tables anyway (you cannot modify InnoDB tables under a live server even if the server is read-only). So Peter gave mysqldump the --single-transaction option, and users were enlightened. mysqldump --single-transaction allowed to avoid FTWRL, but there was a few catches:

  • one cannot perform any schema modifications or updates to non-InnoDB tables while mysqldump --single-transaction is in progress, because those operations are not transactional and thus would ignore the data snapshot created by --single-transaction;
  • one cannot get binary log coordinates with --master-data or
    --dump-slave, because in that case FTWRL would still be used to ensure that the binary log coordinates are consistent with the data dump;

Which makes --single-transaction similar to the --no-lock option in Percona XtraBackup: it shifts the responsibility for backup consistency to the user. Any change in the workload violating the prerequisites for those options may result in a broken backup without any signs for the user to take action.

Present

Fast forward to present day. MySQL is capable of handling over a million queries per second, MyISAM is certainly not a popular choice to store data, and there are many backup solutions to choose from. Yet all of them still rely on FLUSH TABLES WITH READ LOCK in one way or another to guarantee consistency of .frm files, non-transactional tables and binary log coordinates.

To some extent, the problem with concurrent DDL + mysqldump --single-transaction has been alleviated with metadata locks in MySQL 5.5, which however made some users unhappy, and that behavior was partially reverted in MySQL 5.6.16 with the fix for bug #71017. But the fundamental problem is still there: mysqldump --single-transaction does not guarantee consistency with concurrent DDL statements and updates to non-transactional tables.

So the fact that FTWRL is an overkill for backups has been increasingly obvious for the reasons described below.

What’s the problem with FTWRL anyway?

A lot has been written on what FLUSH TABLES WITH READ LOCK really does. Here’s yet another walk-through in a bit more detail than described elsewhere:

  1. It first invalidates the Query Cache.
  2. It then waits for all in-flight updates to complete and at the same time it blocks all incoming updates. This is one problem for busy servers.
  3. It then closes all open tables (the FLUSH part) and expels them from the table cache. This is also when FTWRL has to wait for all SELECT queries to complete. And this is another, even bigger problem for busy servers, because that wait happens to occur with all updates blocked. What’s even worse, the server at this stage is essentially offline, because even incoming SELECT queries will get blocked.
  4. Finally, it blocks COMMITs.

Action #4 is not required for the original purpose of FTWRL, but is rather a kludge implemented due to the fact that FTWRL is (mis)used by backup utilities.

Actions #1-3 make perfect sense for the original reasons why FTWRL has been implemented. If we are going to access and possibly modify tables outside of the server, we want the server to forget everything it knows about both schema and data for all tables, and flush all in-memory buffers to make the on-disk data representation consistent.

And that’s what makes it an overkill for MySQL database backup utilities: they don’t require #1, because they never modify data. #2 is only required for non-InnoDB tables, because InnoDB provides other ways to ensure consistency for both logical and physical backups. And #3 is certainly not a problem for logical backup utilities like mysqldump or mydumper, because they don’t even access on-disk data directly. As we will see, it is not a big problem for physical backup solutions either.

To FLUSH or not to FLUSH?

So what exactly is flushed by FLUSH TABLES WITH READ LOCK?

Nothing for InnoDB tables, and no physical backup solution require it to flush anything.

For MyISAM it is more complicated. MyISAM key caches are normally write-through, i.e. by the time each update to a MyISAM table completes, all index updates are written to disk. The only exception is delayed key writing feature which you should not be using anyway, if you care about your data. MyISAM may also do data buffering for bulk inserts, e.g. while executing multi-row INSERTs or LOAD DATA statements. Those buffers, however, are flushed between statements, so have no effect on physical backups as long as we block all statements updating MyISAM tables.

The point is that without flushing each storage engine is not any less backup-safe as it is crash-safe, with the only difference that backups are guaranteed to wait for all currently executing INSERT/REPLACE/DELETE/UPDATE statements to complete.

Backup locks

Enter the backup locks feature. The following 3 new SQL statements have been introduced in Percona Server:

  • LOCK TABLES FOR BACKUP
  • LOCK BINLOG FOR BACKUP
  • UNLOCK BINLOG

LOCK TABLES FOR BACKUP

Quoting the documentation page from the manual:

LOCK TABLES FOR BACKUP uses a new MDL lock type to block updates to non-transactional tables and DDL statements for all tables. More specifically, if there’s an active LOCK TABLES FOR BACKUP lock, all DDL statements and updates to MyISAM, CSV, MEMORY and ARCHIVE tables will be blocked in the “Waiting for backup lock” status as visible in PERFORMANCE_SCHEMA or PROCESSLIST. SELECT queries for all tables and INSERT/REPLACE/UPDATE/DELETE against InnoDB, Blackhole and Federated tables are not affected by LOCK TABLES FOR BACKUP. Blackhole tables obviously have no relevance for backups, and Federated tables are ignored by both logical and physical backup tools.

Like FTWRL, the LOCK TABLES FOR BACKUP statement:

  • blocks updates to MyISAM, MEMORY, CSV and ARCHIVE tables;
  • blocks DDL against any tables;
  • does not block updates to temporary and log tables.

Unlike FTWRL, the LOCK TABLES FOR BACKUP statement:

  • does not invalidate the Query Cache;
  • never waits for SELECT queries to complete regardless of the storage engines involved;
  • never blocks SELECTs, or updates to InnoDB, Blackhole and Federated tables.

In other words, it does exactly what backup utilities need: block non-transactional changes that are included into the backup, and leave everything else to InnoDB MVCC and crash recovery.

With the only exception of binary log coordinates obtained with SHOW MASTER STATUS and SHOW SLAVE STATUS.

LOCK BINLOG FOR BACKUP

This is when LOCK BINLOG FOR BACKUP comes in handy. It blocks all updates to binary log coordinates as reported by SHOW MASTER/SLAVE STATUS and used by backup utilities. It has no effect when all of the following conditions apply:

  • when the binary log is disabled. If it is disabled globally, then all connections will not be affected by LOCK BINLOG FOR BACKUP. If it is enabled globally, but disabled for specific connections via sql_log_bin, only those connections are allowed to commit;
  • the server is not a replication slave;

Even if binary logging is used, LOCK BINLOG FOR BACKUP will allow DDL and updates to any tables to proceed until they will be written to binlog (i.e. commit), and/or advance Exec_Master_Log_* / Exec_Gtid_Set when executed by a replication thread, provided that no other global locks are acquired.

To release the lock acquired by LOCK TABLES FOR BACKUP there’s already UNLOCK TABLES. And the LOCK BINLOG FOR BACKUP lock is released with UNLOCK BINLOG.

Let’s look how these statements can be used by MySQL backup utilities.

mysqldump

mysqldump got a new option, --lock-for-backup which along with --single-transaction essentially obsoletes --lock-all-tables (i.e. FLUSH TABLES WITH READ LOCK). It makes mysqldump use LOCK TABLES FOR BACKUP before it starts dumping tables to block all “unsafe” statement that might otherwise interfere with backup consistency.

Of course, that requires backup locks support by the target server, so mysqldump checks if they are indeed supported and fails with an error if they are not.

However, at the moment if binary lock coordinates are requested with --master-data, FTWRL is still used even if --lock-for-backup is specified. mysqldump could use LOCK BINLOG FOR BACKUP, but there’s a better solution for logical backups implemented in MariaDB, which has already been ported to Percona Server and queued for the next release.

There is also another important difference between just mysqldump --single-transaction and mysqldump --lock-for-backup --single-transaction. As of MySQL 5.5 mysqldump --single-transaction acquires shared metadata locks on all tables processed within the transaction. Which will also block DDL statements on those tables when they will try to acquire an exclusive lock. So far, so good. The problems start when there’s an incoming SELECT query against a table that already has a pending DDL statement. It will also be blocked on a pending exclusive MDL request for no apparent reasons. Which was one of the complaints in bug #71017.

It’s better illustrated with an example. Suppose there are 3 sessions: one created by mysqldump, and 2 user sessions.

user1> CREATE TABLE t1 (a INT);
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the MDL created by mysqldump
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this blocks on a pending MDL request by user1
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

This is what would happen with mysqldump --lock-for-backup --single-transaction:

user1> CREATE TABLE t1 (a INT);
mysqldump> LOCK TABLES FOR BACKUP;
mysqldump> START TRANSACTION WITH CONSISTENT SNAPSHOT;
mysqldump> SELECT * FROM t1; # this acquires a table MDL
user1> ALTER TABLE t1 ADD COLUMN b INT; # this blocks on the backup MDL lock
user2> SET lock_wait_timeout=1;
user2> SELECT * FROM t1; # this one is not blocked

This immediate problem was partially fixed in MySQL 5.6.16 by releasing metadata locks after processing each table with the help of savepoints. There is a couple of issues with this approach:

  • there is still a table metadata lock for the duration of SELECT executed by mysqldump. Which, as before, blocks DDL. So there is still a chance that mysqldump --single-transaction may eventually block SELECT queries.
  • after the table is processed and the metadata lock is released, there is now an opportunity for RENAME to break the backup, see bug #71214.

Both issues above along with bug #71215 and bug #71216 do not exist with mysqldump --lock-for-backup --single-transaction as all kinds of DDL statements are properly isolated by backup locks, which do not block SELECT queries at the same time.

Percona XtraBackup

Percona XtraBackup 2.2 will support backup locks and use them automatically if supported by the server being backed up.

The current locking used by XtraBackup is:

# copy InnoDB data
FLUSH TABLES WITH READ LOCK;
# copy .frm, MyISAM, etc.
# get the binary log coordinates
# finalize the background copy of REDO log
UNLOCK TABLES;

With backup locks it becomes:

# copy InnoDB data
LOCK TABLES FOR BACKUP;
# copy .frm, MyISAM, etc
LOCK BINLOG FOR BACKUP;
# finalize the background copy of REDO log
UNLOCK TABLES;
# get the binary log coordinates
UNLOCK BINLOG;

Note that under the following conditions, no blocking occurs at any stage in the server:

  • no updates to non-transactional tables;
  • no DDL;
  • binary log is disabled;

They may look familiar, because they are essentially prerequisites for the --no-lock option. Except that with backup locks, you don’t have to take chances and take responsibility for backup consistency. All the locking will be handled automatically by the server, if and when it is necessary.

mylvmbackup

mylvmbackup takes the server read-only with FLUSH TABLES WITH READ LOCK while the snapshot is being created for two reasons:

  • flush non-transactional tables
  • ensure consistency with the binary log coordinates

For exactly the same reasons as with XtraBackup, it can use backup locks instead of FTWRL.

mydumper

mydumper developers may want to add support for backup locks as well. mydumper relies on START TRANSACTION WITH CONSISTENT SNAPSHOT to ensure InnoDB consistency, but has to resort to FLUSH TABLES WITH READ LOCK to ensure consistency of non-InnoDB tables and binary log coordinates.

Another problem is that START TRANSACTION WITH CONSISTENT SNAPSHOT is not supposed to be used by multi-threaded logical backup utilities. But that is an opportunity for another server-side improvement and probably a separate blog post.

The post Introducing backup locks in Percona Server appeared first on MySQL Performance Blog.

Viewing all 11 articles
Browse latest View live