用JDBC做批量INSERTS的有效方法

在我的应用程序中,我需要做很多插入。 它是一个Java应用程序,我使用普通的JDBC执行查询。 DB是Oracle。 我已经启用批处理,所以它节省了我的networking延迟执行查询。 但是查询作为单独的INSERT被串行执行:

insert into some_table (col1, col2) values (val1, val2) insert into some_table (col1, col2) values (val3, val4) insert into some_table (col1, col2) values (val5, val6) 

我想知道下面的INSERTforms是否更有效率:

 insert into some_table (col1, col2) values (val1, val2), (val3, val4), (val5, val6) 

即折叠多个INSERT到一个。

任何其他技巧,使批量插入更快?

这是前面两个答案的组合:

  PreparedStatement ps = c.prepareStatement("INSERT INTO employees VALUES (?, ?)"); ps.setString(1, "John"); ps.setString(2,"Doe"); ps.addBatch(); ps.clearParameters(); ps.setString(1, "Dave"); ps.setString(2,"Smith"); ps.addBatch(); ps.clearParameters(); int[] results = ps.executeBatch(); 

Statement给你以下选项:

 Statement stmt = con.createStatement(); stmt.addBatch("INSERT INTO employees VALUES (1000, 'Joe Jones')"); stmt.addBatch("INSERT INTO departments VALUES (260, 'Shoe')"); stmt.addBatch("INSERT INTO emp_dept VALUES (1000, 260)"); // submit a batch of update commands for execution int[] updateCounts = stmt.executeBatch(); 

虽然这个问题要求使用JDBC高效地插入到Oracle中 ,但是我正在使用DB2(在IBM大型机上),从概念上讲,插入操作是相似的,所以认为可能有助于查看我的指标

  • 一次插入一条logging

  • 插入一批logging(非常有效)

这里去指标

1)一次插入一条logging

 public void writeWithCompileQuery(int records) { PreparedStatement statement; try { Connection connection = getDatabaseConnection(); connection.setAutoCommit(true); String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" + " VALUES" + "(?, ?, ?, ?, ?)"; statement = connection.prepareStatement(compiledQuery); long start = System.currentTimeMillis(); for(int index = 1; index < records; index++) { statement.setInt(1, index); statement.setString(2, "emp number-"+index); statement.setInt(3, index); statement.setInt(4, index); statement.setString(5, "username"); long startInternal = System.currentTimeMillis(); statement.executeUpdate(); System.out.println("each transaction time taken = " + (System.currentTimeMillis() - startInternal) + " ms"); } long end = System.currentTimeMillis(); System.out.println("total time taken = " + (end - start) + " ms"); System.out.println("avg total time taken = " + (end - start)/ records + " ms"); statement.close(); connection.close(); } catch (SQLException ex) { System.err.println("SQLException information"); while (ex != null) { System.err.println("Error msg: " + ex.getMessage()); ex = ex.getNextException(); } } } 

100个交易的指标:

 each transaction time taken = 123 ms each transaction time taken = 53 ms each transaction time taken = 48 ms each transaction time taken = 48 ms each transaction time taken = 49 ms each transaction time taken = 49 ms ... .. . each transaction time taken = 49 ms each transaction time taken = 49 ms total time taken = 4935 ms avg total time taken = 49 ms 

第一笔交易大约需要120-150ms时间进行查询parsing ,然后执行,后续交易只需要50ms左右。 (这仍然很高,但我的数据库是在不同的服务器上(我需要排除networking故障))

2)插入批处理(高效) – 通过preparedStatement.executeBatch()实现

 public int[] writeInABatchWithCompiledQuery(int records) { PreparedStatement preparedStatement; try { Connection connection = getDatabaseConnection(); connection.setAutoCommit(true); String compiledQuery = "INSERT INTO TESTDB.EMPLOYEE(EMPNO, EMPNM, DEPT, RANK, USERNAME)" + " VALUES" + "(?, ?, ?, ?, ?)"; preparedStatement = connection.prepareStatement(compiledQuery); for(int index = 1; index <= records; index++) { preparedStatement.setInt(1, index); preparedStatement.setString(2, "empo number-"+index); preparedStatement.setInt(3, index+100); preparedStatement.setInt(4, index+200); preparedStatement.setString(5, "usernames"); preparedStatement.addBatch(); } long start = System.currentTimeMillis(); int[] inserted = preparedStatement.executeBatch(); long end = System.currentTimeMillis(); System.out.println("total time taken to insert the batch = " + (end - start) + " ms"); System.out.println("total time taken = " + (end - start)/records + " s"); preparedStatement.close(); connection.close(); return inserted; } catch (SQLException ex) { System.err.println("SQLException information"); while (ex != null) { System.err.println("Error msg: " + ex.getMessage()); ex = ex.getNextException(); } throw new RuntimeException("Error"); } } 

一批100笔交易的指标是

 total time taken to insert the batch = 127 ms 

和1000个交易

 total time taken to insert the batch = 341 ms 

所以,在〜5000ms(一次一个trxn)下进行100次交易的时间减less到〜150ms(一批100个logging)。

注 – 忽略我的networking超慢,但度量值将是相对的。

显然,你必须进行基准testing,但是如果使用PreparedStatement而不是Statement,通过JDBC发出多个插入将会快得多。

如何使用INSERT ALL语句?

 INSERT ALL INTO table_name VALUES () INTO table_name VALUES () ... SELECT Statement; 

我记得最后一个select语句是强制性的,为了使这个请求成功。 不记得为什么。 您也可以考虑使用PreparedStatement 。 很多优点!

法里德

如果迭代次数低,使用PreparedStatements将比语句慢得多。 为了从语句上使用PrepareStatement获得性能优势,您需要在迭代至less为50或更高的循环中使用它。

您可以使用addBatch和executeBatch在java中进行批量插入请参阅示例: 批量插入Java

批量插入使用语句

 int a= 100; try { for (int i = 0; i < 10; i++) { String insert = "insert into usermaster" + "(" + "userid" + ")" + "values(" + "'" + a + "'" + ");"; statement.addBatch(insert); System.out.println(insert); a++; } dbConnection.commit(); } catch (SQLException e) { System.out.println(" Insert Failed"); System.out.println(e.getMessage()); } finally { if (statement != null) { statement.close(); } if (dbConnection != null) { dbConnection.close(); } }