I have written a method insert() in which I am trying to use JDBC Batch for inserting half a million records into a MySQL database:
public void insert(int nameListId, String[] names) {
String sql = "INSERT INTO name_list_subscribers (name_list_id, name, date_added)"+" VALUES (?, ?, NOW())";
Connection conn = null;
PreparedStatement ps = null;
try{
conn = getConnection();
ps = conn.prepareStatement(sql);
for(String s : names ){
ps.setInt(1, nameListId);
ps.setString(2, s);
ps.addBatch();
}
ps.executeBatch();
}catch(SQLException e){
throw new RuntimeException(e);
}finally{
closeDbResources(ps, null, conn);
}
}
But whenever I try to run this method, I get the following error:
java.lang.OutOfMemoryError: Java heap space
com.mysql.jdbc.ServerPreparedStatement$BatchedBindValues.<init>(ServerPreparedStatement.java:72)
com.mysql.jdbc.ServerPreparedStatement.addBatch(ServerPreparedStatement.java:330)
org.apache.commons.dbcp.DelegatingPreparedStatement.addBatch(DelegatingPreparedStatement.java:171)
If I replace ps.addBatch() with ps.executeUpdate() and remove ps.executeBatch(), it works fine, though it takes some time. Please let me know if you know if using Batch is appropriate in this situation, and if it is, then why does it give OurOfMemoryError?
[color=green]It is out of memory because it hold all the transaction in memory and only send it over to the database when you call executeBatch.
If you don't need it to be atomic and would like the get better performance, you can keep a counter and call executeBatch every n number of records.[/color]
public void insert(int nameListId, String[] names) {
String sql = "INSERT INTO name_list_subscribers (name_list_id, name, date_added)"+" VALUES (?, ?, NOW())";
Connection conn = null;
PreparedStatement ps = null;
try{
conn = getConnection();
ps = conn.prepareStatement(sql);
int icount = 0;
for(String s : names ){
ps.setInt(1, nameListId);
ps.setString(2, s);
ps.addBatch();
if(icount%1000 == 0){
ps.executeBatch();
ps.clearBatch();
}
icount++;
}
ps.executeBatch();
}catch(SQLException e){
throw new RuntimeException(e);
}finally{
closeDbResources(ps, null, conn);
}
}
标签:Insert,JDBC,name,ps,executeBatch,OutOfMemoryError,null,conn,String
From: https://blog.51cto.com/u_16261339/7467821