我需要在我的MySQL数据库中进行批量插入(近10000个)。我正在使用JPA/Hibernate和spring Boot。我从hibernate文档中读到了在hibernate中执行大容量插入/更新,我认为我的代码不能正常工作,因为它顺序地插入hibernate查询,而不是在批处理插入中执行它们。下面是我的代码。我是不是漏掉了什么?
这是我的数据源配置。
@Component
public class Datasource {
@Autowired
EnvConfiguration configuration;
private static final Logger logger = LoggerFactory.getLogger(Datasource.class);
@Bean
public DataSource dataSource(){
logger.info("DataSource Bean creation...");
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(configuration.getDBDriver());
dataSource.setUrl("jdbc:mysql://"+configuration.getDBIp()+":"+configuration.getDBPort()+"/"+configuration.getDBName()+"?autoReconnect=true&useSSL=false");
dataSource.setUsername(configuration.getDBUser());
dataSource.setPassword(configuration.getDBPass().trim());
return dataSource;
}
@Bean
public HibernateJpaSessionFactoryBean sessionFactory() {
return new HibernateJpaSessionFactoryBean();
}
}
//Role.java
@Entity
@Table(name = "Role",uniqueConstraints = @UniqueConstraint(
columnNames = { "roleName"}))
public class Role {
@Id
@GeneratedValue(strategy = GenerationType.AUTO)
private Long roleId;
@NotNull
private String roleName;
public Role(){}
public Role(String roleName){
this.roleName = roleName;
}
public Long getRoleId() {
return roleId;
}
public void setRoleId(Long roleId) {
this.roleId = roleId;
}
public String getRoleName() {
return roleName;
}
public void setRoleName(String roleName) {
this.roleName = roleName;
}
}
@Service
public class RoleService{
@Autowired
private SessionFactory factory;
@Autowired
private DataSource source;
private static final Logger logger = LoggerFactory.getLogger(WalletService.class);
public void insertRole(Collection<RegisterWallet> walletMetaCollection){
if(factory==null){
System.out.println("factory is null");
}else{
System.out.println("factory is working");
Session session = factory.openSession();
Transaction tx = session.beginTransaction();
for ( int i=0; i<100000; i++ ) {
Role role=new Role(""+i);
session.persist(role);
System.out.println("this is the role id "+role.getRoleId());
try {
Thread.sleep(1000);
} catch (InterruptedException e){
// TODO Auto-generated catch block
e.printStackTrace();
}
if ( i % 10 == 0 ) { //20, same as the JDBC batch size
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
}
tx.commit();
session.close();
}
}
}
//This is log of the above code.
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 14
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 15
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 16
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 17
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 18
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 19
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 20
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 21
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 22
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 23
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 24
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 25
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 26
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 27
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 28
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 29
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 30
Hibernate: insert into role (active, role_description, role_name) values (?, ?, ?)
this is the role id 31
</pre>
-------------------------------------------------------------
spring.jpa.properties.hibernate.current_session_context_class=org.springframework.orm.hibernate4.SpringSessionContext
spring.jpa.properties.hibernate.jdbc.batch_size=10
我是不是漏掉了什么?
请帮帮忙。
您没有遗漏任何内容。您的输出非常正常。您可以通过以下链接获得更多信息:[1]http://www.dineshonjava.com/2012/06/hibernate-batch-processing_10.html
我正在使用hibernate jpa执行批处理更新。 更新:我得到了解决方案:问题是我正在刷新已经刷新的事务,因此它没有给我任何事务正在进行中的错误以及我上面发布的错误,所以我只是删除了`getem().flush();和getEm().clear();从finally块开始工作:)
问题内容: 我有一个dao,它基本上使用hibernate将记录插入到一个表中,该dao用标记为注释,并且我有一个服务,该服务会生成其他一些东西,然后调用我的dao。我的服务也标注了使用。 我叫服务循环。我在dao上的插入内容是否可以批量或一个接一个地工作?我如何确定它们可以批量工作?hibernateTransaction Manager是否管理批处理插入? 我正在使用Oracle DB。
问题内容: 我需要从每日CSV文件中消耗大量数据。CSV包含约12万条记录。使用hibernate模式时,这会减慢爬行速度。基本上,当使用saveOrUpdate()时,hibernate似乎在每个单独的INSERT(或UPDATE)之前执行SELECT。对于使用saveOrUpdate()持久存储的每个实例,在实际的INSERT或UPDATE之前发出SELECT。我能理解为什么要这样做,但是在进
我有一个批处理过程,它正在为一组实体重新计算数据。通过Hibernate从DB获取实体列表: 当流程运行时,某些实体似乎正在分离,导致两种症状: 当尝试获取惰性数据时,我得到一个异常: 在我的第一次尝试中,我试图通过调用inside
我很难让Hibernate在MySQL上执行大容量插入。 我有要排序的主键生成类型,下面是我的dao.xml 这使得每次都可以执行单个查询。我也相应地刷新和清除实体管理器。 技术堆栈:Spring 4,Hibernate 5.2。 编辑1:我也浏览了下面的链接,但没有运气 https://vladmihalcea.com/how-to-batch-insert-and-update-stateme
问题内容: 我有一个产品对象,它属于某些类别,即经典的多对一关系。 我想插入和更新产品而不预先选择类别。像这样: 要么 是否可以在不选择类别的情况下进行更新和插入?我不想为此使用HQL或直接查询。 问题答案: session.load()专门用于此类情况。以下: 不会打数据库。但是,如果没有提供给定ID的类别,它将在稍后阶段(刷新期间或多或少)引发异常。 使用速度快且没有副作用(级联等)。