我正在创建一个Spring批处理作业,将数据从给定的CSV文件填充到数据库表中。我创建了一个定制的FlatFileItemReader。我的问题是read()方法只被调用一次,所以只有CSV文件的第一行被插入到数据库中。
@Configuration
@EnableBatchProcessing
public class SpringBatchConfig {
private MultipartFile[] files;
@Bean
public Job job(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
ItemReader<MyModelEntity> itemReader,
ItemWriter<MyModelEntity> itemWriter) {
Step step = stepBuilderFactory.get("Load-CSV-file_STP")
.<MyModelEntity, MyModelEntity > chunk(12)
.reader(itemReader)
.writer(itemWriter).build();
return jobBuilderFactory.get("Load-CSV-Files").
incrementer(new RunIdIncrementer()) /
.start(step)
.build();
}
@Bean
ItemReader<MyModelEntity> myModelCsvReader() throws Exception {
return new MyModelCsvReader();
}
}
myModelCsvReader
@Component
@StepScope
public class MyModelCsvReader implements ItemReader<MyModelEntity>{
@Value("#{jobParameters['SDH']}")
private String sdhPath;
private boolean batchJobState= false;
@Autowired
MyModelFieldSetMapper myModelFieldSetMapper;
public LineMapper<MyModelEntity> lineMapper() throws Exception {
DefaultLineMapper<MyModelEntity> defaultLineMapper = new
DefaultLineMapper<MyModelEntity>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]
{
"clientId","ddId","institName","progName",
"qual","startDate","endDate","eType", "country","comments"
});
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(myModelFieldSetMapper);
return defaultLineMapper;}
@Override
public MyModelEntity read()
throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
//if(!batchJobState )
{
FlatFileItemReader<MyModelEntity> flatFileItemReader = new
FlatFileItemReader<MyModelEntity>();
flatFileItemReader.setMaxItemCount(2000);
flatFileItemReader.setResource(new UrlResource("file:\\"+sdhPath));
flatFileItemReader.setName("CSV-Reader");
flatFileItemReader.setLinesToSkip(1);
flatFileItemReader.setLineMapper(lineMapper());
flatFileItemReader.open(new ExecutionContext());
batchJobState=true;
return flatFileItemReader.read();
}
// return null;
}
}
@Component
public class MyModelFieldSetMapper implements FieldSetMapper<MyModelEntity> {
//private SiteService siteService =BeanUtil.getBean(SiteServiceImpl.class);
@Autowired
private SiteService siteService;
@Override
public MyModelEntity mapFieldSet(FieldSet fieldSet ) throws BindException {
if(fieldSet == null){
return null;
}
MyModelEntity educationHistory = new MyModelEntity();
// setting MyModelAttributes Values
return myModel;
}
}
欢迎任何一致意见。感谢//扩展FlatFileItemReader后的thereader
@Component
@StepScope
public class CustomUserItemReader extends FlatFileItemReader<User> {
@Value("#{jobParameters['UserCSVPath']}")
private String UserCSVPath;
private boolean batchJobState;
public CustomUserItemReader() throws Exception {
super();
setResource(new UrlResource("file:\\"+UserCSVPath));
setLineMapper(lineMapper());
afterPropertiesSet();
setStrict(false);
}
public LineMapper<User> lineMapper() throws Exception {
DefaultLineMapper<User> defaultLineMapper =
new DefaultLineMapper<User>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]{"name", "dept",
"salary","endDate"});
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(new CustomUserFieldSetMapper());
//defaultLineMapper.setFieldSetMapper(fieldSetMapper);
return defaultLineMapper;}
@Override
public User read()
throws Exception, UnexpectedInputException, ParseException,
NonTransientResourceException {
//if(!batchJobState )
{
flatFileItemReader).size())
// flatFileItemReader.setMaxItemCount(2000);
this.setResource(new UrlResource("file:\\"+UserCSVPath));
this.setName("CSV-Reader");
this.setLinesToSkip(1);
//flatFileItemReader.setLineMapper(lineMapper());
this.open(new ExecutionContext());
User e = this.read();
batchJobState = true;
return e ;
}
// return null;
}
public
String getUserCSVPath() {
return UserCSVPath;
}
public
void setUserCSVPath(String userCSVPath) {
UserCSVPath = userCSVPath;
}
}
感谢您的所有建议,即使我已经实现了ItemReader<>。我通过将FlatFileItemReader的实例化从read()方法移出来解决这个问题。这就是在每个循环中创建一个新的FlatFileItemReader,并且只读取创建的每个对象的第一行。谢谢
问题内容: 我正在尝试使用FlatFileItemReader解析CSV文件。此CSV包含一些带引号的换行符,如下所示。 但是此解析失败,必填字段为2,而实际字段为1。 FlatFileReader配置中缺少什么? 问题答案: 开箱即用的FlatFileItemReader使用SimpleRecordSeparatorPolicy,用于您的用例 注释部分超过2行或更多行 您需要设置DefaultR
我是R的新手,想读一个csv文件。但是当我试图阅读它时,我遇到了错误。我的csv文件如下: 当我在RStudio中使用此命令时,我得到了错误:命令: 错误: 读取时出错。表(file=file,header=header,sep=sep,quote=quote,:不允许重复的“row.names” 我还尝试删除错误并使用此命令: 但是当我查看输出时,它不能保持方阵的结构。你能帮我做什么吗?
问题内容: 我有一个CSV文件,下面是其外观示例: 我知道如何读取文件并打印每列(例如- )。但是我真正想做的是读取行,就像这样,然后依此类推。 然后,我想将这些数字存储到变量中,以便稍后将它们总计(例如): 。那我可以做。 我将如何在Python 3中做到这一点? 问题答案: 您可以执行以下操作: 要么 : 编辑:
读者是这样的, 我想要实现的是,如果在读取CSV文件时,读取器/作业失败,那么下一个计划的jobinstance应该从上次JobExecution失败的行启动读取器。我可以在CustomProcessor process()函数中跟踪计数器处理的行。
到目前为止,我的方法是: 挑战是:不能使用。如何正确使用?