在C中将大数据集合插入卡桑德拉的最快方法#

Question

我对将大型集合插入cassandra数据库的最快方法有点困惑。我了解到我不应该使用批量插入，因为它是为原子性而创建的。甚至Cassandra也给了我一个信息，让我使用异步写来提高性能。我使用了没有“batch”关键字的最快插入代码:

var cluster = Cluster.Builder()
 .AddContactPoint(“127.0.0.1")
 .Build();
 var session = cluster.Connect();

 //Save off the prepared statement you’re going to use
 var statement = session.Prepare (“INSERT INTO tester.users (userID, firstName, lastName) VALUES (?,?,?)”);

 var tasks = new List<Task>();
 for (int i = 0; i < 1000; i++) 
 {

 //please bind with whatever actually useful data you’re importing
 var bind = statement.Bind (i, “John”, “Tester”);
 var resultSetFuture = session.ExecuteAsync (bind);
 tasks.Add (resultSetFuture);
 }

 Task.WaitAll(tasks.ToArray());
 cluster.Shutdown();

出发地：https://medium.com/@fondev/cassandra-batch-loading-non-the-batch-keyword-40f00e35e23e
但它仍然比我使用的批处理选项慢得多。我当前的代码如下所示：

IList<Movie> moviesList = Movie.CreateMoviesCollectionForCassandra(collectionEntriesNumber);
            var preparedStatements = new List<PreparedStatement>();
            foreach (var statement in preparedStatements)
            {
                statement.SetConsistencyLevel(ConsistencyLevel.One);
            }
            var statementBinding = new BatchStatement();
            statementBinding.SetBatchType(BatchType.Unlogged);

        for (int i = 0; i < collectionEntriesNumber; i++)
        {
            preparedStatements.Add(Session.Prepare("INSERT INTO Movies (id, title, description, year, genres, rating, originallanguage, productioncountry, votingsnumber, director) VALUES (?,?,?,?,?,?,?,?,?,?)"));
        }
        for (int i = 0; i < collectionEntriesNumber; i++)
        {
            statementBinding.Add(preparedStatements[i].Bind(moviesList[i].Id, moviesList[i].Title,
                moviesList[i].Description, moviesList[i].Year, moviesList[i].Genres, moviesList[i].Rating,
                moviesList[i].OriginalLanguage, moviesList[i].ProductionCountry, moviesList[i].VotingsNumber,
                new Director(moviesList[0].Director.Id, moviesList[i].Director.Firstname,
                    moviesList[i].Director.Lastname, moviesList[i].Director.Age)));
        }
        watch.Start();
        Session.ExecuteAsync(statementBinding);
        watch.Stop();

它确实工作得更快，但我只能插入大约2500个准备好的语句，仅此而已，我想测量大约100000个对象插入的时间。

我的代码正确吗？也许我应该增加插入门槛？请解释我如何正确地做这件事。

Answer 1

请记住，您应该准备一次，然后重用相同的准备语句来绑定到不同的参数。

如果您的目标是同一个分区，您可以使用小批量，如果不是，您应该使用单个请求。

使用单个请求时，您可以并行计划执行，并使用信号量限制未完成的请求量。

像这样的东西：

    public async Task<long> Execute(
        IStatement[] statements, int parallelism, int maxOutstandingRequests)
    {
        var semaphore = new SemaphoreSlim(maxOutstandingRequests);
        var tasks = new Task<RowSet>[statements.Length];
        var chunkSize = statements.Length / parallelism;
        if (chunkSize == 0)
        {
            chunkSize = 1;
        }
        var statementLength = statements.Length;
        var launchTasks = new Task[parallelism + 1];
        var watch = new Stopwatch();
        watch.Start();
        for (var i = 0; i < parallelism + 1; i++)
        {
            var startIndex = i * chunkSize;
            //start to launch in parallel
            launchTasks[i] = Task.Run(async () =>
            {
                for (var j = 0; j < chunkSize; j++)
                {
                    var index = startIndex + j;
                    if (index >= statementLength)
                    {
                        break;
                    }
                    await semaphore.WaitAsync();
                    var t = _session.ExecuteAsync(statements[index]);
                    tasks[index] = t;
                    var rs = await t;
                    semaphore.Release();
                }
            });
        }
        await Task.WhenAll(launchTasks);
        await Task.WhenAll(tasks);
        watch.Stop();
        return watch.ElapsedMilliseconds;
    }