This week in Databend #19

柴声

2023-12-01

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

add query log table write by @BohuTANG. (#3273)
return nodes list in metasrv server by @sunisdown. (#2943)
support create stage statement by @GrapeBaBa. (#3253)
default expression support by @sundy-li. (#3282)
add system.query_log by @SGZW. (#3234)
add revoke statement by @flaneur2020. (#3194)
add strings functions
- INSTR, FIELD, SUBSTRING, SUBSTR, MID, SUBSTRING_INDEX, REVERSE, REPLACE, STRCMP, SPACE, LEFT, RIGHT, LPAD, RPAD, CHARACTER_LENGTH, CHAR_LENGTH, ORD by @mshauneu (#3217, #3215, #3231, etc.)
- bin_length by @radmirnovii (#3232)
- octet_length by @unconsolable (#3227)
- trim by @QuantumBear (#3196)
- CONCAT, CONCAT_WS by @Veeupup (#3214, #3260)

Improvement

replace AppendOperationLogEntry vec[] with stream by @junli1026 (#3296)
support create table like statement by @junli1026 (#3292)
show table options by @Junnplus (#3252)
make StateMachine::apply use transaction to perform operations by @ariesdevil. (#3102)
complete dates function monotonicity check by @zhyass. (#3228)
http: refine the mod import and add http usage function by @BohuTANG. (#3246)
refactoring insert interpreter by @dantengsky. (#3226)
[common/function] simplify get_layout_offsets by @drmingdrmer (#3223)
add monotonicity in range filter by @zhyass. (#3195)
introduce compact statement by @dantengsky. (#3182)

Performance Improvement

string performence improvement by @Veeupup. (#3288)

Bug fixes

fix try_from_literal by @dust1. (#3197)
fix github table engine inits by @BohuTANG. (#3204)

Tips

Let's learn a weekly tip from Databend.

How to explore github repos via Databend

Databend now supports GitHub as a data source, and you can read the relevant code at storages/github.

create github-engine based database

Before running databend, your Github Access Token should be set.

export GITHUB_TOKEN=<your_token>;

Create a Github powered database.

databend :) create database datafuselabs engine=github;

0 rows in set. Elapsed: 2.611 sec.

show all tables

Show all tables in this database, which are currently flattened. This means that Repos, issues and PRs are all in the form of tables.

databend :) use datafuselabs;

0 rows in set. Elapsed: 0.013 sec.

databend :) show tables;

+---------------------------------+
| name                            |
+---------------------------------+
| .github                         |
| .github_comments                |
| .github_issues                  |
| .github_prs                     |
| databend                        |
| databend-playground             |
| databend-playground_comments    |
| databend-playground_issues      |
| databend-playground_prs         |
| databend_comments               |
| databend_issues                 |
| databend_prs                    |
| ...                             |
+---------------------------------+

36 rows in set. Elapsed: 0.053 sec.

View basic information about a repo

databend :) select * from databend;

+------------+----------+------------+------------+-------------+----------------+-------------------+-------------------+
| reposiroty | language | license    | star_count | forks_count | watchers_count | open_issues_count | subscribers_count |
+------------+----------+------------+------------+-------------+----------------+-------------------+-------------------+
| databend   | Rust     | apache-2.0 |       2661 |         252 |           2661 |               349 |                63 |
+------------+----------+------------+------------+-------------+----------------+-------------------+-------------------+

1 rows in set. Elapsed: 1.368 sec.

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

arrow2 #641 & #664: Add lower and upper support for string. by @Xuanwo
arrow2 #651: Improved performance in cast Primitive to Binary/String again (4x): Memcpy style write, no extra copy. by @sundy-li

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

如果想了解更多，可以关注公众号： Databend