Popular New Releases in DB Client
doobie
typegoose
v9.8.1
p6spy
3.9.1
SchemaCrawler
v16.16.14
scalikejdbc
version 4.0.0
Popular Libraries in DB Client
by brettwooldridge java
16232 Apache-2.0
光 HikariCP・A solid, high-performance, JDBC connection pool at last.
by nestjsx typescript
2929 MIT
NestJs CRUD for RESTful APIs
by jprante java
2834 NOASSERTION
JDBC importer for Elasticsearch
by developit javascript
2390 MIT
:battery: Starter project for an ES6 RESTful Express API.
by tpolecat scala
1917 MIT
Functional JDBC layer for Scala.
by Nealyang javascript
1829
:fire: React full stack+Express+Mongo implementation blog website tutorial :new_moon_with_face:
by typegoose typescript
1708 MIT
Typegoose - Define Mongoose models using TypeScript classes.
by DreamOfTheRedChamber java
1522
Preparing for system design interview questions
by p6spy java
1458 Apache-2.0
P6Spy is a framework that enables database data to be seamlessly intercepted and logged with no code changes to the application.
Trending New libraries in DB Client
by basir javascript
1165
Build ECommerce Website Like Amazon By React & Node & MongoDB
by techreagan javascript
453
VueTube is a YouTube clone built with nodejs, expressjs & mongodb. This is the RESTful API repository.
by basir javascript
237
Build Ecommerce Like Amazon By MERN Stack
by fast-crud javascript
155 MIT
面向配置的crud框架,开发crud 快如闪电;Options-oriented crud framework, develop crud as fast as lightning;based on vue3
by naver java
145 Apache-2.0
Spring JDBC Plus
by dynamic-datasource java
138 NOASSERTION
by codeplutos c++
110
MySQL JDBC Deserialization Payload / MySQL客户端jdbc反序列化漏洞payload
by premieroctet typescript
91 MIT
Full-featured CRUD routes for Next.js
by devsuperior java
80
Top Authors in DB Client
1
11 Libraries
97
2
11 Libraries
415
3
9 Libraries
51
4
9 Libraries
369
5
8 Libraries
174
6
6 Libraries
286
7
5 Libraries
98
8
4 Libraries
98
9
4 Libraries
66
10
4 Libraries
89
1
11 Libraries
97
2
11 Libraries
415
3
9 Libraries
51
4
9 Libraries
369
5
8 Libraries
174
6
6 Libraries
286
7
5 Libraries
98
8
4 Libraries
98
9
4 Libraries
66
10
4 Libraries
89
Trending Kits in DB Client
No Trending Kits are available at this moment for DB Client
Trending Discussions on DB Client
Closing MongoClient connection on exit when using MongoDB Native driver?
Bypassing Firestore Security Rules in jest tests
How to use condition and multiple steps in a reactive function
WordPress Nginx Configuration - GoDaddy Managed WP
Check if user input exist in database using php and mongodb
Strange behaviour insert_one mongocxx 3.6
Simplest way to override an endpoint in google guice when writing functional tests
How to bind lifetimes of Futures to fn arguments in Rust
Encoding UTF8 string to latin1/iso-8859-1 with Go and MySQL
Where is routing implemented (Request routing to the different replicas) when making request to the DynamoDB server?
QUESTION
Closing MongoClient connection on exit when using MongoDB Native driver?
Asked 2022-Apr-07 at 09:49Should the MongoClient connection be closed every time the server shuts down?
I have seen the following code snippet and wanted to know if this is actually valid and should be done or if it's completely unnecessary to perform a closing on exit:
1// Adding listeners
2function setupListeners(client: MongoClient){
3 client.addListener('topologyClosed', ()=>{
4 isTopologyConnected = false;
5 console.warn("topologyClosed");
6 })
7}
8process.on("exit", () => {
9 console.log("EXIT - MongoDB Client disconnected");
10 closeConnection()
11});
12
13//Cleanups
14//catching signals and doing cleanup
15['SIGHUP', 'SIGINT', 'SIGQUIT', 'SIGILL', 'SIGTRAP', 'SIGABRT',
16 'SIGBUS', 'SIGFPE', 'SIGUSR1', 'SIGSEGV', 'SIGUSR2', 'SIGTERM'
17].forEach(function (signal) {
18 process.on(signal, function () {
19 if (isTopologyConnected){
20 client.close();
21 }
22 process.exit(1);
23 });
24});
25
Thanks a lot.
ANSWER
Answered 2022-Apr-07 at 09:49Should the MongoClient connection be closed every time the server shuts down?
Yes, it is a good practice to close the connection. As for every connection, mongo DB does assign a thread for its execution. If you won't close it, it keeps using the resources on the DB server.
Node.js connections use the pool to connect to DB and it can be reused while it is not being used, but it is good practice to close the connection if you are exiting the script as it won't close the connection automatically.
QUESTION
Bypassing Firestore Security Rules in jest tests
Asked 2022-Mar-24 at 08:37Currently working on a React/Typescript/Firebase Firestore project. When writing Jest-tests for some actions/functions that are called from the UI, I ran into the following problem:
In the test file I'm able to setup the firestore client using the v9 api and make it talk to emulator
1const app = initializeApp(config.firebase);
2const firestore = getFirestore(app);
3connectFirestoreEmulator(firestore, "localhost", 8080);
4
In addition I also found out how to setup the admin client and make it talk to emulator
1const app = initializeApp(config.firebase);
2const firestore = getFirestore(app);
3connectFirestoreEmulator(firestore, "localhost", 8080);
4process.env.FIRESTORE_EMULATOR_HOST = "localhost:8080";
5const serviceAccount = require("../../../my-key.json");
6admin.initializeApp({
7 credential: admin.credential.cert(serviceAccount),
8 ...config.firebase
9});
10
The test itself looks something like this:
1const app = initializeApp(config.firebase);
2const firestore = getFirestore(app);
3connectFirestoreEmulator(firestore, "localhost", 8080);
4process.env.FIRESTORE_EMULATOR_HOST = "localhost:8080";
5const serviceAccount = require("../../../my-key.json");
6admin.initializeApp({
7 credential: admin.credential.cert(serviceAccount),
8 ...config.firebase
9});
10describe("createCompanyAndRating action", () => {
11 test("call createCompanyAndRating and make sure it creates a proper rating entity", async () => {
12 // omitted: set testRatingFormState and other test data that are passed as args and
13 // pass in the firestore db client
14 const {
15 ratingId,
16 companyId,
17 } = await createCompanyAndRating({
18 ratingFormState: testRatingFormState,
19 visitorId: testVisitorId,
20 firestore,
21 });
22 // verify result by fetching the rating entity from the emulator db using the admin client
23 const ratingPath = `companies/${companyId}/ratings/${ratingId}`;
24 const ratingSnap = await admin.firestore().doc(ratingPath).withConverter(ratingConverter).get();
25 const rating: Rating | undefined = ratingSnap.data();
26 // omitted: verify result with some Jest expect-statetments...
27 });
28})
29
My problem is now that the Firestore security rules apply and only authenticated users can write docs in the collections used in the createCompanyAndRating function, so the test already throws an error when calling that function.
In this scenario I'm not interested in testing the security rules per se.
- Is there a way to bypass the security rules for the test?
- If yes, how do I have to setup the firestore client?
- Is there even the possibility to somehow impersonate a user in the test?
In addition, please note that I can't to pass the admin client into the createCompanyAndRating function as the admin client API is different from the v9 firebase API that I'm relying on in the createCompanyAndRating function implementation (tried and it didn't work and not only because some type errors in the way).
Maybe my whole approach is a little misguided and I should rather concentrate on testing the internals of the createCompanyAndRating function where I do a lot of factory stuff that could be tested without db interaction.
Anyway, any help/guidance is much appreciated.
ANSWER
Answered 2022-Mar-24 at 08:37Thanks for confirming that I was looking in the right place (i.e. @firebase/rules-unit-testing). Finally figured out what the problem was, missed an "await" in createCompanyAndRating, so the firestore admin instance wasn't getting the data (and I though it was a admin config issue...) Thanks!
QUESTION
How to use condition and multiple steps in a reactive function
Asked 2022-Mar-16 at 08:22I'm currently building a reactive app using kotlin quarkus and mutiny, being new to reactive programming i wonder what's the best way to handle the following workflow :
- try to find an item in the database (dynamodb)
- if it exists return a value indicating this to the caller
- if it does not exists call the save service
- call an external service to get some data
- update the item in database with those data from the external service
- return a value indicating that the object has been created
Here's my code for now :
1fun createCard(creationOrder: CreationOrder): Uni<CardCreationResult> {
2 return creationOrdersRepository.findByOrderId(creationOrder.orderId)
3 .onItem().transform {item ->
4 if (item != null) {
5 CardCreationResult.AlreadyCreated
6 } else {
7 creationOrdersRepository.save(creationOrder)
8
9 //TODO external webservice call
10
11 val cardNumber = UUID.randomUUID().toString()
12 creationOrdersRepository.updateCardNumberAndStatus(externalServiceCallResult)
13 CardCreationResult.Created
14 }
15 }
16}
17
This method will eventually be called by a rest endpoint.
creationOrdersRepository.save
and creationOrdersRepository.updateCardNumberAndStatus
returns a CompletableFuture (i'm using the quarkus amazon dynamodb client).
Is this the right way to do it ? Should i wrap the save and updateCardNumberAndStatus results in Uni (i have been trying to but keep getting type error) ?
ANSWER
Answered 2022-Mar-16 at 08:22I don't believe your code does what you expect. If I understand correctly you need to "save" and "update" before emitting your result. So, you need something like (in Java, as my Kotlin is not great):
1fun createCard(creationOrder: CreationOrder): Uni<CardCreationResult> {
2 return creationOrdersRepository.findByOrderId(creationOrder.orderId)
3 .onItem().transform {item ->
4 if (item != null) {
5 CardCreationResult.AlreadyCreated
6 } else {
7 creationOrdersRepository.save(creationOrder)
8
9 //TODO external webservice call
10
11 val cardNumber = UUID.randomUUID().toString()
12 creationOrdersRepository.updateCardNumberAndStatus(externalServiceCallResult)
13 CardCreationResult.Created
14 }
15 }
16}
17return creationOrdersRepository.findByOrderId(creationOrder.orderId)
18 // We will be returning a Uni as the save, update and the web service are
19 // async
20 .onItem().transformToUni(item -> {
21 if (item != null) {
22 return Uni.createFrom().item(CardCreationResult.AlreadyCreated);
23 } else {
24 // Create an Uni from the save operation
25 Uni<Void> save = Uni.createFrom().completionStage(() ->
26 creationOrdersRepository.save(creationOrder));
27 // This uni represents the invocation of the remote service
28 Uni<String> webService = callWebService();
29
30 // we chain the operations save -> web service -> update -> Created
31 return save
32 .chain(webService)
33 .chain(externalServiceCallResult ->
34 Uni.createFrom().completionStage(() -> creationOrdersRepository
35 .updateCardNumberAndStatus(externalServiceCallResult)
36 )
37 .replaceWith(CardCreationResult.Created);
38 }
39
QUESTION
WordPress Nginx Configuration - GoDaddy Managed WP
Asked 2022-Mar-01 at 22:02_Managed WP, GoDaddy _Tools available: in GD: "File Browser", "phpMyAdmin" (takes me to a URL UI). _phpMyAdmin:
- Has things like "Databases" "Status" "Ex/Import" "Variables" "Charsets" "Engines".
- Left tab has info_schema & server/site id in a drop down menu setup. One drop per item (schema & site/server). Two hierarchies.
- A console/terminal
- db client libmysql - mysqlnd 7.4.16
- I don't know how to control the webserver. I am trying to setup multisite network. Currently testing/configuring pretty permalinks. Instructions unclear. Can't find GoDaddy docs on server interaction. Console in phpMyAdmin only sends SQL queries, totally not helpful. This documentation assumes I am big-brain Dev.
- states
However, we can set our really cool permalink configuration directly by editing Nginx. We will use try_files directive so WordPress can start using pretty permalinks. Let’s see the configuration for WordPress installed on the root of your domain, and also on a directory called /blog
WordPress Nginx Configuration for root installationsSearch for the location / block inside nginx configuration and add the following line inside:
1try_files $uri $uri/ /index.php?$args;
2
My questions / issues
- Search for location?? What location?
- Search using what? How do I get to the webserver/search util? ?SSH to the ip? Currently working with GD to figure out why SSH won't enable as it should...
Everything available to me and working properly does not produced desired returns when searching...
ANSWER
Answered 2021-Dec-07 at 08:55Problem solved! There is a reason so many online articles talk about Apache and few talk about nginx (concerning GoDaddy, Managed WordPress, Multisite Network Install). The reason is because A) Managed WordPress from G-Diddy is incompatible with Multi-site Network installation. Why? The parts of the server and the control over directories needed to achieve Multisite Network install are x) not accessible x) non-existent do to lack of tools (this is why the monthly cost is lower). {I keep mentioning Multisite Network, that was the end-goal; Pretty Permalinks setup was a prerequisite to Multi-site). B) All Managed WordPress offers from GDiddy come with nginx so: no Apache, meaning !no!htaccess.php file exist! C) No Apache also means no cPanel, meaning, a bunch of other s**t doesn't exist too...
Resolution:- Upgraded to "WordPress with cPanel" (this is delivered on a Linux box, with Apache webserver + cPanel).
Be sure to check my next post which is already halfway to a self answer but I don't believe I will be able to fully answer it so the need to post (and partially) self answer is not just so I can pat my own back, still need big time help!
QUESTION
Check if user input exist in database using php and mongodb
Asked 2022-Feb-11 at 07:04I want to check first if the email exists then check the password but I only want to use PHP and MongoDB not any other language or tool because I'm just a beginner.
when I enter an input the page change to an empty page and nothing happen. I know my error is from the line of the array $findemail
and downward.
am I using the $cursor in the right way and if so am I accessing the $findemail
in the right way using $d['email']
to compare it with $email
1<?php
2//Include libraries
3require __DIR__ . '/vendor/autoload.php';
4
5//Create instance of MongoDB client
6$mongoClient = (new MongoDB\Client);
7
8//Select a database
9$db = $mongoClient->EcommerceWeb;
10
11//Select a collection
12$collection = $db->Employees_Data;
13
14//Extract the data that was sent to the server
15$email = filter_input(INPUT_POST, 'email', FILTER_SANITIZE_STRING);
16$password = filter_input(INPUT_POST, 'password', FILTER_SANITIZE_STRING);
17
18
19
20$findemail = [
21"email" => $email,
22"password" => $password
23 ];
24
25
26$cursor = $db->$collection->find($findemail);
27
28foreach ($cursor as $d ){
29
30if($d['email'] == $email and $d['passwprd'] == $password){
31 echo "success";
32 }
33else {
34 echo "fail";
35}
36 }
37
ANSWER
Answered 2022-Feb-09 at 22:34Try to this:
1<?php
2//Include libraries
3require __DIR__ . '/vendor/autoload.php';
4
5//Create instance of MongoDB client
6$mongoClient = (new MongoDB\Client);
7
8//Select a database
9$db = $mongoClient->EcommerceWeb;
10
11//Select a collection
12$collection = $db->Employees_Data;
13
14//Extract the data that was sent to the server
15$email = filter_input(INPUT_POST, 'email', FILTER_SANITIZE_STRING);
16$password = filter_input(INPUT_POST, 'password', FILTER_SANITIZE_STRING);
17
18
19
20$findemail = [
21"email" => $email,
22"password" => $password
23 ];
24
25
26$cursor = $db->$collection->find($findemail);
27
28foreach ($cursor as $d ){
29
30if($d['email'] == $email and $d['passwprd'] == $password){
31 echo "success";
32 }
33else {
34 echo "fail";
35}
36 }
37$cursor = $collection->findOne($findemail);
38
39if($cursor){
40 if($cursor->email == $email and $cursor->password == $password){
41 echo "success";
42 }
43}
44
QUESTION
Strange behaviour insert_one mongocxx 3.6
Asked 2022-Feb-10 at 06:49Working with mongocxx I am trying to retrieve the object id assigned by mongodb when I insert a new object in a collection(insert_one method), and convert this id into a string. This is the code:
1 const mongocxx::database& db = _pClient->database(_dbName.c_str());
2 mongocxx::collection& collection = db.collection(collectionName.c_str());
3 auto retval = collection.insert_one(view);
4 bsoncxx::oid oid = retval->inserted_id().get_oid().value;
5 std::string str = oid.to_string()
6
Unfortunately it looks like when I try to convert the object id to a string the string is unreadable (corrupted like). I am using mongocxx version 3.4 and mongodb 4.0.28 (the two versions should be compatible according to the mongodb website). Do you know what could be the problem here?
Here what I can see through the debugger: visual studio debugger
Here what I can see thorugh the mongodb client: mongodb client
ANSWER
Answered 2022-Feb-10 at 06:49The debugger is showing you the individual bytes in decimal.
I used the mongo shell to convert these to hex, you can see that it is indeed the ObjectID you were looking for, it just looks strange in decimal
1 const mongocxx::database& db = _pClient->database(_dbName.c_str());
2 mongocxx::collection& collection = db.collection(collectionName.c_str());
3 auto retval = collection.insert_one(view);
4 bsoncxx::oid oid = retval->inserted_id().get_oid().value;
5 std::string str = oid.to_string()
6mongos> [98,3,-24,41,-88,89,0,0,-93,0,88,-78].map(n=>("0" +((n & 255).toString(16))).slice(-2)).join("")
76203e829a8590000a30058b2
8
QUESTION
Simplest way to override an endpoint in google guice when writing functional tests
Asked 2022-Feb-04 at 12:02I have an application module that installs a DynamoDB module
1install(new DynamoDBModule());
2
In DynamoDBModule we have some code to build the DynamoDb client and initialize the mapper
1install(new DynamoDBModule());
2 AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
3 .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://prod-endpoint:8000", "us-west-2"))
4 .build();
5
Now, when I am writing tests, I have to replace this dynamoDB endpoint with a local endpoint and I was wondering what is the simplest way to do this. I am aware of this stackoverflow question, but it would mean writing a lot of code for just a single small change. I would have to create a mock dynamodb module, a mock application module and then I can run something like this in my tests
1install(new DynamoDBModule());
2 AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
3 .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://prod-endpoint:8000", "us-west-2"))
4 .build();
5Guice.createInjector(Modules.override(new AppModule()).with(new TestAppModule()));
6
Is there a simple way to somehow use or override the test endpoint when running tests and continue using the prod endpoint otherwise.
ANSWER
Answered 2022-Feb-04 at 12:02Configure an EndpointConfiguration
as binding and override it in TestAppModule
. E.g.:
1install(new DynamoDBModule());
2 AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
3 .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://prod-endpoint:8000", "us-west-2"))
4 .build();
5Guice.createInjector(Modules.override(new AppModule()).with(new TestAppModule()));
6class DynamoDBModule {
7 @Provides
8 @Singleton
9 AmazonDynamoDB provideAmazonDynamoDB(EndpointConfiguration endpointConfiguration) {
10 return AmazonDynamoDBClientBuilder.standard()
11 .withEndpointConfiguration(endpointConfiguration)
12 .build()
13 }
14
15 @Provides
16 @Singleton
17 EndpointConfiguration provideEndpointConfiguration() {
18 return new AwsClientBuilder.EndpointConfiguration("http://prod-endpoint:8000", "us-west-2");
19 }
20}
21
22class TestAppModule {
23
24 @Provides
25 @Singleton
26 EndpointConfiguration provideTestEndpointConfiguration() {
27 return new AwsClientBuilder.EndpointConfiguration("test-value", "us-west-2");
28 }
29}
30
Then use you approach with Modules.override
and it should work:
1install(new DynamoDBModule());
2 AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
3 .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://prod-endpoint:8000", "us-west-2"))
4 .build();
5Guice.createInjector(Modules.override(new AppModule()).with(new TestAppModule()));
6class DynamoDBModule {
7 @Provides
8 @Singleton
9 AmazonDynamoDB provideAmazonDynamoDB(EndpointConfiguration endpointConfiguration) {
10 return AmazonDynamoDBClientBuilder.standard()
11 .withEndpointConfiguration(endpointConfiguration)
12 .build()
13 }
14
15 @Provides
16 @Singleton
17 EndpointConfiguration provideEndpointConfiguration() {
18 return new AwsClientBuilder.EndpointConfiguration("http://prod-endpoint:8000", "us-west-2");
19 }
20}
21
22class TestAppModule {
23
24 @Provides
25 @Singleton
26 EndpointConfiguration provideTestEndpointConfiguration() {
27 return new AwsClientBuilder.EndpointConfiguration("test-value", "us-west-2");
28 }
29}
30Guice.createInjector(Modules.override(new AppModule()).with(new TestAppModule()));
31
QUESTION
How to bind lifetimes of Futures to fn arguments in Rust
Asked 2022-Jan-27 at 19:43Im trying to write a simple run_transaction
function for the Rust MongoDB Driver
This function tries to execute a transaction through the mongo db client and retries the transaction if it encounters a retryable error
Here is a minimum reproducible example of the function.
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76
But the borrow checker keeps complaining with this message:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89
Is there a way I can solve this?
ANSWER
Answered 2022-Jan-27 at 19:43What you really need is something like:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
90 where
91 for<'a>
92 F: Fn(&'a mut ClientSession) -> Fut,
93 Fut: Future<Output = Result<T, TransactionError<E>>> + 'a {
94
Unfortunately this does not work because the "Higer Rank Trait Bound" (HRTB) defined in for<'a>
only applies to the very next bound, not to every one of them, and there is no way to connect both lifetimes...
But not everything is lost! I've found this question in the Rust support forum with a similar problem, that can be adapted to your issue. The basic idea is that you create a trait that wraps the Fn
and the Future
bounds with the same lifetime:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
90 where
91 for<'a>
92 F: Fn(&'a mut ClientSession) -> Fut,
93 Fut: Future<Output = Result<T, TransactionError<E>>> + 'a {
94pub trait XFn<'a, I: 'a, O> {
95 type Output: Future<Output = O> + 'a;
96 fn call(&self, session: I) -> Self::Output;
97}
98
99impl<'a, I: 'a, O, F, Fut> XFn<'a, I, O> for F
100where
101 F: Fn(I) -> Fut,
102 Fut: Future<Output = O> + 'a,
103{
104 type Output = Fut;
105 fn call(&self, x: I) -> Fut {
106 self(x)
107 }
108}
109
And now your bound function is trivial:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
90 where
91 for<'a>
92 F: Fn(&'a mut ClientSession) -> Fut,
93 Fut: Future<Output = Result<T, TransactionError<E>>> + 'a {
94pub trait XFn<'a, I: 'a, O> {
95 type Output: Future<Output = O> + 'a;
96 fn call(&self, session: I) -> Self::Output;
97}
98
99impl<'a, I: 'a, O, F, Fut> XFn<'a, I, O> for F
100where
101 F: Fn(I) -> Fut,
102 Fut: Future<Output = O> + 'a,
103{
104 type Output = Fut;
105 fn call(&self, x: I) -> Fut {
106 self(x)
107 }
108}
109pub async fn run_transaction<T, E, F>(f: F) -> Result<T, TransactionError<E>>
110 where for<'a>
111 F: XFn<'a, &'a mut ClientSession, Result<T, TransactionError<E>>>
112
Just remember that to call the function you have to write f.call(&mut session)
.
Unfortunately, the call to run_transaction()
, as it is, does not compile, saying that implementation of FnOnce
is not general enough. I think that it is a limitation/bug of the async move
as async closures are unstable. But you can instead use a proper async function:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
90 where
91 for<'a>
92 F: Fn(&'a mut ClientSession) -> Fut,
93 Fut: Future<Output = Result<T, TransactionError<E>>> + 'a {
94pub trait XFn<'a, I: 'a, O> {
95 type Output: Future<Output = O> + 'a;
96 fn call(&self, session: I) -> Self::Output;
97}
98
99impl<'a, I: 'a, O, F, Fut> XFn<'a, I, O> for F
100where
101 F: Fn(I) -> Fut,
102 Fut: Future<Output = O> + 'a,
103{
104 type Output = Fut;
105 fn call(&self, x: I) -> Fut {
106 self(x)
107 }
108}
109pub async fn run_transaction<T, E, F>(f: F) -> Result<T, TransactionError<E>>
110 where for<'a>
111 F: XFn<'a, &'a mut ClientSession, Result<T, TransactionError<E>>>
112 async fn do_the_thing(session: &mut ClientSession) -> Result<Document, TransactionError<Never>> {
113 let document = collection().find_one_with_session(None, None, session).await?.unwrap();
114 let r: Result<Document, TransactionError<Never>> = Ok(document);
115 return r;
116 }
117 run_transaction(do_the_thing).await;
118
If you think that this is too complicated, and you don't mind paying the very small runtime price there is a simpler option: you can box the returned future, avoiding the second generic altogether:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
90 where
91 for<'a>
92 F: Fn(&'a mut ClientSession) -> Fut,
93 Fut: Future<Output = Result<T, TransactionError<E>>> + 'a {
94pub trait XFn<'a, I: 'a, O> {
95 type Output: Future<Output = O> + 'a;
96 fn call(&self, session: I) -> Self::Output;
97}
98
99impl<'a, I: 'a, O, F, Fut> XFn<'a, I, O> for F
100where
101 F: Fn(I) -> Fut,
102 Fut: Future<Output = O> + 'a,
103{
104 type Output = Fut;
105 fn call(&self, x: I) -> Fut {
106 self(x)
107 }
108}
109pub async fn run_transaction<T, E, F>(f: F) -> Result<T, TransactionError<E>>
110 where for<'a>
111 F: XFn<'a, &'a mut ClientSession, Result<T, TransactionError<E>>>
112 async fn do_the_thing(session: &mut ClientSession) -> Result<Document, TransactionError<Never>> {
113 let document = collection().find_one_with_session(None, None, session).await?.unwrap();
114 let r: Result<Document, TransactionError<Never>> = Ok(document);
115 return r;
116 }
117 run_transaction(do_the_thing).await;
118pub async fn run_transaction<T, E, F>(f: F) -> Result<T, TransactionError<E>>
119 where for<'a>
120 F: Fn(&'a mut ClientSession) -> Pin<Box<dyn Future<Output = Result<T, TransactionError<E>>> + 'a>>
121
And then, to call it:
1use mongodb::{Client, Collection, ClientSession};
2use mongodb::bson::Document;
3use std::future::Future;
4
5pub enum Never {}
6
7fn main() {
8 run_transaction(|mut session| async move {
9 let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
10 let r: Result<Document, TransactionError<Never>> = Ok(document);
11 return r;
12 });
13}
14
15fn collection() -> Collection<Document> {
16 unimplemented!();
17}
18
19fn client() -> Client {
20 unimplemented!();
21}
22
23pub enum TransactionError<E> {
24 Mongodb(mongodb::error::Error),
25 Custom(E)
26}
27
28impl<T> From<mongodb::error::Error> for TransactionError<T> {
29 fn from(e: mongodb::error::Error) -> Self {
30 TransactionError::Mongodb(e)
31 }
32}
33
34// declaration
35pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
36 where for<'a>
37 F: Fn(&'a mut ClientSession) -> Fut + 'a,
38 Fut: Future<Output = Result<T, TransactionError<E>>> {
39
40
41 let mut session = client().start_session(None).await?;
42 session.start_transaction(None).await?;
43
44 'run: loop {
45 let r = f(&mut session).await;
46
47 match r {
48 Err(e) => match e {
49 TransactionError::Custom(e) => return Err(TransactionError::Custom(e)),
50 TransactionError::Mongodb(e) => {
51 if !e.contains_label(mongodb::error::TRANSIENT_TRANSACTION_ERROR) {
52 return Err(TransactionError::Mongodb(e));
53 } else {
54 continue 'run;
55 }
56 }
57 },
58
59 Ok(v) => {
60 'commit: loop {
61 match session.commit_transaction().await {
62 Ok(()) => return Ok(v),
63 Err(e) => {
64 if e.contains_label(mongodb::error::UNKNOWN_TRANSACTION_COMMIT_RESULT) {
65 continue 'commit;
66 } else {
67 return Err(TransactionError::Mongodb(e))
68 }
69 }
70 }
71 }
72 }
73 }
74 }
75}
76error: lifetime may not live long enough
77 --> src/main.rs:8:35
78 |
798 | run_transaction(|mut session| async move {
80 | ______________________------------_^
81 | | | |
82 | | | return type of closure `impl Future` contains a lifetime `'2`
83 | | has type `&'1 mut ClientSession`
849 | | let document = collection().find_one_with_session(None, None, &mut session).await?.unwrap();
8510 | | let r: Result<Document, TransactionError<Never>> = Ok(document);
8611 | | return r;
8712 | | });
88 | |_____^ returning this value requires that `'1` must outlive `'2`
89pub async fn run_transaction<T, E, F, Fut>(f: F) -> Result<T, TransactionError<E>>
90 where
91 for<'a>
92 F: Fn(&'a mut ClientSession) -> Fut,
93 Fut: Future<Output = Result<T, TransactionError<E>>> + 'a {
94pub trait XFn<'a, I: 'a, O> {
95 type Output: Future<Output = O> + 'a;
96 fn call(&self, session: I) -> Self::Output;
97}
98
99impl<'a, I: 'a, O, F, Fut> XFn<'a, I, O> for F
100where
101 F: Fn(I) -> Fut,
102 Fut: Future<Output = O> + 'a,
103{
104 type Output = Fut;
105 fn call(&self, x: I) -> Fut {
106 self(x)
107 }
108}
109pub async fn run_transaction<T, E, F>(f: F) -> Result<T, TransactionError<E>>
110 where for<'a>
111 F: XFn<'a, &'a mut ClientSession, Result<T, TransactionError<E>>>
112 async fn do_the_thing(session: &mut ClientSession) -> Result<Document, TransactionError<Never>> {
113 let document = collection().find_one_with_session(None, None, session).await?.unwrap();
114 let r: Result<Document, TransactionError<Never>> = Ok(document);
115 return r;
116 }
117 run_transaction(do_the_thing).await;
118pub async fn run_transaction<T, E, F>(f: F) -> Result<T, TransactionError<E>>
119 where for<'a>
120 F: Fn(&'a mut ClientSession) -> Pin<Box<dyn Future<Output = Result<T, TransactionError<E>>> + 'a>>
121 run_transaction(|mut session| Box::pin(async move {
122 let document = collection().find_one_with_session(None, None, session).await?.unwrap();
123 let r: Result<Document, TransactionError<Never>> = Ok(document);
124 return r;
125 }));
126
QUESTION
Encoding UTF8 string to latin1/iso-8859-1 with Go and MySQL
Asked 2022-Jan-25 at 22:48I have a MySQL database with German "Umlaute öäü" in a table and I need to write a Go app that reads the table, encode it to ISO-8859-1 and write it to a text file.
So far so good, but the encoding to iso-8859-1 is not working. I have tried to debug this.
Here some details and information:
MySQL The MySQL database is UTF8, also the table itself. Also other character sets should be fine, except the character_set_server, but I think this is not relevant here, it should be just a default for new databases as far as I know.
When I query the database with the following SQL, I get the correct UTF8 encoded text:
1select street, hex(street) from test_table where id = '36'
2
3Result: (in real it is called Fröbelstraße)
4Fröbelstraße, 4672C3B662656C73747261C39F65
5
So from the hex string it is basically exact what I have expected. OK.
Go App Just the relevant parts....
1select street, hex(street) from test_table where id = '36'
2
3Result: (in real it is called Fröbelstraße)
4Fröbelstraße, 4672C3B662656C73747261C39F65
5db, err := sql.Open("mysql", "...<connection string>...")
6res, err := db.Query("select street from from test_table where id = '36'")
7
8for res.Next() {
9var pb Phonebook
10 err := res.Scan(&pb.Street)
11 fmt.Println(hex.EncodeToString([]byte(pb.Street)))
12}
13
The output is
4672c383c2b662656c73747261c383c5b865
And that's the problem why my encoding to ISO-8859-1 is not working because the string from the database is not correct. The hex from the db direct query is correct and also working the the encoding.
But I don't understand why I get a different string from the go client.
In the original string "Fröbelstraße" are 2 characters "ö" which is C3B6 and "ß" which is C39F. The hex from the query with a db client is ok, but the one from the go app is too long, because I get a 2 bytes more per character.
When I feed my latin1 converter, with the correct hex string, it is working fine, I get an iso-8859-1 string. But not from the other one I query directly from Go.
I do this with
1select street, hex(street) from test_table where id = '36'
2
3Result: (in real it is called Fröbelstraße)
4Fröbelstraße, 4672C3B662656C73747261C39F65
5db, err := sql.Open("mysql", "...<connection string>...")
6res, err := db.Query("select street from from test_table where id = '36'")
7
8for res.Next() {
9var pb Phonebook
10 err := res.Scan(&pb.Street)
11 fmt.Println(hex.EncodeToString([]byte(pb.Street)))
12}
13 d := charmap.ISO8859_1.NewEncoder()
14 out, err := d.String(inp)
15
Also just a snippet, I actually call a function with a string, but I never got the correct iso8859-1 result. So I tried it with the hex code from the MySQL client query, converted back to a string, and bingo the iso8859-1 result is correct.
I also tried to query from python and get there the same strange hex from the queried string, so I am completely lost of what is wrong here. Cannot be go, because it is the same in python. But in the db it is stored correctly in my opinion and the MySQL character sets are all set the utf8mb4 or utf8, except the one I mentioned in above.
ANSWER
Answered 2022-Jan-22 at 22:58After two days of completely lost, I found the root cause myself. Strange that it happened shortly after I posted the question here.
I wanted to try a different mysql server and therefore dumped the table. And then I have seen in the dump that each field has its own character set definition which was latin1 in my case.
So that explains why it was a strange result. I just created a correct encoded test table and it works now as expected.
Now I have to think how I can "repair" these encodings, maybe a dump/restore will do it, but that's a another story.
QUESTION
Where is routing implemented (Request routing to the different replicas) when making request to the DynamoDB server?
Asked 2022-Jan-21 at 12:16DynamoDB has multiple nodes and all nodes are equal: there is no single leader. I am assuming we have a DynamoDB instance and all the data can be stored on a single node. But we have several replicas for availability and redundancy.
So writing and reading can happen on any replica node. I am just curious, how DynamoDB client decides to which node the request should be sent? Or the application developer has to take care of this routing themselves and the DyanmoDB client simply forwards these requests to the DB node.
If someone could explain to me where this routing happens and could point me to the code where this routing is implemented, that would be a great help?
I am new to the DynamoDB and trying to understand how things work.
ANSWER
Answered 2022-Jan-21 at 12:15Fundamentally this is an implementation detail that AWS handles - you won't find it in your Java code.
DynamoDB has multiple nodes and all nodes are equal: there is no single leader.
This is a misconception, the table is split up into partitions and each partition consists of three storage nodes. For each partition a storage node is selected that is the primary contact point for writes.
A few years ago there was an interesting talk by Jason Sorensen about the internals of DynamoDB at re:invent, you can see it on YouTube.
To summarise it, there is a fleet of request router instances that are aware of the storage nodes and how the hashing works. The storage router selects the leader node of the partition and writes data to it. As soon as two of the storage nodes acknowledge the write, the data is considered to be persisted.
Community Discussions contain sources that include Stack Exchange Network
Tutorials and Learning Resources in DB Client
Tutorials and Learning Resources are not available at this moment for DB Client