orm | Doctrine Object Relational Mapper | Object-Relational Mapping library
kandi X-RAY | orm Summary
Support
Quality
Security
License
Reuse
- Load class metadata for a class .
- Creates a new entity .
- Get the Doctrine Schema based on class metadata .
- Validate class metadata .
- Export class metadata .
- Generate the doc block for an association mapping .
- Walks down a join association declaration .
- Load class metadata .
- Get a select expression .
- Validate and complete association mapping .
orm Key Features
orm Examples and Code Snippets
Trending Discussions on orm
Trending Discussions on orm
QUESTION
Currently, I am writing a laravel application with a part for sending messages between the staff at the company and the clients.
So, I have a field named "status" in the database. In this field, a value of one indicates that the message is waiting for an answer, a value of two indicates that it has been answered, and a value of three indicates that the message has been closed. There is a problem here, however. It's not clear what these numbers do when someone looks at my code.
Would there be any way for me to define this number or any other way to make my code more readable?
(I'm using laravel eloquent ORM) The code below is for the method that closes a conversation:
public function close(Request $request)
{
$message = Message::find($request->message_id);
// Status one indicates that a conversation has been closed
$message->status = 1;
$message->save();
return \response($message, 200);
}
ANSWER
Answered 2022-Mar-18 at 09:36Use constants in your Message model
class Message
{
const STATUS_PENDING = 1;
const STATUS_ANSWERED = 2;
const STATUS_CLOSED = 3;
//...
}
Then your code will be readable
public function close(Request $request)
{
$message = Message::find($request->message_id);
$message->status = Message::STATUS_CLOSED;
$message->save();
return \response($message, 200);
}
Or Even better, make it a method in your model on top of the constants values
public function close(Request $request)
{
$message = Message::find($request->message_id);
$message->close();
return \response($message, 200);
}
That way you can in the future upgrade the method, for example
class Message
{
public function close()
{
if ($this->status != self::STATUS_ANSWERED) {
//log message closed without client answer
}
$this->status = STATUS_CLOSED;
$this->save();
}
}
QUESTION
In react native, I'm extending an ORM class (according to its documentation) but I'm getting following error in VSCode TypeScript checker:
Class static side 'typeof Animal' incorrectly extends base class static side 'typeof BaseModel'.
Types of property 'database' are incompatible.
Type '() => Promise' is not assignable to type 'void'.ts(2417)
In the end it actually works, but I would like to know if there's a way how to define it more properly or more loose so such checker error is not generated.
The ORM module is just JS (not TypeScript) EDIT: and it's 3rd party, so I can't really edit it
This is the parent class method in BaseModel
:
static get database() {
throw new Error('DB not defined')
}
This is the extending method of Animal
model:
static get database() {
return async () => SQLite.openDatabase('database.db')
}
ANSWER
Answered 2022-Mar-07 at 06:49Typescript infers BaseModel
's database
getter to be of type void
. This is because you neither return a value, nor do you have an explicit type on that getter. Then Animal
tries to extend that, and it returns an async function, which is not void, and you get the type error.
The correct fix here is to properly type the BaseModel.database
return value. In this case, I believe it should return an async function, which returns a promise, which wraps your database object.
class BaseModel {
static get database(): () => Promise {
throw new Error('DB not defined')
}
}
Now Animal
works without type errors:
class Animal extends BaseModel {
static get database() {
return async () => SQLite.openDatabase('database.db')
}
}
And you can get a db reference:
const db = await Animal.database()
If different subclasses would return different databases with different interfaces, then you can instead let the subclasses define that return type:
class BaseModel {
static get database(): () => Promise { // unknown here
throw new Error('DB not defined')
}
}
QUESTION
I am developing a Django app (Django v3.2.10, pytest v7.0.1, pytest-django v4.5.2) which uses cursor to perform raw queries to my secondary DB: my_db2, but when running tests, all the queries return empty results, like if they were running on parallel transactions.
My test file:
@pytest.mark.django_db(transaction=True, databases=['default', 'my_db2'])
class TestItems:
def test_people(self):
person1 = PeopleFactory() # Adds 1 person to my_db2
assert fetch_all_persons() == 1 # Fails Returns 0
My Factory:
class PeopleFactory(factory.django.DjangoModelFactory):
id = factory.Sequence(lambda x: x + 1)
name = factory.Faker('first_name')
class Meta:
model = People
My function:
from django.db import connections
def fetch_all_persons():
with connections['my_db2'].cursor() as cursor:
cursor.execute(f"SELECT * FROM Persons")
return len(list(cursor.fetchall())):
According documentation transaction=True
should prevent this issue, but it doesn't, does somebody know how to fix it?
Note.- Using the ORM is not an option, this is just a simplified example to represent the issue. The real queries used are way more complex.
ANSWER
Answered 2022-Feb-24 at 05:47@hoefling and @Arkadiusz Łukasiewicz were right, I just needed to add the corresponding DB within the factories:
class PeopleFactory(factory.django.DjangoModelFactory):
id = factory.Sequence(lambda x: x + 1)
name = factory.Faker('first_name')
class Meta:
model = People
database = 'my_db2'
Thank you both.
QUESTION
This worked fine for me be building under Java 8. Now under Java 17.01 I get this when I do mvn deploy.
mvn install works fine. I tried 3.6.3 and 3.8.4 and updated (I think) all my plugins to the newest versions.
Any ideas?
[ERROR] Failed to execute goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy (injected-nexus-deploy) on project persism: Execution injected-nexus-deploy of goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:de
ploy failed: An API incompatibility was encountered while executing org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy: java.lang.ExceptionInInitializerError: null
[ERROR] import: Entry[import from realm ClassRealm[maven.api, parent: null]]
[ERROR]
[ERROR] -----------------------------------------------------
[ERROR] : Unable to make field private final java.util.Comparator java.util.TreeMap.comparator accessible: module java.base does not "opens java.util" to unnamed module @149f5761
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy (injected-nexus-deploy) on project persism: Execution injected-nexus-deploy of goal org.sona
type.plugins:nexus-staging-maven-plugin:1.6.8:deploy failed: An API incompatibility was encountered while executing org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy: java.lang.ExceptionInInitializerError: null
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution injected-nexus-deploy of goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy failed: An API incompatibility was encountered while executing org.son
atype.plugins:nexus-staging-maven-plugin:1.6.8:deploy: java.lang.ExceptionInInitializerError: null
POM:
4.0.0
io.github.sproket
persism
2.0.0
jar
./src
./test
./test
org.apache.maven.plugins
maven-compiler-plugin
3.8.1
17
17
org.apache.maven.plugins
maven-jar-plugin
3.2.0
sproket.github.io.persism
org.apache.maven.plugins
maven-source-plugin
3.2.1
attach-sources
jar-no-fork
org.apache.maven.plugins
maven-javadoc-plugin
3.2.0
net.sf.persism.log*;net.sf.persism.logging.*
17
attach-javadocs
jar
maven-surefire-plugin
3.0.0-M5
net.sf.persism.categories.ExternalDB,net.sf.persism.categories.TestContainerDB
org.sonatype.plugins
nexus-staging-maven-plugin
1.6.8
true
ossrh
https://s01.oss.sonatype.org/
true
persism
A zero ceremony ORM for Java
https://github.com/sproket/Persism
17
17
UTF-8
BSD-3-Clause License
https://github.com/sproket/Persism/blob/master/license.txt
Dan Howard
--------------------------
io.github
https://sproket.github.io/Persism/
ossrh
https://s01.oss.sonatype.org/content/repositories/snapshots
ossrh
https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/
scm:git:git://github.com/sproket/Persism.git
scm:git:ssh://github.com/sproket/Persism.git
https://github.com/sproket/Persism
include-test-containers-db
false
maven-surefire-plugin
3.0.0-M5
net.sf.persism.categories.ExternalDB
exclude-test-containers-db
false
maven-surefire-plugin
3.0.0-M5
net.sf.persism.categories.TestContainerDB
release
org.apache.maven.plugins
maven-jar-plugin
3.2.0
sproket.github.io.persism
org.apache.maven.plugins
maven-source-plugin
3.2.1
attach-sources
jar-no-fork
org.apache.maven.plugins
maven-javadoc-plugin
3.2.0
attach-javadocs
jar
17
org.apache.maven.plugins
maven-gpg-plugin
3.0.1
sign-artifacts
verify
sign
junit
junit
4.13.2
test
com.carrotsearch
junit-benchmarks
0.7.2
test
org.testcontainers
testcontainers
1.15.2
test
ch.qos.logback
logback-classic
1.2.7
provided
log4j
log4j
1.2.17
provided
org.apache.logging.log4j
log4j-api
2.14.1
provided
org.apache.logging.log4j
log4j-core
2.14.1
provided
commons-dbcp
commons-dbcp
1.4
test
org.firebirdsql.jdbc
jaybird
4.0.2.java8
test
org.firebirdsql
firebird-testcontainers-java
1.1.0
test
com.h2database
h2
1.4.200
test
org.hsqldb
hsqldb
2.5.1
test
org.apache.derby
derby
10.8.2.2
test
com.microsoft.sqlserver
mssql-jdbc
8.4.1.jre8
test
org.testcontainers
mssqlserver
1.15.2
test
mysql
mysql-connector-java
8.0.23
test
org.testcontainers
mysql
1.15.2
test
net.sourceforge.jtds
jtds
1.3.1
test
com.oracle.database.jdbc
ojdbc8
21.3.0.0
test
org.postgresql
postgresql
9.2-1004-jdbc41
test
org.testcontainers
postgresql
1.15.2
test
org.xerial
sqlite-jdbc
3.34.0
test
net.sf.ucanaccess
ucanaccess
5.0.1
test
com.ibm.informix
informix-jdbc-complete
4.50.4.1
test
com.toddfast.typeconverter
typeconverter
1.0
test
org.reflections
reflections
0.9.11
test
javax.persistence
javax.persistence-api
2.2
test
ANSWER
Answered 2022-Feb-11 at 22:39Update: Version 1.6.9 has been released and should fix this issue! 🎉
This is actually a known bug, which is now open for quite a while: OSSRH-66257. There are two known workarounds:
1. Open ModulesAs a workaround, use --add-opens
to give the library causing the problem access to the required classes:
export MAVEN_OPTS="--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED"
mvn deploy
Or you can update the library that causes the problem:
org.sonatype.plugins
nexus-staging-maven-plugin
1.6.8
true
ossrh
https://s01.oss.sonatype.org/
true
com.thoughtworks.xstream
xstream
1.4.15
QUESTION
I'm trying to figure out how to setup a login via Discord Oauth2 while using Dapper as my ORM.
Microsoft has a guide here that I have followed to setup all of my stores. I infact can call CreateAsync()
method and a user gets created in my database, so I believe that side of things is completely setup.
My issues lie within external login. Below you will find what I have tried.
Program.cs:
//omitted code that binds interfaces and classes - this code works and is fully tested. it is not related to problem at hand.
builder.Services.AddIdentity()
.AddDefaultTokenProviders();
builder.Services.AddAuthentication()
.AddCookie(options =>
{
options.LoginPath = "/signin";
options.LogoutPath = "/signout";
})
.AddDiscord(options =>
{
options.ClientId = "some id";
options.ClientSecret = "some secret";
options.ClaimActions.MapCustomJson("urn:discord:avatar:url", user =>
string.Format(
CultureInfo.InvariantCulture,
"https://cdn.discordapp.com/avatars/{0}/{1}.{2}",
user.GetString("id"),
user.GetString("avatar"),
user.GetString("avatar")!.StartsWith("a_") ? "gif" : "png"));
});
builder.Services.AddRazorPages();
var app = builder.Build();
app.UseDeveloperExceptionPage();
app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseRouting();
app.UseAuthentication();
app.MapControllerRoute("default", "{controller=Home}/{action=Index}/{id?}");
app.Run();
Here is the Account Controller Code:
public class AccountController : Controller
{
private readonly ISignInService _signInService;
private readonly IUserService _userService;
public AccountController(ISignInService signInService, IUserService userService)
{
_signInService = signInService;
_userService = userService;
}
[HttpGet("~/signin")]
public async Task SignIn() => View("SignIn", await HttpContext.GetExternalProvidersAsync());
[HttpPost("~/signin")]
public async Task SignIn([FromForm] string provider, string returnUrl)
{
if (string.IsNullOrWhiteSpace(provider))
{
return BadRequest();
}
if (!await HttpContext.IsProviderSupportedAsync(provider))
{
return BadRequest();
}
var redirectUrl = Url.Action(nameof(LoginCallback), "Account", new { returnUrl });
var properties = _signInService.ConfigureExternalAuthenticationProperties(provider, redirectUrl, null);
properties.Items.Add("XsrfKey", "Test");
return Challenge(properties, provider);
}
[HttpGet("~/signout")]
[HttpPost("~/signout")]
public IActionResult SignOutCurrentUser()
{
return SignOut(new AuthenticationProperties {RedirectUri = "/"},
CookieAuthenticationDefaults.AuthenticationScheme);
}
//[HttpGet("~/Account/LoginCallback")]
[HttpGet]
public async Task LoginCallback(string returnUrl = null, string remoteError = null)
{
if (remoteError != null)
{
return RedirectToAction("Index", "Home");
}
var info = await _signInService.GetExternalLoginInfoAsync("Test");
if (info == null)
{
return RedirectToAction("Index", "Home");
}
var result = await _signInService.ExternalLoginSignInAsync(info.LoginProvider, info.ProviderKey, isPersistent: false, bypassTwoFactor: true);
if (result.Succeeded)
{
return RedirectToLocal(returnUrl);
}
if (result.IsLockedOut)
{
return RedirectToAction("Index", "Home");
}
else
{
// If the user does not have an account, then ask the user to create an account.
ViewData["ReturnUrl"] = returnUrl;
ViewData["LoginProvider"] = info.LoginProvider;
var email = info.Principal.FindFirstValue(ClaimTypes.Email);
return RedirectToAction("Index", "Home");
}
}
private IActionResult RedirectToLocal(string returnUrl)
{
if (Url.IsLocalUrl(returnUrl))
{
return Redirect(returnUrl);
}
else
{
return RedirectToAction(nameof(HomeController.Index), "Home");
}
}
}
Here is what happens:
- I click on the Login via discord button.
- I am taken to Discord Website
- I login via the discord website
- I am redirected back to my website
- info is never retrieved.
var info = await _signInService.GetExternalLoginInfoAsync("Test");
that line is always null.
I've been struggling to figure out what I have overlooked in my setup as I don't have any errors about anything.
ANSWER
Answered 2022-Jan-29 at 17:34Firstly... We need to take a look at the implementation of the internal method GetExternalLoginInfoAsync inside SignInManager.cs and take note of all the conditions that could possibly lead to null being returned.
I will provide my answer as comments within the code below:
///
/// Gets the external login information for the current login, as an asynchronous operation.
///
/// Flag indication whether a Cross Site Request Forgery token was expected in the current request.
/// The task object representing the asynchronous operation containing the
/// for the sign-in attempt.
public virtual async Task GetExternalLoginInfoAsync(string expectedXsrf = null)
{
var auth = await Context.AuthenticateAsync(IdentityConstants.ExternalScheme);
var items = auth?.Properties?.Items;
if (auth?.Principal == null || items == null || !items.ContainsKey(LoginProviderKey))
{
// What cases can lead us here?
// * The authentication was unsuccessful maybe due to
// - Login cancellation
// - Project not running on a secured environment (https)
// - SignInScheme property of auth options not
// equal to IdentityConstants.ExternalScheme
return null;
}
if (expectedXsrf != null)
{
// It is important to note that XsrfKey is a constant
// declared above in this class whose value is "XsrfId".
if (!items.ContainsKey(XsrfKey))
{
// What cases can lead us here?
// * You passed an argument for expectedXsrf but
// the initialized key-value pairs does not contain
// any key for XsrfKey ("XsrfId").
// In your case the below is wrong:
// properties.Items.Add("XsrfKey", "Test"); <= remove
// Pass the value as 3rd parameter in
// "ConfigureExternalAuthenticationProperties" method call instead
// _signInService.ConfigureExternalAuthenticationProperties(provider, redirectUrl, "Test")
return null;
}
var userId = items[XsrfKey] as string;
if (userId != expectedXsrf)
{
// What cases can lead us here?
// * The argument passed for expectedXsrf does not
// match the value of initialized key-value pair
// for XsrfKey ("XsrfId").
// Ensure "Test" should go with "XsrfId" as key
// by passing the value as 3rd parameter in
// "ConfigureExternalAuthenticationProperties" method call instead.
return null;
}
}
var providerKey = auth.Principal.FindFirstValue(ClaimTypes.NameIdentifier);
var provider = items[LoginProviderKey] as string;
if (providerKey == null || provider == null)
{
return null;
}
var providerDisplayName = (await GetExternalAuthenticationSchemesAsync()).FirstOrDefault(p => p.Name == provider)?.DisplayName
?? provider;
return new ExternalLoginInfo(auth.Principal, provider, providerKey, providerDisplayName)
{
AuthenticationTokens = auth.Properties.GetTokens()
};
}
So from the code review these are some possible causes for null:
The authentication was unsuccessful maybe due to
Login cancellation
Project not running on a secured environment (https)
SignInScheme property of auth options under StartUp.cs or appsettings.json not equal to IdentityConstants.ExternalScheme
You passed an argument for expectedXsrf but the
initialized key-value pair
does not contain any key forXsrfKey ("XsrfId")
.In your case the below is wrong:
properties.Items.Add("XsrfKey", "Test"); <= remove this line as "XsrfKey" is unknown.
Instead you pass the value as 3rd parameter in "ConfigureExternalAuthenticationProperties" method call:
_signInService.ConfigureExternalAuthenticationProperties(provider, redirectUrl, "Test");
QUESTION
How can I define a Doctrine property in a parent class and override the association in a class which extends the parent class? When using annotation, this was implemented by using AssociationOverride, however, I don't think they are available when using PHP 8 attributes
Why I want to:
I have a class AbstractTenantEntity
whose purpose is to restrict access to data to a given Tenant
(i.e. account, owner, etc) that owns the data, and any entity which extends this class will have tenant_id
inserted into the database when created and all other requests will add the tenant_id
to the WHERE clause. Tenant
typically does not have collections of the various entities which extend AbstractTenantEntity
, but a few do. When using annotations, I handled it by applying Doctrine's AssociationOverride
annotation to the extended classes which should have a collection in Tenant
, but I don't know how to accomplish this when using PHP 8 attributes?
My attempt described below was unsuccessful as I incorrectly thought that the annotation class would magically work with attributes if modified appropriately, but now I see other code must be able to apply the appropriate logic based on the attributes. As such, I abandoned this approach and just made the properties protected and duplicated them in the concrete class.
My attempt:
Tenant entity
use Doctrine\ORM\Mapping\Entity;
use Doctrine\ORM\Mapping\Column;
use Doctrine\ORM\Mapping\Id;
use Doctrine\ORM\Mapping\OneToMany;
use Doctrine\Common\Collections\ArrayCollection;
use Doctrine\Common\Collections\Collection;
#[Entity()]
class Tenant
{
#[Id, Column(type: "integer")]
#[GeneratedValue]
private ?int $id = null;
#[OneToMany(targetEntity: Asset::class, mappedBy: 'tenant')]
private array|Collection|ArrayCollection $assets;
// Other properties and typical getters and setters
}
AbstractTenantEntity entity
use Doctrine\ORM\Mapping\ManyToOne;
use Doctrine\ORM\Mapping\JoinColumn;
abstract class AbstractTenantEntity implements TenantInterface
{
/**
* inversedBy performed in child where required
*/
#[ManyToOne(targetEntity: Tenant::class)]
#[JoinColumn(nullable: false)]
protected ?Tenant $tenant = null;
// Typical getters and setters
}
This is the part which has me stuck. When using annotation, my code would be as follows:
use Doctrine\ORM\Mapping as ORM;
/**
* @ORM\Entity()
* @ORM\AssociationOverrides({
* @ORM\AssociationOverride(name="tenant", inversedBy="assets")
* })
*/
class Asset extends AbstractTenantEntity
{
// Various properties and typical getters and setters
}
But AssociationOverrides
hasn't been modified to work with attributes, so based on the official class, I created my own class similar to the others which Doctrine has updated:
namespace App\Mapping;
use Attribute;
use Doctrine\Common\Annotations\Annotation\NamedArgumentConstructor;
use Doctrine\ORM\Mapping\Annotation;
/**
* This annotation is used to override association mapping of property for an entity relationship.
*
* @Annotation
* @NamedArgumentConstructor()
* @Target("ANNOTATION")
*/
#[Attribute(Attribute::TARGET_CLASS | Attribute::IS_REPEATABLE)]
final class AssociationOverride implements Annotation
{
/**
* The name of the relationship property whose mapping is being overridden.
*
* @var string
*/
public $name;
/**
* The join column that is being mapped to the persistent attribute.
*
* @var array<\Doctrine\ORM\Mapping\JoinColumn>
*/
public $joinColumns;
/**
* The join table that maps the relationship.
*
* @var \Doctrine\ORM\Mapping\JoinTable
*/
public $joinTable;
/**
* The name of the association-field on the inverse-side.
*
* @var string
*/
public $inversedBy;
/**
* The fetching strategy to use for the association.
*
* @var string
* @Enum({"LAZY", "EAGER", "EXTRA_LAZY"})
*/
public $fetch;
public function __construct(
?string $name = null,
?array $joinColumns = null,
?string $joinTable = null,
?string $inversedBy = null,
?string $fetch = null
) {
$this->name = $name;
$this->joinColumns = $joinColumns;
$this->joinTable = $joinTable;
$this->inversedBy = $inversedBy;
$this->fetch = $fetch;
//$this->debug('__construct',);
}
private function debug(string $message, string $file='test.json', ?int $options = null)
{
$content = file_exists($file)?json_decode(file_get_contents($file), true):[];
$content[] = ['message'=>$message, 'object_vars'=>get_object_vars($this), 'debug_backtrace'=>debug_backtrace($options)];
file_put_contents($file, json_encode($content, JSON_PRETTY_PRINT));
}
}
When validating the mapping, Doctrine complains that target-entity does not contain the required inversedBy
. I've spent some time going through the Doctrine source code but have not made much progress.
Does my current approach have merit and if so please fill in the gaps. If not, however, how would you recommend meeting this need?
ANSWER
Answered 2021-Oct-11 at 18:30Override Field Association Mappings In Subclasses
Sometimes there is a need to persist entities but override all or part of the mapping metadata. Sometimes also the mapping to override comes from entities using traits where the traits have mapping metadata. This tutorial explains how to override mapping metadata, i.e. attributes and associations metadata in particular. The example here shows the overriding of a class that uses a trait but is similar when extending a base class as shown at the end of this tutorial.
Suppose we have a class ExampleEntityWithOverride. This class uses trait ExampleTrait:
The docblock is showing metadata override of the attribute and association type. It basically changes the names of the columns mapped for a property foo and for the association bar which relates to Bar class shown above. Here is the trait which has mapping metadata that is overridden by the annotation above:
The case for just extending a class would be just the same but:
Overriding is also supported via XML and YAML (examples).
QUESTION
I have a django app running in production. Its database has main write instance and a few read replicas. I use DATABASE_ROUTERS
to route between the write instance and the read replicas based on whether I need to read or write.
I encountered a situation where I have to do some async processing on an object due to a user request. The order of actions is:
- User submits a request via HTTPS/REST.
- The view creates an Object and saves it to the DB.
- Trigger a celery job to process the object outside of the request-response cycle and passing the object ID to it.
- Sending an OK response to the request.
Now, the celery job may kick in in 10 ms or 10 minutes depending on the queue. When it finally tuns, the celery job first tries to load the object based on the ID provided. Initially I had issues doing a my_obj = MyModel.objects.get(pk=given_id)
because the read replica would be used at this point, if the queue is empty and the celery job runs immediately after being triggered, the object may have not propagated to the read-replicas yet.
I resolved that issue by replacing my_obj = MyModel.objects.get(pk=given_id)
with my_obj = MyModel.objects.using('default').get(pk=given_id)
-- this ensures the object is read from my write-db-instance and is always available.
however, now I have another issue I did not anticipate.
calling my_obj.certain_many_to_many_objects.all()
triggers another call to the database as the ORM is lazy. That call IS being done on the read-replica. I was hoping it would stick to the database I defined with using
but that's not the case. Is there a way to force all sub-element objects to use the same write-db-instance?
ANSWER
Answered 2021-Sep-08 at 07:19Model managers and the QuerySet API reference can be used to change the database replica There is a way to specify which DB connection to use with Django. For each model manager, Django's BaseManager
class uses a private property self._db
to hold the DB connection, you may specify another value as well.
class MyModelRelationQuerySet(models.QuerySet):
def filter_on_my_obj(self, given_id):
# preform the base query set you want
return self.filter(relation__fk=given_id)
class MyModelManager(models.Manager):
# bypass self._db on BaseManager class
def get_queryset(self):
# proper way to pass "using" would be using=self._db
# for your case you may pass your 'master db connection'
return MyModelRelationQuerySet (self.model, using=your_write_replica)
def my_obj_filter(self, given_id):
return self.get_queryset().get_my_obj(given_id)
# pass the model manager to model
class MyModel(models.Model):
# ...
objects = MyModelManager()
documents on making custom QuerySet for model managers in Django.
and reading Django's models.Manger source code and the QuerySet source code can be insightful for such advanced issues with querying the data bese.
QUESTION
I want to use Testcontainers with @DataJpaTest
(and @SpringBootTest
) using JUnit 5. I have the basic setup working using the @Testcontainers
and @Container
annotation like this:
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import static org.assertj.core.api.Assertions.assertThat;
@DataJpaTest
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Testcontainers
public class AtleteRepositoryTest {
@Container
private static final PostgreSQLContainer CONTAINER = new PostgreSQLContainer<>("postgres:11");
@DynamicPropertySource
static void registerProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", CONTAINER::getJdbcUrl);
registry.add("spring.datasource.username", CONTAINER::getUsername);
registry.add("spring.datasource.password", CONTAINER::getPassword);
}
@Autowired
private AtleteRepository repository;
@Test
void testSave() {
repository.save(new Atlete("Wout Van Aert", 0, 1, 0));
assertThat(repository.count()).isEqualTo(1);
}
}
See https://github.com/wimdeblauwe/blog-example-code/tree/feature/testcontainers-datajpatest/testcontainers-datajpatest for the full example code (AtleteRepositoryTest, TeamRepositoryTest and TestcontainersDatajpatestApplicationTests).
To avoid the repetition of declaring the PostgreSQL container and the dynamic properties, I tried the following:
JUnit 5 extensionBaeldung has a blog about how you can use a JUnit 5 extension to avoid the duplication.
import org.junit.jupiter.api.extension.AfterAllCallback;
import org.junit.jupiter.api.extension.BeforeAllCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.testcontainers.containers.PostgreSQLContainer;
public class PostgreSQLExtension implements BeforeAllCallback, AfterAllCallback {
private PostgreSQLContainer postgres;
@Override
public void beforeAll(ExtensionContext context) {
postgres = new PostgreSQLContainer<>("postgres:11");
postgres.start();
System.setProperty("spring.datasource.url", postgres.getJdbcUrl());
System.setProperty("spring.datasource.username", postgres.getUsername());
System.setProperty("spring.datasource.password", postgres.getPassword());
}
@Override
public void afterAll(ExtensionContext context) {
postgres.stop();
}
}
It works if you only have 1 test, but not if you run multiple at the same time (using IntelliJ or with Maven). In that case, one of the tests will fail because there is no connection with the database that can be made. Also note that this extension does not use the DynamicPropertyRegistry
, but plain environment variables. See the feature/testcontainers-datajpatest_baeldung-extension branch for the code.
On the branch feature/testcontainers-datajpatest_database-base-test, I tried using a common superclass:
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.PostgreSQLContainer;
public class DatabaseBaseTest {
private static final PostgreSQLContainer CONTAINER = new PostgreSQLContainer<>("postgres:11");
@BeforeAll
static void start() {
CONTAINER.start();
}
@AfterAll
static void stop() {
CONTAINER.stop();
}
@DynamicPropertySource
static void registerProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", () -> {
String jdbcUrl = CONTAINER.getJdbcUrl();
System.out.println("jdbcUrl = " + jdbcUrl);
return jdbcUrl;
});
registry.add("spring.datasource.username", CONTAINER::getUsername);
registry.add("spring.datasource.password", CONTAINER::getPassword);
}
}
Unfortunately that also does not work. I noticed in the logging that the @DynamicPropertySource
annotated method was only called once and not for each test, which led me to try option 3:
@DynamicPropertySource
in subclasses
When using the common superclass, but adding the @DynamicPropertySource
method in each subclass, it works again.
Example code of such a subclass:
@DataJpaTest
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
public class AtleteRepositoryTest extends DatabaseBaseTest {
@DynamicPropertySource
static void registerProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", () -> {
String jdbcUrl = CONTAINER.getJdbcUrl();
System.out.println("jdbcUrl = " + jdbcUrl);
return jdbcUrl;
});
registry.add("spring.datasource.username", CONTAINER::getUsername);
registry.add("spring.datasource.password", CONTAINER::getPassword);
}
@Autowired
private AtleteRepository repository;
@Test
void testSave() {
repository.save(new Atlete("Wout Van Aert", 0, 1, 0));
assertThat(repository.count()).isEqualTo(1);
}
}
See branch feature/testcontainers-datajpatest_database-base-test_subclasses for that version.
So while it works, there is still a lot of duplication in each test class.
Are there any other options for avoiding the duplication?
ANSWER
Answered 2021-Aug-23 at 09:35To avoid Testcontainers code repetition I generally follow 2 approaches:
- Using ApplicationContextInitializer with @ContextConfiguration
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.test.util.TestPropertyValues;
import org.springframework.context.ApplicationContextInitializer;
import org.springframework.context.ConfigurableApplicationContext;
import org.testcontainers.containers.PostgreSQLContainer;
@Slf4j
public class PostgreSQLContainerInitializer
implements ApplicationContextInitializer {
private static PostgreSQLContainer sqlContainer = new PostgreSQLContainer("postgres:10.7");
static {
sqlContainer.start();
}
public void initialize (ConfigurableApplicationContext configurableApplicationContext){
TestPropertyValues.of(
"spring.datasource.url=" + sqlContainer.getJdbcUrl(),
"spring.datasource.username=" + sqlContainer.getUsername(),
"spring.datasource.password=" + sqlContainer.getPassword()
).applyTo(configurableApplicationContext.getEnvironment());
}
}
import com.sivalabs.myservice.common.PostgreSQLContainerInitializer;
import com.sivalabs.myservice.entities.User;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest;
import org.springframework.test.context.ContextConfiguration;
import javax.persistence.EntityManager;
import java.util.Optional;
import static org.assertj.core.api.Assertions.assertThat;
@DataJpaTest
@AutoConfigureTestDatabase(replace= AutoConfigureTestDatabase.Replace.NONE)
@ContextConfiguration(initializers = {PostgreSQLContainerInitializer.class})
class UserRepositoryTest {
@Autowired
EntityManager entityManager;
@Autowired
private UserRepository userRepository;
@Test
void shouldReturnUserGivenValidCredentials() {
User user = new User(null, "test@gmail.com", "test", "Test");
entityManager.persist(user);
Optional userOptional = userRepository.login("test@gmail.com", "test");
assertThat(userOptional).isNotEmpty();
}
}
- Using @DynamicPropertySource in Java 8+ Interface
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
@Testcontainers
public interface PostgreSQLContainerInitializer {
@Container
PostgreSQLContainer postgres = new PostgreSQLContainer<>("postgres:12.3");
@DynamicPropertySource
static void registerPgProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", postgres::getJdbcUrl);
registry.add("spring.datasource.username", postgres::getUsername);
registry.add("spring.datasource.password", postgres::getPassword);
}
}
@DataJpaTest
@AutoConfigureTestDatabase(replace= AutoConfigureTestDatabase.Replace.NONE)
class UserRepositoryTest implements PostgreSQLContainerInitializer {
....
....
}
With these approaches we don't have to repeat PostgreSQLContainer declarations and Spring property settings.
Whether to use PostgreSQLContainer as a static field or not depends on whether you want to spin up a new container for every test or 1 container per test class.
PS: I avoided using common base class approach because sometime one test needs only 1 container and another test needs multiple containers. If we follow add all the containers in common base class then for every test/class all those containers will be started irrespective of their usage which makes tests very slow.
QUESTION
The documentation for Django 2.2, which I'm using, gives the following example usage for select_for_update
:
from django.db import transaction
entries = Entry.objects.select_for_update().filter(author=request.user)
with transaction.atomic():
for entry in entries:
...
Using this approach, one would presumably mutate the model instances assigned to entry
and call save
on these.
There are cases where I'd prefer the alternative approach below, but I'm unsure whether it would work (or even make sense) with select_for_update
.
with transaction.atomic():
Entry.objects.select_for_update().filter(author=request.user).update(foo="bar", wobble="wibble")
The documentation states that the lock is created when the queryset is evaluated, so I doubt the update
method would work. As far as I'm aware update
just performs an UPDATE ... WHERE
query, with no SELECT
before it. However, I would appreciate it if someone more experienced with this aspect of the Django ORM could confirm this.
A secondary question is whether a lock even adds any protection against race conditions if one makes a single UPDATE
query against the locked rows. (I've entered this train of thought because I'm refactoring code that uses a lock when updating the values of two columns of a single row.)
ANSWER
Answered 2021-Jul-27 at 02:49As far as I'm aware update just performs an UPDATE ... WHERE query, with no SELECT before it
Yes, that's correct. You could confirm this by looking at the actual queries made. Using the canonical django tutorial "polls" app as an example:
with transaction.atomic():
qs = polls.models.Question.objects.select_for_update().all()
qs.update(question_text='test')
print(connection.queries)
# {'sql': 'UPDATE "polls_question" SET "question_text" = \'test\'', 'time': '0.008'}
So, as you expect, there is no SELECT
.
Though, ensuring the lock is acquired would be as simple as doing anything to cause the queryset to be evaluated.
with transaction.atomic():
qs = polls.models.Question.objects.select_for_update().all()
list(qs) # cause evaluation, locking the selected rows
qs.update(question_text='test')
print(connection.queries)
#[...
# {'sql': 'SELECT "polls_question"."id", "polls_question"."question_text", "polls_question"."pub_date" FROM "polls_question" FOR UPDATE', 'time': '0.003'},
# {'sql': 'UPDATE "polls_question" SET "question_text" = \'test\'', 'time': '0.001'}
#]
A secondary question is whether a lock even adds any protection against race conditions if one makes a single UPDATE query against the locked rows
In general, yes. Whether it is necessary in a particular situation depends what kind of race condition you're worried about. The lock will prevent race conditions where another transaction may try to update the same row, for example.
Race conditions can be avoided without locks, too, depending on the nature of the update/race condition. Sometimes a transaction is sufficient, sometimes it's not. You may also use expressions which are evaluated server-side on the db to prevent race conditions (e.g. using Django's F()
expressions).
There are also other considerations, like your db dialect, isolation levels, and more.
Additional reference on race condition thoughts: PostgreSQL anti-patterns: read-modify-write cycles (archive)
QUESTION
Trying to make the populate
parameter in MikroORM strict for the variant with string paths. I managed to implement (more like adjust the one I found on TS discord) the AuthPath
type that works based on my needs. It works ok when used with a single parameter, but when used with array, it won't validate correctly if one of the array items is valid.
So the question is, how can I make it work with arrays? Is it even possible? I was trying to use tuple types to get around this, but failed to make it work. I kinda understand the problem is the shared generic type P
- I am planning to leverage it in the return type so I need something to bare the actual (inferred) types in the signature from params to return type.
The full playground in here.
Here is the demonstration of the problem:
declare const user: User;
declare function get1(obj: O, path: AutoPath): void;
declare function get2(obj: O, path: AutoPath[]): void;
// works fine with single item
get1(user, "friend.books.title")
get1(user, "friend.books.ref1.age")
get1(user, "friend.friend.name")
// @ts-expect-error
get1(user, "friend.friend.www")
// @ts-expect-error
get1(user, "friend.books.www")
// @ts-expect-error
get1(user, "friend.books.ref1.www")
// works fine with array when there is just one item
get2(user, ["friend.name"])
get2(user, ["friend.books.ref1.age"])
// @ts-expect-error
get2(user, ["friend.books.ref1.www"])
// if there are more items it works only sometimes
// @ts-expect-error
get2(user, ["friend.name", "books.author.www"])
// if we add one more item that is valid and on the root level, it will make it pass
get2(user, ["friend.name", "books.author.www", "age"])
Here is the code for AutoPath
and the entity type definitions:
class Collection { items?: T[] }
class Reference { item?: T }
type Book = {
id: string,
title: string,
author: User,
ref1: Reference,
}
type User = {
id: string,
name: string,
age: number,
friend: User,
friends: Collection,
books: Collection,
}
type ExtractType = T extends Collection ? U : (T extends Reference ? U : T)
type StringKeys = T extends Collection
? `${Exclude, symbol>}`
: T extends Reference
? `${Exclude, symbol>}`
: `${Exclude}`
type GetStringKey> = K extends keyof T ? ExtractType : never
type AutoPath =
(P & `${string}.` extends never ? P : P & `${string}.`) extends infer Q
? Q extends `${infer A}.${infer B}`
? A extends StringKeys
? `${A}.${AutoPath, B>}`
: never
: Q extends StringKeys
? (GetStringKey extends unknown ? Exclude : never) | (StringKeys> extends never ? never : `${Q}.`)
: StringKeys
: never
(the AutoPath
type still has some issues, but that is not really important - this question is about how to use it with array of strings instead of a single string parameter)
ANSWER
Answered 2021-Jul-08 at 13:49I think the issue here is that you want AutoPath
to distribute over unions in P
. That is, you want AutoPath
to be equivalent to AutoPath | AutoPath
. And sometimes it does not seem to work out that way.
If so, then you can use distributive conditional types to get this behavior. All you need to do is wrap your original definition with P extends any ? ... : never
:
type AutoPath =
P extends any ?
/* ORIGINAL IMPLEMENTATION */
(P & `${string}.` extends never ? P : P & `${string}.`) extends infer Q
? Q extends `${infer A}.${infer B}`
? A extends StringKeys
? `${A}.${AutoPath, B>}`
: never
: Q extends StringKeys
? (GetStringKey extends unknown ? Exclude : never) | (StringKeys> extends never ? never : `${Q}.`)
: StringKeys
: never
/* END ORIGINAL IMPLEMENTATION */
: never
And you should hopefully get the behavior you want:
get2(user, ["friend.name", "books.author.www", "age"]); // error!
// ----------------------> ~~~~~~~~~~~~~~~~~~
It's always possible that some inference or other behavior you were relying on will be altered by this change, but because the particular implementation of AutoPath
and what it's being used for is out of scope for the question, I'll leave it up to you to deal with any such issues.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install orm
PHP requires the Visual C runtime (CRT). The Microsoft Visual C++ Redistributable for Visual Studio 2019 is suitable for all these PHP versions, see visualstudio.microsoft.com. You MUST download the x86 CRT for PHP x86 builds and the x64 CRT for PHP x64 builds. The CRT installer supports the /quiet and /norestart command-line switches, so you can also script it.
Support
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesExplore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits
Save this library and start creating your kit
Share this Page